Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] update autodiff tutorial #3836

Merged
merged 4 commits into from
Oct 7, 2024
Merged

[docs] update autodiff tutorial #3836

merged 4 commits into from
Oct 7, 2024

Conversation

gdalle
Copy link
Contributor

@gdalle gdalle commented Oct 6, 2024

Two modifications to the tutorial on autodiff of user-defined operators:

  1. Upgrade to Enzyme v0.13 (no need for autodiff_deferred anymore). I didn't necessarily go for the most efficient solutions, you may want to ask @wsmoses for that, I just performed the minimal changes to gain compatibility with the breaking release.
  2. Add a section demonstrating DifferentiationInterface. For some reason it doesn't work with AutoEnzyme() but it's a start.

Related issues:

@gdalle
Copy link
Contributor Author

gdalle commented Oct 6, 2024

I had a question about the use of DI. For many backends, performance is greatly improved by the preparation mechanism, which allows you to record a tape / preallocate a cache / etc. But to apply it, you need an example input that has the same type and size as the ones you'll encounter during optimization. Is there any way we can use this mechanism in JuMP?

@odow
Copy link
Member

odow commented Oct 6, 2024

You could just use the first call to build the tape, and then error if there are calls from different inputs. JuMP doesn't provide anything special. You can do everything in user-land.

# [DifferentiationInterface.jl](https://github.com/gdalle/DifferentiationInterface.jl)
# is a package that provides an abstraction layer across multiple underlying
# autodiff libaries.

Copy link
Contributor

@wsmoses wsmoses Oct 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add a disclaimer:

While DI makes it easier to experiment with different autodiff tools, this comes at a cost. In particular, DI may produce slower derivatives than using an AD tool directly, and sometimes fail to work at all. For example, in this code passing AutoEnzyme to DI fails to differentiate the code which is supported when directly calling Enzyme, like above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way do you have any idea why it fails here? As usual it's a segfault so not particularly user friendly

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea, I haven’t even tried to run it or seen what the error message, just took your word that it failed xD.

open an issue with the mwe and error and can take a look

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and sometimes fail to work at all

I don't want to mention something in our documentation if this is the case.

What is the failure mode? Will we gracefully report errors? Segfaults are not good.

Copy link
Contributor

@wsmoses wsmoses Oct 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh agreed in that segfaults are bad, in that case @gdalle just drop DI docs then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized my changes mean we never actually mention Enzyme in the context of DI, so perhaps that's okay.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that it is a segfault is not due to DI per se, it's Enzyme not catching the error gracefully. Sure, DI might be using Enzyme wrong in this particular case (perhaps because of a function returning two closures), but I too would very much prefer a clean error message from Enzyme to a segfault. Will try to reproduce tomorrow.

Again haven't seen the error message so not clear what's up, but I'm unaware of any segfaults caused by Enzyme atm. Most the time people saw segfaults in the past were actually bugs in the Julia compiler itself (e.g. JuliaLang/julia#55306 ).

That said unfortunately even if you're not the underlying cause of an error, the top level package you use is going to be the one people blame for issues (especially if code works otherwise without it). Analogously, there's a ton of code in Enzyme working around bugs in the Julia and LLVM compiler errors =/. Perhaps something similar would be good in DI?

Copy link
Contributor Author

@gdalle gdalle Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not DI's job to work around LLVM errors, it's Enzyme's or Julia's. DI is a fancy argument-passer

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here's the MWE: EnzymeAD/Enzyme.jl#1942

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, pure Enzyme also fails on this one when one uses splat(f)

Copy link

codecov bot commented Oct 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.90%. Comparing base (288dca8) to head (23c82e8).
Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3836   +/-   ##
=======================================
  Coverage   97.90%   97.90%           
=======================================
  Files          44       44           
  Lines        6019     6019           
=======================================
  Hits         5893     5893           
  Misses        126      126           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@odow odow merged commit 1aa0ff7 into jump-dev:master Oct 7, 2024
10 checks passed
wsmoses added a commit to wsmoses/JuMP.jl that referenced this pull request Oct 8, 2024
Looks like the warning got dropped before merging
@gdalle gdalle deleted the gd/di_docs branch October 8, 2024 06:58
odow pushed a commit that referenced this pull request Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants