-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Temporarily disable custom forward/backward diff rules in Enzyme-based interpreter #321
Conversation
It seems like this doesn't circumvent the issue 🥲 |
sad |
x/ref EnzymeAD/Enzyme.jl#2152 if that is the same place things are beign slow [and we can borrow the same things I was using to debug perf here] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reactant.jl Benchmarks
Benchmark suite | Current: 4f93c5f | Previous: 45ae14f | Ratio |
---|---|---|---|
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) |
6775831093 ns |
6749076193 ns |
1.00 |
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant |
6962689336 ns |
5078740247 ns |
1.37 |
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) |
6661316973 ns |
5013817961 ns |
1.33 |
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) |
7668593077 ns |
7197691815 ns |
1.07 |
ViT base (256 x 256 x 3 x 32)/forward/CPU/Lux |
29090804314 ns |
35464964244 ns |
0.82 |
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) |
1611585726 ns |
1572590140 ns |
1.02 |
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant |
1603076438 ns |
1559474266 ns |
1.03 |
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) |
1605022182 ns |
1557501067 ns |
1.03 |
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) |
3462398317 ns |
3290628669 ns |
1.05 |
ViT small (256 x 256 x 3 x 4)/forward/CPU/Lux |
2966881116 ns |
2876354148 ns |
1.03 |
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) |
2182413214 ns |
2154505585 ns |
1.01 |
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant |
2160966244 ns |
2139776302 ns |
1.01 |
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) |
2156321409 ns |
2123332313 ns |
1.02 |
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) |
3947774259 ns |
3879039560 ns |
1.02 |
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Lux |
6732934936.5 ns |
5729200009 ns |
1.18 |
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) |
1504823325 ns |
1424258021 ns |
1.06 |
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant |
1490440471 ns |
1421721118 ns |
1.05 |
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) |
1485869889 ns |
1420742881 ns |
1.05 |
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) |
3304622051 ns |
3162578762 ns |
1.04 |
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Lux |
1056207342 ns |
2138106366 ns |
0.49 |
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) |
1761607099 ns |
1721006513 ns |
1.02 |
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant |
1794126244 ns |
1711405549 ns |
1.05 |
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) |
1742471268 ns |
1704835369 ns |
1.02 |
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) |
3532317713 ns |
3443971150 ns |
1.03 |
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Lux |
3052575121 ns |
3110298785 ns |
0.98 |
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) |
2218916991 ns |
2169947879 ns |
1.02 |
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant |
2246423502 ns |
2163945294 ns |
1.04 |
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) |
2235978635 ns |
2151891950 ns |
1.04 |
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) |
4006048902 ns |
3946269320 ns |
1.02 |
ViT small (256 x 256 x 3 x 16)/forward/CPU/Lux |
6449456243 ns |
6287057122 ns |
1.03 |
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) |
3048692753 ns |
3050356994 ns |
1.00 |
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant |
3098350261 ns |
3082997102 ns |
1.00 |
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) |
3122950600 ns |
2965563203 ns |
1.05 |
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) |
5078377116 ns |
4841087626 ns |
1.05 |
ViT small (256 x 256 x 3 x 32)/forward/CPU/Lux |
17246174602 ns |
8484129480 ns |
2.03 |
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) |
3249527673 ns |
3125511597 ns |
1.04 |
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant |
3234462974 ns |
3098530069 ns |
1.04 |
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) |
3330731868 ns |
3115589553 ns |
1.07 |
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) |
5239910158 ns |
5036626230 ns |
1.04 |
ViT base (256 x 256 x 3 x 16)/forward/CPU/Lux |
10391102697 ns |
11289651474 ns |
0.92 |
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) |
1934213262 ns |
1859519475 ns |
1.04 |
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant |
2060174442 ns |
1869845638 ns |
1.10 |
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) |
2038338573 ns |
1850101657 ns |
1.10 |
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) |
3731746639 ns |
3593739548 ns |
1.04 |
ViT base (256 x 256 x 3 x 4)/forward/CPU/Lux |
3198658520.5 ns |
3325189113.5 ns |
0.96 |
This comment was automatically generated by workflow using github-action-benchmark.
fixed in #341 |
Let's see if this circumvents the compilation time issue while @vchuravy and @wsmoses fix the issue in Enzyme