Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporarily disable custom forward/backward diff rules in Enzyme-based interpreter #321

Closed
wants to merge 1 commit into from

Conversation

mofeing
Copy link
Collaborator

@mofeing mofeing commented Dec 3, 2024

Let's see if this circumvents the compilation time issue while @vchuravy and @wsmoses fix the issue in Enzyme

…d interpreter

Let's see if this circumvents the compilation time issue while @vchuravy and @wsmoses fix the issue in Enzyme
@mofeing
Copy link
Collaborator Author

mofeing commented Dec 3, 2024

It seems like this doesn't circumvent the issue 🥲

@wsmoses
Copy link
Member

wsmoses commented Dec 3, 2024

sad

@wsmoses
Copy link
Member

wsmoses commented Dec 3, 2024

x/ref EnzymeAD/Enzyme.jl#2152 if that is the same place things are beign slow [and we can borrow the same things I was using to debug perf here]

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reactant.jl Benchmarks

Benchmark suite Current: 4f93c5f Previous: 45ae14f Ratio
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 6775831093 ns 6749076193 ns 1.00
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant 6962689336 ns 5078740247 ns 1.37
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 6661316973 ns 5013817961 ns 1.33
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 7668593077 ns 7197691815 ns 1.07
ViT base (256 x 256 x 3 x 32)/forward/CPU/Lux 29090804314 ns 35464964244 ns 0.82
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1611585726 ns 1572590140 ns 1.02
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant 1603076438 ns 1559474266 ns 1.03
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1605022182 ns 1557501067 ns 1.03
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3462398317 ns 3290628669 ns 1.05
ViT small (256 x 256 x 3 x 4)/forward/CPU/Lux 2966881116 ns 2876354148 ns 1.03
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 2182413214 ns 2154505585 ns 1.01
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant 2160966244 ns 2139776302 ns 1.01
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 2156321409 ns 2123332313 ns 1.02
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 3947774259 ns 3879039560 ns 1.02
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Lux 6732934936.5 ns 5729200009 ns 1.18
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1504823325 ns 1424258021 ns 1.06
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant 1490440471 ns 1421721118 ns 1.05
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1485869889 ns 1420742881 ns 1.05
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3304622051 ns 3162578762 ns 1.04
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Lux 1056207342 ns 2138106366 ns 0.49
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 1761607099 ns 1721006513 ns 1.02
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant 1794126244 ns 1711405549 ns 1.05
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 1742471268 ns 1704835369 ns 1.02
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 3532317713 ns 3443971150 ns 1.03
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Lux 3052575121 ns 3110298785 ns 0.98
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 2218916991 ns 2169947879 ns 1.02
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant 2246423502 ns 2163945294 ns 1.04
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 2235978635 ns 2151891950 ns 1.04
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 4006048902 ns 3946269320 ns 1.02
ViT small (256 x 256 x 3 x 16)/forward/CPU/Lux 6449456243 ns 6287057122 ns 1.03
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 3048692753 ns 3050356994 ns 1.00
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant 3098350261 ns 3082997102 ns 1.00
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 3122950600 ns 2965563203 ns 1.05
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 5078377116 ns 4841087626 ns 1.05
ViT small (256 x 256 x 3 x 32)/forward/CPU/Lux 17246174602 ns 8484129480 ns 2.03
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 3249527673 ns 3125511597 ns 1.04
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant 3234462974 ns 3098530069 ns 1.04
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 3330731868 ns 3115589553 ns 1.07
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 5239910158 ns 5036626230 ns 1.04
ViT base (256 x 256 x 3 x 16)/forward/CPU/Lux 10391102697 ns 11289651474 ns 0.92
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1934213262 ns 1859519475 ns 1.04
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant 2060174442 ns 1869845638 ns 1.10
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 2038338573 ns 1850101657 ns 1.10
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3731746639 ns 3593739548 ns 1.04
ViT base (256 x 256 x 3 x 4)/forward/CPU/Lux 3198658520.5 ns 3325189113.5 ns 0.96

This comment was automatically generated by workflow using github-action-benchmark.

@mofeing
Copy link
Collaborator Author

mofeing commented Dec 8, 2024

fixed in #341

@mofeing mofeing closed this Dec 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants