Add all-gather coalescing for FSDP/ZeRO1#5624

Closed

jeffhataws wants to merge 9 commits intopytorch:masterfrom jeffhataws:cc_coalesce_revival

+323-45

Commits on Nov 18, 2023

Add all-gather and reduce-scatter coalescence support for FSDP.
jeffhataws
committed

Commits on Nov 20, 2023

clang-format-7 and python lint fixes
jeffhataws
committed

Commits on Nov 21, 2023

Fix "SyntaxError: 'return' outside function" error
jeffhataws
committed

Commits on Nov 22, 2023

Code/test fixes to get run_tests.sh to run on CPU
jeffhataws
committed

Commits on Nov 27, 2023

Fix allgather to be compatible with openxla allgather tuple change without token
jeffhataws
committed

Commits on Nov 28, 2023

Fix reduce-scatter-coalesce to be compatible with openxla reduce-scatter tuple change without token
jeffhataws
committed

Commits on Nov 29, 2023

Commits on Nov 30, 2023

Add separate BuildAllGatherCoalesced builder and AllGatherCoalesced class
jeffhataws
committed