[Zerobubble] merge main. #6142

duanjunwen · 2024-11-18T10:18:50Z

📌 Checklist before creating the PR

I have created an issue for this PR for traceability
The title follows the standard format: [doc/gemini/tensor/...]: A concise description
I have added relevant tags if possible for us to better distinguish different PRs
I have installed pre-commit: pip install pre-commit && pre-commit install

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

I have linked my PR to an issue (instruction)
My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
I have performed a self-review of my code
I have added thorough tests.
I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

🌝 Yes, I do.
🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

…add test for zerobubble;

…ffer error;

…p.grad assert close test & all pass;

…ributes; fix schedule loop "while"--> "for"; add communication dict;

…nfo;

…marking for llama hybrid;

.github/workflows/build_on_pr.yml

.github/workflows/build_on_schedule.yml

colossalai/pipeline/schedule/v_schedule.py

colossalai/pipeline/schedule/zero_bubble_pp.py

duanjunwen added 30 commits August 22, 2024 10:25

[feat] add zerobubble pp (just a frame now); add POC test for dx_dw; …

ee9baed

…add test for zerobubble;

[feat] add dw test;

c18ef06

[fix] fix weight not close;

203033e

[update] update text;

107230d

Merge branch 'main' into dev/zero_bubble

fd5526b

[feat] add test run_fwd_bwd automatic scheduling;

1d75045

[feat] split communication and calculation; fix pop empty send_bwd_bu…

5e09c8b

…ffer error;

[feat] add test for p & p grad;

f1c1a87

[feat] add comments for ZBV func;

1b4bb2b

[fix] rm useless assign and comments;

283c9ff

[fix] fix ci test; add pytest;

9e0bd1a

[feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&…

8b37323

…p.grad assert close test & all pass;

[feat] add apply v_schedule graph; p & p.grad assert err exist;

fe20916

[fix] update

29383b2

[feat] fix ci; add assert;

d6e3d7d

[feat] fix poc format

b5f7b4d

[feat] fix func name & ci; add comments;

582ba0d

[fix] fix poc test; add comments in poc;

b1419ef

[feat] add optim backward_b_by_grad

4c4b01b

[feat] fix optimizer bwd b & w; support return accum loss & output

48ba22d

[feat] add fwd_bwd_step, run_fwd_only;

6af81d8

[fix] fix optim bwd; add license for v_schedule; remove redundant att…

8eb6eac

…ributes; fix schedule loop "while"--> "for"; add communication dict;

[fix] fix communication_map;

a7b767b

[feat] update test; rm comments;

6d18d38

[fix] rm zbv in hybridplugin

77fe442

[fix] fix optim bwd;

591a13b

[fix] fix optim bwd;

a48afc4

[fix] rm output.data after send fwd;

ab643c9

[fix] fix bwd step if condition; remove useless comments and format i…

4c1f81c

…nfo;

[fix] fix detach output & release output;

b4103f1

duanjunwen added 20 commits November 1, 2024 05:27

[fix] fix hybridparall use_fp8 config

5b5fbcf

[fix] fix use_fp8 flag

0218e67

[fix] fix model zoo init

8e40087

[feat] support no_tp Linear for sharderformer.llama

4fc92aa

[fix] fix zbv llama pp4

0d6d40c

[fix] fix send_tensor_metadata & send_grad_metadata;

12919de

[feat] fix testcase;

337debc

[feat] support mixtral policy with zbv tp_Linear & non_tp_Linear

80b04d7

[feat] update mixtral policy & bert policy for zerobubble

b6d5e61

[fix] fix p2p error in zbv

1bc4dba

[fix] fix attn

014afbd

[fix] fix mixtral modeling & policy; update wait handles; doing bench…

5c2ebbf

…marking for llama hybrid;

[fix] fix zbv wait_handle

cf86c1b

[fix] rm debug info; update llama policy; update wait handle

0fb500c

[fix] fix test_lora

2980da5

[fix] fix test_lora in llama policy

f48a85e

[fix] fix wait handle in run_fwd_bwd

9a21f87

[fix] remove debug info;

dafda0f

[fix] rm unused comments

41fdd21

Merge branch 'main' into dev/zero_bubble

cb9e5cc

duanjunwen requested a review from a team as a code owner November 18, 2024 10:18

duanjunwen requested a review from ver217 November 18, 2024 10:19

[fix] fix fp8 overlap code

8a0bad9

ver217 reviewed Nov 19, 2024

View reviewed changes

[fix] fix yml file & v_schedule comments

9aa4c67

duanjunwen requested a review from ver217 November 19, 2024 08:22

[fix] rm fwd only meta cache comments;

e4488b1

ver217 approved these changes Nov 19, 2024

View reviewed changes

duanjunwen requested a review from ver217 November 19, 2024 10:32

duanjunwen merged commit e0c68ab into hpcaitech:main Nov 19, 2024
6 of 7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Zerobubble] merge main. #6142

[Zerobubble] merge main. #6142

duanjunwen commented Nov 18, 2024

[Zerobubble] merge main. #6142

[Zerobubble] merge main. #6142

Conversation

duanjunwen commented Nov 18, 2024

📌 Checklist before creating the PR

🚨 Issue number

📝 What does this PR do?

💥 Checklist before requesting a review

⭐️ Do you enjoy contributing to Colossal-AI?