Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DS] Deepspeed rolling batch support #1295

Merged
merged 2 commits into from
Nov 15, 2023
Merged

[DS] Deepspeed rolling batch support #1295

merged 2 commits into from
Nov 15, 2023

Conversation

maaquib
Copy link
Contributor

@maaquib maaquib commented Nov 8, 2023

Description

Brief description of what this PR is about

  • If this change is a backward incompatible change, why must this change be made?
  • Interesting edge cases to note here

@maaquib maaquib changed the title [WIP] Deepspeed rolling batch support [WIP] [DS] Deepspeed rolling batch support Nov 9, 2023
@lanking520 lanking520 marked this pull request as ready for review November 13, 2023 18:43
@lanking520 lanking520 requested review from zachgk, frankfliu and a team as code owners November 13, 2023 18:43
@lanking520
Copy link
Contributor 

git rebase -i upstream/master

Try to squash the commits to keep the commit tree clean

@maaquib maaquib force-pushed the deepspeed_kvcache branch 2 times, most recently from 7c839f5 to 053662a Compare November 14, 2023 17:00
@maaquib maaquib force-pushed the deepspeed_kvcache branch 2 times, most recently from 60fff17 to f1a4db9 Compare November 15, 2023 17:32
@lanking520 lanking520 changed the title [WIP] [DS] Deepspeed rolling batch support [DS] Deepspeed rolling batch support Nov 15, 2023
@maaquib
Copy link
Contributor Author

maaquib commented Nov 15, 2023

Nightly build: https://github.com/deepjavalibrary/djl-serving/actions/runs/6882455410
Large model integration tests: https://github.com/deepjavalibrary/djl-serving/actions/runs/6882839289
Rolling Batch Integration tests: https://github.com/deepjavalibrary/djl-serving/actions/runs/6883468941

@maaquib maaquib merged commit 2e6accc into deepjavalibrary:master Nov 15, 2023
39 of 51 checks passed
@maaquib maaquib deleted the deepspeed_kvcache branch November 15, 2023 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lanking520 lanking520 lanking520 approved these changes

@zachgk zachgk Awaiting requested review from zachgk zachgk is a code owner

@frankfliu frankfliu Awaiting requested review from frankfliu

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@maaquib @lanking520