Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[neo] Allow setting GPU memory-related LMI options in sharding jobs #2546

Merged
merged 4 commits into from
Nov 12, 2024
Merged

[neo] Allow setting GPU memory-related LMI options in sharding jobs #2546

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e215b7d
[neo] Increase gpu_memory_utilization of lmi-dist engine in sharding …
ethnzhng Nov 12, 2024
bca3148
[neo] Support option.gpu_memory_utilization for sharding jobs
ethnzhng Nov 12, 2024
0d1b11d
[neo] Support option.enforce_eager for sharding jobs
ethnzhng Nov 12, 2024
1776eb6
Add max_num_seqs & max_model_len
ethnzhng Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions serving/docker/partition/sm_neo_shard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ def shard_lmi_dist_model(self, input_dir: str, output_dir: str,
enforce_eager=True,
disable_custom_all_reduce=True,
distributed_executor_backend="mp",
gpu_memory_utilization=0.99,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this may allow for the sharding of 405b on a p4de, will the resulting converted artifacts even be usable by the customer on the same instance? Do we also provide the necessary configs in the output to ensure the model can be loaded on the instance for inference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct, it would have to be set to the same value during runtime. I updated the PR to simply allow customers to specify option.gpu_memory_utilization during sharding jobs, and with #2545 that value will be propagated to the serving.properties. Same with option.enforce_eager, since it also affects GPU mem utilization.

Thus customers wanting to use a model/instance combination which requires a change to gpu_memory_utilization or enforce_eager will have to be aware of this at the time of converting the artifacts.

Copy link
Contributor

@siddvenk siddvenk Nov 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, this sounds like the right approach!

Lets also add:

  • option.max_rolling_batch_size -> max_num_seqs
  • option.model_max_len -> model_max_len

That should be good for this PR - we should look at the engine configs and determine the full set, but with these 4 (in total) we can do the rest in a fast folllow

)
engine = engine_from_args(engine_args)

Expand Down
Loading