Add examples/gke/tgi-multi-lora-deployment
#102
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To also include modifications on `examples`, `Makefile`, and `docs/scripts` or anything under `docs/`
Still pending on the official release of the latest TGI DLC on Google Cloud
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
TODO(alvarobartt): automate this shortly as it's not so straight forward :/
alvarobartt
commented
Oct 1, 2024
alvarobartt
commented
Oct 1, 2024
…run,gke,vertex-ai}/README.md` Update example listing via `python scripts/internal/update_example_tables.py` so as to automatically generate those listings (respecting the previous content within the file, to alphabetically sort those as Vertex AI > GKE > Cloud Run, and some more fixes and improvements
To include the `examples/` prefix within the paths to the examples from the root directory i.e. in the `README.md` file
Fix `examples/<service>/examples/<service>/...`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds an example on how to deploy TGI via the Hugging Face DLC for Gemma2 using multiple LoRA adapters for inference on a single NVIDIA L4 instance.
The three adapters have been fine-tuned in collaboration with @Jofthomas and can be found under the https://hf.co/google-cloud-partnership org on the Hub (still private, datasets can be moved there too):
cc @philschmid for a potential Cloud Tuesday post, @Jofthomas for his presentation on the upcoming Gemma Developer Day in Tokyo, and @pagezyhf for visibility on the example itself
And kudos to @Narsil for support on reviewing and merging huggingface/text-generation-inference#2567, and @datavistics et al for their post at https://huggingface.co/blog/multi-lora-serving
Additionally
This PR also includes the
scripts/internal/update_example_tables.pyscript, which is being internally used to automatically generate the tables with the examples across the different files within this repository, to be automated on another PR.This would temporarily make things easier to maintain, as when adding a new example one can just
python scripts/internal/update_example_tables.pyin the meantime to update those.