Skip to content

Commit

Permalink
Add default-nvidia-tensorrtllm variant
Browse files Browse the repository at this point in the history
  • Loading branch information
Atinoda committed Jul 26, 2024
1 parent 4676bf2 commit df01c0c
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,19 @@ RUN echo "Nvidia Extended (No AVX2)" > /variant.txt
ENV EXTRA_LAUNCH_ARGS=""
CMD ["python3", "/app/server.py"]

# Extended with TensorRT-LLM
FROM run_base AS default-nvidia-tensorrtllm
# Copy venv
COPY --from=app_nvidia_x $VIRTUAL_ENV $VIRTUAL_ENV
# Install TensorRT-LLM
RUN apt install -y openmpi-bin libopenmpi-dev
RUN pip3 install tensorrt_llm==0.10.0 -U --pre --extra-index-url https://pypi.nvidia.com
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
# Variant parameters
RUN echo "Nvidia Extended (TensorRT-LLM)" > /variant.txt
ENV EXTRA_LAUNCH_ARGS=""
CMD ["python3", "/app/server.py"]


# ROCM
# Base
Expand Down

0 comments on commit df01c0c

Please sign in to comment.