Upstream sync 2024 07 01 #350

robertgshaw2-neuralmagic · 2024-07-01T02:29:40Z

Upstream sync 2024 07 01 (#350) - not a release candidate

SUMMARY:

Merge commits from vllm-project@6c916ac to vllm-project@8e0817c
Note that vllm-project@6c916ac is NOT included in this merge.

COMPARE vs UPSTREAM:

https://github.com/neuralmagic/nm-vllm/compare/upstream-sync-2024-07-01..8e0817c262da5c104f651a0ce4ac9ee0cd76f4ce

…rement (vllm-project#5772)

…ct#5783)

…vllm-project#5795)

Signed-off-by: kevin <[email protected]>

…el size than target model (vllm-project#5414)

Signed-off-by: Thomas Parnell <[email protected]>

…ommunication (vllm-project#5408) Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: Stephanie <[email protected]> Co-authored-by: Stephanie <[email protected]>

…#5832)

…llm-project#5801)

…roject#5560) Co-authored-by: Chih-Chieh-Yang <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]>

…Weight Loading) (vllm-project#5940) Co-authored-by: Robert Shaw <rshaw@neuralmagic>

…-project#6007)

… Spec Decode Worker (vllm-project#5348)

…ct#6041) Co-authored-by: Simon Mo <[email protected]>

…roupPool (vllm-project#6039)

vllm-project#6029)

derekk-nm

all checks passed. approving

andoorve and others added 30 commits July 1, 2024 02:01

[Distributed] Add send and recv helpers (vllm-project#5719)

54348ea

[Bugfix] Add phi3v resize for dynamic shape and fix torchvision requi…

5ae4430

…rement (vllm-project#5772)

[doc][faq] add warning to download models for every nodes (vllm-proje…

1601d82

…ct#5783)

[Doc] Add "Suggest edit" button to doc pages (vllm-project#5789)

20faeb6

[Doc] Add Phi-3-medium to list of supported models (vllm-project#5788)

62ecd68

[Bugfix] Fix FlexibleArgumentParser replaces _ with - for actual args (…

ca8bc83

…vllm-project#5795)

[ci] Remove aws template (vllm-project#5757)

91b2d1d

Signed-off-by: kevin <[email protected]>

[Doc] Add notice about breaking changes to VLMs (vllm-project#5818)

21450bc

[Speculative Decoding] Support draft model on different tensor-parall…

1d55e23

…el size than target model (vllm-project#5414)

[Misc] Remove useless code in cpu_worker (vllm-project#5824)

980c10b

[Core] Add fault tolerance for RayTokenizerGroupPool (vllm-project#…

3b261da

…5748)

[doc][distributed] add both gloo and nccl tests (vllm-project#5834)

8d6c12f

[CI/Build] Add unit testing for FlexibleArgumentParser (vllm-project#…

c3bc8c6

…5798)

[Misc] Update w4a16 compressed-tensors support to include w8a16 (…

a9e34b9

…vllm-project#5794)

[Hardware][TPU] Refactor TPU backend (vllm-project#5831)

21f69d1

resolved

ece7c7f

[Hardware][TPU] Raise errors for unsupported sampling params (vllm-pr…

e6935bd

…oject#5850)

[CI/Build] Add E2E tests for MLPSpeculator (vllm-project#5791)

81a21d2

Signed-off-by: Thomas Parnell <[email protected]>

[Bugfix] Fix assertion in NeuronExecutor (vllm-project#5841)

f9775e9

[Core] Refactor Worker and ModelRunner to consolidate control plane c…

fb41934

…ommunication (vllm-project#5408) Signed-off-by: Stephanie Wang <[email protected]> Signed-off-by: Stephanie <[email protected]> Co-authored-by: Stephanie <[email protected]>

[Misc][Doc] Add Example of using OpenAI Server with VLM (vllm-project…

ce9da79

…#5832)

[bugfix][distributed] fix shm broadcast when the queue size is full (v…

cb364ef

…llm-project#5801)

[Bugfix] Fix embedding to support 2D inputs (vllm-project#5829)

9744700

[Bugfix][TPU] Fix KV cache size calculation (vllm-project#5860)

2f7eba7

[CI/Build] Refactor image test assets (vllm-project#5821)

74952fd

[Kernel] Adding bias epilogue support for cutlass_scaled_mm (vllm-p…

1d1929b

…roject#5560) Co-authored-by: Chih-Chieh-Yang <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]>

[Frontend] Add tokenize/detokenize endpoints (vllm-project#5054)

5095252

[Hardware][TPU] Support parallel sampling & Swapping (vllm-project#5855)

1653293

[Bugfix][TPU] Fix CPU cache allocation (vllm-project#5869)

e423b2c

Support CPU inference with VSX PowerPC ISA (vllm-project#5652)

698f968

robertgshaw2-neuralmagic and others added 27 commits July 1, 2024 02:18

[ Misc ] Refactor w8a8 to use process_weights_after_load (Simplify …

27a711a

…Weight Loading) (vllm-project#5940) Co-authored-by: Robert Shaw <rshaw@neuralmagic>

format

53655b2

isort

07abe05

format

0f0fec4

updated skipping

1cc7c46

added skipping to compressed-tensors

a699814

updated

9a4be7f

format

b4eec34

[misc][cuda] use nvml to avoid accidentally cuda initialization (vllm…

08dedd5

…-project#6007)

[Speculative Decoding 2/2 ] Integrate typical acceptance sampler into…

dac4bb3

… Spec Decode Worker (vllm-project#5348)

[ CI ] Re-enable Large Model LM Eval (vllm-project#6031)

87a4288

[doc][misc] remove deprecated api server in doc (vllm-project#6037)

81e1c3e

[Misc] update benchmark backend for scalellm (vllm-project#6018)

2c3c43b

[doc][misc] further lower visibility of simple api server (vllm-proje…

cf4e758

…ct#6041) Co-authored-by: Simon Mo <[email protected]>

[Bugfix] Use RayActorError for older versions of Ray in RayTokenizerG…

9c7608c

…roupPool (vllm-project#6039)

[Bugfix] adding chunking mechanism to fused_moe to handle large inputs (

1b7245f

vllm-project#6029)

add FAQ doc under 'serving' (vllm-project#5946)

fa05042

[Bugfix][Doc] Fix Doc Formatting (vllm-project#6048)

484a2e3

Update conftest.py

99f1474

format

afb93b9

make _ImageAssets type hint py3.8 compatible

fcb4dd3

fixed sampler

ceaf019

switched to compressed tensors instrad of sparseml

655389d

formatted

206af82

fix sampler test again

cd2aa72

Merge branch 'main' into upstream-sync-2024-07-01

f43cb06

Remove errant __commit__ definition

7a45bfa

derekk-nm approved these changes Jul 3, 2024

View reviewed changes

robertgshaw2-neuralmagic merged commit 7144d20 into main Jul 3, 2024
28 checks passed

robertgshaw2-neuralmagic deleted the upstream-sync-2024-07-01 branch July 3, 2024 20:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upstream sync 2024 07 01 #350

Upstream sync 2024 07 01 #350

robertgshaw2-neuralmagic commented Jul 1, 2024 •

edited

Loading

derekk-nm left a comment

Upstream sync 2024 07 01 #350

Upstream sync 2024 07 01 #350

Conversation

robertgshaw2-neuralmagic commented Jul 1, 2024 • edited Loading

derekk-nm left a comment

Choose a reason for hiding this comment

robertgshaw2-neuralmagic commented Jul 1, 2024 •

edited

Loading