Code

Add KV-Cache int8 quant support#10354

Code

Open

YanyunDuanIEI wants to merge 15 commits intovllm-project:mainvllm-project/vllm:mainfrom YanyunDuanIEI:int8-kv-cacheYanyunDuanIEI/vllm:int8-kv-cacheCopy head branch name to clipboard

+11,873-133

Commits

Commits on Nov 15, 2024

• int8 kv-cache support

  Show description for b8e7779
  

  Yanyun Duan

  committed
  b8e7779

Commits on Nov 18, 2024

• Merge branch 'main' into int8-kv-cache

  YanyunDuanIEI

  authored
  6309f86

Commits on Nov 20, 2024

• Update dtype_fp8.cuh

  YanyunDuanIEI

  authored
  3b3d784

• Update quant_utils.cuh

  YanyunDuanIEI

  authored
  f09c559

• Update quant_utils.cuh

  YanyunDuanIEI

  authored
  03476e5

• Update cache_kernels.cu

  YanyunDuanIEI

  authored
  9a426c0

• Update attention_kernels.cuh

  YanyunDuanIEI

  authored
  2a67ed4

• Update paged_attention_v1.cu

  YanyunDuanIEI

  authored
  83b3d41

• Update paged_attention_v2.cu

  YanyunDuanIEI

  authored
  1056a5f

• Update layer.py

  YanyunDuanIEI

  authored
  e870565

• Update selector.py

  YanyunDuanIEI

  authored
  95effce

• Update config.py

  YanyunDuanIEI

  authored
  4e8ddb6

• Update utils.py

  YanyunDuanIEI

  authored
  cf98d49

• Update model_runner.py

  YanyunDuanIEI

  authored
  c5da7f5

• merge the k_scale/v_scale and k_scaling_factor/v_scaling_factor into tensors

  YanyunDuanIEI

  committed
  36afe99