Code

Use continuous batching by default#882

Code

Closed

Wovchena wants to merge 90 commits intoopenvinotoolkit:masteropenvinotoolkit/openvino.genai:masterfrom Wovchena:use-continuos-batching-by-defaultWovchena/openvino.genai-public:use-continuos-batching-by-defaultCopy head branch name to clipboard

+27-4

Commits

Commits on Sep 19, 2024

• Use continuous batching by default

  Wovchena

  committed
  ec5f305

  Copy full SHA for ec5f305

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  dd7a5cf

  Copy full SHA for dd7a5cf

Commits on Sep 20, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  229b7c5

  Copy full SHA for 229b7c5

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  41d1fe7

  Copy full SHA for 41d1fe7

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  36150c4

  Copy full SHA for 36150c4

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  4a4a09e

  Copy full SHA for 4a4a09e

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  1a58b5e

  Copy full SHA for 1a58b5e

• Reorder cout

  Wovchena

  committed
  90d81e6

  Copy full SHA for 90d81e6

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  6dc43a3

  Copy full SHA for 6dc43a3

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  03e2f32

  Copy full SHA for 03e2f32

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  e561e93

  Copy full SHA for e561e93

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  37ea2ad

  Copy full SHA for 37ea2ad

Commits on Sep 23, 2024

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  b62aee9

  Copy full SHA for b62aee9

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  07505b3

  Copy full SHA for 07505b3

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  e078818

  Copy full SHA for e078818

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  001d3a0

  Copy full SHA for 001d3a0

Commits on Sep 25, 2024

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  a0a964f

  Copy full SHA for a0a964f

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  3cb2105

  Copy full SHA for 3cb2105

• Limit max new tokens.

  popovaan

  authored
  40ea516

  Copy full SHA for 40ea516

• Fixed error

  popovaan

  authored
  193df7e

  Copy full SHA for 193df7e

Commits on Sep 30, 2024

• Clean up

  Wovchena

  committed
  1704548

  Copy full SHA for 1704548

• Default destructors

  Wovchena

  committed
  086c7b8

  Copy full SHA for 086c7b8

• Merge branch 'master' into use-continuos-batching-by-default

  Wovchena

  committed
  607d90d

  Copy full SHA for 607d90d

• Default ~PerfTime

  Wovchena

  committed
  741c13b

  Copy full SHA for 741c13b

Commits on Oct 10, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  06d1b1e

  Copy full SHA for 06d1b1e

Commits on Oct 11, 2024

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  8d7d39d

  Copy full SHA for 8d7d39d

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  c4e8e05

  Copy full SHA for c4e8e05

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  8116342

  Copy full SHA for 8116342

• Update src/cpp/src/llm_pipeline.cpp

  andrei-kochin

  authored
  b87d0f6

  Copy full SHA for b87d0f6

• CB: fix deadlock (#71)

  Wovchena

  authored
  1806fa0

  Copy full SHA for 1806fa0

Commits on Oct 12, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  c9dc107

  Copy full SHA for c9dc107

• Increase timeouts for tests

  ilya-lavrenov

  authored
  4bbcd0e

  Copy full SHA for 4bbcd0e

• Update causal_lm_cpp.yml

  ilya-lavrenov

  authored
  743e018

  Copy full SHA for 743e018

• Use split_core_complile_config for CB

  ilya-lavrenov

  authored
  cfccefa

  Copy full SHA for cfccefa

• Update causal_lm_cpp.yml

  ilya-lavrenov

  authored
  03965d6

  Copy full SHA for 03965d6

Commits on Oct 13, 2024

• Drop request if it's aborted by streamer

  ilya-lavrenov

  authored
  784c331

  Copy full SHA for 784c331

• Update src/cpp/src/continuous_batching_impl.cpp

  ilya-lavrenov

  authored
  93b8c38

  Copy full SHA for 93b8c38

Commits on Oct 14, 2024

• Drop request in case of exceptions, etc

  ilya-lavrenov

  authored
  043d842

  Copy full SHA for 043d842

• Turned off prefix caching

  ilya-lavrenov

  authored
  fdad63c

  Copy full SHA for fdad63c

• Apply suggestions from code review

  ilya-lavrenov

  authored
  a21f725

  Copy full SHA for a21f725

• Apply suggestions from code review

  ilya-lavrenov

  authored
  a66be9e

  Copy full SHA for a66be9e

• Update continuous_batching_impl.cpp

  ilya-lavrenov

  authored
  82fceb5

  Copy full SHA for 82fceb5

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  a246c1c

  Copy full SHA for a246c1c

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  4ee8f12

  Copy full SHA for 4ee8f12

• Apply suggestions from code review

  ilya-lavrenov

  authored
  73a8872

  Copy full SHA for 73a8872

• Apply suggestions from code review

  ilya-lavrenov

  authored
  4019678

  Copy full SHA for 4019678

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  ed7668e

  Copy full SHA for ed7668e

• Update causal_lm_cpp.yml

  ilya-lavrenov

  authored
  feae546

  Copy full SHA for feae546

• Apply suggestions from code review

  ilya-lavrenov

  authored
  5bdf779

  Copy full SHA for 5bdf779

Commits on Oct 16, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  e3f2949

  Copy full SHA for e3f2949

Commits on Oct 17, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  f1a9ab5

  Copy full SHA for f1a9ab5

Commits on Oct 18, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  debbdd4

  Copy full SHA for debbdd4

Commits on Oct 21, 2024

• Apply suggestions from code review

  ilya-lavrenov

  authored
  7827199

  Copy full SHA for 7827199

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  3de57d3

  Copy full SHA for 3de57d3

Commits on Oct 22, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  42d26df

  Copy full SHA for 42d26df

• Apply suggestions from code review

  ilya-lavrenov

  authored
  467ab86

  Copy full SHA for 467ab86

Commits on Oct 24, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  5b7f94a

  Copy full SHA for 5b7f94a

Commits on Oct 30, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  5a391a8

  Copy full SHA for 5a391a8

Commits on Nov 6, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  b4a4174

  Copy full SHA for b4a4174

Commits on Nov 7, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  c3d55eb

  Copy full SHA for c3d55eb

Commits on Nov 11, 2024

• Update linux.yml

  ilya-lavrenov

  authored
  9fad1d2

  Copy full SHA for 9fad1d2

• Update windows.yml

  ilya-lavrenov

  authored
  35f4ff2

  Copy full SHA for 35f4ff2

• Update mac.yml

  ilya-lavrenov

  authored
  ad78839

  Copy full SHA for ad78839

• Update linux.yml

  ilya-lavrenov

  authored
  c5201e4

  Copy full SHA for c5201e4

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  d8d397a

  Copy full SHA for d8d397a

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  dc2673c

  Copy full SHA for dc2673c

• Update llm_pipeline.cpp

  ilya-lavrenov

  authored
  4dd053c

  Copy full SHA for 4dd053c

Commits on Nov 12, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  31ea070

  Copy full SHA for 31ea070

• Update llm_pipeline.cpp

  ilya-lavrenov

  authored
  7847060

  Copy full SHA for 7847060

• Apply suggestions from code review

  ilya-lavrenov

  authored
  d11db7e

  Copy full SHA for d11db7e

• Update llm_pipeline.cpp

  ilya-lavrenov

  authored
  9acf368

  Copy full SHA for 9acf368

• Update causal_lm_cpp.yml

  ilya-lavrenov

  authored
  3c835af

  Copy full SHA for 3c835af

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  046c017

  Copy full SHA for 046c017

Commits on Nov 21, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  badaa5b

  Copy full SHA for badaa5b

Commits on Nov 22, 2024

• Fix validation

  Wovchena

  committed
  c807011

  Copy full SHA for c807011

Commits on Nov 25, 2024

• Update linux.yml

  ilya-lavrenov

  authored
  7097164

  Copy full SHA for 7097164

• Update windows.yml

  ilya-lavrenov

  authored
  3538bbe

  Copy full SHA for 3538bbe

Commits on Nov 26, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  andrei-kochin

  authored
  9eb4176

  Copy full SHA for 9eb4176

• Update linux.yml

  ilya-lavrenov

  authored
  a510e77

  Copy full SHA for a510e77

• Update windows.yml

  ilya-lavrenov

  authored
  eb0b0f4

  Copy full SHA for eb0b0f4

Commits on Dec 4, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  d223042

  Copy full SHA for d223042

Commits on Dec 17, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  cbd1ba8

  Copy full SHA for cbd1ba8

Commits on Dec 18, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  62304ef

  Copy full SHA for 62304ef

Commits on Dec 23, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  44a9249

  Copy full SHA for 44a9249

Commits on Dec 24, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  f7b2ac8

  Copy full SHA for f7b2ac8

Commits on Dec 25, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  5f1417a

  Copy full SHA for 5f1417a

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  131f782

  Copy full SHA for 131f782

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  e517917

  Copy full SHA for e517917

Commits on Dec 26, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  17d4333

  Copy full SHA for 17d4333

Commits on Dec 28, 2024

• Merge branch 'master' into use-continuos-batching-by-default

  ilya-lavrenov

  authored
  ca58d3a

  Copy full SHA for ca58d3a