Squashed commit of the following:

commit adec0e0 Author: Irina Efode <[email protected]> Date: Tue Jun 11 14:32:45 2024 +0400 Remove extra token desc commit a64f30a Author: Irina Efode <[email protected]> Date: Tue Jun 11 13:36:01 2024 +0400 Working sampler commit 05048ff Author: Irina Efode <[email protected]> Date: Tue Jun 11 13:23:43 2024 +0400 check commit e349418 Merge: bfaa55a 0b1ce98 Author: Irina Efode <[email protected]> Date: Mon Jun 10 23:11:58 2024 +0400 Merge remote-tracking branch 'ilavrenov_upstream/ct-beam-search' into penalties commit 0b1ce98 Merge: 16d857e 2da1556 Author: Ilya Lavrenov <[email protected]> Date: Mon Jun 10 18:52:20 2024 +0400 Merge pull request openvinotoolkit#21 from iefode/n_support Support num_return_seq for multinomial case commit bfaa55a Author: Irina Efode <[email protected]> Date: Mon Jun 10 17:42:01 2024 +0400 Fix tests commit fa0efb6 Author: Irina Efode <[email protected]> Date: Mon Jun 10 16:41:04 2024 +0400 Config tests commit 7551303 Author: Irina Efode <[email protected]> Date: Mon Jun 10 15:34:14 2024 +0400 Implement LogitTransformers. todo config check commit 16d857e Merge: 76148c5 1ee4f38 Author: Ilya Lavrenov <[email protected]> Date: Mon Jun 10 10:41:27 2024 +0200 Merge remote-tracking branch 'upstream/master' into ct-beam-search commit 1ee4f38 Author: guozhong wang <[email protected]> Date: Sun Jun 9 18:26:57 2024 +0800 Add option --prompt_index (openvinotoolkit#481) Run the corresponding prompt according to the option prompt index commit 9902928 Author: Pavel Esir <[email protected]> Date: Fri Jun 7 20:57:47 2024 +0200 Generate pipeline (openvinotoolkit#334) LLM return logits with probabilities of each token, these probabilities can be converted to tokens/words with different technics: greedy decoding, beam search decoding, random sampling, etc. This requires writing user unfriendly post-processing even for the simplest scenario of greedy decoding. In order to make live easier we we combined all decoding scenarios into a single function call, where the decoding method and parameters are specified by arguments. In this PR we provide a user friendly API for text generation inspired by `generate` method from HuggingFace transformers library. - [x] enable calling tokenizers/detokenizers from LLMPipeline - [ ] add callback for streaming mode - done partially, need to improve - [x] rewritten samples with the current approach: [causal_lm/cpp/generate_pipeline/generate_sample.cpp#L73-L83](https://github.com/pavel-esir/openvino.genai/blob/generate_pipeline/text_generation/causal_lm/cpp/generate_pipeline/generate_sample.cpp#L73-L83) - [x] Multibatch greedy decoding - [ ] Speculative decoding - [ ] Grouped Beam Search decoding: ready for batch 1, need to rebase multibatch support after merging openvinotoolkit#349 - [x] Random sampling Example 1: Greedy search generation ``` LLMPipeline pipe(model_path, device); // Will try to load config from generation_config.json. // but if not found default velues for gready search will be used GenerationConfig config = pipe.generation_config(); cout << pipe(prompt, config.max_new_tokens(20)); ``` Example 2: TextStreaming mode ``` LLMPipeline pipe(model_path, device); GenerationConfig config = pipe.generation_config(); auto text_streamer = TextStreamer{pipe}; auto text_streamer_callback = [&text_streamer](std::vector<int64_t>&& tokens, LLMPipeline& pipe){ text_streamer.put(tokens[0]); }; pipe(prompt, config.max_new_tokens(20).set_callback(text_streamer_callback)); text_streamer.end(); ``` CVS-132907 CVS-137920 --------- Co-authored-by: Wovchena <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]> Co-authored-by: Alexander Suvorov <[email protected]> Co-authored-by: Yaroslav Tarkan <[email protected]> Co-authored-by: Xiake Sun <[email protected]> Co-authored-by: wenyi5608 <[email protected]> Co-authored-by: Ekaterina Aidova <[email protected]> Co-authored-by: guozhong wang <[email protected]> Co-authored-by: Chen Peter <[email protected]> commit 2da1556 Author: Irina Efode <[email protected]> Date: Thu Jun 6 19:24:45 2024 +0400 library/src/continuous_batching_pipeline.cpp commit 7b48fa4 Author: Irina Efode <[email protected]> Date: Thu Jun 6 15:03:05 2024 +0400 enable streaming for greedy commit 5c601e0 Author: Irina Efode <[email protected]> Date: Thu Jun 6 13:29:47 2024 +0400 Comments commit 4f73d36 Author: Irina Efode <[email protected]> Date: Wed Jun 5 22:46:04 2024 +0400 Enable frequency and presence penalties commit 5e49c46 Author: Irina Efode <[email protected]> Date: Wed Jun 5 11:56:31 2024 +0400 Fix python tests commit eb4a219 Author: Irina Efode <[email protected]> Date: Tue Jun 4 22:38:43 2024 +0400 fix assert place commit f4d8461 Author: Irina Efode <[email protected]> Date: Tue Jun 4 22:22:37 2024 +0400 Correct accumulation commit 55448a1 Merge: 1128792 76148c5 Author: Irina Efode <[email protected]> Date: Tue Jun 4 18:56:42 2024 +0400 Merge remote-tracking branch 'ilavrenov_upstream/ct-beam-search' into n_support commit 1128792 Author: Irina Efode <[email protected]> Date: Tue Jun 4 18:52:38 2024 +0400 test commit e245041 Author: Irina Efode <[email protected]> Date: Tue Jun 4 18:52:03 2024 +0400 Apply comments commit 561cde0 Author: guozhong wang <[email protected]> Date: Tue Jun 4 16:27:08 2024 +0800 using sdpa for statble diffusion (openvinotoolkit#458) Co-authored-by: Chen Peter <[email protected]> commit 04510d4 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Jun 3 17:37:41 2024 +0000 Bump optimum[openvino] from 1.19.2 to 1.20.0 in /text_generation/causal_lm/cpp (openvinotoolkit#467) commit db4a88f Merge: e5d33f5 b63bda2 Author: Irina Efode <[email protected]> Date: Mon Jun 3 13:17:32 2024 +0400 Merge remote-tracking branch 'ilavrenov_upstream/ct-beam-search' into n_support commit e5d33f5 Merge: fe29df9 bcdcefc Author: Irina Efode <[email protected]> Date: Fri May 31 14:11:13 2024 +0400 Merge remote-tracking branch 'ilavrenov_upstream/ct-beam-search' into n_support commit fe29df9 Author: Irina Efode <[email protected]> Date: Fri May 31 14:06:51 2024 +0400 Tests + Readme commit 7af72aa Author: Irina Efode <[email protected]> Date: Wed May 29 15:16:23 2024 +0400 Squashed commit of the following: commit 28af66d Author: Anastasiia Pnevskaia <[email protected]> Date: Tue May 28 15:40:15 2024 +0200 Added cache_size to python binding of scheduler config. commit 65a793a Author: Anastasiia Pnevskaia <[email protected]> Date: Tue May 28 15:12:16 2024 +0200 Fixed tests. commit 033558e Author: Irina Efode <[email protected]> Date: Wed May 29 00:40:48 2024 +0400 One more change commit dbae0bf Merge: f992591 2c2799f Author: Irina Efode <[email protected]> Date: Wed May 29 00:38:52 2024 +0400 Merge master, without py tests commit a5b14c7 Author: Lyalyushkin Nikolay <[email protected]> Date: Tue May 28 16:15:42 2024 +0200 grammar corrector support in WWB (openvinotoolkit#462) This PR introduces support for `AutoForSeq2SeqLM` models in WWB. Previously, WWB only supported `AutoForCasualLM`, assuming that the `generate` method copies the prompt to the output. But AutoForSeq2SeqLM generates output differently: there is no copy of the prompt, and it directly generates output. The fix was checked on the [example](https://gist.github.com/ljaljushkin/5a489a27cd0020ddbd42ea7ae54be688). It evaluates grammar correction with Seq2Seq model using WWB. commit f992591 Author: Irina Efode <[email protected]> Date: Tue May 28 17:39:17 2024 +0400 tmp commit 7e771f1 Author: Liwenke <[email protected]> Date: Tue May 28 15:24:15 2024 +0800 Note for wikitext data set connection issue (openvinotoolkit#452) Co-authored-by: Chen Peter <[email protected]> commit 24ef06e Author: guozhong wang <[email protected]> Date: Tue May 28 14:23:19 2024 +0800 Force to generate more tokens (openvinotoolkit#457) commit 1ed7539 Author: guozhong wang <[email protected]> Date: Tue May 28 09:44:45 2024 +0800 Correct flan-t5 output size (openvinotoolkit#451) openvinotoolkit#358 --------- Co-authored-by: Chen Peter <[email protected]> commit b5a9f28 Author: Irina Efode <[email protected]> Date: Mon May 27 23:48:03 2024 +0400 Extend in beam support commit edc53e5 Author: Irina Efode <[email protected]> Date: Fri May 24 17:59:48 2024 +0400 remove extra commit 9038308 Author: Irina Efode <[email protected]> Date: Fri May 24 16:20:13 2024 +0400 Improve multinomial commit c453e3e Author: Irina Efode <[email protected]> Date: Fri May 24 15:42:48 2024 +0400 Support num_return_seq for multinomial case commit e6f05c6 Author: guozhong wang <[email protected]> Date: Thu May 23 11:36:50 2024 +0800 Output median min and avg values to csv (openvinotoolkit#450) Co-authored-by: Chen Peter <[email protected]> commit 25909cc Author: guozhong wang <[email protected]> Date: Thu May 23 11:12:27 2024 +0800 verify beam search 1st token optimization (openvinotoolkit#426) The minimum version of transformers to get 1st and 2nd tokens latency is v4.40-release. commit 03e78fe Author: Chen Peter <[email protected]> Date: Wed May 22 13:06:11 2024 +0800 Revert "Force to generate "inference count" tokens" (openvinotoolkit#455) Reverts openvinotoolkit#289 to unblock the release. Since it causes the performance regression of some models. (WIP to investigate the reason) commit 05a0f36 Author: Ekaterina Aidova <[email protected]> Date: Tue May 21 20:33:26 2024 +0400 fix path based configuration (openvinotoolkit#456) commit 41b07d3 Author: Ekaterina Aidova <[email protected]> Date: Fri May 17 06:20:18 2024 +0400 Fix md5 hash for env that does not support usedforsecurity arg (openvinotoolkit#445) I got an error running benchmarking on my working machine (python3.8, ubuntu20) due to unsupported args for hashlib. ``` [ ERROR ] An exception occurred [ INFO ] Traceback (most recent call last): File "benchmark.py", line 532, in main iter_data_list, pretrain_time = CASE_TO_BENCH[model_args['use_case']](model_path, framework, args.device, model_args, args.num_iters) File "benchmark.py", line 194, in run_text_generation_benchmark run_text_generation(input_text, num, model, tokenizer, args, iter_data_list, warmup_md5, prompt_idx, bench_hook, model_precision, proc_id) File "benchmark.py", line 131, in run_text_generation result_md5_list.append(hashlib.md5(result_text.encode(), usedforsecurity=False).hexdigest()) TypeError: openssl_md5() takes at most 1 argument (2 given) ``` Based on this [StackOverflow issue](https://stackoverflow.com/questions/54717862/how-do-i-know-if-the-usedforsecurity-flag-is-supported-by-hashlib-md5), not all clients support this argument and usage hashlib.new("md5") vs hashlib.md5 should be safe for usage in both cases commit d473e96 Author: guozhong wang <[email protected]> Date: Fri May 17 10:09:27 2024 +0800 output no hook data warning when it is text gen model (openvinotoolkit#449) commit cad3abb Author: guozhong wang <[email protected]> Date: Thu May 16 17:28:49 2024 +0800 Fix an attempt to add a string value to a numerical value (openvinotoolkit#447) commit 93f7670 Author: Ekaterina Aidova <[email protected]> Date: Thu May 16 11:49:08 2024 +0400 update optimum intel commit in llm bench (openvinotoolkit#444) commit d73346c Author: Yaroslav Tarkan <[email protected]> Date: Wed May 15 14:24:30 2024 +0300 Fix noise images generated for '--num' > 1 in Stable Diffusion sample (openvinotoolkit#441) Fixes openvinotoolkit#405
iefode · Jun 11, 2024 · e5aa6b9 · e5aa6b9
1 parent 86cc68f
commit e5aa6b9
Show file tree

Hide file tree

Showing 103 changed files with 6,113 additions and 2,068 deletions.
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -1,5 +1,9 @@
 version: 2
 updates:
+  - package-ecosystem: "pip"
+    directory: "./"
+    schedule:
+      interval: "weekly"
   - package-ecosystem: "pip"
     directory: "image_generation/stable_diffusion_1_5/cpp/scripts/"
     schedule:
@@ -8,6 +12,10 @@ updates:
     directory: "image_generation/lcm_dreamshaper_v7/cpp/scripts/"
     schedule:
       interval: "weekly"
+  - package-ecosystem: "pip"
+    directory: "./tests/python_tests/"
+    schedule:
+      interval: "weekly"
   - package-ecosystem: "pip"
     directory: "text_generation/causal_lm/cpp/"
     schedule:

diff --git a/.github/workflows/causal_lm_cpp.yml b/.github/workflows/causal_lm_cpp.yml
diff --git a/.github/workflows/genai_package.yml b/.github/workflows/genai_package.yml
@@ -0,0 +1,98 @@
+name: genai_package
+on: pull_request
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}
+  cancel-in-progress: true
+jobs:
+  ubuntu_genai_package:
+    strategy:
+      matrix:
+        build-type: [Release, Debug]
+    runs-on: ubuntu-20.04
+    env:
+      CMAKE_BUILD_PARALLEL_LEVEL: null
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.8
+      - run: mkdir ./ov/
+      - run: curl https://storage.openvinotoolkit.org/repositories/openvino/packages/pre-release/2024.2.0rc1/linux/l_openvino_toolkit_ubuntu20_2024.2.0.dev20240524_x86_64.tgz | tar --directory ./ov/ --strip-components 1 -xz
+      - run: sudo ./ov/install_dependencies/install_openvino_dependencies.sh
+      - run: source ./ov/setupvars.sh && cmake -DCMAKE_BUILD_TYPE=${{ matrix.build-type }} -S ./ -B ./build/
+      - run: source ./ov/setupvars.sh && cmake --build ./build/ --config ${{ matrix.build-type }} --target package -j
+      - run: source ./ov/setupvars.sh && cmake --install ./build/ --config ${{ matrix.build-type }} --prefix ov
+      - run: ov/samples/cpp/build_samples.sh -i ${{ github.workspace }}/s\ pace
+        if: ${{ 'Release' == matrix.build-type }}  # build_samples enforces Release build
+      - run: source ./ov/setupvars.sh && cmake -DCMAKE_BUILD_TYPE=${{ matrix.build-type }} -S ./ov/samples/cpp/ -B ./samples\ build/ && cmake --build ./samples\ build/ --config ${{ matrix.build-type }} -j && cmake --install ./samples\ build/ --config ${{ matrix.build-type }} --component samples_bin --prefix s\ pace
+        if: ${{ 'Release' != matrix.build-type }}
+      - run: source ./ov/setupvars.sh && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release
+      - run: source ./ov/setupvars.sh && python -m pip install --upgrade-strategy eager -r ./samples/cpp/requirements.txt
+      - run: source ./ov/setupvars.sh && optimum-cli export openvino --trust-remote-code --weight-format fp16 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0
+      - run: source ./ov/setupvars.sh && timeout 50s ${{ github.workspace }}/s\ pace/samples_bin/greedy_causal_lm ./TinyLlama-1.1B-Chat-v1.0/ ""
+
+  macos_genai_package:
+    strategy:
+      matrix:
+        build-type: [Release, Debug]
+    runs-on: macos-12
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.8
+      - run: mkdir ./ov/
+      - run: curl https://storage.openvinotoolkit.org/repositories/openvino/packages/pre-release/2024.2.0rc2/macos/m_openvino_toolkit_macos_12_6_2024.2.0.dev20240529_x86_64.tgz | tar --directory ./ov/ --strip-components 1 -xz
+      - run: brew install coreutils scons
+      - run: source ./ov/setupvars.sh && cmake -DCMAKE_BUILD_TYPE=${{ matrix.build-type }} -S ./ -B ./build/
+      - run: source ./ov/setupvars.sh && cmake --build ./build/ --config ${{ matrix.build-type }} --target package -j
+      - run: source ./ov/setupvars.sh && cmake --install ./build/ --config ${{ matrix.build-type }} --prefix ov
+      - run: ov/samples/cpp/build_samples.sh -i ${{ github.workspace }}/s\ pace
+        if: ${{ 'Release' == matrix.build-type }}  # build_samples enforces Release build
+      - run: source ./ov/setupvars.sh && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release
+        if: ${{ 'Release' == matrix.build-type }}
+      - run: source ./ov/setupvars.sh && python -m pip install --upgrade-strategy eager -r ./samples/cpp/requirements.txt
+        if: ${{ 'Release' == matrix.build-type }}
+      - run: source ./ov/setupvars.sh && optimum-cli export openvino --trust-remote-code --weight-format fp16 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0
+        if: ${{ 'Release' == matrix.build-type }}
+      - run: source ./ov/setupvars.sh && timeout 50s ${{ github.workspace }}/s\ pace/samples_bin/greedy_causal_lm ./TinyLlama-1.1B-Chat-v1.0/ ""
+        if: ${{ 'Release' == matrix.build-type }}
+
+  windows_genai_package:
+    strategy:
+      matrix:
+        build-type: [Release, Debug]
+    runs-on: windows-latest
+    env:
+      CMAKE_BUILD_PARALLEL_LEVEL: null
+    defaults:
+      run:
+        shell: cmd
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.8
+      - run: curl --output ov.zip https://storage.openvinotoolkit.org/repositories/openvino/packages/pre-release/2024.2.0rc1/windows/w_openvino_toolkit_windows_2024.2.0.dev20240524_x86_64.zip
+      - run: unzip ov.zip
+      # Shorten the next setupvars calls.
+      - run: mklink /D ov w_openvino_toolkit_windows_2024.2.0.dev20240524_x86_64
+      - run: call ov\setupvars.bat && cmake -DCMAKE_BUILD_TYPE=${{ matrix.build-type }} -S ./ -B ./build/
+      - run: call ov\setupvars.bat && cmake --build ./build/ --config ${{ matrix.build-type }} --target package -j
+      - run: call ov\setupvars.bat && cmake --install ./build/ --config ${{ matrix.build-type }} --prefix ov
+      - run: call ov\samples\cpp\build_samples_msvc.bat -i "${{ github.workspace }}/samples_install"
+        if: ${{ false && 'Release' == matrix.build-type }}  # build_samples enforces Release build
+      - run: call ov\setupvars.bat && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release
+        if: ${{ false && 'Release' == matrix.build-type }}
+      - run: call ov\setupvars.bat && python -m pip install --upgrade-strategy eager -r ./samples/cpp/requirements.txt
+        if: ${{ false && 'Release' == matrix.build-type }}
+      - run: call ov\setupvars.bat && optimum-cli export openvino --trust-remote-code --weight-format fp16 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0
+        if: ${{ false && 'Release' == matrix.build-type }}
+      - run: call ov\setupvars.bat && "${{ github.workspace }}/samples_install/samples_bin/greedy_causal_lm" .\TinyLlama-1.1B-Chat-v1.0\ ""
+        if: ${{ false && 'Release' == matrix.build-type }}
diff --git a/.github/workflows/genai_python_lib.yml b/.github/workflows/genai_python_lib.yml
@@ -0,0 +1,85 @@
+name: genai_python_lib
+on: pull_request
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}
+  cancel-in-progress: true
+jobs:
+  ubuntu_genai_python_lib:
+    # A tokenizers' dependency fails to compile on ubuntu-20 n CenOS7 env.
+    runs-on: ubuntu-22.04
+    env:
+      # A tokenizers' dependency fails to compile with Ninja in CenOS7 env.
+      CMAKE_GENERATOR: Unix Makefiles
+      CMAKE_BUILD_PARALLEL_LEVEL: null
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.8
+      - run: mkdir ./ov/
+      # Install CentOS7 instead of Ubuntu to match PyPI distribution ABI.
+      - run: curl https://storage.openvinotoolkit.org/repositories/openvino/packages/pre-release/2024.2.0rc1/linux/l_openvino_toolkit_centos7_2024.2.0.dev20240524_x86_64.tgz | tar --directory ./ov/ --strip-components 1 -xz
+      - run: sudo ./ov/install_dependencies/install_openvino_dependencies.sh
+      - run: source ./ov/setupvars.sh && cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build/
+      - run: source ./ov/setupvars.sh && cmake --build ./build/ --config Release -j
+      # GitHub Actions already provides what is listed in ./requirements-build.txt but the internal
+      # build system doesn't. Install ./requirements-build.txt to detect possible conflicts.
+      - run: source ./ov/setupvars.sh && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] -r ./requirements-build.txt -r ./tests/python_tests/requirements.txt --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release --upgrade-strategy eager
+      - run: source ./ov/setupvars.sh && PYTHONPATH=./build/:$PYTHONPATH python -m pytest ./tests/python_tests/test_generate_api.py -m precommit
+      - run: source ./ov/setupvars.sh && python -m pip install . --config-settings=build-dir="build" --verbose
+      - run: python -m pytest ./tests/python_tests/test_generate_api.py -m precommit
+
+  macos_genai_python_lib:
+    runs-on: macos-12
+    env:
+      # A tokenizers' dependency fails to compile with Ninja.
+      CMAKE_GENERATOR: Unix Makefiles
+      CMAKE_BUILD_PARALLEL_LEVEL: null
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.8
+      - run: mkdir ./ov/
+      - run: curl https://storage.openvinotoolkit.org/repositories/openvino/packages/pre-release/2024.2.0rc2/macos/m_openvino_toolkit_macos_12_6_2024.2.0.dev20240529_x86_64.tgz | tar --directory ./ov/ --strip-components 1 -xz
+      - run: brew install coreutils scons
+      - run: source ./ov/setupvars.sh && cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build/
+      - run: source ./ov/setupvars.sh && cmake --build ./build/ --config Release -j
+      # GitHub Actions already provides what is listed in ./requirements-build.txt but the internal
+      # build system doesn't. Install ./requirements-build.txt to detect possible conflicts.
+      - run: source ./ov/setupvars.sh && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] -r ./requirements-build.txt -r ./tests/python_tests/requirements.txt --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release --upgrade-strategy eager
+      - run: source ./ov/setupvars.sh && PYTHONPATH=./build/:$PYTHONPATH python -m pytest ./tests/python_tests/test_generate_api.py -m precommit
+      - run: source ./ov/setupvars.sh && python -m pip install . --config-settings=build-dir="build" --verbose
+      - run: python -c "from openvino_genai import LLMPipeline"
+      - run: python -m pytest ./tests/python_tests/test_generate_api.py -m precommit
+
+  windows_genai_python_lib:
+    if: false
+    runs-on: windows-latest
+    env:
+      CMAKE_BUILD_PARALLEL_LEVEL: null
+    defaults:
+      run:
+        shell: cmd
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.8
+      - run: curl --output ov.zip https://storage.openvinotoolkit.org/repositories/openvino/packages/pre-release/2024.2.0rc1/windows/w_openvino_toolkit_windows_2024.2.0.dev20240524_x86_64.zip
+      - run: unzip ov.zip
+      # Shorten the next setupvars calls.
+      - run: mklink /D ov w_openvino_toolkit_windows_2024.2.0.dev20240524_x86_64
+      - run: call ./ov/setupvars.bat && cmake -DCMAKE_BUILD_TYPE=Release -S ./ -B ./build/
+      - run: call ./ov/setupvars.bat && cmake --build ./build/ --config Release -j
+      - run: call ./ov/setupvars.bat && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] -r ./requirements-build.txt -r ./tests/python_tests/requirements.txt --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release --upgrade-strategy eager
+      # cmd evaluates variables in a different way. Setting PYTHONPATH before setupvars.bat instead of doing that after solves that.
+      - run: set "PYTHONPATH=./build/" && call ./ov/setupvars.bat && python -m pytest ./tests/python_tests/test_generate_api.py -m precommit
+      - run: call ./ov/setupvars.bat && python -m pip install . --config-settings=build-dir="build" --verbose
+      - run: python -m pytest ./tests/python_tests/test_generate_api.py -m precommit
diff --git a/.github/workflows/lcm_dreamshaper_cpp.yml b/.github/workflows/lcm_dreamshaper_cpp.yml
@@ -40,15 +40,15 @@ jobs:
         run: |
           conda activate openvino_lcm_cpp
           conda update -c conda-forge --all
-          conda install -c conda-forge openvino=2024.1.0 c-compiler cxx-compiler git make cmake
+          conda install -c conda-forge -c conda-forge/label/openvino_dev openvino==2024.2.0.dev20240513 c-compiler cxx-compiler git make cmake
           conda env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
       
       - name: Install python dependencies
         working-directory: ${{ env.working_directory }}
         run: |
           conda activate openvino_lcm_cpp
-          python -m pip install -r requirements.txt
           python -m pip install ../../../thirdparty/openvino_tokenizers/[transformers]
+          python -m pip install -r requirements.txt
           
       - name: Download and convert model and tokenizer
         working-directory: ${{ env.working_directory }}
@@ -85,15 +85,15 @@ jobs:
         run: |
           conda activate openvino_lcm_cpp
           conda update -c conda-forge --all
-          conda install -c conda-forge openvino=2024.1.0 c-compiler cxx-compiler git make cmake
+          conda install -c conda-forge -c conda-forge/label/openvino_dev openvino==2024.2.0.dev20240513 c-compiler cxx-compiler git make cmake
           conda env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
       
       - name: Install python dependencies
         working-directory: ${{ env.working_directory }}
         run: |
           conda activate openvino_lcm_cpp
-          python -m pip install -r requirements.txt
           python -m pip install ../../../thirdparty/openvino_tokenizers/[transformers]
+          python -m pip install -r requirements.txt
 
       - name: Download and convert model and tokenizer
         working-directory: ${{ env.working_directory }}

diff --git a/.github/workflows/llm_bench-python.yml b/.github/workflows/llm_bench-python.yml
@@ -39,7 +39,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         python -m pip install flake8 pytest black
-        pip install -r ${{ env.LLM_BENCH_PYPATH }}/requirements.txt
+        GIT_CLONE_PROTECTION_ACTIVE=false pip install -r ${{ env.LLM_BENCH_PYPATH }}/requirements.txt
         pip install openvino-nightly
 
     - name: Lint with flake8
@@ -73,7 +73,7 @@ jobs:
           python-version: 3.8
       - name: Test stateful
         run: |
-          python -m pip install -r llm_bench/python/requirements.txt
+          GIT_CLONE_PROTECTION_ACTIVE=false python -m pip install -r llm_bench/python/requirements.txt
           python -m pip uninstall --yes openvino
           python -m pip install openvino-nightly
           python llm_bench/python/convert.py --model_id TinyLlama/TinyLlama-1.1B-Chat-v1.0 --output_dir . --stateful

diff --git a/.github/workflows/stable_diffusion_1_5_cpp.yml b/.github/workflows/stable_diffusion_1_5_cpp.yml
@@ -39,15 +39,15 @@ jobs:
       - name: Install OpenVINO and other conda dependencies
         run: |
           conda activate openvino_sd_cpp
-          conda install -c conda-forge openvino=2024.1.0 c-compiler cxx-compiler git make cmake
+          conda install -c conda-forge -c conda-forge/label/openvino_dev openvino==2024.2.0.dev20240513 c-compiler cxx-compiler git make cmake
           conda env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
 
       - name: Install python dependencies
         working-directory: ${{ env.working_directory }}
         run: |
           conda activate openvino_sd_cpp
-          python -m pip install -r requirements.txt
           python -m pip install ../../../thirdparty/openvino_tokenizers/[transformers]
+          python -m pip install -r requirements.txt
 
       - name: Download and convert model and tokenizer
         working-directory: ${{ env.working_directory }}
@@ -83,14 +83,14 @@ jobs:
         - name: Install OpenVINO and other conda dependencies
           run: |
             conda activate openvino_sd_cpp
-            conda install -c conda-forge openvino=2024.1.0 c-compiler cxx-compiler git make cmake
+            conda install -c conda-forge -c conda-forge/label/openvino_dev openvino==2024.2.0.dev20240513 c-compiler cxx-compiler git make cmake
   
         - name: Install python dependencies
           working-directory: ${{ env.working_directory }}
           run: |
             conda activate openvino_sd_cpp
-            python -m pip install -r requirements.txt
             python -m pip install ../../../thirdparty/openvino_tokenizers/[transformers]
+            python -m pip install -r requirements.txt
   
         - name: Download and convert model and tokenizer
           working-directory: ${{ env.working_directory }}

diff --git a/.gitignore b/.gitignore
@@ -39,4 +39,4 @@ CMakeUserPresets.json
 *.?env*
 *.pyc
 .env
-__pycache__
+__pycache__
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -0,0 +1,46 @@
+# Copyright (C) 2018-2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+#
+
+cmake_minimum_required(VERSION 3.23.0)  # The requirement comes from Jinja2Cpp
+
+# Multi config generators such as Visual Studio ignore CMAKE_BUILD_TYPE. Multi config generators are configured with
+# CMAKE_CONFIGURATION_TYPES, but limiting options in it completely removes such build options
+get_property(GENERATOR_IS_MULTI_CONFIG_VAR GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)
+if(CMAKE_GENERATOR STREQUAL "Ninja Multi-Config")
+    # 'Ninja Multi-Config' specific, see:
+    # https://cmake.org/cmake/help/latest/variable/CMAKE_DEFAULT_BUILD_TYPE.html
+    set(CMAKE_DEFAULT_BUILD_TYPE "Release" CACHE STRING "CMake default build type")
+elseif(NOT GENERATOR_IS_MULTI_CONFIG_VAR AND NOT DEFINED CMAKE_BUILD_TYPE)
+    message(STATUS "CMAKE_BUILD_TYPE is not defined, 'Release' will be used")
+    # Setting CMAKE_BUILD_TYPE as CACHE must go before project(). Otherwise project() sets its value and set() doesn't take an effect
+    set(CMAKE_BUILD_TYPE Release CACHE STRING "Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel ...")
+endif()
+
+project(OpenVINOGenAI VERSION 2024.2.0.0)
+
+add_subdirectory(./thirdparty/)
+add_subdirectory(src)
+add_subdirectory(samples/cpp/beam_search_causal_lm/)
+add_subdirectory(samples/cpp/chat_sample/)
+add_subdirectory(samples/cpp/greedy_causal_lm/)
+add_subdirectory(samples/cpp/multinomial_causal_lm/)
+add_subdirectory(samples/cpp/prompt_lookup_decoding_lm/)
+add_subdirectory(samples/cpp/speculative_decoding_lm/)
+
+add_subdirectory(text_generation/causal_lm/cpp/continuous_batching)
+
+install(DIRECTORY
+        ./samples/cpp/beam_search_causal_lm
+        ./samples/cpp/chat_sample
+        ./samples/cpp/greedy_causal_lm
+        ./samples/cpp/multinomial_causal_lm
+        # Don't install prompt_lookup_decoding_lm and speculative_decoding_lm because they don't use openvino_genai library and arent verifyed yet.
+    DESTINATION samples/cpp/ COMPONENT cpp_samples_genai)
+install(FILES ./samples/cpp/requirements.txt DESTINATION samples/cpp/ COMPONENT cpp_samples_genai)
+install(FILES LICENSE DESTINATION licensing COMPONENT licensing_genai RENAME LICENSE-GENAI)
+install(FILES third-party-programs.txt DESTINATION licensing COMPONENT licensing_genai RENAME third-party-programs-genai.txt)
+if(MSVC AND NOT DEFINED CPACK_GENERATOR)
+    set(CPACK_GENERATOR "ZIP")
+endif()
+include(CPack)
diff --git a/image_generation/lcm_dreamshaper_v7/cpp/README.md b/image_generation/lcm_dreamshaper_v7/cpp/README.md
@@ -18,7 +18,7 @@ Prepare a python environment and install dependencies:
 conda create -n openvino_lcm_cpp python==3.10
 conda activate openvino_lcm_cpp
 conda update -c conda-forge --all
-conda install -c conda-forge openvino=2024.1.0 c-compiler cxx-compiler git make cmake
+conda install -c conda-forge openvino=2024.2.0 c-compiler cxx-compiler git make cmake
 # Ensure that Conda standard libraries are used
 conda env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
 ```

diff --git a/image_generation/lcm_dreamshaper_v7/cpp/requirements.txt b/image_generation/lcm_dreamshaper_v7/cpp/requirements.txt
@@ -1,4 +1,4 @@
 --extra-index-url https://download.pytorch.org/whl/cpu
 torch==2.2.2+cpu
 diffusers==0.27.2
-optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel.git@fb1b35bef23242d65b2fb057c4a7ac78a7cfd4c3
+optimum-intel[openvino]==1.17.0
-Original file line number
+Diff line change
@@ Expand Up / @@ -39,4 +39,4 @@ CMakeUserPresets.json @@
     *.?env*
     *.pyc
     .env
-    __pycache__
+    __pycache__