Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace export_MiniCPM-V-2_6.py with default optimum-intel #988

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
6151231
Hide VLM files and API
Wovchena Oct 12, 2024
7d94e1a
Remove unused concatenate_mid_dim
Wovchena Oct 12, 2024
eeb818d
Initialize m_image_id in constructor similar to the reset of the fields
Wovchena Oct 12, 2024
20a6954
Retrigger
Wovchena Oct 12, 2024
0737db2
Move to visual_language
Wovchena Oct 14, 2024
0bddfba
Correct py_vlm_pipeline.cpp include
Wovchena Oct 14, 2024
1b2da2d
fix
Wovchena Oct 14, 2024
7f0ef7a
Move vision_encoder, pipeline.hpp
Wovchena Oct 14, 2024
457024c
Replace export_MiniCPM-V-2_6.py
Wovchena Oct 14, 2024
d11f18d
Downgrade optimum
Wovchena Oct 14, 2024
a82fe79
Everywhere python -m pip install -U optimum<1.23 --no-dependencies
Wovchena Oct 14, 2024
6d37b64
Remove duplicates
Wovchena Oct 14, 2024
b8fd628
Fix dtype
Wovchena Oct 14, 2024
b5bad1f
Merge branch 'master' into replace-export_MiniCPM-V-2_6.py
Wovchena Oct 15, 2024
7bdce55
fix merge
Wovchena Oct 15, 2024
ff4f4be
delete src/cpp/src/visual_language/vlm_pipeline.cpp
Wovchena Oct 15, 2024
4112edf
fix conversion in test
Wovchena Oct 15, 2024
c4573b8
dont print in test
Wovchena Oct 15, 2024
8c67805
skip
Wovchena Oct 15, 2024
24015da
cleanup
Wovchena Oct 15, 2024
8410b22
Put torchvision back
Wovchena Oct 15, 2024
1fea50f
update tests requirements
Wovchena Oct 15, 2024
65644db
Merge branch 'master' into replace-export_MiniCPM-V-2_6.py
Wovchena Oct 15, 2024
d1448ef
remove wwb req
Wovchena Oct 15, 2024
67e60ac
wwb reqs
Wovchena Oct 15, 2024
f67ce00
req
Wovchena Oct 15, 2024
e2ac30e
int8
Wovchena Oct 15, 2024
e084e79
xfail
Wovchena Oct 15, 2024
d868e0f
Merge branch 'master' into replace-export_MiniCPM-V-2_6.py
Wovchena Oct 16, 2024
509fb2f
Move common model parts
Wovchena Oct 16, 2024
7513752
Merge branch 'master' into replace-export_MiniCPM-V-2_6.py
Wovchena Oct 16, 2024
8dc91ab
A copy of
Wovchena Oct 16, 2024
c0df11d
correct branch
Wovchena Oct 16, 2024
dc32fc2
correct
Wovchena Oct 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/causal_lm_cpp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -703,18 +703,18 @@ jobs:
source ./ov/setupvars.sh
python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
python -m pip install --upgrade-strategy eager -r ./samples/requirements.txt --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
python ./samples/cpp/visual_language_chat/export_MiniCPM-V-2_6.py ./miniCPM-V-2_6/
python -m pip install git+https://github.com/eaidova/optimum-intel.git@ea/minicpmv
python -m pip install -U "optimum<1.23" --no-dependencies
optimum-cli export openvino -m openbmb/MiniCPM-V-2_6 MiniCPM-V-2_6 --trust-remote-code
wget https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/d5fbbd1a-d484-415c-88cb-9986625b7b11 --output-document cat.jpg
- name: Run visual_language_chat sample - MiniCPM-V-2_6
run: >
source ./ov/setupvars.sh
&& timeout 120s ./build/samples/cpp/visual_language_chat/visual_language_chat ./miniCPM-V-2_6/ cat.jpg
&& timeout 120s ./build/samples/cpp/visual_language_chat/visual_language_chat ./MiniCPM-V-2_6/ cat.jpg
<<< $'What is on the image?\nWhat is special on the image?'
- name: Download and convert LLaVa 1.5 model and an image
run: |
source ./ov/setupvars.sh
python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
python -m pip install --upgrade-strategy eager -r ./samples/requirements.txt --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
optimum-cli export openvino --model llava-hf/llava-1.5-7b-hf ./llava_1_5_7b_ov/
wget https://llava-vl.github.io/static/images/monalisa.jpg
- name: Run visual_language_chat sample - LLaVa 1.5
Expand All @@ -729,7 +729,7 @@ jobs:
source ./ov/setupvars.sh
export PYTHONPATH=./build/:$PYTHONPATH
printf 'What is on the image?\nWhat is special on the image?\n' > ./input.txt
timeout 120s python ./samples/python/visual_language_chat/visual_language_chat.py ./miniCPM-V-2_6/ cat.jpg < input.txt > ./pred.txt
timeout 120s python ./samples/python/visual_language_chat/visual_language_chat.py ./MiniCPM-V-2_6/ cat.jpg < input.txt > ./pred.txt

cpp-continuous-batching-ubuntu:
runs-on: ubuntu-20.04-8-cores
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,13 @@ jobs:
run: |
source ${OV_INSTALL_DIR}/setupvars.sh
python -m pip install . --verbose --find-links ${OV_INSTALL_DIR}/wheels
python -m pytest ./tests/python_tests --ignore ./tests/python_tests/test_whisper_generate_api.py
python -m pytest ./tests/python_tests --ignore ./tests/python_tests/test_whisper_generate_api.py --ignore ./tests/python_tests/test_vlm_api.py

- run: python -m pip install git+https://github.com/eaidova/optimum-intel.git@ea/minicpmv
- run: python -m pip install -U "optimum<1.23" --no-dependencies
- run: >
source ${OV_INSTALL_DIR}/setupvars.sh
&& python -m pytest ./tests/python_tests/test_vlm_api.py

genai_python_lib_whisper:
name: OpenVINO genai extension whisper tests (cmake + wheel)
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,8 @@ jobs:
run: |
. "${{ env.OV_INSTALL_DIR }}/setupvars.ps1"
python -m pip install ./thirdparty/openvino_tokenizers/[transformers] -r ./tests/python_tests/requirements.txt --find-links ${env:OV_INSTALL_DIR}/wheels --upgrade-strategy eager
python -m pip install git+https://github.com/eaidova/optimum-intel.git@ea/minicpmv
python -m pip install -U "optimum<1.23" --no-dependencies
python -m pytest ./tests/python_tests/test_vlm_api.py
env:
PYTHONPATH: "./build/" # cmd evaluates variables in a different way. Setting PYTHONPATH before setupvars.bat instead of doing that after solves that.
Expand Down
2 changes: 1 addition & 1 deletion samples/cpp/visual_language_chat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ It's not required to install [../../requirements.txt](../../requirements.txt) fo

```sh
pip install --upgrade-strategy eager -r ../../requirements.txt
export_MiniCPM-V-2_6.py miniCPM-V-2_6
optimum-cli export openvino --model openbmb/MiniCPM-V-2_6 --trust-remote-code MiniCPM-V-2_6
```

## Run
Expand Down
Loading
Loading