Enable LLaVa-1.5 in VLM Pipeline #917
Conversation
…for llava and minicpm
yatarkan
force-pushed
the
yt/add-llava-1-5
branch
from
42d275f to
790981f
Compare
|
Please, resolve the conflict |
| /// @brief A model for image encoding. | ||
| ov::InferRequest m_encoder; | ||
| /// @brief A config to follow. | ||
| ProcessorConfig m_processor_config; | ||
|
|
||
| // LLaVa specific members | ||
| ov::InferRequest m_vision_embeddings; |
There was a problem hiding this comment.
should we define base class for VisionEncoder ? and create inherited classes for MiniCPM and LLava specifically
In this case, we don't need to put all the fields for different models into common class. Similarly, processor config can be model specific class.
| } | ||
| } | ||
|
|
||
| ov::Tensor VLMPipeline::get_inputs_embeds_minicpm(const std::string& prompt, const std::vector<ov::Tensor>& images) { |
There was a problem hiding this comment.
can we create a dedicated class responsible for Inputs embedding?
Such class will be used in Continuous batching implementation
In this case, VLM Pipeline will compute input embeddings w/o knowing any model / pipeline specific and then perform inference of LLM part in auto-regressive mode
github-merge-queue
bot
removed this pull request from the merge queue due to a conflict with the base branch
|
There's a conflict again |
|
Please, update a list of supported models https://github.com/openvinotoolkit/openvino.genai/blob/master/src/docs/SUPPORTED_MODELS.md#visual-language-models |
ilya-lavrenov
added
category: visual language
Ticket: CVS-153333