support Yi-Vl

[Lzhang-hub]

Is any plan support Yi-Vl?
https://huggingface.co/01-ai/Yi-VL-34B

[ySingularity]

It seems to have the same architecture as llava, you can try to load it by setting MODEL_TYPE=llava?

[Lzhang-hub]

I had test with MODEL_TYPE=llava, it can not work. Is it closed because it is already supported?

[ySingularity]

It is not supported specially yet, we will have a try and find the problem.
It would be appreciated if you could show us some error logs.

[Lzhang-hub]

this is error logs:

Traceback (most recent call last):
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/rtp-llm-0227/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/nfs15/nfs/bigdata/zhanglei/conda/envs/rtp-llm-0227/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "<maga_transformer-0.1.2+cuda121>/maga_transformer/start_server.py", line 388, in <module>
  File "<maga_transformer-0.1.2+cuda121>/maga_transformer/start_server.py", line 382, in main
  File "<maga_transformer-0.1.2+cuda121>/maga_transformer/start_server.py", line 350, in local_rank_main
  File "<maga_transformer-0.1.2+cuda121>/maga_transformer/start_server.py", line 347, in local_rank_main
  File "<maga_transformer-0.1.2+cuda121>/maga_transformer/start_server.py", line 67, in start
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/inference.py", line 28, in __init__
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/model_factory.py", line 172, in create_from_env
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/model_factory.py", line 70, in from_model_config
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/model_factory.py", line 49, in _create_model
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/models/gpt.py", line 173, in from_config
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/models/llava.py", line 82, in __init__
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/models/llava_vit.py", line 17, in __init__
  File "<maga_transformer-0.1.4+cuda121>/maga_transformer/models/llava_vit.py", line 161, in build_vision_projector
ValueError: Unknown projector type: mlp2x_gelu_Norm

yi-vl have little difference with lalva, We add LayNorm in the two-layer MLP of the projection module.
https://github.com/01-ai/Yi/tree/main/VL#major-difference-with-llava

I try resolve.

[ySingularity]

yi-vl have little difference with lalva, We add LayNorm in the two-layer MLP of the projection module.

Current implementation matches '^mlp(\d+)x_gelu$', and generates gelu and MLP. So that there should be a new branch to match the suffix '_Norm' and a norm layer attached.