yolo-world 源码解析

Tianheng Cheng^2,3,*, Lin Song^1,📧,*, Yixiao Ge^1,🌟,2, Wenyu Liu³, Xinggang Wang^3,📧, Ying Shan^1,2

* Equal contribution 🌟 Project lead 📧 Corresponding author

¹ Tencent AI Lab, ² ARC Lab, Tencent PCG ³ Huazhong University of Science and Technology

Updates

🔥[2024-3-3]: We add the high-resolution YOLO-World, which supports 1280x1280 resolution with higher accuracy and better performance for small objects!
🔥[2024-2-29]: We release the newest version of YOLO-World-v2 with higher accuracy and faster speed! We hope the community can join us to improve YOLO-World!
🔥[2024-2-28]: Excited to announce that YOLO-World has been accepted by CVPR 2024! We're continuing to make YOLO-World faster and stronger, as well as making it better to use for all.
🔥[2024-2-22]: We sincerely thank RoboFlow and @Skalskip92 for the Video Guide about YOLO-World, nice work!
🔥[2024-2-18]: We thank @Skalskip92 for developing the wonderful segmentation demo via connecting YOLO-World and EfficientSAM. You can try it now at the 🤗 HuggingFace Spaces.
[2024-2-17]: The largest model X of YOLO-World is released, which achieves better zero-shot performance!
[2024-2-17]: We release the code & models for YOLO-World-Seg now! YOLO-World now supports open-vocabulary / zero-shot object segmentation!
[2024-2-15]: The pre-traind YOLO-World-L with CC3M-Lite is released!
[2024-2-14]: We provide the image_demo for inference on images or directories.
[2024-2-10]: We provide the fine-tuning and data details for fine-tuning YOLO-World on the COCO dataset or the custom datasets!
[2024-2-3]: We support the Gradio demo now in the repo and you can build the YOLO-World demo on your own device!
[2024-2-1]: We've released the code and weights of YOLO-World now!
[2024-2-1]: We deploy the YOLO-World demo on HuggingFace 🤗, you can try it now!
[2024-1-31]: We are excited to launch YOLO-World, a cutting-edge real-time open-vocabulary object detector.

TODO

YOLO-World is under active development and please stay tuned ☕️!

Gradio demo!
Complete documents for pre-training YOLO-World.
COCO & LVIS fine-tuning.
Extra pre-trained models on more data, such as CC3M.
Deployment toolkits, e.g., ONNX or TensorRT.
Inference acceleration and scripts for speed evaluation.
Automatic labeling framework for image-text pairs, such as CC3M.

Highlights

This repo contains the PyTorch implementation, pre-trained weights, and pre-training/fine-tuning code for YOLO-World.

YOLO-World is pre-trained on large-scale datasets, including detection, grounding, and image-text datasets.
YOLO-World is the next-generation YOLO detector, with a strong open-vocabulary detection capability and grounding ability.
YOLO-World presents a prompt-then-detect paradigm for efficient user-vocabulary inference, which re-parameterizes vocabulary embeddings as parameters into the model and achieve superior inference speed. You can try to export your own detection model without extra training or fine-tuning in our online demo!

Abstract

The You Only Look Once (YOLO) series of detectors have established themselves as efficient and practical tools. However, their reliance on predefined and trained object categories limits their applicability in open scenarios. Addressing this limitation, we introduce YOLO-World, an innovative approach that enhances YOLO with open-vocabulary detection capabilities through vision-language modeling and pre-training on large-scale datasets. Specifically, we propose a new Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate the interaction between visual and linguistic information. Our method excels in detecting a wide range of objects in a zero-shot manner with high efficiency. On the challenging LVIS dataset, YOLO-World achieves 35.4 AP with 52.0 FPS on V100, which outperforms many state-of-the-art methods in terms of both accuracy and speed. Furthermore, the fine-tuned YOLO-World achieves remarkable performance on several downstream tasks, including object detection and open-vocabulary instance segmentation.

Main Results

We've pre-trained YOLO-World-S/M/L from scratch and evaluate on the LVIS val-1.0 and LVIS minival. We provide the pre-trained model weights and training logs for applications/research or re-producing the results.

Zero-shot Inference on LVIS dataset

model	Pre-train Data	Size	AP^mini	AP_r	AP_c	AP_f	AP^val	AP_r	AP_c	AP_f	weights
YOLO-World-S	O365+GoldG	640	24.3	16.6	22.1	27.7	17.8	11.0	14.8	24.0	HF Checkpoints 🤗
YOLO-Worldv2-S 🔥	O365+GoldG	640	22.7	16.3	20.8	25.5	17.3	11.3	14.9	22.7	HF Checkpoints 🤗
YOLO-World-M	O365+GoldG	31.0	28.6	19.7	26.6	31.9	22.3	16.2	19.0	28.7	HF Checkpoints 🤗
YOLO-Worldv2-M 🔥	O365+GoldG	640	30.0	25.0	27.2	33.4	23.5	17.1	20.0	30.1	HF Checkpoints 🤗
YOLO-World-L	O365+GoldG	640	32.5	22.3	30.6	36.1	24.8	17.8	22.4	32.5	HF Checkpoints 🤗
YOLO-World-L	O365+GoldG+CC3M-Lite	640	33.0	23.6	32.0	35.5	25.3	18.0	22.1	32.1	HF Checkpoints 🤗
YOLO-Worldv2-L 🔥	O365+GoldG	640	33.0	22.6	32.0	35.8	26.0	18.6	23.0	32.6	HF Checkpoints 🤗
YOLO-Worldv2-L 🔥	O365+GoldG	1280 🔸	34.6	29.2	32.8	37.2	27.6	21.9	24.2	34.0	HF Checkpoints 🤗
YOLO-Worldv2-L 🔥	O365+GoldG+CC3M-Lite	640	32.9	25.3	31.1	35.8	26.1	20.6	22.6	32.3	HF Checkpoints 🤗
YOLO-World-X	O365+GoldG+CC3M-Lite	640	33.4	24.4	31.6	36.6	26.6	19.2	23.5	33.2	HF Checkpoints 🤗
YOLO-Worldv2-X 🔥	O365+GoldG+CC3M-Lite	640	35.4	28.7	32.9	38.7	28.4	20.6	25.6	35.0	HF Checkpoints 🤗

NOTE:

The evaluation results of AP^fixed are tested on LVIS minival with fixed AP.
The evaluation results of AP^mini are tested on LVIS minival.
The evaluation results of AP^val are tested on LVIS val 1.0.
HuggingFace Mirror provides the mirror of HuggingFace, which is a choice for users who are unable to reach.
🔸: fine-tuning models with the pre-trained data.

Pre-training Logs:

We provide the pre-training logs of YOLO-World-v2. Due to the unexpected errors of the local machines, the training might be interrupted several times.

Model	Pre-training Log
YOLO-World-v2-S	Part-1, Part-2
YOLO-World-v2-L	Part-1, Part-2
YOLO-World-v2-M	Part-1, Part-2
YOLO-World-v2-X	Final part

YOLO-World-Seg: Open-Vocabulary Instance Segmentation

We fine-tune YOLO-World on LVIS (LVIS-Base) with mask annotations for open-vocabulary (zero-shot) instance segmentation.

We provide two fine-tuning strategies YOLO-World towards open-vocabulary instance segmentation:

fine-tuning all modules: leads to better LVIS segmentation accuracy but affects the zero-shot performance.
fine-tuning the segmentation head: maintains the zero-shot performanc but lowers LVIS segmentation accuracy.

Model	Fine-tuning Data	Fine-tuning Modules	AP^mask	AP_r	AP_c	AP_f	Weights
YOLO-World-Seg-M	`LVIS-Base`	`all modules`	25.9	13.4	24.9	32.6	HF Checkpoints 🤗
YOLO-World-Seg-L	`LVIS-Base`	`all modules`	28.7	15.0	28.3	35.2	HF Checkpoints 🤗
YOLO-World-Seg-M	`LVIS-Base`	`seg head`	16.7	12.6	14.6	20.8	HF Checkpoints 🤗
YOLO-World-Seg-L	`LVIS-Base`	`seg head`	19.1	14.2	17.2	23.5	HF Checkpoints 🤗

NOTE:

The mask AP are evaluated on the LVIS val 1.0.
All models are fine-tuned for 80 epochs on LVIS-Base (866 categories, common + frequent).
The YOLO-World-Seg with only seg head fine-tuned maintains the original zero-shot detection capability and segments objects.

Getting started

1. Installation

YOLO-World is developed based on torch==1.11.0 mmyolo==0.6.0 and mmdetection==3.0.0.

Clone Project

git clone --recursive https://github.com/AILab-CVC/YOLO-World.git

Install

pip install torch wheel -q
pip install -e .

2. Preparing Data

We provide the details about the pre-training data in docs/data.

Training & Evaluation

We adopt the default training or evaluation scripts of mmyolo. We provide the configs for pre-training and fine-tuning in configs/pretrain and configs/finetune_coco. Training YOLO-World is easy:

chmod +x tools/dist_train.sh
# sample command for pre-training, use AMP for mixed-precision training
./tools/dist_train.sh configs/pretrain/yolo_world_l_t2i_bn_2e-4_100e_4x8gpus_obj365v1_goldg_train_lvis_minival.py 8 --amp

NOTE: YOLO-World is pre-trained on 4 nodes with 8 GPUs per node (32 GPUs in total). For pre-training, the node_rank and nnodes for multi-node training should be specified.

Evaluating YOLO-World is also easy:

chmod +x tools/dist_test.sh
./tools/dist_test.sh path/to/config path/to/weights 8

NOTE: We mainly evaluate the performance on LVIS-minival for pre-training.

Fine-tuning YOLO-World

We provide the details about fine-tuning YOLO-World in docs/fine-tuning.

Deployment

We provide the details about deployment for downstream applications in docs/deployment. You can directly download the ONNX model through the online demo in Huggingface Spaces 🤗.

Demo

Gradio Demo

We provide the Gradio demo for local devices:

pip install gradio
python demo.py path/to/config path/to/weights

Additionaly, you can use a Dockerfile to build an image with gradio. As a prerequisite, make sure you have respective drivers installed alongside nvidia-container-runtime. Replace MODEL_NAME and WEIGHT_NAME with the respective values or ommit this and use default values from the Dockerfile

docker build --build-arg="MODEL=MODEL_NAME" --build-arg="WEIGHT=WEIGHT_NAME" -t yolo_demo .
docker run --runtime nvidia -p 8080:8080

Image Demo

We provide a simple image demo for inference on images with visualization outputs.

python image_demo.py path/to/config path/to/weights image/path/directory 'person,dog,cat' --topk 100 --threshold 0.005 --output-dir demo_outputs

Notes:

The image can be a directory or a single image.
The texts can be a string of categories (noun phrases) which is separated by a comma. We also support txt file in which each line contains a category ( noun phrases).
The topk and threshold control the number of predictions and the confidence threshold.

Google Golab Notebook

We sincerely thank Onuralp for sharing the Colab Demo, you can have a try 😊！

Acknowledgement

We sincerely thank mmyolo, mmdetection, GLIP, and transformers for providing their wonderful code to the community!

Citations

If you find YOLO-World is useful in your research or applications, please consider giving us a star 🌟 and citing it.

@article{cheng2024yolow,
  title={YOLO-World: Real-Time Open-Vocabulary Object Detection},
  author={Cheng, Tianheng and Song, Lin and Ge, Yixiao and Liu, Wenyu and Wang, Xinggang and Shan, Ying},
  journal={arXiv preprint arXiv:2401.17270},
  year={2024}
}

Licence

YOLO-World is under the GPL-v3 Licence and is supported for comercial usage.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

yolo-world 源码解析

Updates

TODO

Highlights

Abstract

Main Results

Zero-shot Inference on LVIS dataset

YOLO-World-Seg: Open-Vocabulary Instance Segmentation

Getting started

1. Installation

Clone Project

Install

2. Preparing Data

Training & Evaluation

Fine-tuning YOLO-World

Deployment

Demo

Gradio Demo

Image Demo

Google Golab Notebook

Acknowledgement

Citations

Licence

Files

README.md

Latest commit

History

README.md

File metadata and controls

yolo-world 源码解析

Updates

TODO

Highlights

Abstract

Main Results

Zero-shot Inference on LVIS dataset

YOLO-World-Seg: Open-Vocabulary Instance Segmentation

Getting started

1. Installation

Clone Project

Install

2. Preparing Data

Training & Evaluation

Fine-tuning YOLO-World

Deployment

Demo

Gradio Demo

Image Demo

Google Golab Notebook

Acknowledgement

Citations

Licence