Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom Dataset for tracking #1030

Closed
1 task done
MahejabeenNidhi opened this issue Jul 30, 2023 · 6 comments
Closed
1 task done

Custom Dataset for tracking #1030

MahejabeenNidhi opened this issue Jul 30, 2023 · 6 comments
Labels
question Further information is requested Stale

Comments

@MahejabeenNidhi
Copy link

MahejabeenNidhi commented Jul 30, 2023 •
edited
Loading

Search before asking

  • I have searched the Yolo Tracking issues and found no similar bug report.

Question

Thank you so much for such a valuable repository!

I am trying to run the tracking on a custom dataset.

The dataset is from the following paper
Nguyen, C., Wang, D., Von Richter, K., Valencia, P., Alvarenga, F. A. P., & Bishop-Hurley, G. (2021). Video-based cattle identification and action recognition. 2021 Digital Image Computing: Techniques and Applications (DICTA), 01–05. https://doi.org/10.1109/DICTA52665.2021.9647417

I converted their JSON COCO annotation to MOT annotations. I followed the same structure suggested in the tutorial. I then put the folder under yolo_tracking/examples/val_utils/data
For added reproducibility, you can download the CustomCow MOT dataset that I created from this link

I also converted to YOLO annotations. I then trained my own YOLO model and added the weights as best_Chuong.pt. You can also find the weight here

I made the following changes to the mot_challenge_2d_box.py

        # Get classes to eval
        self.valid_classes = ['cow_1', 'cow_2', 'cow_3', 'cow_4', 'cow_5', 'cow_6', 'cow_7', 'cow_8']
        self.class_list = [cls.lower() if cls.lower() in self.valid_classes else None
                           for cls in self.config['CLASSES_TO_EVAL']]
        if not all(self.class_list):
            raise TrackEvalException('Attempted to evaluate an invalid class. Only pedestrian class is valid.')
        self.class_name_to_class_id = {'cow 1': 0, 'cow 2': 1, 'cow 3': 2, 'cow 4': 3, 'cow 5': 4, 'cow 6': 5, 'cow 7': 6, 'cow 8': 7}
        self.valid_class_numbers = list(self.class_name_to_class_id.values())

I ran the following

python val.py --yolo-model best_Chuong.pt --benchmark CustomCow --split test --tracking-method strongsort

Then I got this error message

val: yolo_model=best_Chuong.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['0'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-30 18:28:23.734 | INFO     | __main__:download_mot_eval_tools:59 - Eval repo already downloaded Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v -- https://github.com/JonathonLuiten/TrackEval /media/D2/public/mae/yolo_tracking/examples/val_utils
  stderr: 'fatal: destination path '/media/D2/public/mae/yolo_tracking/examples/val_utils' already exists and is not an empty directory.
'
[PosixPath('/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1'), PosixPath('/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1')]
2023-07-30 18:28:23.735 | INFO     | __main__:eval:189 - Staring evaluation process on /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1
2023-07-30 18:28:23.740 | INFO     | __main__:eval:189 - Staring evaluation process on /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1
2023-07-30 18:28:26.549 | INFO     | boxmot.utils.torch_utils:select_device:73 - Yolo Tracking v10.0.17 🚀 Python-3.8.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3090, 24265MiB)

2023-07-30 18:28:26.549 | INFO     | __main__:run:65 - {'yolo_model': PosixPath('best_Chuong.pt'), 'reid_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt'), 'tracking_method': 'strongsort', 'source': '/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1', 'imgsz': [1280], 'conf': 0.45, 'iou': 0.7, 'device': '', 'show': False, 'save': True, 'classes': [0], 'project': '/media/D2/public/mae/yolo_tracking/examples/runs/val', 'name': 'exp4', 'exist_ok': True, 'half': False, 'vid_stride': 1, 'show_labels': True, 'show_conf': True, 'save_txt': False, 'save_id_crops': False, 'save_mot': True, 'line_width': None, 'per_class': False}
2023-07-30 18:28:26.587 | INFO     | boxmot.utils.torch_utils:select_device:73 - Yolo Tracking v10.0.17 🚀 Python-3.8.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3090, 24265MiB)

2023-07-30 18:28:26.587 | INFO     | __main__:run:65 - {'yolo_model': PosixPath('best_Chuong.pt'), 'reid_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt'), 'tracking_method': 'strongsort', 'source': '/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1', 'imgsz': [1280], 'conf': 0.45, 'iou': 0.7, 'device': '', 'show': False, 'save': True, 'classes': [0], 'project': '/media/D2/public/mae/yolo_tracking/examples/runs/val', 'name': 'exp4', 'exist_ok': True, 'half': False, 'vid_stride': 1, 'show_labels': True, 'show_conf': True, 'save_txt': False, 'save_id_crops': False, 'save_mot': True, 'line_width': None, 'per_class': False}
2023-07-30 18:28:28.867 | SUCCESS  | boxmot.appearance.reid_model_factory:load_pretrained_weights:187 - Successfully loaded pretrained weights from "/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt"
2023-07-30 18:28:28.867 | WARNING  | boxmot.appearance.reid_model_factory:load_pretrained_weights:191 - The following layers are discarded due to unmatched keys or layer size: ('classifier.weight', 'classifier.bias')
Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 280, in <module>
    main(opt)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 275, in main
    run(vars(opt))
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 99, in run
    model = Yolo(
TypeError: 'NoneType' object is not callable
2023-07-30 18:28:28.908 | SUCCESS  | boxmot.appearance.reid_model_factory:load_pretrained_weights:187 - Successfully loaded pretrained weights from "/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt"
2023-07-30 18:28:28.908 | WARNING  | boxmot.appearance.reid_model_factory:load_pretrained_weights:191 - The following layers are discarded due to unmatched keys or layer size: ('classifier.weight', 'classifier.bias')
Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 280, in <module>
    main(opt)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 275, in main
    run(vars(opt))
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 99, in run
    model = Yolo(
TypeError: 'NoneType' object is not callable
2023-07-30 18:28:29.564 | SUCCESS  | __main__:eval:214 - Evaluation succeeded
val: yolo_model=best_Chuong.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['0'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-30 18:28:29.827 | ERROR    | __main__:eval:244 - Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 84, in <module>
    dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 75, in __init__
    raise TrackEvalException('Attempted to evaluate an invalid class. Only pedestrian class is valid.')
trackeval.utils.TrackEvalException: Attempted to evaluate an invalid class. Only pedestrian class is valid.

2023-07-30 18:28:29.827 | ERROR    | __main__:eval:245 - Error importing BURST due to missing underlying dependency: No module named 'pycocotools'

Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 4                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /media/D2/public/mae/yolo_tracking/examples/val_utils/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : True                          

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test
TRACKERS_FOLDER      : /media/D2/public/mae/yolo_tracking/examples/runs/val/exp4
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['labels']                    
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            :                               
SPLIT_TO_EVAL        : train                         
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   :                               
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : {'CustomCow-01': None, 'CustomCow-02': None}
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : True

I am confused why the only valid class is 'pedestrian' even after changing the mot_challenge_2d_box.py
I also created seqinfo.ini files for each CustomCow-0X folder. I wonder why
SEQ_INFO : {'CustomCow-01': None, 'CustomCow-02': None}

One possible issue:
You had mentioned

Don't forget to change the classes to evaluate: https://github.com/mikel-brostrom/yolo_tracking/blob/1e266d982c84620e0f8319df7b7f5233292132ae/examples/val.py#L204

In the current val.py, it doesn't specify str(0) so I just left it as is. Would this be here I went wrong as well?

 p = subprocess.Popen(
                    args=[
                        sys.executable, str(EXAMPLES / "track.py"),
                        "--yolo-model", self.opt.yolo_model,
                        "--reid-model", self.opt.reid_model,
                        "--tracking-method", self.opt.tracking_method,
                        "--conf", str(self.opt.conf),
                        "--imgsz", str(self.opt.imgsz[0]),
                        "--classes", *self.opt.classes,
                        "--name", save_dir.name,
                        "--save-mot",
                        "--project", self.opt.project,
                        "--device", str(tracking_subprocess_device),
                        "--source", seq_path,
                        "--exist-ok",
                        "--save",
                    ],

I would greatly appreciate any insight!
@MahejabeenNidhi MahejabeenNidhi added the question Further information is requested label Jul 30, 2023
@MahejabeenNidhi MahejabeenNidhi changed the title Custom Tracking Custom Dataset for tracking Jul 30, 2023
@mikel-brostrom
Copy link
Owner

I really recommend you to switch to this branch: #959 for evaluation on multi-class datasets. Trackeval is not design for being flexible with respect to which dataset is evaluated. The only reason it is still used in this repo is that the current main MOT metric, HOTA, is not available anywhere else.

@mikel-brostrom
Copy link
Owner

This may help you out: #927 😄

@MahejabeenNidhi
Copy link
Author

Hi, I appreciate your response a lot!

I realise I have made an error! I realised that I don't need multiclass detection since I only have cows being detected.

Therefore, I used the default yolo model. Since that is trained on the COCO dataset, I specified str(19) in val.py for classes

args=[
                        sys.executable, str(EXAMPLES / "track.py"),
                        "--yolo-model", self.opt.yolo_model,
                        "--reid-model", self.opt.reid_model,
                        "--tracking-method", self.opt.tracking_method,
                        "--conf", str(self.opt.conf),
                        "--imgsz", str(self.opt.imgsz[0]),
                        "--classes", str(19),
                        "--name", save_dir.name,
                        "--save-mot",
                        "--project", self.opt.project,
                        "--device", str(tracking_subprocess_device),
                        "--source", seq_path,
                        "--exist-ok",
                        "--save",
                    ]

I changed the mot_challenge_2d_box.py as follows,

        # Get classes to eval
        self.valid_classes = ['cow']
        self.class_list = [cls.lower() if cls.lower() in self.valid_classes else None
                           for cls in self.config['CLASSES_TO_EVAL']]
        if not all(self.class_list):
            raise TrackEvalException('Attempted to evaluate an invalid class. Only cow class is valid.')
        self.class_name_to_class_id = {'cow': 1}
        self.valid_class_numbers = list(self.class_name_to_class_id.values())

I ran the following command

python val.py --benchmark CustomCow --split test --tracking-method strongsort --classes 19

Then I got the following error. I deleted some of the tracking outputs but kept a handful for you to see that it repeated over all the CustomCow-01 and CustomCow-02 img1 files.

val: yolo_model=/media/D2/public/mae/yolo_tracking/examples/weights/yolov8n.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['19'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-31 11:30:43.212 | INFO     | __main__:download_mot_eval_tools:59 - Eval repo already downloaded Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v -- https://github.com/JonathonLuiten/TrackEval /media/D2/public/mae/yolo_tracking/examples/val_utils
  stderr: 'fatal: destination path '/media/D2/public/mae/yolo_tracking/examples/val_utils' already exists and is not an empty directory.
'
[PosixPath('/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1'), PosixPath('/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1')]
2023-07-31 11:30:43.217 | INFO     | __main__:eval:189 - Staring evaluation process on /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1
2023-07-31 11:30:43.225 | INFO     | __main__:eval:189 - Staring evaluation process on /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1
2023-07-31 11:30:57.190 | INFO     | boxmot.utils.torch_utils:select_device:73 - Yolo Tracking v10.0.17 🚀 Python-3.8.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3090, 24265MiB)

2023-07-31 11:30:57.190 | INFO     | __main__:run:65 - {'yolo_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/yolov8n.pt'), 'reid_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt'), 'tracking_method': 'strongsort', 'source': '/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1', 'imgsz': [1280], 'conf': 0.45, 'iou': 0.7, 'device': '', 'show': False, 'save': True, 'classes': [19], 'project': '/media/D2/public/mae/yolo_tracking/examples/runs/val', 'name': 'exp8', 'exist_ok': True, 'half': False, 'vid_stride': 1, 'show_labels': True, 'show_conf': True, 'save_txt': False, 'save_id_crops': False, 'save_mot': True, 'line_width': None, 'per_class': False}
2023-07-31 11:30:57.190 | INFO     | boxmot.utils.torch_utils:select_device:73 - Yolo Tracking v10.0.17 🚀 Python-3.8.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3090, 24265MiB)

2023-07-31 11:30:57.190 | INFO     | __main__:run:65 - {'yolo_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/yolov8n.pt'), 'reid_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt'), 'tracking_method': 'strongsort', 'source': '/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1', 'imgsz': [1280], 'conf': 0.45, 'iou': 0.7, 'device': '', 'show': False, 'save': True, 'classes': [19], 'project': '/media/D2/public/mae/yolo_tracking/examples/runs/val', 'name': 'exp8', 'exist_ok': True, 'half': False, 'vid_stride': 1, 'show_labels': True, 'show_conf': True, 'save_txt': False, 'save_id_crops': False, 'save_mot': True, 'line_width': None, 'per_class': False}
2023-07-31 11:30:59.601 | SUCCESS  | boxmot.appearance.reid_model_factory:load_pretrained_weights:187 - Successfully loaded pretrained weights from "/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt"
2023-07-31 11:30:59.601 | WARNING  | boxmot.appearance.reid_model_factory:load_pretrained_weights:191 - The following layers are discarded due to unmatched keys or layer size: ('classifier.weight', 'classifier.bias')
2023-07-31 11:30:59.613 | SUCCESS  | boxmot.appearance.reid_model_factory:load_pretrained_weights:187 - Successfully loaded pretrained weights from "/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt"
2023-07-31 11:30:59.614 | WARNING  | boxmot.appearance.reid_model_factory:load_pretrained_weights:191 - The following layers are discarded due to unmatched keys or layer size: ('classifier.weight', 'classifier.bias')
2023-07-31 11:30:59.677 | INFO     | __main__:run:203 - image 1/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000001.jpg: 1280x1280 2 cows, YOLO 4.8ms, TRACKING 11.0ms
2023-07-31 11:30:59.687 | INFO     | __main__:run:203 - image 1/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000001.jpg: 1280x1280 3 cows, YOLO 6.2ms, TRACKING 10.8ms
/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000304.jpg: 1280x1280 3 cows, YOLO 9.5ms, TRACKING 305.8ms
2023-07-31 11:31:27.590 | INFO     | __main__:run:203 - image 349/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000349.jpg: 1280x1280 2 cows, YOLO 15.1ms, TRACKING 56.8ms
2023-07-31 11:31:27.665 | INFO     | __main__:run:203 - image 305/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000305.jpg: 1280x1280 3 cows, YOLO 5.4ms, TRACKING 47.1ms
2023-07-31 11:31:27.753 | INFO     | __main__:run:203 - image 350/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000350.jpg: 1280x1280 3 cows, YOLO 5.5ms, TRACKING 108.8ms
2023-07-31 11:31:27.889 | INFO     | __main__:run:203 - image 306/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000306.jpg: 1280x1280 3 cows, YOLO 4.8ms, TRACKING 168.4ms
2023-07-31 11:31:27.960 | INFO     | __main__:run:203 - image 351/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000351.jpg: 1280x1280 3 cows, YOLO 10.0ms, TRACKING 152.7ms
2023-07-31 11:31:28.048 | INFO     | __main__:run:203 - image 352/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000352.jpg: 1280x1280 3 cows, YOLO 14.2ms, TRACKING 26.3ms
2023-07-31 11:31:28.136 | INFO     | __main__:run:203 - image 353/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000353.jpg: 1280x1280 3 cows, YOLO 5.1ms, TRACKING 35.7ms
2023-07-31 11:31:28.216 | INFO     | __main__:run:203 - image 354/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000354.jpg: 1280x1280 3 cows, YOLO 7.1ms, TRACKING 29.6ms
2023-07-31 11:31:28.283 | INFO     | __main__:run:203 - image 307/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000307.jpg: 1280x1280 3 cows, YOLO 9.4ms, TRACKING 321.6ms
2023-07-31 11:31:28.316 | INFO     | __main__:run:203 - image 355/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000355.jpg: 1280x1280 3 cows, YOLO 14.6ms, TRACKING 44.5ms
2023-07-31 11:31:28.410 | INFO     | __main__:run:203 - image 356/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000356.jpg: 1280x1280 3 cows, YOLO 5.2ms, TRACKING 47.4ms
2023-07-31 11:31:28.489 | INFO     | __main__:run:203 - image 357/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1/000357.jpg: 1280x1280 3 cows, YOLO 4.8ms, TRACKING 32.6ms
2023-07-31 11:32:33.631 | INFO     | __main__:run:203 - image 903/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000903.jpg: 1280x1280 1 cow, YOLO 5.2ms, TRACKING 22.9ms
2023-07-31 11:32:33.662 | INFO     | __main__:run:203 - image 904/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000904.jpg: 1280x1280 1 cow, YOLO 5.2ms, TRACKING 2.5ms
2023-07-31 11:32:33.690 | INFO     | __main__:run:203 - image 905/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000905.jpg: 1280x1280 (no detections), YOLO 5.1ms, TRACKING 2.8ms
2023-07-31 11:32:33.735 | INFO     | __main__:run:203 - image 906/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000906.jpg: 1280x1280 1 cow, YOLO 4.8ms, TRACKING 20.4ms
2023-07-31 11:32:33.765 | INFO     | __main__:run:203 - image 907/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000907.jpg: 1280x1280 1 cow, YOLO 5.1ms, TRACKING 1.6ms
2023-07-31 11:32:33.821 | INFO     | __main__:run:203 - image 908/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000908.jpg: 1280x1280 1 cow, YOLO 5.0ms, TRACKING 26.6ms
2023-07-31 11:32:33.821 | INFO     | __main__:run:217 - Results saved to /media/D2/public/mae/yolo_tracking/examples/runs/val/exp8
2023-07-31 11:32:34.407 | SUCCESS  | __main__:eval:214 - Evaluation succeeded
val: yolo_model=/media/D2/public/mae/yolo_tracking/examples/weights/yolov8n.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['19'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-31 11:32:34.655 | ERROR    | __main__:eval:244 - Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 84, in <module>
    dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 75, in __init__
    raise TrackEvalException('Attempted to evaluate an invalid class. Only cow class is valid.')
trackeval.utils.TrackEvalException: Attempted to evaluate an invalid class. Only cow class is valid.

2023-07-31 11:32:34.655 | ERROR    | __main__:eval:245 - 
Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 4                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /media/D2/public/mae/yolo_tracking/examples/val_utils/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : True                          

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test
TRACKERS_FOLDER      : /media/D2/public/mae/yolo_tracking/examples/runs/val/exp8
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['labels']                    
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            :                               
SPLIT_TO_EVAL        : train                         
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   :                               
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : {'CustomCow-01': None, 'CustomCow-02': None}
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : True

This time, I could see that it was tracking and it was able to detect cows. However, then it threw an error at the end because of invalid class. I did make changes in the mot_challenge_2d_box.py as I previously states. I made no changes to the run_mot_challenge.py file.

Could you please advise?

@MahejabeenNidhi
Copy link
Author

I trained a YOLOv8 model that only had one class - cow. My reasoning was that if it had only one type of detection then it wouldn't affect the multiclass issue. For this trial, I did not change the val.py or the mot_challenge_2d_box.py file.

I ran the following code

python val.py --yolo-model best_OnlyCow.pt --benchmark CustomCow --split test --tracking-method strongsort

I get the following error,

val: yolo_model=best_OnlyCow.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['0'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, gsi=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-31 15:56:20.376 | INFO     | __main__:download_mot_eval_tools:59 - Eval repo already downloaded Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v -- https://github.com/JonathonLuiten/TrackEval /media/D2/public/mae/yolo_tracking/examples/val_utils
  stderr: 'fatal: destination path '/media/D2/public/mae/yolo_tracking/examples/val_utils' already exists and is not an empty directory.
'
2023-07-31 15:56:20.381 | INFO     | __main__:eval:189 - Staring evaluation process on /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1
2023-07-31 15:56:20.391 | INFO     | __main__:eval:189 - Staring evaluation process on /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1
2023-07-31 15:56:33.433 | INFO     | boxmot.utils.torch_utils:select_device:73 - Yolo Tracking v10.0.17 🚀 Python-3.8.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3090, 24265MiB)

2023-07-31 15:56:33.433 | INFO     | __main__:run:65 - {'yolo_model': PosixPath('best_OnlyCow.pt'), 'reid_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt'), 'tracking_method': 'strongsort', 'source': '/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-01/img1', 'imgsz': [1280], 'conf': 0.45, 'iou': 0.7, 'device': '', 'show': False, 'save': True, 'classes': [0], 'project': '/media/D2/public/mae/yolo_tracking/examples/runs/val', 'name': 'exp7', 'exist_ok': True, 'half': False, 'vid_stride': 1, 'show_labels': True, 'show_conf': True, 'save_txt': False, 'save_id_crops': False, 'save_mot': True, 'line_width': None, 'per_class': False}
2023-07-31 15:56:33.433 | INFO     | boxmot.utils.torch_utils:select_device:73 - Yolo Tracking v10.0.17 🚀 Python-3.8.17 torch-2.0.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3090, 24265MiB)

2023-07-31 15:56:33.434 | INFO     | __main__:run:65 - {'yolo_model': PosixPath('best_OnlyCow.pt'), 'reid_model': PosixPath('/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt'), 'tracking_method': 'strongsort', 'source': '/media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1', 'imgsz': [1280], 'conf': 0.45, 'iou': 0.7, 'device': '', 'show': False, 'save': True, 'classes': [0], 'project': '/media/D2/public/mae/yolo_tracking/examples/runs/val', 'name': 'exp7', 'exist_ok': True, 'half': False, 'vid_stride': 1, 'show_labels': True, 'show_conf': True, 'save_txt': False, 'save_id_crops': False, 'save_mot': True, 'line_width': None, 'per_class': False}
2023-07-31 15:56:35.951 | SUCCESS  | boxmot.appearance.reid_model_factory:load_pretrained_weights:187 - Successfully loaded pretrained weights from "/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt"
2023-07-31 15:56:35.951 | WARNING  | boxmot.appearance.reid_model_factory:load_pretrained_weights:191 - The following layers are discarded due to unmatched keys or layer size: ('classifier.weight', 'classifier.bias')
Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 280, in <module>
    main(opt)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 275, in main
    run(vars(opt))
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 99, in run
    model = Yolo(
TypeError: 'NoneType' object is not callable
2023-07-31 15:56:35.975 | SUCCESS  | boxmot.appearance.reid_model_factory:load_pretrained_weights:187 - Successfully loaded pretrained weights from "/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt"
2023-07-31 15:56:35.975 | WARNING  | boxmot.appearance.reid_model_factory:load_pretrained_weights:191 - The following layers are discarded due to unmatched keys or layer size: ('classifier.weight', 'classifier.bias')
Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 280, in <module>
    main(opt)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 275, in main
    run(vars(opt))
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/media/D2/public/mae/yolo_tracking/examples/track.py", line 99, in run
    model = Yolo(
TypeError: 'NoneType' object is not callable
2023-07-31 15:56:36.714 | SUCCESS  | __main__:eval:214 - Evaluation succeeded
val: yolo_model=best_OnlyCow.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['0'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, gsi=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-31 15:56:37.792 | ERROR    | __main__:eval:249 - Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 84, in <module>
    dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 124, in __init__
    raise TrackEvalException(
trackeval.utils.TrackEvalException: Tracker file not found: labels//CustomCow-01.txt

2023-07-31 15:56:37.792 | ERROR    | __main__:eval:250 - 
Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 4                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /media/D2/public/mae/yolo_tracking/examples/val_utils/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : True                          

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test
TRACKERS_FOLDER      : /media/D2/public/mae/yolo_tracking/examples/runs/val/exp7
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['labels']                    
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            :                               
SPLIT_TO_EVAL        : train                         
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   :                               
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : {'CustomCow-01': None, 'CustomCow-02': None}
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : True                          
Tracker file not found: /media/D2/public/mae/yolo_tracking/examples/runs/val/exp7/labels/CustomCow-01.txt

When I run it without the --yolo-model best_OnlyCow.pt,

python val.py --benchmark CustomCow --split test --tracking-method strongsort

it iterates over the files with no detections and then throws the error

2023-07-31 16:08:25.640 | INFO     | __main__:run:217 - Results saved to /media/D2/public/mae/yolo_tracking/examples/runs/val/exp8
2023-07-31 16:08:25.666 | INFO     | __main__:run:203 - image 905/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000905.jpg: 1280x1280 (no detections), YOLO 4.8ms, TRACKING 0.1ms
2023-07-31 16:08:25.701 | INFO     | __main__:run:203 - image 906/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000906.jpg: 1280x1280 (no detections), YOLO 4.8ms, TRACKING 0.1ms
2023-07-31 16:08:25.737 | INFO     | __main__:run:203 - image 907/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000907.jpg: 1280x1280 (no detections), YOLO 4.8ms, TRACKING 0.1ms
2023-07-31 16:08:25.773 | INFO     | __main__:run:203 - image 908/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000908.jpg: 1280x1280 (no detections), YOLO 4.8ms, TRACKING 0.1ms
2023-07-31 16:08:25.774 | INFO     | __main__:run:217 - Results saved to /media/D2/public/mae/yolo_tracking/examples/runs/val/exp8
2023-07-31 16:08:26.484 | SUCCESS  | __main__:eval:214 - Evaluation succeeded
val: yolo_model=/media/D2/public/mae/yolo_tracking/examples/weights/yolov8n.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['0'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, gsi=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-31 16:08:27.782 | ERROR    | __main__:eval:249 - Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 84, in <module>
    dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 124, in __init__
    raise TrackEvalException(
trackeval.utils.TrackEvalException: Tracker file not found: labels//CustomCow-01.txt

2023-07-31 16:08:27.782 | ERROR    | __main__:eval:250 - 
Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 4                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /media/D2/public/mae/yolo_tracking/examples/val_utils/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : True                          

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test
TRACKERS_FOLDER      : /media/D2/public/mae/yolo_tracking/examples/runs/val/exp8
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['labels']                    
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            :                               
SPLIT_TO_EVAL        : train                         
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   :                               
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : {'CustomCow-01': None, 'CustomCow-02': None}
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : True                          
Tracker file not found: /media/D2/public/mae/yolo_tracking/examples/runs/val/exp8/labels/CustomCow-01.txt

It also iterates if I specify --classes 19

python val.py --benchmark CustomCow --split test --tracking-method strongsort --classes 19

It iterates over the files and detects cows. Then it throws the error of only detecting pedestrian classes

2023-07-31 16:16:45.041 | INFO     | __main__:run:203 - image 899/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000899.jpg: 1280x1280 2 cows, YOLO 5.1ms, TRACKING 25.9ms
2023-07-31 16:16:45.107 | INFO     | __main__:run:203 - image 900/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000900.jpg: 1280x1280 1 cow, YOLO 5.1ms, TRACKING 22.0ms
2023-07-31 16:16:45.183 | INFO     | __main__:run:203 - image 901/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000901.jpg: 1280x1280 1 cow, YOLO 14.7ms, TRACKING 23.1ms
2023-07-31 16:16:45.249 | INFO     | __main__:run:203 - image 902/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000902.jpg: 1280x1280 1 cow, YOLO 5.1ms, TRACKING 27.3ms
2023-07-31 16:16:45.300 | INFO     | __main__:run:203 - image 903/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000903.jpg: 1280x1280 1 cow, YOLO 4.8ms, TRACKING 12.6ms
2023-07-31 16:16:45.344 | INFO     | __main__:run:203 - image 904/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000904.jpg: 1280x1280 1 cow, YOLO 5.2ms, TRACKING 2.5ms
2023-07-31 16:16:45.380 | INFO     | __main__:run:203 - image 905/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000905.jpg: 1280x1280 (no detections), YOLO 4.8ms, TRACKING 2.4ms
2023-07-31 16:16:45.440 | INFO     | __main__:run:203 - image 906/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000906.jpg: 1280x1280 1 cow, YOLO 4.8ms, TRACKING 21.4ms
2023-07-31 16:16:45.484 | INFO     | __main__:run:203 - image 907/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000907.jpg: 1280x1280 1 cow, YOLO 5.2ms, TRACKING 1.6ms
2023-07-31 16:16:45.551 | INFO     | __main__:run:203 - image 908/908 /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test/CustomCow-02/img1/000908.jpg: 1280x1280 1 cow, YOLO 4.8ms, TRACKING 27.8ms
2023-07-31 16:16:45.552 | INFO     | __main__:run:217 - Results saved to /media/D2/public/mae/yolo_tracking/examples/runs/val/exp9
2023-07-31 16:16:46.220 | SUCCESS  | __main__:eval:214 - Evaluation succeeded
val: yolo_model=/media/D2/public/mae/yolo_tracking/examples/weights/yolov8n.pt, reid_model=/media/D2/public/mae/yolo_tracking/examples/weights/mobilenetv2_x1_4_dukemtmcreid.pt, tracking_method=strongsort, name=exp, classes=['19'], project=/media/D2/public/mae/yolo_tracking/examples/runs/val, exist_ok=False, gsi=False, benchmark=CustomCow, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2
2023-07-31 16:16:47.396 | ERROR    | __main__:eval:249 - multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap
    return f(*args, **kw)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/eval.py", line 222, in eval_sequence
    data = dataset.get_preprocessed_seq_data(raw_data, cls)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap
    return f(*args, **kw)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 353, in get_preprocessed_seq_data
    raise TrackEvalException(
trackeval.utils.TrackEvalException: Evaluation is only valid for pedestrian class. Non pedestrian class (19) found in sequence CustomCow-01 at timestep 44.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/eval.py", line 102, in evaluate
    results = pool.map(_eval_sequence, seq_list)
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
trackeval.utils.TrackEvalException: Evaluation is only valid for pedestrian class. Non pedestrian class (19) found in sequence CustomCow-01 at timestep 44.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap
    return f(*args, **kw)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/eval.py", line 222, in eval_sequence
    data = dataset.get_preprocessed_seq_data(raw_data, cls)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap
    return f(*args, **kw)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 353, in get_preprocessed_seq_data
    raise TrackEvalException(
trackeval.utils.TrackEvalException: Evaluation is only valid for pedestrian class. Non pedestrian class (19) found in sequence CustomCow-01 at timestep 44.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 91, in <module>
    evaluator.evaluate(dataset_list, metrics_list)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap
    return f(*args, **kw)
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/eval.py", line 207, in evaluate
    raise err
  File "/media/D2/public/mae/yolo_tracking/examples/val_utils/trackeval/eval.py", line 102, in evaluate
    results = pool.map(_eval_sequence, seq_list)
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/media/D2/public/mae/anaconda3/envs/yolo-tracking/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
trackeval.utils.TrackEvalException: Evaluation is only valid for pedestrian class. Non pedestrian class (19) found in sequence CustomCow-01 at timestep 44.

2023-07-31 16:16:47.396 | ERROR    | __main__:eval:250 - 
Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 4                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /media/D2/public/mae/yolo_tracking/examples/val_utils/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : True                          

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /media/D2/public/mae/yolo_tracking/examples/val_utils/data/CustomCow/test
TRACKERS_FOLDER      : /media/D2/public/mae/yolo_tracking/examples/runs/val/exp9
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['labels']                    
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            :                               
SPLIT_TO_EVAL        : train                         
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   :                               
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : {'CustomCow-01': None, 'CustomCow-02': None}
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : True                          

CLEAR Config:
METRICS              : ['HOTA', 'CLEAR', 'Identity'] 
THRESHOLD            : 0.5                           
PRINT_CONFIG         : True                          

Identity Config:
METRICS              : ['HOTA', 'CLEAR', 'Identity'] 
THRESHOLD            : 0.5                           
PRINT_CONFIG         : True                          

Evaluating 1 tracker(s) on 2 sequence(s) for 1 class(es) on MotChallenge2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count


Evaluating labels

Tracker labels was unable to be evaluated.
Evaluation is only valid for pedestrian class. Non pedestrian class (19) found in sequence CustomCow-01 at timestep 44.

I am currently going through issues #883 and #889 and they were very helpful.

@mesllo-bc mentioned something that stuck out to me about the det.txt files for MOT20 and I have no idea what they are. Have you figured out how to get them?

My next steps will be to fix my gt.txt files according to @Hunter-v1 's suggestion.

<frame> <id> <bb_left> <bb_top> <bb_width> <bb_height> <conf> <class_id> <visibility_ratio>

Should I just set my class_id to 1 to avoid the issue of only one class valid since I don't have multiple classes in my class? Would that simplify my work?
Should I set my conf as -1 since it is ground truth as I have done before?
How do I calculate visibility_ratio?
It's crazy that none of this is specified in the MOTChallenge or TrackEval documents.

I will continue to work on this comment. Just commenting now to keep you updated on where I am with this issue.

@mikel-brostrom
Copy link
Owner

Just commenting now to keep you updated on where I am with this issue.

👍

It's crazy that none of this is specified in the MOTChallenge or TrackEval documents.

It is absolutely not production ready code. That is why I suggest you to try out pymotmetrics where we have proven successful multi-class tracking evaluation. But again, it has no HOTA metric which is delaying its full integration

@github-actions
Copy link

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

@github-actions github-actions bot added the Stale label Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikel-brostrom @MahejabeenNidhi