Meet error when training DINO, Cannot import 'detrex._C', therefore 'MultiScaleDeformableAttention' is not available.

[Mr-Bigworth]

Traceback (most recent call last):
File "/share/home/wut_xujw/detrex/tools/train_net.py", line 229, in
launch(
File "/share/home/wut_xujw/detrex/detectron2/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "/share/home/wut_xujw/detrex/tools/train_net.py", line 224, in main
do_train(args, cfg)
File "/share/home/wut_xujw/detrex/tools/train_net.py", line 159, in do_train
model = instantiate(cfg.model)
File "/share/home/wut_xujw/detrex/detectron2/detectron2/config/instantiate.py", line 67, in instantiate
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/share/home/wut_xujw/detrex/detectron2/detectron2/config/instantiate.py", line 67, in
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/share/home/wut_xujw/detrex/detectron2/detectron2/config/instantiate.py", line 67, in instantiate
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/share/home/wut_xujw/detrex/detectron2/detectron2/config/instantiate.py", line 67, in
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/share/home/wut_xujw/detrex/detectron2/detectron2/config/instantiate.py", line 83, in instantiate
return cls(**cfg)
File "/share/home/wut_xujw/detrex/projects/dino/modeling/dino_transformer.py", line 45, in init
attn=MultiScaleDeformableAttention(
File "/share/home/wut_xujw/detrex/detrex/layers/multi_scale_deform_attn.py", line 373, in init
raise ImportError(err)
ImportError: Cannot import 'detrex._C', therefore 'MultiScaleDeformableAttention' is not available. detrex is not compiled successfully, please build following the instructions!

[Mr-Bigworth]

I have installed detectron2 and build an editable version of detrex like
$ python -m pip install -e detectron2
$ pip install -e .
And I can train dab_detr_r50_50ep successfully, but I meet this error when training DINO

[rentainhe]

I have installed detectron2 and build an editable version of detrex like $ python -m pip install -e detectron2 $ pip install -e . And I can train dab_detr_r50_50ep successfully, but I meet this error when training DINO

You should make sure you have CUDA Runtime and set the CUDA_HOME environments variable as (an example)

export CUDA_HOME=/path/to/cuda-11.3/

You can refer to this issue for more details: #109

[rentainhe]

I have installed detectron2 and build an editable version of detrex like $ python -m pip install -e detectron2 $ pip install -e . And I can train dab_detr_r50_50ep successfully, but I meet this error when training DINO

You should make sure you have CUDA Runtime and set the CUDA_HOME environments variable as (an example)

export CUDA_HOME=/path/to/cuda-11.3/

You can refer to this issue for more details: #109

CUDA Runtime is necessary for the customized MultiScaleDeformableAttention operator

[Mr-Bigworth]

I have installed detectron2 and build an editable version of detrex like $ python -m pip install -e detectron2 $ pip install -e . And I can train dab_detr_r50_50ep successfully, but I meet this error when training DINO

You should make sure you have CUDA Runtime and set the CUDA_HOME environments variable as (an example)

export CUDA_HOME=/path/to/cuda-11.3/

You can refer to this issue for more details: #109

Thanks for your help! And I have another question about training with fp16. Can I modify the config in command line directly?

[rentainhe]

I have installed detectron2 and build an editable version of detrex like $ python -m pip install -e detectron2 $ pip install -e . And I can train dab_detr_r50_50ep successfully, but I meet this error when training DINO

You should make sure you have CUDA Runtime and set the CUDA_HOME environments variable as (an example)

export CUDA_HOME=/path/to/cuda-11.3/

You can refer to this issue for more details: #109

Thanks for your help! And I have another question about training with fp16. Can I modify the config in command line directly?

I'm not sure if the MultiScaleDeformableAttention works with fp16 training. To the best of my knowledge, MultiScaleDeformableAttention did not support fp16 training now. So the fp16 mode is still under test now in detrex.

You can directly enable fp16 training in command line like this:

python tools/train_net.py --config-file projects/dino/configs/dino_r50_4scale_12ep.py --num-gpus 8 train.amp.enabled=True

[Mr-Bigworth]

I have installed detectron2 and build an editable version of detrex like $ python -m pip install -e detectron2 $ pip install -e . And I can train dab_detr_r50_50ep successfully, but I meet this error when training DINO

You should make sure you have CUDA Runtime and set the CUDA_HOME environments variable as (an example)

export CUDA_HOME=/path/to/cuda-11.3/

You can refer to this issue for more details: #109

Thanks for your help! And I have another question about training with fp16. Can I modify the config in command line directly?

I'm not sure if the MultiScaleDeformableAttention works with fp16 training. To the best of my knowledge, MultiScaleDeformableAttention did not support fp16 training now. So the fp16 mode is still under test now in detrex.

You can directly enable fp16 training in command line like this:

python tools/train_net.py --config-file projects/dino/configs/dino_r50_4scale_12ep.py --num-gpus 8 train.amp.enabled=True

Thanks! I will close this issue.

[rentainhe]

You're welcome! Feel free to reopen it or leave a new issue if you meet other problems~