RuntimeWarning: invalid value encountered in greater is_positive = p_agree[similar_idx] > self.threshold.item()

[queenie88]

Iter: [600/1348] Freq 248.2 loss_source 2.152 loss_st 0.781 loss_ml 3458.865 loss_target 0.000 loss_total 116.117 [2019-10-31 09:07:41]
Iter: [700/1348] Freq 248.9 loss_source 2.127 loss_st 0.782 loss_ml 3387.675 loss_target 0.000 loss_total 114.824 [2019-10-31 09:08:53]
Iter: [800/1348] Freq 250.2 loss_source 2.104 loss_st 0.784 loss_ml 3329.171 loss_target 0.000 loss_total 113.704 [2019-10-31 09:10:04]
Iter: [900/1348] Freq 250.9 loss_source 2.078 loss_st 0.784 loss_ml 3280.807 loss_target 0.000 loss_total 112.419 [2019-10-31 09:11:16]
Iter: [1000/1348] Freq 251.3 loss_source 2.060 loss_st 0.786 loss_ml 3250.651 loss_target 0.000 loss_total 111.489 [2019-10-31 09:12:28]
Iter: [1100/1348] Freq 252.0 loss_source 2.040 loss_st 0.786 loss_ml 3223.414 loss_target 0.000 loss_total 110.485 [2019-10-31 09:13:39]
Iter: [1200/1348] Freq 252.7 loss_source nan loss_st nan loss_ml nan loss_target 0.000 loss_total nan [2019-10-31 09:14:49]
Iter: [1300/1348] Freq 253.6 loss_source nan loss_st nan loss_ml nan loss_target 0.000 loss_total nan [2019-10-31 09:15:59]
Train loss_source nan loss_st nan loss_ml nan loss_target 0.000 loss_total nan
Test r1 0.000 r5 0.119 r10 0.208 MAP 5.396

==>>[2019-10-31 09:19:46] [Epoch=001/020] Stage 1, [Need: 05:18:16]
/data/qli/Person_Re-Identification/MAR/utils.py:165: RuntimeWarning: invalid value encountered in greater
is_positive = p_agree[similar_idx] > self.threshold.item()
Iter: [000/1348] Freq 96.7 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:19:48]
Iter: [100/1348] Freq 213.6 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:21:14]
Iter: [200/1348] Freq 211.7 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:22:41]
Iter: [300/1348] Freq 211.6 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:24:08]
Iter: [400/1348] Freq 211.8 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:25:35]
Iter: [500/1348] Freq 211.5 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:27:02]
Iter: [600/1348] Freq 211.2 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:28:30]
Iter: [700/1348] Freq 211.0 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:29:58]
Iter: [800/1348] Freq 211.2 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:31:24]
Iter: [900/1348] Freq 211.1 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:32:52]
Iter: [1000/1348] Freq 211.0 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:34:19]
Iter: [1100/1348] Freq 211.2 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:35:46]
Iter: [1200/1348] Freq 210.9 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:37:14]
Iter: [1300/1348] Freq 210.8 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:38:42]
Train loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan
Test r1 0.000 r5 0.119 r10 0.208 MAP 5.396

==>>[2019-10-31 09:42:20] [Epoch=002/020] Stage 1, [Need: 05:56:28]
Iter: [000/1348] Freq 93.9 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:42:22]
Iter: [100/1348] Freq 214.4 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:43:47]
Iter: [200/1348] Freq 212.8 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:45:14]
Iter: [300/1348] Freq 212.4 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:46:41]
Iter: [400/1348] Freq 212.3 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:48:08]
Iter: [500/1348] Freq 212.1 loss_source nan loss_st nan loss_ml nan loss_target nan loss_total nan [2019-10-31 09:49:35]
Anybody face this problem, please tell me how to solve this problem？

[KovenYu]

Sorry for the late reply. Do you use the default setting? Actually I have seen a similar issue before and I tried to reproduce it, but I failed. Could you use the default setting to see if any problem?