Problems when training mar model from scratch

[swasler]

Hi TianHong,

I've been trying to train MAR from scratch recently, but I encountered an issue. If I don’t use the pre-trained parameters you provided and train from scratch, the generated images during training (visualized using the online evaluation command) turn out completely black. I only trained for two epochs on the ImageNet dataset since it’s so large, but I feel like the output shouldn’t be entirely black—at least there should be some noise in the images, right?

To reduce the computational cost, I set num_class=1 and trained on just one class of the ImageNet dataset. Even after training for dozens of epochs, the generated images are still completely black. However, if I use your pre-trained parameters (modified due to num_class=1), the training works fine and I get the expected results.

Did you encounter this issue when training? I’m a bit confused about why training from scratch leads to black images. Do you have any suggestions or things I should pay attention to?

Thank you for your answering!

[LTH14]

The randomly initialized model will result in NAN value if you use autocast during inference. If you want to see noise instead of black image, just turn off the autocast during inference https://github.com/LTH14/mar/blob/main/engine_mar.py#L155

[LTH14]

Since we use ema=0.9999 by default, you would need at least 100k iterations to get a reasonable result. If you want to see it early, you can disable the ema here https://github.com/LTH14/mar/blob/main/engine_mar.py#L124, but it will harm the final performance.