Is it possible to save the trained model and resume later?

[BoggyBB]

[SOLVED] - Use "Snapshot" in the GUI

Hello,

First of all, I'm fascinated by how well this works so far.

I've been experimenting lately with some very large, high-res models that take quite a lot of time to train up to the quality that will allow me to render a nice looking video. For example I have a 650 images (1080p) model that started to look good (fine details appearing) after about 30-40minutes (RTX4090).

Would it be possible to somehow save the progress of the trained model for a given dataset, to then be able to reload it from that point and not wait again for 30-40min before being able to do a -nice- render?

P.S. I see Colmap 3.8 is released, would that be updated automatically by Instant-NGP? I have no clue about coding/Python, to be able to update it myself at this point. Any benefits for the updated version compared to 3.7? (asked here so I won't open another thread)

Thanks!
Bogdan

[Tom94]

Hi, yes, you can save the trained model in the "Snapshot" part of the GUI. To load, you can drag the resulting file back into the window, use the GUI, or provide it as command line argument.

If you want to resume training later, check the "w/ optimizer state" checkbox before saving the trained model. And then, to resume training, first load your dataset, then the snapshot, and then start training again.

[BoggyBB]

This is great, thank you Thomas! In my ignorance I thought "shapshot" is some sort of image export/screenshot function that I didn't even touch.