How do I continue a training which was disrupted?

[Utsa7]

I was finetuing a model but due to some reason I had to stop at the middle. Now I want to continue from where I left but when I start training even with the same configuration it starts from the beginning. I definitely don't want to start all over again as it took me 4+ hours to get to that point with my low end gpu.
I hope it's possible to continue earlier training otherwise it might not possible to finetune models. Please help me

[erew123]

Hi @Utsa7

You aren't meant to break training session BUT I may be able to think of a way you could resume it.

In the tmp-trn folder you will have your training session folder (You will need to identify the correct folder for your training session). In my example I have a XTTS_FT-April-02-2024_02+43PM-5aecb5b folder.

Within that folder you have a best_model.pth file and a best_model_XX.pth the XX should be a number which kind of relates to how many epochs it went through.

best_model.pth is the file is working ON currently and COULD be damaged IF you broke the training.
best_model_XX.pth is a copy of the the LAST epoch it worked on successfully.

So the theory I have goes like this:

In that folder you can rename your best_model.pth to something else e.g. best_model.pth.OLD

Then re-name your best_model_XX.pth to best_model.pth

Start the finetuning application and you should be able to go to Step 3 and load that model now and test it.

Assuming that works, you can go to "What to do next" tab and save that model out. Just follow the instructions given there to save it to your finetuned model folder. (Note this will over-write any other model in that location if it already exists).

In theory, if that's all done, you should be able to exit and reload finetuning now re-using that model to continue its training, re-using the same training data.

Thanks