Resume from last correct checkpoint when restarting a pytorch job

[ppnaik1890]

Describe the bug

Incase the pytorch job is restarted it should choose the last clean checkpoint instead of the latest checkpoint folder since it may be of an incomplete checkpoint.

Sample Steps

Start a pytorch job
Stop and start the job

Expected behavior

It should cleanly start from the last checkpoint.

Observed behavior

It is unable to start the job and the job errors out with following error

FileNotFoundError: [Errno 2] No such file or directory: '/cos1/output/sg/fine-tuned-tiny-llama-new3/checkpoint-25/trainer_state.json'

This is because it tries to start from a last checkpoint folder even if its not the completed checkpoint folder

[SilverSoldier]

Seems to be HF Trainer related. Added a feature request in Transformers: huggingface/transformers#35525