Training ongoing for 60+ hours on a RTX 4090, when will it be done

[ericq]

Thanks for the good work.

We're trying to re-create the training process here. We're using a i9, 64G, RTX 4090 GPU. It has been running for more than 60 hours. the current iteration is 31000. The process took about 9G GPU mem and the GPU utilization is at 84%.

The ML dashboard cannot be run for some package reasons.

When will the training be done?

Thanks.