How can i estimate training time of deepspeech

[JRMeyer]

>>> rao_jjn
[January 11, 2018, 10:45am]

I am running deepspeech training on Tedlium-2 and Mozzila open voice. slash
in approximate 12 hours it reached

I used batch size 8 for training, validation and test data with
-use_warpctc option. Except these I am using default options. slash
Approximately how much time my training should take. slash
I am using 1 GeForce GTX 1080 GPU.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/how-can-i-estimate-training-time-of-deepspeech]

[JRMeyer]

>>> lissyx
[January 11, 2018, 11:22am]

How big is your overall dataset? TED-LIUM2 + CommonVoice ? That would be
slash ~1000 hours of audio ?

[Archived Post]

[JRMeyer]

>>> rao_jjn
[January 11, 2018, 12:15pm]

It should be between 300 - 500 hours. I have used valid sets only

[Archived Post]

[JRMeyer]

>>> lissyx
[January 11, 2018, 1:28pm]

I think that ballpark for slash ~500 hours on our cluster with 16 TITAN X
GPUs is around 2h for one epoch, so 12 hours for one epoch on your 1080
seems consistent.

[Archived Post]

[JRMeyer]

>>> rao_jjn
[January 12, 2018, 7:22am]

What total training time I can expect ? Any idea ?

[Archived Post]

[JRMeyer]

>>> rao_jjn
[January 15, 2018, 5:33am]

Hello,

Can you provide some hints about the total training time in my case ?
Thanks a lot.

[Archived Post]

[JRMeyer]

>>> lissyx
[January 15, 2018, 8:21am]

I think I just did ...

[Archived Post]

[JRMeyer]

>>> rao_jjn
[January 16, 2018, 7:34am]

Does it mean that on my hardware, to complete 75 epochs it will take
slash ~38 days ?

[Archived Post]

[JRMeyer]

>>> Tilman_Kamp
[January 16, 2018, 10:12am]

Yes. Running one full epoch will give you a better estimate.

[Archived Post]

[JRMeyer]

>>> adryan
[February 5, 2018, 11:51am]

How much time an epoch will last on TED-LIUM2 if using newly NVIDIA
Titan V ? Or what would be the speed up over Titan X Pascal? Will the
tensor cores help ?

[Archived Post]