logging shows lesser samples compare to the actual given training samples

[Renuk9390]

Hi there,
We are finetuning Minilm and Roberta on some domain-specific data, which we have kept in sQUAD format:
Minilm is finetuning on 100 thousand samples and Roberta on 200 thousand samples.

Issue: However, during the finetuning, logging information shows it only 30 thousand (for Minilm) and 29 thousand (for Roberta).
Reference figure is attached (a line with 35%... where it is showing).

Is someone aware of this issue?
The issue might be something silly, but we are unaware of it.
Answers will be really appreciated.
Thank you.

Renuk

[Renuk9390]

Is that a number that shows a number of steps by chance?? if I am not wrong!

[TuanaCelik]

Hey @Renuk9390 sorry for making you wait for an answer here. As you suggested, the progress bar shows the number of training steps (number of batches), not the number of samples. Here you can see the relevant line of code

Hope that helps, let me know 😊

[Renuk9390]

Hi Tuana Celik, Thank you very much for this information. Sure, I will connect to you in future, we are also working on domain specific QA datasets and we made it up to 84% F1 with the models from Haystack and the suggested way of fine tuning. We would like to evaluate the complete QA pipeline including both Reader and Retriever. And we are aiming to compare it to the benchmarks what Haystack claimed, however one doubt is that both are different datasets and don’t how it could be valid comparison. But our is to produce this dataset and release to the community in future through an article. Thanks Renuk

On Fri 12. Aug 2022 at 3:28 PM, Tuana Celik ***@***.***> wrote: Hey @Renuk9390 <https://github.com/Renuk9390> sorry for making you wait for an answer here. As you suggested, the progress bar shows the number of training steps (number of batches), not the number of samples. Here you can see the relevant line of code <https://github.com/deepset-ai/haystack/blob/26c938a8e6ac4fa76a2ab96ae5eafa201172ce24/haystack/modeling/training/base.py#L269> Hope that helps, let me know 😊 — Reply to this email directly, view it on GitHub <#2987 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEU6POHPW5NZWHKQUL4ANTVYZGN7ANCNFSM554S7MHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Best, Renuk