Is there a way to know "Haystack model's overfitting/not" on our data?

[Renuk9390]

Hello There,
We are working on SquaD formatted custom data, in the logs during training what I could see is the training loss.
As a conventional approach, if there is a way for noting down "Validation loss", then I can somehow investigate overfit.

In order to get those eval details, I have added the following parameters:
param evaluate_every: Perform dev set evaluation after this many steps of training.
:param eval_report: If evaluate_every is not 0, specifies if an eval report should be generated when evaluating

haystack/haystack/modeling/training/base.py

Line 158 in bc6f71b

:param eval_report: If evaluate_every is not 0, specifies if an eval report should be generated when evaluating

However, I get the following error:
TypeError: train() got an unexpected keyword argument 'eval_report'

Please let me know if there is a way to know the overfit case for Haystack models.
Thank you,
Renuk

[bogdankostic]

Hi @Renuk9390! I assume you are using the FARMReader, please correct me if I am wrong. You are getting the TypeError, as the FARMReader's train method does not allow an argument eval_report. eval_report is a property of the Trainer that is used inside the FARMReader's train method, and it is set to True by default, so the eval report should be produced. The problem here probably is that the eval report is logged at the logging level INFO, while the default logging level in python is WARNING. In order to see the logs at the INFO level produced by Haystack, you need to add the following code snippet:

import logging
logging.basicConfig(format="%(levelname)s - %(name)s - %(message)s", level=logging.WARNING)
logging.getLogger("haystack").setLevel(logging.INFO)

Please let me know if this helps :)

[Renuk9390]

Hi @bogdankostic, thank you very much for your answer.