[Question] How the answers are aggregated (from different documents) ?

[gabinguo]

Hey all,

I have a simple question that bugs me a lot... Hope to get some help from the community of Haystack...

Context:

Benefit from the Haystack library, we are able to use Reader to infer multiple documents retrieved by Retriever in order to get the final answers. And when the document is too long we chunkify them into chunks that fit the max_seq_len limitation. However, the final goal is to provide the best-sorted answers for the end-users... And for doing that, we will need to aggregate the answers from different chunks of the same document (possibly, only when the document is longer than max_seq_len), and also aggregate the answers from different documents...

My question will be:

• How do we aggregate the answers?
  • based on the start_logit + end_logit for each answer?
• How do we softmax the logits to get probabilities?
  • softmax the chunks from the same document?
  • softmax all the start_logit + end_logit from all answers?

Haystack FARMReader currently outputs answers with reasonable probabilities from different documents, I would like to ask under the hood what's the mechanism?

Thanks in advance for all the advice : )

[julian-risch]

Hi @gabinguo in this context I would like to quote the famous saying "A line of code is worth a thousand words." 😉 So I would recommend that you go through the lines of code in the logits_to_preds method of the QuestionAnsweringHead here:

haystack/haystack/modeling/model/prediction_head.py

Line 444 in f7d0047

# get scores for all combinations of start and end logits => candidate answers

The other method you want to look at is aggregate_preds:

haystack/haystack/modeling/model/prediction_head.py

Line 666 in f7d0047

def aggregate_preds(self, preds, passage_start_t, ids, seq_2_start_t=None, labels=None):

In short, softmax is calculated only for all answers in the same passage and then there is only one more complex trick we do to compare answers from different passages and no_answers in reduce_preds here:

haystack/haystack/modeling/model/prediction_head.py

Line 790 in f7d0047

# "no answer" scores and positive answers scores are difficult to compare, because

[gabinguo]

Thanks a lot : ) Exactly what I need.

[ghost]

really thanks :) its perfect