Textual Entailment in Haystack!

[anakin87]

Hello everybody!

Inspiration

As you may know, I recently released a HF Space about fact-checking using Haystack.
The idea is very simple:

• the user enters a factual statement
• the relevant passages are retrieved from the document store using dense retrieval
• the system computes the textual entailment between each relevant passage and the statement, using a Natural Language Inference model
• the entailment scores are aggregated to produce a summary score.

You can find more details here.

Textual entailment node

For the textual entailment task, using NLI models, I developed a simple custom Haystack node (EntailmentChecker).
My implementation is very coarse and for the moment it enriches the document's metadata with information about the entailment with the statement to be verified. The summary scores are computed outside of this node.

Potential use cases

• Fact-checking
• Yes/No questions

If you think that it can be interesting to integrate this node into Haystack, I can work on it, refactor it and with your help make it suitable for the project.
Otherwise, if a specific node is not needed, but the task is interesting we can figure out how to perform this task with already existing nodes.

[vblagoje]

Hello @anakin87, first of all, apologies for missing your post. And thank you for such a detailed write-up. I'll consult with my colleagues and get back to you in a few days. The EntailmentChecker node looks 👍