Might NSP be another method to retrieve documents?

[nicir]

Hi there,

currently I am working with Haystack to train a Retriever for context retrieval in a custom domain. I am using the "DensePassageRetriever" that offers "dot_product" and "cosine" as similarity functions for training.

Important Note: I am not interested in a QA-System with a Generator like RAG. It's all about the retrieval.

--

I was wondering if it would be beneficial to use some kind of next sentence prediction as function for the retriver. An example to explain my thoughts:

Query:

• "A house is building that people, usually one family, live in."

Retrievals:

• "It takes about a year to build a house."
• "A normal family home has up to 2 storeys."
• "In the UK, over 80% of people live in detached houses."
• ...

So the structure is always a definition of something as query and a second sentence that specifies the subject (in this case of 'house') and gives extra information. Something Next Sentence Prediction would be suitable.

--

Is anything like this planned @haystack (integrating custom metrics to retrieve documents)?
Is there a recommended way to implement this function on my own, e.g. by modifying the DPR-class at this point:

I would appreciate a second opinion or an assessment if my idea would be feasible with moderate effort and if it would be beneficial at all. Thank you in advance!

[TuanaCelik]

Hey @nicir interesting!
I don't quite understand this though "a definition of something as query and a second sentence that specifies the subject"
In your example what would this second sentence be? Would this be something you expect the 'user' to input? Or are you suggesting NSP as a similarity function and we would apply it on the 'retrieval' candidates?

As a side note. I am moving this to the Discussions section :) It will make it easier to refer to in the future.

[nicir]

Thank you @TuanaCelik,

If someone queries a sentence (this could be a question or a definition of something) it would be likely that you would find a sentence that answers the question or gives extra information to the definition by using NSP.

Example 1:
Query: "Who is the president of the United States?" - (question)
It would be likely that out of the document store a sentence like this could be found by NSP:
"The president of the US is Joe Biden."
As it is reasonable this sentence follows the question.

Example 2:
Query: "The president of the United States is the head of state and head of government of the USA." - (definition)
It would be likely that out of the document store a sentence like this could be found by NSP:
"The president also plays a leading role in federal legislation and domestic policymaking."
As it is reasonable this sentence follows the definition.

So yes, I am suggesting to use NSP as similarity function (altough I know it has practically nothing to do with similiarity itself).

Do you think this would be worth a try and could be implemented with reasonable effort?

[TuanaCelik]

Ok now I get what you're proposing.. Maybe @julian-risch can shed more light here but from what I understand of NSP I would bring up a few things:

• If you're suggesting using a pre-trained model that can predict NSP, it would mean that each document and sentences in it has to go through an additional inference step. I can imagine this bringing down the speed of retrieval.
• BERT models are trained using NSP. So, say you're using models to create embeddings (as you would with dpr or embedding retrieval) - you're already feeding your documents through models that have been trained with NSP to create embeddings :)
• NSP is trained for a different purpose to retrieval. The purpose is like you say, predict if sentence a would have b after it. Therefore the score you get don't tell you much about document relevance per se.

It's an interesting idea to think about though! Julian had a good example. Imagine a document starting with the sentence 'That is a goof question'. -> likely to follow any question. And NSP is likely to say 'yes, this sentence follows your query: "Who is the president of the United States?"'

[nicir]

I am using a BERT model.

I think @julian-risch's example is right. For my specific use case I think it would still be worth trying. But I see that for general usage it is no feasible way. Thanks for the discussion, though!

Do you still see a way to easily implement it into Haystack? I would have to deeply dig into the libs, right? Probably a complicated thing to get it running.

[TuanaCelik]

The simplest way I can think of for testing purposes is going to the specific retriever you want to use and creating an 'nsp' similarity function option, catching it in an if statement and taking it from there: https://github.com/deepset-ai/haystack/blob/main/haystack/nodes/retriever/dense.py#L100

Or you can also try to add it in prediction_head.py
Then you add the option to all retrievers. So you would have to add some checks to make sure the option is only useable where appropriate.

[TuanaCelik]

On another note @nicir - I don't know if you're already a part of it but we have a Haystack discord server. I'm sure people would be happy to chat about ideas there too so if you're up for it, come join us :)