How to make search engine as good as google with haystack?

[superchargez]

I have tried both generative (tutorial 12) and question answering models (tutorial 1) but neither of them is good enough. You can simply ask google "who won battle of the bastards" and it gives you correct answer, or ask something like "how high is the north wall" but the models do not yield such good (or correct) answers.

[brandenchan]

Hi @superchargez , nice to see another Game of Thrones fan here! Sorry that Haystack hasn't been performing as well as you might have expected.

I tested out both those questions and what I noticed first was that the database from which Google and Haystack are pulling from are very different corpora. To the question about the battle of the bastards, my Google search returns a result from www.screenrant.com and to the question about the wall, Google returns a page from the Game of Thrones Fandom. By contrast, Tutorial 1 and 12 uses only a subset of the Game of Thrones Wikipedia pages.

If you want to properly evaluate these questions using Haystack, I would recommend scraping those pages and indexing them into your document store. It's worth noting also that there many model options for Retriever and Reader that could boost the performance of your Haystack system. Please keep us updated on how it goes for you!

[superchargez]

@brandenchan oh you are a fan as well, nice to meet you too. :)

Thank you, I have tried DPR as well as retribert (from yjernite) for retrieval and have tried both RAG and Seq2Seq model for generator (and also tried with reader used in QA system (Tutorial 1) with varying answers). It is understandable that generators will generate different answers every time but retriever should obtain the correct files and get their answers (for reader/generator).

It appears that ES's generic retriever (BM25) is surprisingly doing well job even though it is a BoW.

Question now: How comes google know where to look for the answer? I mean there are many sites where google might have searched for information and some of them might have gotten wrong answers, for example it is possible that someone might have typed 1700 instead of 700 for height of the North Wall. So, how comes google knows where to look for the answers? Or is it that google MUST look for the right answer from prescribed sites (for GoT it is screenrant.com, and for medical information it is some other particular site)? If this is the case then perhaps it is impossible to match let alone beat google by a mere student (like myself).
Question 2: (if above question is answered 'yes'); Can we use some algorithm to find relevant information to replace man power with machine power (to find which information is correct and which is wrong, or WHICH/WHERE the information is likely to be true)?

Thank you so much again for the effort.

[PSanni]

I think the answer lies within googles' eco-system. If you search the same query with bing or any other search engine, the answer is not the same. Google uses PageRank (PR) for indexing, which has evolved over years, and with PageRank, google knows which information is more valid than another. For example, search for "Can you run your car just on water?" it comes up with a hypothetical answer posted on some low ranker website, if tesla writes on their website turning down this hypothesis, google will update itself the next day with a more relevant answer which is 'No'. So being a company or student, with a good strategy to identify and validate information, you can achieve the same accuracy as google. Because in the end, Google uses the same algorithms and models we have in ML space.