Search on a subset of documents from DocumentStore

[NormanLYJ]

In many search engines, Faceted search is a very important aspect.
User may want to search for documents that only contains certain meta data tags, products in certain price range, content published in certain time window, etc.

From the underlying DocumentStore perspective, some of them (eg Elasticsearch, Weaviate etc) already have their native api to perform this kind of filters. I am just wondering, in Haystack, is there a feasibility of applying these filters, get a subset of search space before throw it to the retriever?
Looking at the documentation, I have not find it possible yet
document_store = ElasticsearchDocumentStore(host="localhost", username="", password="", index="document")

This is actually becoming a critical part in my project. One possible way is to do the search on all the docs first, then perform the filters on my downstream flow. However, this is inefficient, and also, typically we are obtaining top K outputs from the search, if we apply filtering on theses K result, there is a good chance none of them meet the criteria.

I know this is very search specific use case, but I think it is a big one.
If anyone knows how to do this, please suggest me. Much appreciated

[Timoeller]

Hey @NormanLYJ we think this is a quite common use case and some of our engineers worked on it in our hacky friday last week : )

@erendabanlioglu could you post some of the wireframes you created and see if our solutions could be a fit to the problem described here?

[erendabanlioglu]

Of course! Just a disclaimer, it is an early draft for our idea session, I wouldn't call it a final wireframe :) There are also duplicated and dummy labels for the sake of design.

On the search page of the haystack hub when the user wants to have an advanced filter these metadata filtering options would appear. The groups could be preselected by the user or determined by the use case. On selection of filters the search would rerun and user would get new set of answers. Of course, this is on the Hub UI and it would be supported on the API naturally. I can't assume how it would be implemented on the framework, someone with more experience can give an idea.

Something similar to this would help you @NormanLYJ ? Please feel free to give feedback what would help you.

[NormanLYJ]

Thank you @Timoeller and @erendabanlioglu for sharing this.
That is something similar to what we are trying to build. But at this stage, I am tying to focus more on the API side.
Are you able to share anything on that perspective as well? Is the search performed on a filtered dataset? if so, how was that implemented with the DocumentStore.

Thanks very much

[Timoeller]

Thanks for following up. I also realized that part of your question is just about filtered search.

TL;DR

our retrievers can use filtering in the retriever.retrieve(query, filters, ...) method:
You index the data with a "meta" field attached like

dicts = [{
             'text': "Done have much to say about it. It work fine. Don't see the different between this and any other "
                     "mouse pad. It just look and feel like a plastic that is cut into the shape of a mouse pad.",
             'meta': {'item_id': 'B0017D5Z40', '_split_id': 0}}]

doc_store.write_documents(dicts)

and then your filter should look like: filter = {'item_id': ['B0010T3QT2']}

Details

Please have a look at our blog post about filtering here: https://www.deepset.ai/blog/metadata-filtering-in-haystack

[NormanLYJ]

Thanks @Timoeller
That is what I am looking for.

I will definitely read more articles from this blog, they look insightful, surprisingly I didn't notice it yet...