Copy FAISSDocumentStore to 'production' to have a stable version while new docs get saved and indexed. #7646
Unanswered
MarcSchluperAtIntel
asked this question in
Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a FAISSDocumentStore loaded with documents, with three files: store.db, store.index, and store.json.
I open the store with
FAISSDocumentStore(faiss_index_path=".../faiss/store.index")
I can get the documents matching a question just fine, like so:
matches = retriever.retrieve(question)
While a retriever serves some customers, new documents are added to the store by others. This yields an error message in the next retrieve call:
ValueError: The number of documents in the SQL database (NNN) doesn't match the number of embeddings in FAISS (MMM). Make sure your FAISS configuration file points to the same database that you used when you saved the original index.
To work around this, when the retrieve was working, I made a copy of the three files to a subfolder (production), and open the copy using
FAISSDocumentStore(faiss_index_path=".../faiss/production/store.index")
I was hoping to detach from the loading process (write_documents and update_embeddings), but I still get the error.
Beta Was this translation helpful? Give feedback.
All reactions