How to read in parsed dictionary of document text without getting Duplicate Documents warning?

[asharm0662]

Hi,

I am separating document parsing and question/answer prediction for the following the in-memory tutorial into separate scripts, so I can simply read in all only the dictionary object during time of prediction.

First script for document parsing:

from haystack.utils import clean_wiki_text, convert_files_to_dicts, fetch_archive_from_http
import json

from haystack.document_store.memory import InMemoryDocumentStore
document_store = InMemoryDocumentStore()

doc_dir = "data/article_txt_got"
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt.zip"
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)

# convert files to dicts containing documents that can be indexed to our datastore
# You can optionally supply a cleaning function that is applied to each doc (e.g. to remove footers)
# It must take a str as input, and return a str.
dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)

with open('data.txt', 'w') as outfile:
    json.dump(dicts, outfile)

Second script to read in the parsed data from the JSON file from the first script:

import json
from haystack.document_stores import InMemoryDocumentStore


document_store = InMemoryDocumentStore()



with open('data.txt') as jsonfile:
    dicts = json.load(jsonfile)

document_store.write_documents(dicts, duplicate_documents='overwrite')

This line document_store.write_documents(dicts, duplicate_documents='overwrite'), returns back,
INFO - haystack.document_stores.base - Duplicate Documents: Document with id '9e9a3181b6bc168b4a25429b641e8c86' already exists in index 'document', for all the documents every time I read them in.

Why is this the case? How can I avoid this output when I am reading in the data from the JSON file that has already been parsed?

Is there anyway to write out the document_store variable and read that back in to avoid the warning message?

[ZanSara]

Hello @asharm0662, I'll admit this sounds like a bug. I'm now trying to replicate this behavior, so I can have a closer look at what's going on.

Do you mind if I convert this discussion into an issue? This way we can track it better, especially if it ends up being a real bug.

[asharm0662]

@ZanSara not at all, please convert as you see fit. thank you for your help. I look forward to solution.

[ZanSara]

The issue was handled in #2035. It was due to actual duplicates in the source dataset.

[asharm0662]

@ZanSara i am still getting the same duplicate error with the solution provided in #2035.

from haystack import Document
from haystack.document_stores import InMemoryDocumentStore

document_store = InMemoryDocumentStore()

with open('data.txt') as jsonfile:
    dicts = json.load(jsonfile)

docs = [Document(**content, id_hash_keys=["content", "meta"]) for content in dicts]

document_store.write_documents(docs, duplicate_documents='overwrite')

Is there anyway the InMemoryDocumentStore, can be cleared out after every run of the code? I am trying to wrap this into a serverless.

[ZanSara]

Well technically InMemoryDocumentStore is not persisted. Every run of the code creates a brand new, empty document store. So this can't be due to persisted documents: there must be duplicates in your documents list. If you still see duplicate IDs errors with this code, that can only mean that you have documents with the same content and the same metadata.

Please inspect your data.txt more carefully. If it's not too big, you can also share it here and I'll have a quick look.