Handling duplicates for uploaded files

[vivianonlea]

Hello! So I have been trying to implement the version control feature for files that has been uploaded to haystack. For now, here are some of my observations:

1. If I upload two identical files "A.txt" with content "A", only one "A.txt" will show up
2. If I upload file "A.txt" with content "A", and then upload file "A.txt" with content "B", I will only get the second file that is "A.txt" with content "B"
3. If I upload file "A.txt" with content "A", and then upload file "B.txt" with content "A", "B.txt" will replace "A.txt"

I am just curious why this is happening and what part of the code should I modify so it doesn't replace files with the same name/content by default? Thanks!

[ZanSara]

Hello @vivianonlea! Sorry for the late reply. I moved this issue into Discussions as this sounds more like a general question to me.

The behavior you describe is by design. Haystack is trying to have no duplicate content in the document store, so it does its best to detect them and overwrite them when found. In general I recommend to stick with this, as removing duplicates improves query time significantly.

If you have a good reason to keep the duplicates, there are workarounds. One is id_hash_keys: https://haystack.deepset.ai/reference/primitives#document

• id_hash_keys: Generate the document id from a custom list of strings that refer to the documents attributes. If you want ensure you don't have duplicate documents in your DocumentStore but texts are not unique, you can modify the metadata and pass e.g. "meta" to this field (e.g. ["content", "meta"]). In this case the id will be generated by using the content and the defined metadata.

Hope it helps!