Evaluation of Retrieval with GermanDPR

[Zaker237]

Thear all,
Thank you for the amazing work done to come out with the paper and the 2 datasets GermanDPR and GermanQuAD.
I have come across your valuable paper GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval and I would like to do a replica study on it and I have a few questions, namely how to evalute a model trained on GermanDPR for retrieval:

(1) How can I obtain the corpus you used? I.e. what specific Wikipedia dump was used in combination with GenSim's WikiCorpus(it will be nice if i can get the specific version of the corpus maybe as link)?
(2) How can I use the GermanDPR test set to evaluate retrieval? As far as I can see, there are no passage IDs or similar. Once I have retrieved passages from the corpus, how can I use your labels to compute retrieval metrics (similar to your experiment in the paper, Figure 4). if there is an evaluation script available i will be happy to see it.

I look forward to your feedback and thank you in advance.

[julian-risch]

Hi @Zaker237
Download links for GermanQuAD and GermanDPR are at the bottom of this page, including the test set: https://deepset.ai/germanquad The Wikipedia dump is from June 2019. The latest dump can be found in xml format here: https://dumps.wikimedia.org/dewiki/latest/ Older dumps you would need to find from the source yourself but I think they are also available. Maybe somebody at Wikimedia can help you.

[julian-risch]

Regarding code, I can share with you scripts that we used for training and evaluation. Note that these scripts are for usage with FARM and Haystack. You can use them as a starting point but don't expect them to run out-of-the-box with the current version of Haystack.
Code for GermanDPR.zip

[mrjleo]

Hi, I have some additional questions regarding reproducing the GermanDPR results.
Currently, we are using the script you provided (germanqa_dpr_evaluation.py) as a guideline. Our concrete steps are:

1. Load a dewiki dump into elasticsearch (from https://dumps.wikimedia.org/other/cirrussearch/).
2. Create a DocumentStore, where the doc_index is our dewiki dump. We have to use custom values for search_fields, content_field and name_field as the mappings are a bit different.
3. Use document_store.add_eval_data(filename="GermanDPR/GermanDPR_test_in_squad_format.json", ...) with a PreProcessor.
4. Create a retriever with the pre-trained encoders: "deepset/gbert-base-germandpr-question_encoder" and "deepset/gbert-base-germandpr-ctx_encoder".
5. Run document_store.update_embeddings(retriever, index=doc_index).
6. Evaluate like so: retriever.eval(doc_index=doc_index, label_index=label_index, top_k=10).

This works, but I'm not sure whether we're doing it correctly. Wouldn't there be many false negatives? Going with your example below, we'd now have two docs corresponding to the article "Beleuchtung", one from the dump and one from the test set. However, only the one from the test set would have a relevance label. Am I missing something?

In your code, there is a reference to a file called "german_wiki_passages_short_filtered.csv". I guess it would be good to know how this file is created or what the format is.

Thank you in advance.

[Zaker237]

Thank for the answer and the script. it really help me to understand the evaluation Process. but still now i'm facing some other problem.
I fund the wiki dump and wanted to use it bot for the evaluation and also the training but still yet i was not able to genere the IDs of the article please do you habe any advice for me??
thank you

[julian-risch]

I'd say there is no need to generate the document IDs. Instead you can use the page titles as identifiers. For example,
In GermanDPR_test.json and GermanDPR_train.json, the title is in the field called title and it is also encoded in the text field. In the text field, it is always in the very beginning and then followed by two line breaks and equal signs \n\n===:

[
    {
        "question": "Welches ist die zweitgrößte Stadt in den Alpen?",
        "answers": [
            "Innsbruck"
        ],
        "positive_ctxs": [
            {
                "title": "Alpen",
                "text": "Alpen\n\n=== Städte ===\nInnerhalb ...",
                "passage_id": ""
            }
        ],

In the GermanDPR_test_in_squad_format.json file, there it is encoded in the title field and also in the beginning of the context field:. For example, Beleuchtung\n\n=== refers to the following Wikipedia page: https://de.wikipedia.org/wiki/Beleuchtung