Trec-Covid

Trec Covid Dataset Term Project for CmpE493 Information Retrieval

Download Related Files

topics : https://ir.nist.gov/covidSubmit/data/topics-rnd5.xml

data : https://ai2-semanticscholar-cord-19.s3-us-west-2.amazonaws.com/historical_releases/cord-19_2020-07-16.tar.gz

relevance-data : https://ir.nist.gov/covidSubmit/data/qrels-covid_d5_j0.5-5.txt

Relevance data 1st column topic-id -- 2nd column not relevant -- 3rd column document-id -- 4th column relevancy 0/1/2

result file : query-id Q0 document-id rank score STANDARD

The field query-id is a alphanumeric sequence to identify the query. The second field, with "Q0" value, is currently ignored by trec_eval, just put it in the file. The field document-id is a alphanumeric sequence to identify the retrieved document. The field rank is an integer value which represents the document position in the ranking, but this field is also ignored by trec_eval. The field score can be an integer or float value to indicate the similarity degree between document and query, so the most relevants docs will have higher scores. The last field, with "STANDARD" value, is used only to identify this run (this name is also showed in the output), you can use any alphanumeric sequence.

Relevant Papers :

We will use Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (NDCG), and Precision of top 10 results (P@10) for evaluation. You should use the official evaluation tool available at https://github.com/usnistgov/trec_eval. Some relevant papers are provided below and more papers are avaiable at https://ir. nist.gov/covidSubmit/bib.html. I also suggest you to look for other relevant publications on the Web.

Roberts, Kirk, et al. ”TREC-COVID: Rationale and Structure of an Information Retrieval

Shared Task for COVID-19.” Journal of the American Medical Informatics Association (2020). Available at https://academic.oup.com/jamia/article/27/9/1431/5828938.

Voorhees, Ellen, et al. ”TREC-COVID: Constructing a Pandemic Information Retrieval

Test Collection.” ACM SIGIR Forum (2020). Available at https://ir.nist.gov/ covidSubmit/papers/Forum_TRECCOVID1.pdf.

Chen, Jimmy, and William Hersh. ”A Comparative Analysis of System Features Used in the

TREC-COVID Information Retrieval Challenge.” medRxiv (2020). Available at https: //www.medrxiv.org/content/10.1101/2020.10.15.20213645v1

Zhang, Edwin, et al. ”Covidex: Neural Ranking Models and Keyword Search Infrastructure

for the COVID-19 Open Research Dataset.” Proceedings of the First Workshop on Scholarly 1Wang, Lucy Lu, et al. ”CORD-19: The Covid-19 Open Research Dataset.” ArXiv (2020). Document Processing. 2020. Available at https://www.aclweb.org/anthology/ 2020.sdp-1.5/.

Esteva, Andre, et al. ”Co-search: Covid-19 information retrieval with semantic search, question answering, and abstractive summarization.” arXiv preprint arXiv:2006.09595 (2020).

Available at https://arxiv.org/abs/2006.09595.