ECIR2021.An Enhanced Evaluation Framework for Query Performance Prediction

[soroush-ziaeinejad]

Why did I choose this paper? QPP is concerned with estimating the effectiveness of a query within the context of a retrieval model. Thus, reviewing the evaluation methods for QPP can be highly related to the second part of my thesis.

Main problem:

This paper examines the existing evaluation methods that are commonly used for Query Performance Prediction. In addition, the authors propose a new approach for QPP.
Applications:

• Feedback to the user (how to change the query)
• Feedback to the system (query expansion)
• Conversational search (clarifying question)
• Query suggestion

QPP tasks:

• Classifying queries
• Predicting the effectiveness of retrieval
• ranking queries based on effectiveness

Existing work:

• Pre-retrieval predictors: analyze query and corpus statistics prior to retrieval
• Post-retrieval predictors: analyze the retrieval results
• GAPS
  • Single value as the performance
  • hard to interpret
  • Unable to find the hard queries
  • Cannot completely analyze the effects of different components
  • Hard to generalize

Inputs:

• A list of queries
• A corpus of documents
• Retrieval models

Outputs:

the distribution to show the performance of input query set

Method:

Advantages:

• outputs a distribution for the QPP task instead of relying on point estimates.
• explores the benefits of using multiple query formulations and ANalysis Of VAriance (ANOVA) modeling
• The ability of factor analysis
• The Ability of failure analysis

Experimental Setup:

Dataset:
TREC Robust 2004 (Robust04): 528K documents, 249 topics

Parameters:

• 16 QPP models
• 4 different stoplists + no stop
• 2 different stemmers + no stem
• Average Precision (AP) to measure the effectiveness of the different retrieval pipelines

Metrics:

• Avg Precision (AP) or nDCG
• Correlation
• AP induced scaled Absolute Rank Error (sARE_AP )

Baselines:

• Pre-retrieval
  • SCQ, AvgSCQ, MaxSCQ: CF-iDF to corpus and query terms
  • SumVAR, AvgVAR, MaxVAR: CF-iDF variability to corpus and query terms
  • AvgIDF, MaxIDF: iDF value for query terms.
• Post-retrieval
  • Clarity: Language model divergance (for top documents and whole corpus)
  • NQC: Standard deviation of top documents
  • WIG: mean retrieval score comparison between top documents and whole corpus
  • SMV: scoresو standard deviationو and magnitude (?)
  • UEF: similarity of the initial result list with the re-ranked list

Code:

The code of this paper is available on: https://github.com/Zendelo/QPP-EnhancedEval

Presentation:

The presentation of this paper is available on: https://www.youtube.com/watch?v=TOd1W1rujbg

[hosseinfani]

@soroush-ziaeinejad where is the body?!