Article Dans Une Revue ACM Transactions on Information Systems Année : 2025

Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing

Résumé

Query Performance Prediction (QPP) estimates the effectiveness of retrieval systems for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, a critical gap remains in understanding how well QPPs generalize across diverse retrieval paradigms and collections, a question of robustness that has significant implications for their practical utility. This paper provides the first comprehensive cross-paradigm evaluation of QPP robustness and generalization capabilities, examining state-of-the-art QPPs including NQC, WIG, LETOR-based features, and newly explored dense-based predictors MQPPF and BERT-QPP. We systematically assess their performance across diverse sparse (BM25, DFree with and without query expansion), hybrid (SPLADE), and dense (ColBERT, TCT-ColBERT) rankers on four benchmark collections: TREC Robust, GOV2, WT10G, and MS-MARCO. The results reveal fundamental robustness challenges: predictors exhibit significant variability in accuracy, with collection being the dominant factor, followed by ranker type. Some sparse predictors perform adequately on specific collections such as TREC Robust and GOV2, but critically fail to generalize to other collections like WT10G and MS-MARCO. Dense-based predictors, while showing promise in specific scenarios with dense rankers, similarly lack generalization to sparse contexts. We demonstrate that these generalization failures severely limit practical applications: QPP-driven selective query processing achieves only marginal gains (≈4% NDCG improvement), with reliability varying dramatically across settings. Our findings underscore that current QPP methods lack the robustness necessary for real-world deployment and highlight the urgent need for predictors that generalize reliably across diverse collections, align with modern dense retrieval architectures, and provide consistent utility for downstream applications. We publicly release our data and code to facilitate future research on robust QPP methods 1 .

Fichier principal
Vignette du fichier
3774427.pdf (2.71 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
licence

Dates et versions

hal-05379351 , version 1 (24-11-2025)

Licence

Identifiants

Citer

Adrian-Gabriel Chifu, Sébastien Déjean, Moncef Garouani, Josiane Mothe, Diégo Ortiz, et al.. Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing. ACM Transactions on Information Systems, 2025, ⟨10.1145/3774427⟩. ⟨hal-05379351⟩
4132 Consultations
346 Téléchargements

Altmetric

Partager

  • More