On the Feasibility and Robustness of Pointwise Evaluation of Query Performance Prediction

Document Type

Conference Article

Publication Title

CEUR Workshop Proceedings

Abstract

Despite the retrieval effectiveness of queries being mutually independent of one another, the evaluation of query performance prediction (QPP) systems has been carried out by measuring rank correlation over an entire set of queries. Such a listwise approach has a number of disadvantages, notably that it does not support the common requirement of assessing QPP for individual queries. In this paper, we propose a pointwise QPP framework that allows us to evaluate the quality of a QPP system for individual queries by measuring the deviations between each prediction versus the corresponding true value, and then aggregating the results over a set of queries. Our experiments demonstrate that this new approach leads to smaller variances in QPP evaluations across a range of different target metrics and retrieval models.

First Page

1

Last Page

6

Publication Date

1-1-2023

This document is currently not available here.

Share

COinS