Measuring and Comparing the Consistency of IR Models for Query Pairs with Similar and Different Information Needs

Document Type

Conference Article

Publication Title

International Conference on Information and Knowledge Management, Proceedings

Abstract

A widespread use of supervised ranking models has necessitated an investigation on how consistent their outputs align with user expectations. While a match between the user expectations and system outputs can be sought at different levels of granularity, we study this alignment for search intent transformation across a pair of queries. Specifically, we propose a consistency metric, which for a given pair of queries - one reformulated from the other with at least one term in common, measures if the change in the set of the top-retrieved documents induced by this reformulation is as per a user's expectation. Our experiments led to a number of observations, such as DRMM (an early interaction based IR model) exhibits better alignment with set-level user expectations, whereas transformer-based neural models (e.g., MonoBERT) agree more consistently with the content and rank-based expectations of overlap.

First Page

4449

Last Page

4453

DOI

10.1145/3511808.3557637

Publication Date

10-17-2022

This document is currently not available here.

Share

COinS