Algorithms and methods for evaluating the effectiveness of information retrieval approaches, taking into account the user's browsing behaviour.
Traditional IR evaluation is based on implicit assumptions about the user's interaction with the search system: It is assumed that the user is presented with a ranked list of results and examines the returned documents one after the other in the order they are listed. However, such a model is obsolete in the case of the Web and structured document retrieval, where browsing is an integral part of the user's search strategy.
The goal of this project is to better understand users' post-query browsing behaviour, and to develop appropriate evaluation measures that incorporate richer models of user interaction, taking into account browsing.
Our approach to capture retrieval effectiveness for query and navigation based search is to introduce a measure of retrieval effectiveness that comprises a probabilistic model of the users’ post-query navigation. Building on this, our measure of effort-precision and gain-recall (ep/gr) allows to explicitly model both the benefits and costs of discovering additional relevant information while browsing from a result in the ranked list. Our measure of Structural Relevance (SR) builds on a model of tree retrieval and a Markovian model of user navigation which eliminates the need for the computation of an ideal ranking.
- Our paper on SR, presented at CIKM 2008, won Best paper runner up prize.
Project team and collaborators
- Sadek Ali (University of Toronto)
- Mariano Consens (University of Toronto)
- Gabriella Kazai (Microsoft Research)
- Mounia Lalmas (University of Glasgow)
- Natasa Milic-Frayling (Microsoft Research)
- Benjamin Piwowarski (University of Glasgow)
- Stephen Robertson (Microsoft Research)
- Andrew Trotman (University of Otago)
- M S. Ali, Mariano P. Consens, Gabriella Kazai, and Mounia Lalmas, Structural relevance: A common basis for the evaluation of structured document retrieval, in CIKM '08: Proceeding of the 17th ACM Conference on Information and Knowledge Mining, Association for Computing Machinery, Inc., October 2008.
- Gabriella Kazai, Search and navigation in structured document retrieval: comparison of user behaviour in search on document passages and XML elements, in Proceedings of The Twelfth Australasian Document Computing Symposium (ADCS'07), December 2007.
- Gabriella Kazai and Andrew Trotman, Users' perspectives on the usefulness of structure for XML information retrieval, in Studies in theory of information retrieval (Proceedings of ICTIR 2007), October 2007.