IR evaluation methods for retrieving highly relevant documents
Kalervo Järvelin and Jaana Kekäläinen
Department of Information Studies
University of Tampere
FIN-33014 University of Tampere
Finland
REFERENCE
Järvelin, K. & Kekäläinen, J. (2000). IR evaluation methods for retrieving highly relevant documents.
In: Belkin, N. & Ingwersen, P. & Leong, M-K. (Eds.), Proceedings of the 23th Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval (ACM SIGIR '00), Athens, Greece, July 24-28,
2000. New York, NY: ACM Press, pp. 41-48. (Receiver of the ACM SIGIR '00 Best Paper Award).
Abstract
This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments.
It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents.
This is desirable from the user point-of view in modern large IR environments. The proposed methods are (1) a novel
application of P-R -curves and average precision computations based on separate recall bases for documents of different
degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the
retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study
on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents
of various degrees of relevance. The test was run with a best match retrieval system (InQuery) in a text database
consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving
highly relevant documents. The differences between the query types are practically essential and statistically significant.
More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments
are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.
Return to Kal's home page.
Return to Kal's publication list.
Paluu Kallen kotisivulle.
Paluu Kallen julkaisuluetteloon.