IR seminar, April 4, 2008, (this week at 9:30am in Room 142 )

Topic Evaluation, XingYi and XiaobingXue leading

Background:

This week’s session will discuss the challenges of reliably evaluating different IR systems in the scenarios of incomplete/biased/noisy relevant judgments. The goal is to understand previous endeavors on this topic, some state-of-the-art techniques and results; then we may broadly discuss any interesting ideas related to this topic or suggest plausible solutions. Previous research could be roughly put into three folds: 1) to analyze the problems caused by incomplete relevant judgments and the impact on the existing IR measurements; 2) to design new IR measurements which can be more reliable/robust when judgments are incomplete/noisy; 3) to design new techniques to judge as few docs as possible without scarifying effectiveness and reliability.

Required papers:

We list four required papers here. Since most people are familiar with Ben's work, the required number is still three actually.

• Reliable information retrieval evaluation with incomplete and biased judgements - discuss the problems and effects of incomplete/biased relevant judgments.
• How reliable are the results of large-scale information retrieval experiments? - discuss many issues with incomplete/noisy judgments, and focus on the cons/pros of "pooling"
• Minimal test collections for retrieval evaluation.
• Information retrieval system evaluation: effort, sensitivity, and reliability.

Recommended:

• Retrieval evaluation with incomplete information.
• Robust test collections for retrieval evaluation.
• Estimating average precision with incomplete and imperfect judgments.
• Forming test collections with no system pooling. and Efficient construction of large test collections. - to investigate how to build test collections by using other methods instead of pooling.

Tangential but interesting:

• Strategic system comparisons via targeted relevance judgments.
• Evaluating evaluation metrics based on the bootstrap.
• Ranking retrieval systems without relevance judgments. - an interesting paper: to investigate the limiting case of assessors that don't agree with each other at all (i.e. completely randomly judgments). and even in this case, systems could be ranked fairly reliably.

Background

• The significance of the cranfield tests on index languages. This is an very old paper which describes the cranfield methodology of evaluating different systems. Basically, to evaluate different systems, it discusses how to create query, gather relevance judgments and the role of precision/recall.
• Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness.. This paper is one of the "SWIRL" recommendation papers for IR student. It describes although the relevance judgments could be subjective and noisy, the current TREC pooling evaluation methods will still get very stable and consistent rankings for different system.

Your Questions

Please directly add your questions and thoughts to the wiki (or email to us) about this topic by Friday 6am at the latest. When you save your editing, please select "Release edit lock" so that other people can edit it immediately.

• In Buttcher et al., when SVM classifier is applied for predicting relevance, it is interesting to note that precision is quite high (~75%), while recall is low (~35%). As classifier is based on textual similarity, this seems to support "cluster hypothesis" to some extent. From this the following observation could follow. First, we could use this text-based classifier to rank documents in the collection by their probability of belonging to the "relevant". Then, we could use this ranking to retrieve more documents to judge (according to the results, a high percentage of these documents will be relevant). We could iterate this process, potentially accumulating a fuller sample of a "relevant" class after each step, which could improve the recall as more and more relevant documents are discovered. -- Michael

• The "Minimal Test Collections" and "Reliable Information Retrieval Evaluation" papers each present a method for assigning relevance judgments to unjudged documents. However, this makes the resulting qrels unique, making the reproducibility of experiments difficult. How could these evaluation techniques be integrated into the current IR research framework (e.g. should we have multiple versions of qrels floating around for each corpus, etc.)? --HenryFeild

• I wonder how much domain knowledge can be put to use in this task? Is it easier to determine relevance/nonrelevance when you have a much narrower scope of topicality, or do you start splitting hairs? I would like to see what the effects are of using some of the automatic judging techniques (particularly the ones from "Reliable Information Retrieval Evaluation") when the KL divergence will most likely be dampened due to narrower topicality. I suspect SVMs would perform better here. -- MarcCartright

• How can this traditional evaluation method be extended for evaluating personalized(or contextual) retrieval, where the notion of relevant may dependent on user(or context)? While some of current researches use click-through for mixed rank list of baseline and improved method, would TREC-style evaluation be possible? One way seems to be providing textual representation of user (or context) so that each participant can make the best use of it and have relevant judgment created by the user(or the one who understands the context). -- JinyoungKim

• The papers discuss evaluation issues for document retrieval. What about other tasks, such as passage retrieval? To what extent can judgments for passages be generated from judgments at the document level (that contain these passages)? -- Elif

to top