Questions/Answers/Hints

Q: What is a "reasonable assumption" when the trfile doesnt contain all the relevant files? I have no idea what is reasonable for the retrieval engine used to create the trfile.

A: Your assumptions do not necessarily have to take into account which retrieval engine you're using. You just need to make an assumption as to how the relevant documents that were not retrieved are distributed in the set of documents that were not retrieved. For example, a somewhat unreasonable assumption would be that all relevant documents are retrieved at the very end of the ranked result list.

-- ToniRath - 21 Oct 2003

Q: What is the precision at 0% recall level?

A: Precision at 0% recall level is the highest precision value at any recall point greater than or equal to 0%. That's how interpolation works for all recall values (answer provided by J. Allan).

Q: What is the "final level" mentioned on the slides that describe the expected search length (ESL) evaluation measure?

A: or the calculation of the ESL for a number n, the "final level" is the rank in which the number of retrieved relevant documents reaches n.

Q: How do I calculate the expected search length (ESL) for n, if there are fewer relevant documents than n for a particular query?

A: In that case the ESL is undefined. Please report <undefined> as the value for the ESL of that query (you need to report all other measures that you can calculate).

-- ToniRath - 20 Oct 2003

P3-big??.top files: These files are missing the top 3 lines. Please check before you run your code on them. Update (Oct 20): 3 lines have been added to these files.

-- ToniRath - 15 Oct 2003

Q: Where can I get the total number of relevant documents? I need that to calculate recall.

A: You can find that information in the relevance files on the project page (the final relevance file may not yet be available for download).

-- ToniRath - 14 Oct 2003
to top