Background document for ISMIR 2000 on Music Information Retrieval Evaluation

Tim Crawford 11 October 2000

Contents

1. Candidate Music Test Collections
2. The 'Cranfield' model for Information Retrieval Evaluation

1. Candidate Music Test Collections

NB: appearance in this list means only that we think the collection exists in machine-readable form somewhere and might be available; there are serious copyright as well as other availability issues for most of these!

Exception: collections whose name is prefixed with "*" don't yet exist, as far as we know.

CMN = some form of Conventional Music Notation (perhaps very incompletely encoded)

image = GIF, TIFF, JPEG, etc.; needs Optical Music Recognition to be useful

SMF = Standard MIDI File (the standard encoding of Time-Stamped MIDI)

2. The 'Cranfield' model for Information Retrieval Evaluation

Tim Crawford, 11 October 2000

The model is first described in a classic article from the 1960s:

CLEVERDON, C.W., MILLS, J. and KEEN, M.: Factors Determining the Performance of Indexing Systems, Volume I - Design, Volume II - Test Results, ASLIB Cranfield Project, Cranfield (1966).

This article is reprinted in:

SPARCK JONES, K., and WILLETT, P., eds. Readings in Information Retrieval (San Francisco: Morgan Kaufmann, 1997).

It is also covered in some depth in:

van RIJSBERGEN, C. J.: INFORMATION RETRIEVAL (first edn: London: Butterworths, 1975). (online version at: http://www.dcs.gla.ac.uk/Keith/Preface.html )

and many others (apologies for omissions).

The following excellent work doesn't seem to mention either Cleverdon or the Cranfield model, but it includes a good discussion of IR evaluation in general and of TREC, which is based on the model:

WITTEN, I., MOFFAT, A and BELL, T: Managing Gigabytes (current edn.: Morgan Kaufmann, 1999).

Van Rijsbergen sums up the idea thus (here quoted from http://www.dcs.gla.ac.uk/Keith/Chapter.7/Ch.7.html without permission) :

The ... question (what to evaluate?) boils down to what can we measure that will reflect the ability of the system to satisfy the user. ... Cleverdon ... listed six main measurable quantities:

1. The coverage of the collection, that is, the extent to which the system includes relevant matter;
2. the time lag, that is, the average interval between the time the search request is made and the time an answer is given;
3. the form of presentation of the output;
4. the effort involved on the part of the user in obtaining answers to his search requests;
5. the recall of the system, that is, the proportion of relevant material actually retrieved in answer to a search request;
6. the precision of the system, that is, the proportion of retrieved material that is actually relevant.

It is claimed that (1)-(4) are readily assessed. It is recall and precision which attempt to measure what is now known as the effectiveness of the retrieval system. In other words it is a measure of the ability of the system to retrieve relevant documents while atthe same time holding back non-relevant ones. It is assumed that the more effective the system the more it will satisfy the user. It is also assumed that precision and recall are sufficient for the measurement of effectiveness.

On the all-important definition of 'relevance', van Rijsbergen has this to say:

Relevance is a subjective notion. Different users may differ about the relevance or non-relevance of particular documents to given questions. However, the difference is not large enough to invalidate experiments which have been made with document collections for which test questions with corresponding relevance assessments are available. These questions are usually elicited from bona fide users, that is, users in a particular discipline who have an information need. The relevance assessments are made by a panel of experts in that discipline. So we now have the situation where a number of questions exist for which the 'correct' responses are known. It is a general assumption in the field of IR that should a retrieval strategy fare well under a large number of experimental conditions then it is likely to perform well in an operational situation where relevance is not known in advance.

Musical 'relevance' remains to be defined ...