Measuring Search Effectiveness

Speaker:  Ellen M Voorhees – Gaithersburg, MD, United States
Topic(s):  Information Systems, Search, Information Retrieval, Database Systems, Data Mining, Data Science


In contrast to most areas of computer science research, information retrieval research has a rich tradition of experimentation dating back to the 1960s. This experimental methodology involves comparing retrieval system effectiveness using test collections, a type of benchmark task. Because test collections were introduced in a series of experiments run at the College of Aeronautics, Cranfield, England, UK, the methodology has become known as `Cranfield'.

In 1981, Karen Sparck Jones edited a collection of papers on information retrieval evaluation[1].  The volume was dedicated to Cyril Cleverdon, principal architect of the Cranfield experiments, and it was published to reflect the state of IR experimentation 20 years after Cranfield.  In the introduction to the volume, Sparck Jones observes "It is arguable that our current understanding of information processing is like that of sixteenth century herbalists: it embodies some observation and insight, but lacks detailed analysis and supporting theory."

Ten years after the book was published, the National Institute of Standards and Technology began the TREC project that had the explicit goal of facilitating IR research through building test collections and standardizing evaluation procedures.  Today, another twenty-five years later, search is routinely used in ways unimagined in 1981, and the number of IR investigations is likewise well beyond anything anticipated then.  Nonetheless, fifty-odd years after its introduction, Cranfield remains the dominant experimental paradigm in the field. This talk will describe how the Cranfield methodology has evolved and how it is used today to evaluate search effectiveness.

[1] "Information Retrieval Experiment", edited by Karen Sparck Jones.


Butterworths, 1981. 

About this Lecture

Number of Slides:  52
Duration:  60 minutes
Languages Available:  English
Last Updated: 

Request this Lecture

To request this particular lecture, please complete this online form.

Request a Tour

To request a tour with this speaker, please complete this online form.

All requests will be sent to ACM headquarters for review.