Ben Carterette @ the University of Massachusetts

Research

I am interested in low-cost and robust evaluation of information retrieval systems.

Information retrieval evaluation requires relevance judgments indicating whether each document in a collection is relevant to each query used to test the system. Since we need human assessors to read and judge each documents, acquiring relevance judgments is very expensive (especially for collections of reasonable size, i.e. hundreds of thousands to billions of documents!).

The expense of acquiring relevance judgments limits the types of problems that can be studied and the data they can be studied on. Solving this problem opens the door to vast amounts of data for problems both old and new in information retrieval research.

My work is focused on reducing the cost of acquiring relevance judgments by:

Maximizing the value of each individual judgment by intelligent selection of documents to judge;
Maximizing the amount of information relevant to evaluation questions that can be obtained from any small set of judgments.

Curriculum Vitae

Publications

2008
- Evaluation Over Thousands of Queries
  Ben Carterette, Virgil Pavlu, Evangelos Kanoulas, James Allan, and Javed A. Aslam. SIGIR 2008.
- Evaluation Measures for Preference Judgments
  Ben Carterette & Paul N. Bennett. SIGIR 2008.
- Here or There: Preference Judgments for Relevance
  Ben Carterette, Paul N. Bennett, D. Maxwell Chickering, and Susan T. Dumais. ECIR 2008.
2007
- Million Query Track 2007 Overview
  James Allan, Ben Carterette, Javed A. Aslam, Virgil Pavlu, Blagovest Dachev, and Evangelos Kanoulas. TREC Notebook 2007.
- Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks
  Ben Carterette & Rosie Jones. NIPS 2007.
- Hypothesis Testing with Incomplete Relevance Judgments
  Ben Carterette & Mark D. Smucker. CIKM 2007.
- Semiautomatic Evaluation of Retrieval Systems Using Document Similarity
  Ben Carterette & James Allan. CIKM 2007.
- A Comparison of Statistical Significance Tests for Information Retrieval Evaluation
  Mark D. Smucker, James Allan, and Ben Carterette. CIKM 2007.
- Robust Test Collections for Retrieval Evaluation
  Ben Carterette. SIGIR 2007.
- Research Methodology in Studies of Assessor Effort for Information Retrieval Evaluation
  Ben Carterette & James Allan. RIAO 2007.
2006
- Minimal Test Collections for Retrieval Evaluation
  Ben Carterette, James Allan, and Ramesh Sitaraman. SIGIR 2006.
  Winner of the Best Paper Award at SIGIR 2006!
- Learning a Ranking from Pairwise Preferences
  Ben Carterette & Desislava Petkova. SIGIR 2006.
- N Semantic Classes Are Harder Than Two
  Ben Carterette, Rosie Jones, Wiley Greiner, and Cory Barr. ACL/COLING 2006.
2005
- Incremental Test Collections
  Ben Carterette & James Allan. CIKM 2005.
- When Will Information Retrieval be Good Enough? User Effectiveness as a Function of Retrieval Accuracy
  James Allan, Ben Carterette, and Joshua Lewis. SIGIR 2005.
- Comparing Inverted Files and Signature Files for Searching a Large Lexicon
  Ben Carterette & Fazli Can. Information Process and Management 41(3):613-633 (2005). (abstract)
2003
- Ranking Document Retrieval Systems with BLEU
  Ben Carterette, James Allan, and W. Bruce Croft. CIIR Tech Report.

Code

mtc-eval: an implementation of the evaluation model described in Carterette (2007) and Carterette, Allan, & Sitaraman (2006).
R package for the likelihood maximization described in Carterette & Allan (2007).
To install, start R and type
install.packages("preferences", repos="http://ciir.cs.umass.edu/~carteret")
To get started, see help(maximize). More documentation coming soon...

Experience

Research Assistant, CIIR, UMass Amherst - Sep '03 to present
Advisor: Prof. James Allan
Intern, Microsoft Research - Jun '07 to Aug '07
Supervisor: Dr. Susan Dumais
Intern, Yahoo! Research - Jun '06 to Aug '06
Supervisor: Dr. Rosie Jones
Intern, Yahoo! Research - Jun '05 to Aug '05
Supervisor: Dr. Rosie Jones
Research Assistant, Miami University - '98-'99, '02-'03
Advisor: Prof. Fazli Can

Professional Activities

PC Member for SIGIR '07, ECIR '08, SIGIR '08, EVIA '08.
Reviewer for TOIS, JASIST, IR, IP&M.
Referee for InfoScale '07, InfoScale '08.
Referee for ADVIS '04 and '06.
Member of ACM and ACM SIGIR.

Other

My Erdös Number is no greater than 3 (and almost certainly not less!):
- I wrote "Minimal Test Collections for Retrieval Evaluation" with James Allan and Ramesh Sitaraman;
- Ramesh Sitaraman wrote "Augmented Ring Networks" with William Aiello, Sandeep Bhatt, Fan Chung, and Arnold Rosenberg;
- Fan Chung wrote "On the Decomposition of Graphs into Complete Bipartite Subgraphs" with Paul Erdös.