Research
I am interested in low-cost and robust evaluation of information retrieval systems.
Information retrieval evaluation requires relevance judgments indicating whether each document in a collection is relevant to each query used to test the system. Since we need human assessors to read and judge each documents, acquiring relevance judgments is very expensive (especially for collections of reasonable size, i.e. hundreds of thousands to billions of documents!).
The expense of acquiring relevance judgments limits the types of problems that can be studied and the data they can be studied on. Solving this problem opens the door to vast amounts of data for problems both old and new in information retrieval research.
My work is focused on reducing the cost of acquiring relevance judgments by:
- Maximizing the value of each individual judgment by intelligent selection of documents to judge;
- Maximizing the amount of information relevant to evaluation questions that can be obtained from any small set of judgments.
Curriculum Vitae
Publications
- 2008
- Evaluation Over Thousands of Queries
Ben Carterette, Virgil Pavlu, Evangelos Kanoulas, James Allan, and Javed A. Aslam. SIGIR 2008.
- Evaluation Measures for Preference Judgments
Ben Carterette & Paul N. Bennett. SIGIR 2008.
- Here or There: Preference Judgments for Relevance
Ben Carterette, Paul N. Bennett, D. Maxwell Chickering, and Susan T. Dumais. ECIR 2008.
- 2007
- 2006
- 2005
- 2003
- Ranking Document Retrieval Systems with BLEU
Ben Carterette, James Allan, and W. Bruce Croft. CIIR Tech Report.
Code
- mtc-eval: an implementation of the evaluation model described in Carterette (2007) and Carterette, Allan, & Sitaraman (2006).
- R package for the likelihood maximization described in Carterette & Allan (2007).
To install, start R and type
install.packages("preferences", repos="http://ciir.cs.umass.edu/~carteret")
To get started, see help(maximize). More documentation coming soon...
Experience
- Research Assistant, CIIR, UMass Amherst - Sep '03 to present
Advisor: Prof. James Allan
- Intern, Microsoft Research - Jun '07 to Aug '07
Supervisor: Dr. Susan Dumais
- Intern, Yahoo! Research - Jun '06 to Aug '06
Supervisor: Dr. Rosie Jones
- Intern, Yahoo! Research - Jun '05 to Aug '05
Supervisor: Dr. Rosie Jones
- Research Assistant, Miami University - '98-'99, '02-'03
Advisor: Prof. Fazli Can
Professional Activities
- PC Member for SIGIR '07, ECIR '08, SIGIR '08, EVIA '08.
- Reviewer for TOIS, JASIST, IR, IP&M.
- Referee for InfoScale '07, InfoScale '08.
- Referee for ADVIS '04 and '06.
- Member of ACM and ACM SIGIR.
Other
- My Erdös Number is no greater than 3 (and almost certainly not less!):
- I wrote "Minimal Test Collections for Retrieval Evaluation" with James Allan and Ramesh Sitaraman;
- Ramesh Sitaraman wrote "Augmented Ring Networks" with William Aiello, Sandeep Bhatt, Fan Chung, and Arnold Rosenberg;
- Fan Chung wrote "On the Decomposition of Graphs into Complete Bipartite Subgraphs" with Paul Erdös.
|