The following material is available for download from the CIIR. It is provided without warranty and without support. If there are problems accessing or using any of this material, we would appreciate being told (info at, in case we can address the issue.

Passage Similarity and Diversification in Non-factoid Question Answering - Dataset - ICTIR - July 2021

Lakshmi Vikraman, Ali Montazeralghaem, Helia Hashemi, W. Bruce Croft, James Allan

This consists of two datasets:
(a) NFPassageQA_Sim: Answer Passage Similarity task with 128 questions and 18,314 judgements
(b) NFPassageQA_Div: Answer Passage Diversification task with 93 questions and 1602 judgements

For more details, please refer to the paper given below.

Link to the paper:
Link to the dataset:

Lemur Project Downloads - June 2020

The Lemur Project (an NSF-funded collaboration with CMU and the CIIR) develops search engines, browser toolbars, text analysis tools, and data resources that support research and development of information retrieval and text mining software. The project is best known for its Indri search engine, Lemur Toolbar, and ClueWeb09 dataset. Lemur software and datasets are available at:

Open-Retrieval Conversational Question Answering - May 2020

Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, Mohit Iyyer

The OR-QuAC dataset enhances QuAC by adapting it to an open-retrieval setting. It is an aggregation of three existing datasets:

(1) the QuAC dataset that offers information-seeking conversations,
(2) the CANARD dataset that consists of context-independent rewrites of QuAC questions, and
(3) the Wikipedia corpus that serves as the knowledge source of answering questions.

Link to the dataset:

Incorporating Hierarchical Domain Information to Disambiguate Very Short Queries - Dataset - ICTIR - October 2019

Hamed Bonab, Mohammad Aliannejadi, John Foley, James Allan

This data is prepared based on 50 of NIST relevance assessors queries from TREC 2017 Core Track ( data collection. It has over 1.8 million New York Times (NYT) annotated articles, containing nearly every published article between January 1, 1987, and June 19, 2007. We provide Rel and RetRel query with domain pairs for each query. We also provide the NYT corpus hierarchy in xml file format. For more information please refer to the published paper.

Link to the paper:
Link to the dataset:

Simulating CLIR Translation Resource Scarcity using High-resource Languages - Dataset - ICTIR - October 2019

Hamed Bonab, James Allan, and Ramesh Sitaraman

This data is prepared based on 200 queries of the Cross-Language Evaluation Forum (CLEF) 2000-2003 campaign for bilingual ad-hoc retrieval tracks ( The Swahili and Somali queries are the translation of English queries from C001-C200 topic set. We hired a translation organization to translate the title and description of each topic into Somali and Swahili. For more information please refer to the published paper.

Link to the paper:
Link to the dataset:

ANTIQUE: A Non-Factoid Question Answering Benchmark - May 2019

This dataset contains 2626 non-factoid questions and 34k manual judgments ranged from 1 to 4. All questions and answers are collected from a CQA website. Note that the test relevance labels were obtained through depth-k pooling (k=10).

Link to the dataset:
Link to the paper:

PsgRobust - Answer Passage Retrieval Dataset - December 2018

Keping Bi, Qingyao Ai, W. Bruce Croft

PsgRobust is an answer passage collection built based on the Robust04 collection without manual annotation. It was built for research on iterative relevance feedback based on two assumptions:

1. For the passages that are ranked in the top positions by a powerful ranker, if they are in the relevant documents, we assume that they are relevant.
2. All the passages in the non-relevant document are irrelevant.

There are 383036 passages from 22403 unique documents in total, and 6589 relevant passages in total for the 246 queries, which are from 3544 documents. For detailed information, please check the readme.txt (linked to file and our paper.

Keping Bi, Qingyao Ai, W. Bruce Croft. "Iterative Relevance Feedback for Answer Passage Retrieval with Passage-level Semantic Match." in the Proceedings of the European Conference on Information Retrieval (ECIR 19), Cologne, Germany, April 14-18, 2019, pp. 558-572.

Link to the dataset:
Link to the paper:

Citation Worthiness Dataset - SIGIR - July 2018

Hamed Bonab, Hamed Zamani, Erik Learned-Miller, James Allan

This data is prepared based on SEPID corpus ( All the sentence IDs are consistent with the SEPID corpus, in case you need to do other interesting research with this data. It includes sentence-level segmentation of 10,921 articles from ACL ARC 1.0, up to February 2007.

Citation worthiness dataset contains 85,778 sentences with the "cite" label and 1,142,275 sentences with the "not_cite" label. 10% of data (test data) randomly selected and divided based on the section name into 7 chunks. See the paper for more explanation.

Link to the paper:
Link to the dataset:

Hierarchical Embedding Model for Personal Product Selection - April 2017 Software

Qingyao Ai, Bruce Croft

This software is an implementation of the Hierarchical Embedding Model deep neural network trained to learn personal product selections through user product queries and selections. The probability of a product item being purchased by a user with a query is computed using their corresponding latent representations.

The software is Python and Java based, using Python for neural net processing via Tensorflow and Java Galago indexing and search software from the Lemur Project.

See the README.txt file for this release or the following paper for more details.

Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, W. Bruce Croft. Learning a Hierarchical Embedding Model for Personalized ProductSearch. In Proceedings of SIGIR 2017.

Hierarchical Embedding Model for Personal Product Selections

DeepMerge: Merging Multiple Search Result Lists - December 2015

C.J. Lee, Bruce Croft

DeepMerge Merging of Multiple Search Result Lists

nfL6: Yahoo Non-Factoid Question Dataset - November 2015

Daniel Cohen, Bruce Croft

This dataset contains 87,361 questions derived from the Yahoo Webscope L6 collection and their corresponding best, and additional, answers submitted by users. Only the best answer was reviewed in determining answer quality.

The dataset file is in JSON format and may be downloaded in rar or gzip compressed format.

nfL6: Yahoo Non-Factoid Question Dataset

"Search Engines" Galago Source Code

Trevor Strohman

An old and now obsolete version (1.04) of the Galago Java source code that was referenced as a learning resource in the textbook "Search Engines: Information Retrieval in Practice" by Croft, Metzler and Strohman (2009).

It has been moved from its former Google Code repository to here.

"Search Engines" Galago Source Code

Web Annotated Passages - September 2015 Dataset

Liu Yang, Bruce Croft

This dataset contains 7,499 documents with four grades of sentence level relevance judgment annotations for 82 queries derived from the TREC Gov2 web collection. The dataset is archived in rar or gzipʼed tar file formats. The dataset is described and used in Keikha, Park and Croft, SIGIR 2014.

Web Annotated Passages Dataset

Wikipedia Bullet Points June 2013 Dataset

John Foley, James Allan

This dataset contains over 40,000 bullet-point "facts" mined from English Wikipedia year pages in the June 2013 english XML dump. (3.3M, gzipped JSON)

Wikipedia Bullet Points Dataset

Twitter June - July 2014 Dataset

Nada Naji, James Allan

This dataset consists of 71,564,914 Twitter IDs that have been automatically crawled over the period from mid June through early July 2014.

Wikipedia IDs Dataset

Query Facet and Facet Feedback Annotations

Weize Kong, James Allan

This dataset consists of query facet and facet feedback annotations used in work, "Weize Kong and James Allan. Extending Faceted Search to the General Web. CIKM 2014".

Faceted Web Search Dataset

KB Bridge Entity Linking System

Jeffrey Dalton, Pat Verga, Laura Dietz

KB Bridge is an entity linking system which identifies named entities in free text and links them to entries in a semistructured knowledge base, such as Freebase or Wikipedia. See also Dalton, J. and Dietz, L., "A Neighborhood Relevance Model for Entity Linking," OAIR 2013.

KB Bridge Code and Instructions

Online Appendix for Entity Query Feature Expansion with Knowledge Base Links

Jeffrey Dalton, Laura Dietz, James Allan

This online appendix provides additional material for entity query feature expansion, such as additional gold standard annotations, produces rankings, entity-based features, and software. See also Dalton, J., Dietz, L. and Allan, J., "Entity Query Feature Expansion using Knowledge Base Links," SIGIR 2014.

Entity Query Feature Expansion Appendix

Controversy Annotation Dataset

Shiri Dori-Hacohen, James Allan

This collection consists of controversy annotations for 445 webpages and 2060 Wikipedia articles. See also Dori-Hacohen, S. and Allan, J., "Detecting Controversy on the Web," CIKM 2013; Dori-Hacohen, S. and Allan, J., "Automated Controversy Detection on the Web," ECIR 2015.

Controversy Annotation Dataset

Open Library

Henry Feild

This collection consists of 46,561,553 metadata records crawled from the Open Library on November 30, 2011 and click distributions over records for 22,622 queries recorded over the year October 2010 through September 2011.

Open Library Dataset

RETAS OCR Evaluation Dataset

Zeki Yalniz, R. Manmatha

This dataset was created to evaluate the optical character recognition (OCR) accuracy of scanned books. It is provided here for research purposes. The dataset is extracted from books in Project Gutenberg and the Internet Archive.

RETAS OCR Evaluation Dataset

Book Translation Detection Dataset

Zeki Yalniz, R. Manmatha, Kriste Krstovski, David A. Smith

These datasets were created to evaluate the effectiveness of the translation detection frameworks. The 2K dataset was created by Krstovski and Smith (2011), and the list of translation pairs was updated for use in Yalniz and Manmatha (2012). The 30-book dataset was created for use in Yalniz and Manmatha (2012). This dataset is for research purposes only.

Translation Detection Dataset

Book Duplicate Detection Dataset

Zeki Yalniz, E. F. Can, R. Manmatha

This dataset was created to evaluate the effectiveness of the partial duplicate detection framework for scanned books proposed by Yalniz, Can and Manmatha (2011). This dataset is for research purposes only.

Duplicate Detection Dataset

Searcher Frustration User Study Data

Henry Feild

This is a dataset collected during a user study of frustration during web search at the University of Massachusetts Amherst in October 2009. The study consists of query logs and sensor readings for thirty participants.

This is available under an Open Database/Database Content license. Feel free to use, redistribute, and modify the dataset, but make sure to make it available under the same license and to give due attribution in any public use of the dataset.

Searcher Frustration Data Set

Word Image Data Sets

Toni Rath

Data sets containing word images from the George Washington collection with meta-data for retrieval performance evaluation.

Image Datasets

Stemming Class from Stemming and Cooccurrence on a Larger Corpus

Jeremy Pickens

Three sets of experiments were done, using initial classes created by (1) the Porter stemmer, (2) K-Stem, and (3) the Porter stemmer classes merged in a connected component manner with the K-Stem classes.


Event Threading Experiment

Nallapati, R., Feng, A., Peng, F., and Allan, J.

This is the experimental data from "Event Threading within News Topics" in the Proceedings of CIKM 2004 conference, pp. 446-453.

Event Threading

Novelty Track, TREC

Alvaro Bolivar

This is a collection building toolkit to assemble the training set used by the CIIR in its participation in the Novelty track at TREC 2002. For details check conference proceedings.