CIIR annotation work

This page lists the tasks that annotators are currently assigned.

NOTES:

• Check this Web page every day when you start
• When you complete a task, send an email message to James as well as the contact person
• Please keep the contact person informed of your progress, possibly by sending a message at the end of a day of work

Assigned tasks

• Fannie Dang
  1. Jokes trial (see below)
  2. 10 min Topic Model Evaluation (done)
  3. 1 hour Topic Model Evaluation (done)
  4. Incident
  5. 1MQ (changed recently; see description below)
• Greg Hart
  1. Jokes trial (see below)
  2. 10 min Topic Model Evaluation
  3. 1MQ(changed recently; see description below)
  4. Citation (38/50 completed as of Jan 8, 2008 -- see details below)
• Ghaida Al-Jermozi
  1. Arabic judgments see here
  2. 1MQ (changed recently; see description below), do one query
  3. 1 hour Topic Model Evaluation (done)
  4. ciQA - See Mark
  5. Citation (New Task -- see details below)
• Haotian Liu
  1. Jokes trial (see below)
  2. 10 min Topic Model Evaluation (done)
  3. 1 hour Topic Model Evaluation
  4. Incident
  5. 1MQ (changed recently; see description below)
• Tasnim Ahmed
  1. Arabic judgments see here
  2. 1MQ(changed recently; see description below)
  3. 10 min Topic Model Evaluation (done)
  4. 1 hour Topic Model Evaluation (done)
• Wenglong Ng
  1. Mandarin judgments see here
  2. 10 min Topic Model Evaluation (done)
  3. 1 hour Topic Model Evaluation
  4. Incident
  5. 1MQ(changed recently; see description below)
• Xaoliang Yao
  1. Finish current 1MQ query if one is in progress
  2. Mandarin judgments see here
  3. 1MQ (changed recently; see description below)

Details on tasks

Jokes

• Contact: Lisa Friedland (lfriedl)
• Your task in this one is to help collect groups of nearly-identical jokes from a pool of about 11,000 jokes. You will also help build some queries that might be used to find jokes.
• For the trial, you will try the task with two jokes. If all looks well, you will be given a different (larger) set of jokes to work from.
• Go to http://www.cs.umass.edu/~lfriedl/joke_instructions.html for details

Topic Model Evaluation

• Contact: Xing Yi (yixing)
• Task page -- (10 min) http://ciir.cs.umass.edu/~yixing/Instruction.htm ; (1 hour, full set) http://ciir.cs.umass.edu/~yixing/InstructionStep2.htm
• The goal of this task is to evaluate topic qualities from different topic models. Please read the task description and the detailed instructions on the task page. The first part of the task is a quick trial to ensure that you've got the task figured out. The next part of the task is longer, but it should still take no more than an hour to complete.

ciQA (complex interactive question answersing)

• Contact: Mark Smucker (smucker)
• Completed by Fannie, Greg, and Tasnim.
• Being worked on by Ghaida
• Maybe eventually by others...

1MQ (million query track)

• Contact: Ben Carterette (carteret)
• You will develop a query and judge number of Web pages for relevance to that query
• This task changed in late December. In the earlier version, you had to judge a maximum of 40 pages. The new version requires that you judge a good deal more pages, perhaps more than 100 in some cases. Otherwise it is identical. The site for the evaluation is here.

Incident (Incident threading, currently the annotation interface is ready for all steps) http://maroo.cs.umass.edu/aofeng/violence.php

• Contact: Ao Feng (aofeng)
• Task revised to mark only paragraphs with violent actions: actual description of violent action must appear in the paragraph, related stuff (arrest, methods to stop violence, investigation, charge, trial, etc.) do not qualify. General description of violence does not count either.
• Check with Ao Feng if you are not sure of the description.
• Yes/No decision, no third option. Paragraph does not have to be on-topic.
• Stage 2 is to identify and describe the violent actions, the interface is available now.
• Stage 3 is to find links between these actions
• Current assignment
  • Unassigned queries (AF005, TR048)
  • Greg(), Haotian(), Xiaoliang(), Tasnim(), Fannie(), Wenglong(), Ao()
  • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, TR049, AF001, AF002, AF003, AF004, AF006, AF007)

Arabic queries

• Contact: James Allan (allan)
• http://burnie.cs.umass.edu.edu:7236/varabic/query.html
• Here are some instructions (PDF)

Mandarin queries

• Contact: James Allan (allan)
• http://burnie.cs.umass.edu.edu:7236/vmandarin/query.html
• Here are some instructions (PDF)

Citation Queries

• Contact: Niranjan Balasubramanian (niranjan)
• Stage 1 Task page -- http://burnie.cs.umass.edu:7236/citations/
• The goal of this task is to find restatements of query sentences. Please read the task description and the detailed instructions on the task page.
• STATUS: 38/50 completed as of Jan 08, 2008.

Contact information

• Kate Moruzzi (kate), x5-3415
• James Allan (allan), x5-3640
• Ben Carterette (carteret), x
• Ao Feng (aofeng), x7-0403
• Niranjan Balasubramanian (niranjan), x
• Xing Yi(yixing), x5-0728

• Haotian (haotian) at student.umass.edu
• Tasnim (teahmed) at student.umass.edu
• Xaoliang (xyao) at student.umass.edu
• Greg (Ghart) at student.umass.edu
• Weng (wenglong) at student.umass.edu
• Fannie (fdang) at student.umass.edu
• Ghaida (galjermo) at student.umass.edu