<<O>>  Difference Topic Annotations (r1.74 - 22 Mar 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 84 to 84
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries (AF002, AF005, TR048)
>
>
    • Unassigned queries (AF005, TR048)

    • Greg(), Haotian(), Xiaoliang(), Tasnim(), Fannie(), Wenglong(), Ao()
Changed:
<
<
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, TR049, AF001, AF003, AF004, AF006, AF007)
>
>
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, TR049, AF001, AF002, AF003, AF004, AF006, AF007)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.73 - 09 Mar 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 84 to 84
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries ()
    • Greg(), Haotian(), Xiaoliang(AF006, AF007), Tasnim(), Fannie(AF005), Wenglong(AF002, TR048), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, TR049, AF001, AF003, AF004)
>
>
    • Unassigned queries (AF002, AF005, TR048)
    • Greg(), Haotian(), Xiaoliang(), Tasnim(), Fannie(), Wenglong(), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, TR049, AF001, AF003, AF004, AF006, AF007)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.72 - 11 Feb 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 85 to 85
  • Stage 3 is to find links between these actions
  • Current assignment
    • Unassigned queries ()
Changed:
<
<
    • Greg(), Haotian(TR048), Xiaoliang(AF006, AF007), Tasnim(AF002), Fannie(AF005), Wenglong(TR049), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, AF001, AF003, AF004)
>
>
    • Greg(), Haotian(), Xiaoliang(AF006, AF007), Tasnim(), Fannie(AF005), Wenglong(AF002, TR048), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, TR049, AF001, AF003, AF004)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.71 - 08 Feb 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 84 to 84
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries (AF007)
    • Greg(), Haotian(TR048), Xiaoliang(AF003, AF004), Tasnim(AF002), Fannie(AF005, AF006), Wenglong(TR049), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, AF001)
>
>
    • Unassigned queries ()
    • Greg(), Haotian(TR048), Xiaoliang(AF006, AF007), Tasnim(AF002), Fannie(AF005), Wenglong(TR049), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, AF001, AF003, AF004)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.70 - 29 Jan 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 84 to 84
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries (AF005, AF006, AF007)
    • Greg(), Haotian(TR048), Xiaoliang(AF003, AF004), Tasnim(AF002), Fannie(), Wenglong(TR049), Ao()
>
>
    • Unassigned queries (AF007)
    • Greg(), Haotian(TR048), Xiaoliang(AF003, AF004), Tasnim(AF002), Fannie(AF005, AF006), Wenglong(TR049), Ao()

    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, AF001)

Arabic queries

 <<O>>  Difference Topic Annotations (r1.69 - 28 Jan 2008 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 14 to 14
  • Fannie Dang
    1. Jokes trial (see below)
    2. 10 min Topic Model Evaluation (done)
Changed:
<
<
    1. 1 hour Topic Model Evaluation
>
>
    1. 1 hour Topic Model Evaluation (done)

    1. Incident
    2. 1MQ (changed recently; see description below)
  • Greg Hart
 <<O>>  Difference Topic Annotations (r1.68 - 16 Jan 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 84 to 84
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries (AF003, AF004, AF005, AF006, AF007)
    • Greg(), Haotian(TR048), Xiaoliang(AF001), Tasnim(AF002), Fannie(), Wenglong(TR049), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111)
>
>
    • Unassigned queries (AF005, AF006, AF007)
    • Greg(), Haotian(TR048), Xiaoliang(AF003, AF004), Tasnim(AF002), Fannie(), Wenglong(TR049), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111, AF001)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.67 - 11 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 12 to 12

Assigned tasks

  • Fannie Dang
Added:
>
>
    1. Jokes trial (see below)

    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
    3. Incident
    4. 1MQ (changed recently; see description below)
  • Greg Hart
Added:
>
>
    1. Jokes trial (see below)

    1. 10 min Topic Model Evaluation
    2. 1MQ(changed recently; see description below)
    3. Citation (38/50 completed as of Jan 8, 2008 -- see details below)
Line: 27 to 29
    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
  • Haotian Liu
Added:
>
>
    1. Jokes trial (see below)

    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
    3. Incident
Line: 50 to 53

Details on tasks

Added:
>
>		 Jokes
  • Contact: Lisa Friedland (lfriedl)
  • Your task in this one is to help collect groups of nearly-identical jokes from a pool of about 11,000 jokes. You will also help build some queries that might be used to find jokes.
  • For the trial, you will try the task with two jokes. If all looks well, you will be given a different (larger) set of jokes to work from.
  • Go to http://www.cs.umass.edu/~lfriedl/joke_instructions.html for details

Topic Model Evaluation

 <<O>>  Difference Topic Annotations (r1.66 - 11 Jan 2008 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 27 to 27
    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
  • Haotian Liu
Changed:
<
<
    1. 10 min Topic Model Evaluation
>
>
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation

    1. Incident
    2. 1MQ (changed recently; see description below)
  • Tasnim Ahmed
    1. Arabic judgments see here
    2. 1MQ(changed recently; see description below)
    3. 10 min Topic Model Evaluation (done)
Changed:
<
<
    1. 1 hour Topic Model Evaluation
>
>
    1. 1 hour Topic Model Evaluation (done)

  • Wenglong Ng
    1. Mandarin judgments see here
    2. 10 min Topic Model Evaluation (done)
 <<O>>  Difference Topic Annotations (r1.65 - 11 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 42 to 42
    1. Incident
    2. 1MQ(changed recently; see description below)
  • Xaoliang Yao
Changed:
<
<
    1. 1 hour Topic Model Evaluation (done)
    2. Incident
>
>
    1. Finish current 1MQ query if one is in progress
    2. Mandarin judgments see here

    1. 1MQ (changed recently; see description below)

Details on tasks

 <<O>>  Difference Topic Annotations (r1.64 - 10 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 49 to 49

Details on tasks

Deleted:
<
<		 Template evaluation
  • Contacts: James Allan and Bob Armstrong
  • The goal of this task is to evaluate documents. It is much like the 1MQ judging in spirit, though the interface is dramatically different, you do not get to create the query description, and the notion of what is relevant is more tightly constrained.
  • You will be assigned a template number and a set of queries within that template. You will:
    1. Read the description of what makes a document relevant to a template. That description is in this document (pdf). Look for the line that starts "Template N". To help you find it, here are the starting page numbers for each template: T1 at p2, T2 at p4, T3 at p6, T4 at p8, T5 at p10, T6 at p12, T7 at p17, T8 at p18, T9 at p21, T10 at p23, T11 at p24, T12 at p26, T13 at p27, T14 at p29, T15 at p31, T16 at P34, T17 at p35.
    2. Go to the annotation page here (not yet active)
    3. Select the first query in your topic
    4. Note the query as well as italicized restrictions and/or related words. Those all help you define what is relevant or not. You must honor those as well as the relevance guidelines for the template that you just read.
    5. Judge documents. Anything with a "U" in the tab is unjudged. You should not use the "maybe" option, so when you're done with the query, everything should be "R" or "N". However, if you encounter a document that is in Arabic or Mandarin and you do not read Arabic or Mandarin (respectively), leave the document unjudged and send a message to Bob and James saying that "I couldn't complete query xxx".
    6. Go to the next query in your template. If you're nearing the end of your template (e.g., on the second-to-last query), please drop a note to Bob and James.
    7. When done with that template, start over with your next assigned template
  • TemplateQueries

Topic Model Evaluation

 <<O>>  Difference Topic Annotations (r1.63 - 10 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 17 to 17
    1. Incident
    2. 1MQ (changed recently; see description below)
  • Greg Hart
Deleted:
<
<
    1. Citations until 40 are completed

    1. 10 min Topic Model Evaluation
    2. 1MQ(changed recently; see description below)
    3. Citation (38/50 completed as of Jan 8, 2008 -- see details below)
Line: 37 to 36
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
  • Wenglong Ng
Added:
>
>
    1. Mandarin judgments see here

    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
    3. Incident
 <<O>>  Difference Topic Annotations (r1.62 - 09 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 17 to 17
    1. Incident
    2. 1MQ (changed recently; see description below)
  • Greg Hart
Added:
>
>
    1. Citations until 40 are completed

    1. 10 min Topic Model Evaluation
Deleted:
<
<
    1. Citation (38/50 completed as of Jan 8, 2008 -- see details below)

    1. 1MQ(changed recently; see description below)
Added:
>
>
    1. Citation (38/50 completed as of Jan 8, 2008 -- see details below)

  • Ghaida Al-Jermozi
    1. Arabic judgments see here
    2. 1MQ (changed recently; see description below), do one query
Line: 31 to 32
    1. Incident
    2. 1MQ (changed recently; see description below)
  • Tasnim Ahmed
Added:
>
>
    1. Arabic judgments see here
    2. 1MQ(changed recently; see description below)

    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
Deleted:
<
<
    1. Incident
    2. 1MQ(changed recently; see description below)

  • Wenglong Ng
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
 <<O>>  Difference Topic Annotations (r1.61 - 09 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 21 to 21
    1. Citation (38/50 completed as of Jan 8, 2008 -- see details below)
    2. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
Added:
>
>
    1. Arabic judgments see here

    1. 1MQ (changed recently; see description below), do one query
    2. 1 hour Topic Model Evaluation (done)
    3. ciQA - See Mark
 <<O>>  Difference Topic Annotations (r1.60 - 08 Jan 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 85 to 85
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries ()
    • Greg(), Haotian(), Xiaoliang(), Tasnim(), Fannie(), Wenglong(), Ao()
>
>
    • Unassigned queries (AF003, AF004, AF005, AF006, AF007)
    • Greg(), Haotian(TR048), Xiaoliang(AF001), Tasnim(AF002), Fannie(), Wenglong(TR049), Ao()

    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111)

Arabic queries

 <<O>>  Difference Topic Annotations (r1.59 - 08 Jan 2008 - BobArmstrong)
META TOPICPARENT WebHome

CIIR annotation work

Line: 52 to 52
  • The goal of this task is to evaluate documents. It is much like the 1MQ judging in spirit, though the interface is dramatically different, you do not get to create the query description, and the notion of what is relevant is more tightly constrained.
  • You will be assigned a template number and a set of queries within that template. You will:
    1. Read the description of what makes a document relevant to a template. That description is in this document (pdf). Look for the line that starts "Template N". To help you find it, here are the starting page numbers for each template: T1 at p2, T2 at p4, T3 at p6, T4 at p8, T5 at p10, T6 at p12, T7 at p17, T8 at p18, T9 at p21, T10 at p23, T11 at p24, T12 at p26, T13 at p27, T14 at p29, T15 at p31, T16 at P34, T17 at p35.
Changed:
<
<
    1. Go to the annotation page here (not yet active)
>
>
    1. Go to the annotation page here (not yet active)

    1. Select the first query in your topic
    2. Note the query as well as italicized restrictions and/or related words. Those all help you define what is relevant or not. You must honor those as well as the relevance guidelines for the template that you just read.
    3. Judge documents. Anything with a "U" in the tab is unjudged. You should not use the "maybe" option, so when you're done with the query, everything should be "R" or "N". However, if you encounter a document that is in Arabic or Mandarin and you do not read Arabic or Mandarin (respectively), leave the document unjudged and send a message to Bob and James saying that "I couldn't complete query xxx".
    4. Go to the next query in your template. If you're nearing the end of your template (e.g., on the second-to-last query), please drop a note to Bob and James.
    5. When done with that template, start over with your next assigned template
Added:
>
>

Topic Model Evaluation

 <<O>>  Difference Topic Annotations (r1.58 - 08 Jan 2008 - NiranjanB)
META TOPICPARENT WebHome

CIIR annotation work

Line: 18 to 18
    1. 1MQ (changed recently; see description below)
  • Greg Hart
    1. 10 min Topic Model Evaluation
Changed:
<
<
    1. Citation (17/50 completed as of Dec 30, 2007 -- see details below)
>
>
    1. Citation (38/50 completed as of Jan 8, 2008 -- see details below)

    1. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
    1. 1MQ (changed recently; see description below), do one query
Line: 102 to 102
  • Contact: Niranjan Balasubramanian (niranjan)
  • Stage 1 Task page -- http://burnie.cs.umass.edu:7236/citations/
  • The goal of this task is to find restatements of query sentences. Please read the task description and the detailed instructions on the task page.
Changed:
<
<
  • STATUS: 17/50 completed as of Dec 31, 2007.
>
>
  • STATUS: 38/50 completed as of Jan 08, 2008.

Contact information

 <<O>>  Difference Topic Annotations (r1.57 - 08 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 14 to 14
  • Fannie Dang
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
Deleted:
<
<
    1. ciQA - See Mark

    1. Incident
    2. 1MQ (changed recently; see description below)
  • Greg Hart
    1. 10 min Topic Model Evaluation
Deleted:
<
<
    1. ciQA - See Mark

    1. Citation (17/50 completed as of Dec 30, 2007 -- see details below)
    2. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
Line: 34 to 32
  • Tasnim Ahmed
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
Deleted:
<
<
    1. ciQA - See Mark

    1. Incident
    2. 1MQ(changed recently; see description below)
  • Wenglong Ng
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
Deleted:
<
<
    1. ciQA - See Mark

    1. Incident
    2. 1MQ(changed recently; see description below)
  • Xaoliang Yao
    1. 1 hour Topic Model Evaluation (done)
Deleted:
<
<
    1. ciQA - See Mark

    1. Incident
    2. 1MQ (changed recently; see description below)

Details on tasks

Added:
>
>

Template evaluation

  • Contacts: James Allan and Bob Armstrong
  • The goal of this task is to evaluate documents. It is much like the 1MQ judging in spirit, though the interface is dramatically different, you do not get to create the query description, and the notion of what is relevant is more tightly constrained.
  • You will be assigned a template number and a set of queries within that template. You will:
    1. Read the description of what makes a document relevant to a template. That description is in this document (pdf). Look for the line that starts "Template N". To help you find it, here are the starting page numbers for each template: T1 at p2, T2 at p4, T3 at p6, T4 at p8, T5 at p10, T6 at p12, T7 at p17, T8 at p18, T9 at p21, T10 at p23, T11 at p24, T12 at p26, T13 at p27, T14 at p29, T15 at p31, T16 at P34, T17 at p35.
    2. Go to the annotation page here (not yet active)
    3. Select the first query in your topic
    4. Note the query as well as italicized restrictions and/or related words. Those all help you define what is relevant or not. You must honor those as well as the relevance guidelines for the template that you just read.
    5. Judge documents. Anything with a "U" in the tab is unjudged. You should not use the "maybe" option, so when you're done with the query, everything should be "R" or "N". However, if you encounter a document that is in Arabic or Mandarin and you do not read Arabic or Mandarin (respectively), leave the document unjudged and send a message to Bob and James saying that "I couldn't complete query xxx".
    6. Go to the next query in your template. If you're nearing the end of your template (e.g., on the second-to-last query), please drop a note to Bob and James.
    7. When done with that template, start over with your next assigned template

Topic Model Evaluation
Line: 58 to 67

ciQA (complex interactive question answersing)

  • Contact: Mark Smucker (smucker)
Changed:
<
<
  • New task. See email sent by Mark.
>
>
  • Completed by Fannie, Greg, and Tasnim.
  • Being worked on by Ghaida
  • Maybe eventually by others...

1MQ (million query track)

  • Contact: Ben Carterette (carteret)
 <<O>>  Difference Topic Annotations (r1.56 - 08 Jan 2008 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 24 to 24
    1. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
    1. 1MQ (changed recently; see description below), do one query
Changed:
<
<
    1. 1 hour Topic Model Evaluation
>
>
    1. 1 hour Topic Model Evaluation (done)

    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
  • Haotian Liu
Line: 32 to 32
    1. Incident
    2. 1MQ (changed recently; see description below)
  • Tasnim Ahmed
Changed:
<
<
    1. 10 min Topic Model Evaluation
>
>
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation

    1. ciQA - See Mark
    2. Incident
    3. 1MQ(changed recently; see description below)
  • Wenglong Ng
Changed:
<
<
    1. 10 min Topic Model Evaluation
>
>
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation

    1. ciQA - See Mark
    2. Incident
    3. 1MQ(changed recently; see description below)
  • Xaoliang Yao
Changed:
<
<
    1. 1 hour Topic Model Evaluation (done?)
>
>
    1. 1 hour Topic Model Evaluation (done)

    1. ciQA - See Mark
    2. Incident
    3. 1MQ (changed recently; see description below)
 <<O>>  Difference Topic Annotations (r1.55 - 06 Jan 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 72 to 72
  • Stage 3 is to find links between these actions
  • Current assignment
    • Unassigned queries ()
Changed:
<
<
    • Greg(), Haotian(), Xiaoliang(LDC_TR106, LDC_TR108), Tasnim(), Fannie(), Wenglong(), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR107, LDC_TR109, LDC_TR110, LDC_TR111)
>
>
    • Greg(), Haotian(), Xiaoliang(), Tasnim(), Fannie(), Wenglong(), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR106, LDC_TR107, LDC_TR108, LDC_TR109, LDC_TR110, LDC_TR111)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.54 - 03 Jan 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 72 to 72
  • Stage 3 is to find links between these actions
  • Current assignment
    • Unassigned queries ()
Changed:
<
<
    • Greg(), Haotian(), Xiaoliang(LDC_TR106, LDC_TR111), Tasnim(), Fannie(LDC_TR108), Wenglong(), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR107, LDC_TR109, LDC_TR110)
>
>
    • Greg(), Haotian(), Xiaoliang(LDC_TR106, LDC_TR108), Tasnim(), Fannie(), Wenglong(), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR107, LDC_TR109, LDC_TR110, LDC_TR111)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.53 - 03 Jan 2008 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 42 to 42
    1. Incident
    2. 1MQ(changed recently; see description below)
  • Xaoliang Yao
Changed:
<
<
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation
>
>
    1. 1 hour Topic Model Evaluation (done?)

    1. ciQA - See Mark
    2. Incident
    3. 1MQ (changed recently; see description below)
Line: 62 to 61

 1MQ (million query track)
  • Contact: Ben Carterette (carteret)
  • You will develop a query and judge number of Web pages for relevance to that query
Changed:
<
<
  • This task changed in late December. In the earlier version, you had to judge a maximum of 40 pages. The new version requires that you judge a good deal more pages, perhaps more than 100 in some cases. Otherwise it is identical.
>
>
  • This task changed in late December. In the earlier version, you had to judge a maximum of 40 pages. The new version requires that you judge a good deal more pages, perhaps more than 100 in some cases. Otherwise it is identical. The site for the evaluation is here.

Incident (Incident threading, currently the annotation interface is ready for all steps) http://maroo.cs.umass.edu/aofeng/violence.php

  • Contact: Ao Feng (aofeng)
 <<O>>  Difference Topic Annotations (r1.52 - 03 Jan 2008 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 73 to 73
  • Stage 3 is to find links between these actions
  • Current assignment
    • Unassigned queries ()
Changed:
<
<
    • Greg(), Haotian(LDC_TR109), Xiaoliang(LDC_TR106, LDC_TR107), Tasnim(), Fannie(LDC_TR108), Wenglong(), Ao(LDC_TR111)
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR110)
>
>
    • Greg(), Haotian(), Xiaoliang(LDC_TR106, LDC_TR111), Tasnim(), Fannie(LDC_TR108), Wenglong(), Ao()
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR107, LDC_TR109, LDC_TR110)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.51 - 01 Jan 2008 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 12 to 12

Assigned tasks

  • Fannie Dang
Changed:
<
<
    1. 10 min Topic Model Evaluation
>
>
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation

    1. ciQA - See Mark
    2. Incident
    3. 1MQ (changed recently; see description below)
Line: 23 to 24
    1. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
    1. 1MQ (changed recently; see description below), do one query
Changed:
<
<
    1. continue Topic Model Evaluation after hearing from Xing
>
>
    1. 1 hour Topic Model Evaluation

    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
  • Haotian Liu
Line: 41 to 42
    1. Incident
    2. 1MQ(changed recently; see description below)
  • Xaoliang Yao
Changed:
<
<
    1. 10 min Topic Model Evaluation
>
>
    1. 10 min Topic Model Evaluation (done)
    2. 1 hour Topic Model Evaluation

    1. ciQA - See Mark
    2. Incident
    3. 1MQ (changed recently; see description below)
Line: 50 to 52

Topic Model Evaluation

  • Contact: Xing Yi (yixing)
Changed:
<
<
>
>

  • The goal of this task is to evaluate topic qualities from different topic models. Please read the task description and the detailed instructions on the task page. The first part of the task is a quick trial to ensure that you've got the task figured out. The next part of the task is longer, but it should still take no more than an hour to complete.

ciQA (complex interactive question answersing)

 <<O>>  Difference Topic Annotations (r1.50 - 31 Dec 2007 - NiranjanB)
META TOPICPARENT WebHome

CIIR annotation work

Line: 19 to 19
  • Greg Hart
    1. 10 min Topic Model Evaluation
    2. ciQA - See Mark
Changed:
<
<
    1. Citation (New Task -- see details below)
>
>
    1. Citation (17/50 completed as of Dec 30, 2007 -- see details below)

    1. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
    1. 1MQ (changed recently; see description below), do one query
Line: 86 to 86

Citation Queries

  • Contact: Niranjan Balasubramanian (niranjan)
Changed:
<
<
>
>

  • The goal of this task is to find restatements of query sentences. Please read the task description and the detailed instructions on the task page.
Added:
>
>
  • STATUS: 17/50 completed as of Dec 31, 2007.

Contact information

 <<O>>  Difference Topic Annotations (r1.49 - 30 Dec 2007 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

Line: 22 to 22
    1. Citation (New Task -- see details below)
    2. 1MQ(changed recently; see description below)
  • Ghaida Al-Jermozi
Changed:
<
<
    1. 10 min Topic Model Evaluation
>
>
    1. 1MQ (changed recently; see description below), do one query
    2. continue Topic Model Evaluation after hearing from Xing

    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
Deleted:
<
<
    1. 1MQ(changed recently; see description below)

  • Haotian Liu
    1. 10 min Topic Model Evaluation
    2. Incident
 <<O>>  Difference Topic Annotations (r1.48 - 30 Dec 2007 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 71 to 71
  • Stage 3 is to find links between these actions
  • Current assignment
    • Unassigned queries ()
Changed:
<
<
    • Greg(), Haotian(LDC_TR105, LDC_TR109), Xiaoliang(LDC_TR106, LDC_TR107), Tasnim(), Fannie(LDC_TR008, LDC_TR104), Wenglong(LDC_TR108), aofeng(LDC_TR111)
    • Finished queries(BAE_TR002, LDC_TR110)
>
>
    • Greg(), Haotian(LDC_TR109), Xiaoliang(LDC_TR106, LDC_TR107), Tasnim(), Fannie(LDC_TR108), Wenglong(), Ao(LDC_TR111)
    • Finished queries(BAE_TR002, LDC_TR008, LDC_TR104, LDC_TR105, LDC_TR110)

Arabic queries

  • Contact: James Allan (allan)
 <<O>>  Difference Topic Annotations (r1.47 - 30 Dec 2007 - JamesAllan)
META TOPICPARENT WebHome

CIIR annotation work

This page lists the tasks that annotators are currently assigned.

Added:
>
>		 NOTES:
  • Check this Web page every day when you start
  • When you complete a task, send an email message to James as well as the contact person
  • Please keep the contact person informed of your progress, possibly by sending a message at the end of a day of work

Assigned tasks

  • Fannie Dang
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. ciQA - See Mark
    2. Incident
Changed:
<
<
    1. 1MQ
    2. 10 min Topic Model Evaluation
>
>
    1. 1MQ (changed recently; see description below)

  • Greg Hart
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
Changed:
<
<
    1. 1MQ
    2. 10 min Topic Model Evaluation
>
>
    1. 1MQ(changed recently; see description below)

  • Ghaida Al-Jermozi
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. ciQA - See Mark
Deleted:
<
<
    1. Arabic queries

    1. Citation (New Task -- see details below)
Changed:
<
<
    1. 1MQ
    2. 10 min Topic Model Evaluation
>
>
    1. 1MQ(changed recently; see description below)

  • Haotian Liu
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. Incident
Added:
>
>
    1. 1MQ (changed recently; see description below)

  • Tasnim Ahmed
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. ciQA - See Mark
Deleted:
<
<
    1. Arabic queries

    1. Incident
Changed:
<
<
    1. 1MQ
    2. 10 min Topic Model Evaluation
>
>
    1. 1MQ(changed recently; see description below)

  • Wenglong Ng
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. ciQA - See Mark
    2. Incident
Changed:
<
<
    1. 1MQ
    2. 10 min Topic Model Evaluation
>
>
    1. 1MQ(changed recently; see description below)

  • Xaoliang Yao
Added:
>
>
    1. 10 min Topic Model Evaluation

    1. ciQA - See Mark
Deleted:
<
<
    1. Mandarin queries

    1. Incident
Changed:
<
<
    1. 1MQ
    2. 10 min Topic Model Evaluation
>
>
    1. 1MQ (changed recently; see description below)

Details on tasks

Added:
>
>		 Topic Model Evaluation
  • Contact: Xing Yi (yixing)
  • Task page -- http://ciir.cs.umass.edu/~yixing/Instruction.htm
  • The goal of this task is to evaluate topic qualities from different topic models. Please read the task description and the detailed instructions on the task page. The first part of the task is a quick trial to ensure that you've got the task figured out. The next part of the task is longer, but it should still take no more than an hour to complete.

ciQA (complex interactive question answersing)
  • Contact: Mark Smucker (smucker)
  • New task. See email sent by Mark.

1MQ (million query track)

  • Contact: Ben Carterette (carteret)
Changed:
<
<
  • This is a constant background task. If you do not have something else to do, or if you need a brief break from a more complex task, work on some of the 1MQ queries.
>
>
  • You will develop a query and judge number of Web pages for relevance to that query
  • This task changed in late December. In the earlier version, you had to judge a maximum of 40 pages. The new version requires that you judge a good deal more pages, perhaps more than 100 in some cases. Otherwise it is identical.

Incident (Incident threading, currently the annotation interface is ready for all steps) http://maroo.cs.umass.edu/aofeng/violence.php

  • Contact: Ao Feng (aofeng)
Line: 64 to 74
    • Greg(), Haotian(LDC_TR105, LDC_TR109), Xiaoliang(LDC_TR106, LDC_TR107), Tasnim(), Fannie(LDC_TR008, LDC_TR104), Wenglong(LDC_TR108), aofeng(LDC_TR111)
    • Finished queries(BAE_TR002, LDC_TR110)
Deleted:
<
<		 Photos (first round completed)
  • Contact: James Allan (allan)
  • Only one person should work on this task, so it should be someone planning to put in several hours.
  • This task can be done only on one of the annotator computers: phoenix. You will look at a series of photos of a whiteboard. For each picture, you will answer two questions:
    1. Does this picture contain a substantive difference from the previous one? That could be a change in the contents of the whiteboard or it could be a person appearing in the frame who was not there before or who was in a different position in the previous picture. Simple changes in lighting do not count.
    2. Did material on the whiteboard change compared to the previous picture--or, if someone was occluding part of the whiteboard, compared to the previous picture where the material was visible.
  • To work on this task, you must use "phoenix", the black Dell annotation matchine.
  • On the desktop, in the upper right, you'll find a text file "photos" that if you doubleclick will give you more detailed instructions. It is in that file that you will write the information about what changed. When you leave for the day, please remember to save the file.
  • To look at the photos, doubleclick the "whiteboard photos" link, also in the upper right. You will get a directory of hundreds of photos. Doubleclick on the first one to open the viewer. You can use your arrow keys or the buttons in the interface to move back and forth.

Arabic queries
Line: 89 to 89
Changed:
<
<		 Topic Model Evaluation
  • Contact: Xing Yi (yixing)
  • Task page -- http://ciir.cs.umass.edu/~yixing/Instruction.htm
  • The goal of this task is to evaluate topic qualities from different topic models. Please read the task description and the detailed instructions on the task page.
>
>

Contact information

  • Kate Moruzzi (kate), x5-3415
  • James Allan (allan), x5-3640
 <<O>>  Difference Topic Annotations (r1.46 - 28 Dec 2007 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 24 to 24
    1. 10 min Topic Model Evaluation
  • Haotian Liu
    1. Incident
Deleted:
<
<
    1. 10 min Topic Model Evaluation

  • Tasnim Ahmed
    1. ciQA - See Mark
    2. Arabic queries
 <<O>>  Difference Topic Annotations (r1.45 - 28 Dec 2007 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 10 to 10
    1. ciQA - See Mark
    2. Incident
    3. 1MQ
Added:
>
>
    1. 10 min Topic Model Evaluation

  • Greg Hart
    1. ciQA - See Mark
    2. Citation (New Task -- see details below)
    3. 1MQ
Added:
>
>
    1. 10 min Topic Model Evaluation

  • Ghaida Al-Jermozi
    1. ciQA - See Mark
    2. Arabic queries
    3. Citation (New Task -- see details below)
    4. 1MQ
Added:
>
>
    1. 10 min Topic Model Evaluation

  • Haotian Liu
    1. Incident
Added:
>
>
    1. 10 min Topic Model Evaluation

  • Tasnim Ahmed
    1. ciQA - See Mark
    2. Arabic queries
    3. Incident
    4. 1MQ
Added:
>
>
    1. 10 min Topic Model Evaluation

  • Wenglong Ng
    1. ciQA - See Mark
    2. Incident
    3. 1MQ
Added:
>
>
    1. 10 min Topic Model Evaluation

  • Xaoliang Yao
    1. ciQA - See Mark
    2. Mandarin queries
    3. Incident
    4. 1MQ
Added:
>
>
    1. 10 min Topic Model Evaluation

Details on tasks

 <<O>>  Difference Topic Annotations (r1.44 - 24 Dec 2007 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 85 to 85

Topic Model Evaluation

  • Contact: Xing Yi (yixing)
Changed:
<
<
>
>

  • The goal of this task is to evaluate topic qualities from different topic models. Please read the task description and the detailed instructions on the task page.

Contact information

  • Kate Moruzzi (kate), x5-3415
Line: 93 to 93
  • Ben Carterette (carteret), x
  • Ao Feng (aofeng), x7-0403
  • Niranjan Balasubramanian (niranjan), x
Changed:
<
<
  • Xing Yi(yixing), x7-0403
>
>
  • Xing Yi(yixing), x5-0728

  • Haotian (haotian) at student.umass.edu
  • Tasnim (teahmed) at student.umass.edu
 <<O>>  Difference Topic Annotations (r1.43 - 24 Dec 2007 - XingYi)
META TOPICPARENT WebHome

CIIR annotation work

Line: 83 to 83
Added:
>
>		 Topic Model Evaluation
  • Contact: Xing Yi (yixing)
  • Task page -- http://burnie.cs.umass.edu:7236/citations/
  • The goal of this task is to evaluate topic qualities from different topic models. Please read the task description and the detailed instructions on the task page.

Contact information

  • Kate Moruzzi (kate), x5-3415
  • James Allan (allan), x5-3640
  • Ben Carterette (carteret), x
  • Ao Feng (aofeng), x7-0403
  • Niranjan Balasubramanian (niranjan), x
Added:
>
>
  • Xing Yi(yixing), x7-0403

  • Haotian (haotian) at student.umass.edu
  • Tasnim (teahmed) at student.umass.edu
 <<O>>  Difference Topic Annotations (r1.42 - 19 Dec 2007 - AoFeng)
META TOPICPARENT WebHome

CIIR annotation work

Line: 54 to 54
  • Stage 2 is to identify and describe the violent actions, the interface is available now.
  • Stage 3 is to find links between these actions
  • Current assignment
Changed:
<
<
    • Unassigned queries (LDC_TR111)
    • Greg(), Haotian(LDC_TR105, LDC_TR109), Xiaoliang(LDC_TR106, LDC_TR107), Tasnim(), Fannie(LDC_TR008, LDC_TR104), Wenglong(LDC_TR108), aofeng(LDC_TR110)
    • Finished queries(BAE_TR002)
>
>
    • U