CIIR GALE Y2 annotation work

This page is for the GALE Arabic/Mandarin judging

• Contacts: James Allan and Bob Armstrong

Judgment interface is here.

Project instructions are here

Do not forget to read the template relevance instructions before starting a new template. You should also review them if you end up resuming a template that you've worked on before.

Assignments

Unassigned templates (Mandarin): none

Unassigned templates (Arabic): 2 8 9 13 14 16 17

Ghaida (Arabic)

Please work in this order:

1. Template 3: Queries LDC_TR021 (26/26) LDC_TR014 (31/31) LDC_TR017 (43/58) LDC_TR015 (1/6)
2. Template 12: Queries LDC_TR083 (26/26) LDC_TR082 (24/24) TR032 (44/44) TR033 (20/20) LDC_TR081 (30/30) TR031 (22/22) TR034 (39/39) LDC_TR084 (27/27) LDC_TR085 (48/48) TR035 (49/49)

The following are completed:

• None so far

Tasnim (Arabic)

Please work in this order:

1. Template 5: Queries LDC_TR036 (12/12) LDC_TR035 (13/13) TR013 (37/37) TR012 (10/10) LDC_TR030 (32/32) LDC_TR037 (3/3) LDC_TR034 (19/19) LDC_TR031 (49/49) LDC_TR032 (44/44) LDC_TR033 (25/25)

The following are completed:

• Templates 1, 10

Reem Faraj (Arabic)

Please work in this order:

1. Monitoring templates 11 and 6 for new documents
2. Template 7: Queries BAE_TR011 (4/4) LDC_TR047 (32/32) TR019 (7/7) LDC_TR050 (15/15) LDC_TR048 (40/40) TR017 (33/33) TR015 (44/44) TR016 (25/25) LDC_TR051 (8/8) LDC_TR049 (38/38) TR018 (5/5)
3. Template 4: Queries LDC_TR026 (25/25) LDC_TR024 (14/14) LDC_TR027 (12/12) LDC_TR022 (25/25)

The following are completed:

• Template 15

Wenglong (Mandarin)

Please work in this order:

1. None assigned

The following are completed:

• None so far

Xaoliang (Mandarin)

Please work in this order:

1. Template 3: Queries TR009 (9/31) LDC_TR017 (15/61)
2. Template 1: Queries TR001 (14/36) TR002 (5/13) LDC_TR004 (7/31) LDC_TR007 (2/23) LDC_TR001 (1/43)

The following are completed:

• None So Far

Lin Yang (Mandarin)

Please work in this order:

1. Monitor templates 5, 8, and 11 for unjudged documents.
2. Template 11: Queries LDC_TR080 (12/12) BAE_TR012 (4/5) TR030 (3/4)
3. Template 8: Queries LDC_TR058 (2/11) LDC_TR057 (4/25) LDC_TR054 (3/40) LDC_TR055 (1/36)
4. Template 5: Queries LDC_TR030 (4/36) LDC_TR032 (3/47)

The following are completed:

• None so far

Jun Hu (Andrew) (Mandarin)

Please work in this order:

1. None assigned

The following are completed:

• Template 12, now assigned to Lin Hong

Cheng-Chih Yang (Mandarin)

Please work in this order:

1. None assigned

The following are completed:

• Template 13, now assigned to Lin Hong

Ruiyang Wu (Mandarin)

Please work in this order:

1. Monitor templates 2, 6, and 7 for changes
2. Template 2: Queries BAE_TR002 (1/20)
3. Template 6: Queries LDC_TR045 (2/2) TR014 (4/14)
4. Template 7: Queries TR019 (1/8) LDC_TR051 (1/9) TR016 (1/26)

The following are completed:

• None so far

Ke Jin (Mandarin)

Please work in this order:

1. Template 14: Queries TR041 (20/20) TR042 (20/20) LDC_TR091 (55/55) LDC_TR092 (45/78) LDC_TR094 (47/93) TR045 (16/32) TR043 (29/62) TR044 (11/25) LDC_TR095 (22/57) LDC_TR093 (24/70)

The following are completed:

• Template 16, now assigned to Wei Yun Ma

Wei Wei Guo (Mandarin)

Please work in this order:

1. None assigned

The following are completed:

• Template 8, now assigned to Lin Yang
• Template 17, now assigned to Wei Yun Ma

Yiping Xu (Mandarin)

Please work in this order:

1. Template 10: Queries TR028 (11/11) LDC_TR072 (33/33) LDC_TR071 (37/37) TR027 (33/33) TR026 (31/32) TR025 (30/31) LDC_TR068 (19/21) TR024 (31/35) LDC_TR069 (32/77)

The following are completed:

• Template 2, now assigned to Ruiyang Wu

Wei Yun Ma (Mandarin)

Please work in this order:

1. Monitor templates 15, 16, and 17 for changes
2. Template 15: Queries LDC_TR100 (13/13)
3. Template 17: Queries LDC_TR116 (11/43) LDC_TR115 (4/25) LDC_TR112 (2/37) TR052 (1/29)
4. Template 16: Queries LDC_TR104 (1/48)

Lin Hong (Mandarin)

Please work in this order:

1. Monitor templates 9, 12, and 13
2. Template 9 current nothing unjudged
3. Template 12: Queries TR031 (9/31) TR032 (4/48)
4. Template 13: Queries LDC_TR090 (13/59) TR036 (8/39) TR037 (1/34)

The following are completed:

• Template 5, now assigned to Lin Yang

Liye Fei (Mandarin)

Please work in this order:

1. Template 4: Queries TR010 (39/49) LDC_TR028 (29/43) LDC_TR027 (22/35) LDC_TR022 (37/62) LDC_TR029 (42/72) LDC_TR026 (31/56) LDC_TR023 (30/57) TR011 (26/50) LDC_TR024 (14/28) LDC_TR025 (28/57)

The following are completed:

• Template 9, now assigned to Lin Hong

Details on template evaluation task

The goal of this task is to evaluate documents. For those of you who have done it, this task is much like the 1MQ judging in spirit, though the interface is dramatically different, you do not get to create the query description, and the notion of what is relevant is more tightly constrained.

You will be assigned a template number and a set of queries within that template. You will:

1. Read the description of what makes a document relevant to a template. That description is in this PDF document. Look for the line that starts "Template N". To help you find it, here are the starting page numbers for each template: T1 at p2, T2 at p4, T3 at p6, T4 at p8, T5 at p10, T6 at p12, T7 at p17, T8 at p18, T9 at p21, T10 at p23, T11 at p24, T12 at p26, T13 at p27, T14 at p29, T15 at p31, T16 at P34, T17 at p35.
2. Look at your list of queries for the template and select the next one that you have not done. You'll see in the list a pair of numbers after each query. That indicates how many unjudged documents there were in the query (originally) and how many of them are in "your" language (Arabic or Mandarin). For example, if you're reading Arabic, "10/10" means that all 10 are Arabic documents, but "10/15" means that 10 are in Arabic and that another 5 are in Mandarin.
3. Go to the annotation page here.
4. Look for your query under your template and click on it. You'll get the annotation page for that query. (If by some chance your query is missing, send a message to Bob and James and move on to the next query in your list.)
5. The query is displayed there, but it also includes italicized restrictions and/or related words. Those all help you define what is relevant or not. You must honor those as well as the relevance guidelines for the template that you just read.
6. Judge documents.
   • Anything with a "U" in the tab is unjudged.
   • Start with the first unjudged document and read it to decide if it is relevant or not, then select the appropriate link (do not use the "maybe" option).
   • Note that a document is relevant if any part of it is relevant. So if you find a relevant sentence in a document, you can stop reading, mark the document as relevant, and move on to the next one.
   • Skip any document (leave it "U") that is not in "your" language. Most queries have both Arabic and Mandarin documents, but you are only expected to judge the ones in the language you can read. An automatic machine translation into English is also provided, but you should base your decision of relevance entirely on the original Arabic or Mandarin.
   • If a document is unintelligible (it happens), leave it marked "U" and send a message to Bob and James telling us what you found.
7. Go to the next query in your template. If you're nearing the end of your template (e.g., on the second-to-last query), please drop a note to Bob and James.
8. When done with that template, start over with your next assigned template

We may occasionally put a template back on your to-do list. That means you need to revisit the listed queries because some new documents have been added. All of your old judgments will be preserved, so you'll only be looking at the "U" documents that are in "your" language.
to top