AronCulotta 2 June 2004

Some results with the "Conditional Link" clustering method:

Training data is created by sampling from clusters ("sampled") or by generating one way to cluster the data ("generated").

Testing is done either by randomly choosing nodes to classify with the existing clusters ("random"), or by choosing the node whose decision we're most confident about first ("greedy").

So far, it seems that the average of 5 "random" trials trained on "generated" data performs better than a "greedy" method on "generated" data. This isn't entirely unexpected, as it's hard to get stuck in a local minimum if you're not being greedy.

Preliminary experiments with "sampled" data haven't improved things for either "random" or "greedy." This seems strange, since we have such a large amount of training data. Idea: active learning type approach where we over-sample decisions we're uncertain about.

Also, the sampling performance seems very sensitive to the proportion of positive and negative samples chosen. Currently, .1 - .01 pos/neg ration does best. Is there a better way to set this? Shouldn't it reflect the ratio we'll see at test time??

Trials:

CLRG - Conditional Link - "random", "generated"

CLGG - Conditional Link - "greedy", "generated"

CLRS10 - Conditional Link - "random", "sampled" 10K instances

CLGS10 - Conditional Link - "greedy", "sampled" 10K instances

BEN - Ben's clustering results, reported in UAI

BER - Berkely's clustering results, also reported in UAI paper

The goal is to beat Ben's UAI and Berkely results on clustering citeseer data. Here's where we are:

Pairwise Eval

Cluster Eval

Next: Try clustering venues and papers simultaneously in this framework.

• CharlesSutton skillfully ported and refactored FuchunPeng^? 's and BenWellner^? 's segmentation and coreference code from their users directories to projects/seg_plus_coref

CharlesSutton 25 May 2004

I have gotten annoyed enough to start a ListOfBogusCiteseerLabelings.

• CharlesSutton^? skillfully ported and refactored FuchunPeng^? 's and BenWellner^? 's segmentation and coreference code from their users directories to projects/seg_plus_coref

Obtained baselines for venue coreference (coreference/VenueCoreference.java). Adapted Ben's code to do venue coreference (independent of paper coreference). See coreference/CitationUtils.java. Again, we assume perfect segmentation.

• base: same features as paper coref
• jn-bt-acr: base + one feature for approx journal match and one for booktitle match + acronym feature
• venue-acr: base + one feature for "venue" approx match + acronym
• all: base + jn-bt-acr + venue

Still need to run with noisy CRF output. Then implement joint clustering of papers and venues. How to pass clustering info to segmentation??

Javadocs

Notes:

• Charles skillfully ported and refactored Fuchun's and Ben's segmentation and coreference code from their users directories to projects/seg_plus_coref

• Rewrote SGMLStringOperation to handle tag attributes

• Performed running room tests to see the usefulness of venue clusters. I added a feature that is true if two papers are in the same venue cluster, and another that's true if two papers have the same venue_vol. Expts assume perfect segmentation. Results are in /usr/col/tmp1/culotta/mallet/exp/cite/5.19/{venue.log, novenue.log}

Summary: Venue coref increases cluster precision dramaticall; recall, not so much.

Next steps:

• run trial using CRF output
• train classifier for venue coref
• co-cluster venues and papers

-- AronCulotta - 19 May 2004