Evaluating Tagger Output

This page provide more information to help you evaluate the tagger output in Homework 2. Here is a perl program that can be used to compute the confusion matrices and kappa for the tagger.

Installing the program

Download the program: compare-taggers.pl
Save it into an appropriate directory.
Make the file executable by typing chmod ugo+x compare-taggers.pl
You may have to fix the first line to refer to the local version of perl (which you can find by saying which perl)

Running the program

The basic syntax for compare-taggers.pl is as follows:

./compare-taggers.pl (-b|-h) (-m) (-k) tagger-output gold-standard

The parameters are as follows:

tagger-output: The file containing the output from the tagger
gold-standard: The file containing the "gold standard" tags
-b: Specify that the output is from the Brill tagger
-m: Print out the confusion matrix
-k: Print out kappa

Examples

To print the confusion matrix and kappa for the tagged file wsj_1975.brill, tagged by the Brill tagger, and save the matrix and kappa in the file wsj_1975.results:

./compare-taggers.pl -b -m -k wsj_1975.brill wsj_1975.pos > wsj_1975.results

Notes

Please tag the "raw" files (with the nasty formatting) when you are going to use compare-taggers.pl -- it relies on the formatting being the same in the tagged and gold-standard files.
To compute the confusion matrix or kappa for all of the texts, use the files all.raw and all.pos, which contain all of the texts in one big file.