I2CL Assignment 2

Due: Monday 26 November, in lecture

Please hand in (on paper) all of the questions marked below by SUBMIT. There will be no electronic submission for this assignment.

There is an advice page with answers to commonly-asked questions about this assignment. Please check this page -- your question may already be answered! (If it isn't, you are of course free to ask me your question directly.)

Exercises

  1. Part-of-speech tagging (50 points)

    Evaluate the output of two part-of-speech taggers on a set of documents. The taggers are: the Brill tagger described in Chapter 8 of the textbook, and the Hidden Markov Model tagger from the Language Technology Group here in Edinburgh, called LT-POS. Instructions for running each of these taggers are available on a separate page.

    Questions:

    1. Run each of the taggers on the following texts from the Penn Treebank and compare the output to the "gold standard" texts. In each of the lines below, the link to "Text n" (e.g., "Text 1") is to a version of the text formatted with one sentence per line -- this is easier to read, but you should not use it for the actual tagging experiments. The "Tagged" link is to the tagged file from the Treebank, and the "Untagged" file is formatted the same way as the Tagged one for ease of comparison. (If this is confusing, just look at the files and you'll see what I'm talking about...)

      Choose five tagging errors made by each tagger (i.e., 10 errors in total) and discuss the possible reasons for these errors.

      SUBMIT: Print-outs showing the tagging errors you are discussing, and your discussion of the errors.

    2. Quantitatively compare the performance of the two taggers. To do this, you will use this program to compute the confusion matrices comparing each tagger's output to the gold standard and to compute Kappa for each tagger.

      • Compute Kappa for each tagger. Which one performs better on these data?

      • What is causing the errors? Use the confusion matrices to identify any systematic errors. Describe three of them and show an example of each.

      SUBMIT: Kappa values for each tagger, and answers to the above questions.

    3. MSc Students only Try tagging (with both of the provided taggers) one or more of the following texts from Assignment 1. They have already been tokenised as specified on the tagging instructions page.

      Examine the results. Do you think that having this part-of-speech information would have made the task of locating the date and time expressions on Assignment 1 easier? Why or why not?

      4. SUBMIT: Your answer to the above question, along with any parts of the tagged output that support your answer.

  2. Context-free grammars (50 points)

    In this exercise, you will be writing a context-free grammar. You will use the pstone parser to test your grammar. Information on setting up and running pstone is available on a separate page.

    NEW: Here's a link to an online grammar for those who want a reference for English grammar.

    Questions:

    1. The task is to create a good grammar which is able to handle a large fraction of the sentences in the given input file. You should be able to parse all of the "base" sentences; if your grammar is more powerful than that, you will be eligible for bonus marks. UPDATE 21/11/2001: See advice page for details.

      You can begin by improving on simple-gramm.gr and guardian.le. Note that the lexicon in particular is fairly basic (e.g., all verbs are labelled "Verb") and may not have all necessary categories for the words. You can modify it as much as you want.

      Files:

      SUBMIT:

      • A print-out of your grammar and lexicon.
      • For two of the input sentences for which your grammar produced multiple parses: provide a brief discussion of whether these sentences are truly ambiguous or whether your grammar needs refinement.
      • For the sentences that your grammar could not handle, a discussion of at least three reasons for the problem(s).
      • A print-out of 20 sentences generated at random from your grammar, with indications of which of them you find odd. Create these sentences with 
        ./generate 20 -g my-grammar.gr -l my-lexicon.le
        in your pstone directory. NEW (21/11/2001): See the note on the advice page for an alternative to this question if generate is producing enormously long sentences.

    2. Once you have created your grammar, try it out with the following lexicon and sentence list. The sentences are from "Mother Goose" rhymes.

      SUBMIT: A description of how well your grammar did at parsing these sentences, and a discussion of why it might have done as well (or badly!) as it did.
Last modified: 21 November 2001 at 19:14 by MEF          © 2001 School of Cognitive Science