Project (CS 2731 / ISSP 2230), Fall 2013

The project for this class will be to design, build, and evaluate a Student Response Analysis (SRA) system. This will give you exposure to a cutting edge research area, and experience in building a real NLP system. The task offers the opportunity to evaluate the usefulness of approaches for semantic analysis in a practical application-oriented setting. Current e-learning systems have limited capability for giving students feedback and providing automatic assessment since there is no established technology for assessing natural language responses to questions.

Ideally, the class will be divided into 3-person or 2-person teams for the project. You may form your own team if you know people with whom you'd like to work. Otherwise, I can randomly assign you to a team. If you really want to work by yourself, or have a larger team, that is also possible.

Project Description

Project Resources

Training Set

Tools

Alternatively, the above files can be directly obtained from SemEval_train.zip.

Your course project systems are also free to use any external resources such as WordNet, Wikipedia, Science Dictionary, etc.

Sample Student Response Analysis Papers

Previous approaches to student response analysis include methods based on text classification, latent semantic analysis and other semantic similarity measurements, textual entailment, and, in small domains, parsing and rule-based methods.

In particular, SRA can be cast as a form of recognizing textual entailment (Nielsen et al., 2008). However, educational domains present additional challenges. The text produced by students is often ill-formed and contains many typos and grammar mistakes, and often contains domain terminology that is rare in common training data sets. Collecting and annotating data sets for every new subject domain is expensive. Thus, the task is likely to require adapting existing approaches trained on large amounts of out-of-domain data to a new domain based on a small amount of in-domain training data, and implementing methods to deal with irregularities found in (typed) student input.

Alternatively, approaches used in essay scoring, such as LSA, could be transferred and adapted for use with shorter student answers provided in the task. The likely challenges include the shorter length of student answers, and the need to output categorical labels (chosen intentionally to help guide feedback decisions) rather than numeric grades commonly used in essay scoring.

Below is a sampling of the literature:

Program Submission Instructions

Use the same instructions as for the homeworks. Please read these additional guidelines:

Project Report Instructions

Your writeup should both describe your system (the architecture, components, etc.), and contain a discussion evaluating how well the version turned in for the final evaluation performed on Training Set (and if the results are available, on Test Set). Use the provided programs to compute performance. The conference papers that we have been reading are good models for your project paper. Papers should be NO LONGER THAN 4 pages (excluding references) using ACL 2013 Style Files (available at http://acl2013.org/site/call.html)

Project Presentation Instructions

Each team should prepare a short oral powerpoint presentation to share the contents of your project report with the rest of the class. Presentations should be 10-12 minutes (subject to change once I know how many teams there are). I will cut them off at 12 minutes to allow 3 minutes for questions.