Automated Scoring/Evaluation for oral and written student responses

Content Analysis

• Leacock, C. Scoring free-responses automatically: A case study of a large-scale assessment. Examens, 1(3). 2004. (paraphrase recognition, content assessment)
• Jana Sukkarieh, Stephen Pulman. Information Extraction and Machine Learning: Automarking Short Free Text Responses to Science Questions. Proceedings of 12th International Conference on Artificial Intelligence in Education (AI-ED). 2005. (information extraction, content assessment)

• Rodney D. Nielsen, Wayne Ward, James H. Martin and Martha Palmer. Extracting a Representation from Text for Semantic Analysis. Proceedings of the Association for Computational Linguistics and the Human Language Technologies Conference (ACL-08:HLT), pp 241-244, Columbus, Ohio. 2008 (semantics, intelligent tutoring)
• Rodney D. Nielsen, Wayne Ward, James H. Martin. Automatic Generation of Fine-Grained Representations of Learner Response Semantics. Intelligent Tutoring Systems: 173-183. 2008. (semantics, intelligent tutoring systems)
• OPTIONAL: Rodney D. Nielsen, Wayne Ward, James H. Martin: Learning to Assess Low-Level Conceptual Understanding. FLAIRS Conference 2008: 427-432 (semantics, intelligent tutoring systems)

• D. Higgins, J. Burstein, and Y. Attali. Identifying off-topic student essays without topic-specific training data. Natural Language Engineering 12 (2): 145-159. 2006. (unsupervised text classification, content assessment)

• Maxim Makatchev, Pamela Jordan and Kurt VanLehn. Abductive Theorem Proving for Analyzing Student Explanations and Guiding Feedback in Intelligent Tutoring Systems. Journal of Automated Reasoning for Special Issue on Automated Reasoning and Theorem Proving in Education, vol. 32, issue 3, pp 187-226. 2004. (abduction, intelligent tutoring)
• OPTIONAL: Rose, C. P., & VanLehn, K. An Evaluation of a Hybrid Language Understanding Approach for Robust Selection of Tutoring Goals. International Journal of AI in Education 15(4). 2005.

Grammatical Error Detection

• Na-Rae Han, Martin Chodorow, and Claudia Leacock. Detecting errors in English article usage by non-native speakers. Natural Language Engineering 12 (2): 115-129. 2006. (grammar, language learning)
• Joel Tetreault and Martin Chodorow. The Ups and Downs of Preposition Error Detection in ESL Writing. COLING, Manchester, UK. 2008. (grammar/annotation, language learning)
• Michael Gamon, Jianfeng Gao, Chris Brockett, Alexander Klementiev, William Dolan, Dmitriy Belenko, Lucy Vanderwende. Using Contextual Speller Techniques and Language Modeling for ESL Error Correction. Proceedings of IJCNLP, Hyderabad, India. Jan. 2008. (spelling correction/language modeling, ESL writing)

• Michaud, L.N. and McCoy, K.F. Capturing the Evolution of Grammatical Knowledge in a CALL System for Deaf Learners of English. IJAIED 16, 65-97. 2006. (grammar, language learning)
• J. Lee and S. Seneff. Correcting Misuse of Verb Forms. Proceedings ACL, Columbus, Ohio. 2008. (parsing/n-grams, language learning)

• Links: Microsoft Research ESL Assistant

Discourse Analysis

• Burstein, J., Chodorow, M., & Leacock, C. Criterion: Online essay evaluation: An application for automated evaluation of student essays. Proceedings of the fifteenth annual conference on innovative applications of artificial intelligence, Acapulco, Mexico. (This paper received an AAAI Deployed Application Award.) 2003. (deployed assessment and tutoring system, writing)
• Attali, Y., & Burstein, J. Automated essay scoring with e-rater v.2. Journal of Technology, Learning, and Assessment, 4(3). 2006. (validation, writing)

• Higgins, D., Burstein, J., Marcu, D., & Gentile, C. Evaluating multiple aspects of coherence in student essays Proceedings of the annual meeting of HLT/NAACL, Boston, MA. 2004. (discourse coherence, writing)
• E. Miltsakaki and K. Kukich. Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering 10:1:25-55. 2004. (discourse coherence, writing)

• Christian F. Hempelmann, Vasile Rus, Arthur C. Graesser and Danielle S. McNamara. Evaluating State-of-the-Art Treebank-style Parsers for Coh-Metrix and Other Learning Technology Enviroments. Natural Language Engineering 12(2):131-144. 2006. (parsing, text cohesion assessment)
• McCarthy, P.M., Rus, V., Crossley, S.A., Bigham, S.C., Graesser, A.C., & McNamara, D.S. Assessing entailer with a corpus of natural language. Proceedings of the twentieth International Florida Artificial Intelligence Research Society Conference, pp. 247-252. 2007. (entailment, self-explanation)

• OPTIONAL: Foltz, P. W., Kintsch, W.,& Landauer, T. K. The Measurement of Textual Coherence with Latent Semantic Analysis. Discourse Processes, 25, 285-307. 1998. (coherence, writing/readability)

• Links: Criterion, Coh-Metrix, Entailment Challenge

Spoken Language

• Nigel G. Ward, Rafael Escalante, Yaffa Al Bayyari, and Thamar Solorio. Learning to Show You're Listening. Computer Assisted Language Learning, 20, pp 385 - 407. 2007. (dialogue/prosody, language learning/back-channeling assessment and training)

• Klaus Zechner and Isaac I. Bejar. Towards Automatic Scoring of Non-Native Spontaneous Speech. Proceedings of the HLT-NAACL Conference, New York, NY. 2006. (speech recognition, speaking proficency assessment)
• Beck, J. E., & Sison, J. Using knowledge tracing in a noisy environment to measure student reading proficiencies. International Journal of Artificial Intelligence in Education, 16, 129-143. 2006. (speech recognition, reading proficiency assessment)

• Chao Wang and Stephanie Seneff. High-Quality Speech-to-Speech Translation for Computer-Aided Language Learning. ACM Transactions on Speech and Language Processing:3(2):1-21. 2006. (machine translation/spoken dialogue, language learning)
• Chao Wang and Stephanie Seneff. Automatic Assessment of Student Translations for Foreign Language Tutoring. Proceedings of NAACL-HLT. 2007. (machine translation, language learning)

• Links: Back-Channel Trainer and Project, MIT (Wang) video

Classroom Tools

Automatic identification or generation of materials at a particular readability or grade level

• OPTIONAL: E. Miltsakaki and A. Troutt. READ-X: Automatic evaluation of reading difficulty of web text. Proceedings of E-Learn. 2007. (real-time systems, readability)
• E. Miltsakaki and A. Troutt. Real-Time Web Text Classification and Analysis of Reading Difficulty. The 3rd Workshop on Innovative Use of NLP for Building Educational Applications, ACL. 2008. (real-time systems, readability)
• Emily Pitler and Ani Nenkova. Revisiting Readability: A Unified Framework for Predicting Text Quality. Proceedings of EMNLP. 2008. (lexical/syntax/discourse, readability)

• Sarah E. Petersen and Mari Ostendorf. A Machine Learning Approach to Reading Level Assessment. Computer Speech and Language, Volume 23, pages 89-106. 2009. (statistical NLP (n-grams, parsing, SVMs)/annotation, reading level assessment)

• M. Heilman, K. Collins-Thompson, J. Callan, and M. Eskenazi. Combining lexical and grammatical features to improve readability measures for first and second language texts. NAACL-HLT, 2007. (language modeling and grammar, readability)
• Kulkarni, A., Heilman, M., Eskenazi, M., and Callan, J. Word Sense Disambiguation for Vocabulary Learning. Ninth International Conference on Intelligent Tutoring Systems. 2008. (word sense disambiguation, language learning)

• Links: Penn Discourse Treebank, REAP Demos

Automatic generation of test questions

• J. Brown, G. Frishkoff, and M. Eskenazi. Automatic question generation for vocabulary assessment. Proceedings of HLT/EMNLP. 2005. (lexical semantics, reading)
• Ruslan Mitkov, Le An Ha, and Nikiforos Karamanis. A computer-aided enviroment for generating multiple-choice test items. Natural Language Engineering 12(2): 177-194. 2006. (parsing/information extraction/semantics, multiple choice test generation)

• OPTIONAL: Cox, Barker-Plummer, D., Dale, R. and Etchemendy, J. An Empirical Study of Errors in Translating Natural Language into Logic. Proceedings of the 30th Annual Conference of the Cognitive Science Society Washington, DC. 2008. (Generation of NLP->FOL translation questions: varying difficulties can be predicted via NLP properties)

Processing of and access to online lecture materials

• J. Glass, T. Hazen, S. Cyphers, I. Malioutov, D. Huynh, and R. Barzilay. Recent Progress in the MIT Spoken Lecture Processing Project. Proc. Interspeech, 2553-2556, Antwerp. 2007 (spoken document retrieval/browsing, lecture processing)
• Igor Malioutov, Regina Barzilay. Minimum Cut Model for Spoken Lecture Segmentation. In Proc. of ACL/COLING. 2006. (discourse, lecture processing)

• Links: Lecture Browser

Deployed systems (both commercial products, and research prototypes in classrooms)

• OPTIONAL: Kintsch, E., Steinhart, D., Stahl, G. & LSA research group. Developing Summarization Skills through the Use of LSA-Based Feedback. Interactive learning environments, 8(2), 87-109. 2000. (Latent Semantic Analysis, writing)
• OPTIONAL: Heilman, M., Collins-Thompson, K., Callan, J. & Eskenazi, M. Classroom success of an Intelligent Tutoring System for lexical practice and reading comprehension. Proceedings of the Ninth International Conference on Spoken Language Processing. 2006. (deployment challenges, ITS/language learning/reading)

Affect, Social Agents, and Motivation

Social Language

• Wang, N. and Johnson, W.L. The Politeness Effect in an intelligent foreign language tutoring system. Proceedings of ITS. 2008. (politeness, pedagogical agents/language learning)
• Ning Wang, W. Lewis Johnson, Richard E. Mayer, Paola Rizzo, Erin Shaw, Heather Collins. The politeness effect: Pedagogical agents and learning outcomes. International Journal of Human-Computer Studies, Volume 66 Issue 2. 2008. (politeness, pedagogical agents)

• Gupta, S., Walker, M.A., Romano, D.M. POLLy: A Conversational System that uses a Shared, Representation to Generate Action and Social Language. Third International Joint Conference on Natural Language Processing, Hyderabad, India, 2008. (politeness/ECAs/dialogue generation, language learning)
• Johanna D. Moore, Kaska Porayska-Pomsta, Sebastian Varges, and Claus Zinn. Generating Tutorial Feedback with Affect. Proceedings of the Seventeenth International Florida Artificial Intelligence Research Sociey Conference, AAAI Press. 2004. (affect/generation, tutorial dialogue)

• Links: Tactical Language & Culture Training System, by Alelo

Embodied Agents and Educational Games

• W. Lewis Johnson. Serious use of a serious game for language learning Proceedings Artificial Intelligence in Education. 2007. (spoken communciation, language/culture learning)
• Hjalmarsson, A., Wik, P., & Brusk, J. Dealing with DEAL: a dialogue system for conversation training. Proceedings of SigDial, pp. 132-135. Antwerp, Belgium. 2007. (dialogue, language learning)
• Hjalmarsson, A. Speaking without knowing what to say... or when to end. Proceedings of SIGDial. Columbus, Ohio, USA. 2008 (dialogue/generation/corpora, language learning)

• Jonathan Rowe, Eunyoung Ha, James Lester. Archetype-Driven Character Dialogue Generation for Interactive Narrative. In Proceedings of the Eighth International Conference on Intelligent Virtual Agents, Tokyo, Japan, pp. 45-58. 2008 (unification grammars/dialogue generation, interactive narrative learning environments)
• Scott McQuiggan, Jonathan Rowe, Sunyoung Lee, and James Lester. Story-Based Learning: The Impact of Narrative on Learning Experiences and Outcomes. In Proceedings of the Ninth International Conference on Intelligent Tutoring Systems, Montreal, Canada, pp. 530-539. 2008. (dialogue, interactive narrative learning environments)

• OPTIONAL: Chao Wang and Stephanie Seneff. A Spoken Translation Game for Second Language Learning. Proceedings of AIED. 2007. (machine translation, language learning)
• OPTIONAL: I. McGraw and S. Seneff. Speech-enabled Card Games for Language Learners. Proceedings AAAI, Chicago, Illinois. 2008. (speech recognition, language learning)

• Links: DEAL is developed as a free-standing part of Ville, a framework for language learning developed at KTH. Whereas Ville is a language tutor who provides the user with feedback on performance, the agent in DEAL does not comment on your performance but acts as your conversation partner in a role-playing fashion. DEAL adds the possibility to give conversation training.

Learner Affect

• Aist, G., Kort, B., Reilly, R., Mostow, J., & Picard, R. Experimentally Augmenting an Intelligent Tutoring System with Human-Supplied Capabilities: Adding Human-Provided Emotional Scaffolding to an Automated Reading Tutor that Listens. Proceedings ITS 2002 Workshop on Empirical Methods for Tutorial Dialogue Systems, San Sebastian, Spain. 2002. (affect, tutorial dialogue)
• Nigel Ward and Wataru Tsukahara. A Study in Responsiveness in Spoken Dialog. International Journal of Human-Computer Studies, 59 (6), pp 959-981. 2003. (affect, tutorial dialogue)

• D'Mello, S.K., Craig, S.D., Witherspoon, A., McDaniel, B., & Graesser, A.C. (2008). Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1-2), 45-80. 2008. (affective dialogue, intelligent tutoring)
• Graesser, A.C., D'Mello, S.K, Craig, S.D., Witherspoon, A., Sullins, J., McDaniel, B., & Gholson, B. The relationship between affect states and dialogue patterns during interactions with AutoTutor. Journal of Interactive Learning Research, 19, 293-312. 2008. (affetive dialogue, intelligent tutoring)

• Heather Pon-Barry, Karl Schultz, Elizabeth Owen Bratt, Brady Clark, and Stanley Peters. Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems. International Journal of Artificial Intelligence in Education 16, 171-194. 2006. (adaptive affective dialogue, intelligent tutoring)
• Kristy Elizabeth Boyer, Robert Phillips, Michael Wallis, Mladen Vouk, and James Lester. Balancing Cognitive and Motivational Scaffolding in Tutorial Dialogue. In Proceedings of the Ninth International Conference on Intelligent Tutoring Systems, Montreal, Canada, pp. 239-249. 2008. (motivation, tutorial dialogue)

• OPTIONAL: ITSPOKE paper (see my homepage)

• Links: Project Listen, AutoTutor, SCoT

• Rose, C.P., Wang, Y.C., Cuie, Y., Arguello, J. Fischer, F. Weinberger, A., Stegmann, K. Analyzing Collaborative Learning Processes Automatically: Exploiting the Advances of Computational Linguistics in Computer-Supported Collaborative Learning. International Journal of Computer Supported Collaborative Learning. 2007? (text classification, collaborative learning)
• Kumar, R., Rosé, C. P., Wang, Y. C., Joshi, M., Robinson, A. Tutorial Dialogue as Adaptive Collaborative Learning Support. Proceedings of Artificial Intelligence in Education. 2007. (adaptive dialogue, collaborative learning)

  Links: TagHelper

Classroom Discourse

• Resnick, L., O'Connor, C., and Michaels, S. Classroom Discourse, Mathematical Rigor, and Student Reasoning: An Accountable Talk Literature Review. 2007. (dialog acts, classroom discourse)
• Yamakawa,Y., Forman, E., and Ansell, E. The role of positioning in constructing an identity in a third grade mathematics classroom. 2005.

Observer Learning

• Scotty D. Craig, Barry Gholson, Matthew Ventura, Arthur C. Graesser and the Tutoring Research Group. Overhearing Dialogues and Monologues in Virtual Tutoring Sessions: Effects on Questioning and Vicarious Learning. International Journal of Artificial Intelligence in Education, 11, 242-253. 2000. (tutorial dialogue, vicarious learning)
• Chi, M.T., Roy, M., & Hausmann, R.G. (March, 2008). Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. Cognitive Science: A Multidisciplinary Journal, 32:2, 301-341.

• Piwek, P., H. Hernault, H. Prendinger, M. Ishizuka. T2D: Generating Dialogues between Virtual Agents Automatically from Text. In: Intelligent Virtual Agents: Proceedings of IVA07, LNAI 4722, September 17-19, 2007, Paris, France, pp.161-174. 2007. (generation, vicarious learning)
• Hernault, H., P. Piwek, H. Prendinger and M. Ishizuka. Generating Dialogues for Virtual Agents Using Nested Textual Coherence Relations. In: Proceedings of IVA08: 8th International Conference on Intelligent Virtual Agents, Tokyo, Japan. 2008. (generation, vicarious learning)

Intelligent Tutoring Systems

Tutorial dialogue systems

• K. Vanlehn, A. Graesser, G. T. Jackson, P. Jordan, A. Olney and C. Rose. When are Tutorial Dialogues More Effective than Reading? Cognitive Science 31(1), 3-62. 2007. (dialogue, intelligent tutoring systems)

• OPTIONAL: Papers from Circsim and Beetle projects

Generation

• Di Eugenio, B., Fossati, D., Haller, S., Yu, D. and Glass, M. Be Brief, And They Shall Learn: Generating Concise Language Feedback for a ComputerTutor. IJAIED 18, 317-345. 2008.

Data mining of corpora

• Jack Mostow and Joseph Beck. Some useful tactics to modify, map and mine data from intelligent tutors. Natural Language Engineering 12(2): 195-208. 2006. (data mining, reading/dialogue tutors)

Speech

• Boulder Language Technologies Science Tutors: IES proposal, NSF proposal, similarites/differences document Annual Report
• OPTIONAL: Diane J. Litman, Carolyn P. Rose, Kate Forbes-Riley, Kurt VanLehn, Dumisizwe Bhembe, and Scott Silliman. Spoken Versus Typed Human and Computer Dialogue Tutoring. International Journal of Artificial Intelligence in Education, Volume 16, Pages 145-170. 2006. (spoken dialogue, intelligent tutoring systems)