Introduction to Affective Computing

  • What does it mean for a computer to "have" emotions? Rosalind W. Picard. In Emotions in Humans and Artifacts (edited by R. Trappl, P. Petta and S. Payr). 2003.
  • Affective Computing: Challenges. Rosalind W. Picard. International Journal of Human-Computer Studies 59(1-2). 2003.
  • This computer responds to user frustration: Theory, design, and results. J. Klein, Y. Moon, and R. W. Picard. Interacting with Computers 14. 2002.
  • Picard's Affective Computing Lab

  • Chapters 1-2 from The Media Equation: How People Treat Computers, Television and New Media Like Real People and Places. Byron Reeves and Clifford Nass. 1996.

  • Towards Caring Machines. T. Bickmore and R. W. Picard. Proceedings of CHI. 2004.
  • OPTIONAL: Establishing and Maintaining Long-Term Human-Computer Relationships. T. Bickmore and R. W. Picard. Trans. on Computer-Human Interaction 12(2). 2004.

  • Toward emotion-sensitive multimodal interfaces: the challenge of the European Network of Excellence HUMAINE. Marc Shroder and Roddy Cowie. Proceedings of User Modeling Workshop on Adapting the Interaction Style to Affective Factors (invited talk). 2005.
  • HUMAINE (Human-Machine Interaction Network on Emotion)

  • LINKS: Affective Dialogue Systems: Proceedings Tutorial and Research Workshop (ADS). 2004.
  • LINKS: Affective Computing and Intelligent Interaction: Proceedings First International Conference (ACII). 2005.

    • Emotional Speech

    Introduction

  • Describing the emotional states that are expressed in speech. Roddy Cowie and Randolph R. Cornelius. Speech Communication 40(1-2). 2003.

  • Vocal communication of emotion: A review of research paradigms. Klaus R. Scherer. Speech Communication 40(1-2). 2003.

    • Databases

  • Emotional speech: Towards a new generation of databases. Ellen Douglas-Cowie, Nick Campbell, Roddy Cowie and Peter Roach. Speech Communication 40(1-2). 2003.
  • A list of emotional databases (based on the HUMAINE deliverable D5c)
  • A list of demonstrators. Some of these links have links to speech-processed data (e.g., Jerk-O-Meter).
  • LINKS: International Workshop on Emotion: Corpora for Research on Emotion and Affect

  • Sections 1-3 of Classifying Subject Ratings of Emotional Speech Using Acoustic Features. Jackson Liscombe, Jennifer Venditti, and Julia Hirschberg. Proceedings of Eurospeech. 2003.
  • Emotional Prosody Speech and Transcripts

  • Sections 1-2.2 of Prosody-Based Automatic Detection of Annoyance and Frustration in Human-Computer Dialog. Jeremy Ang, Rajdip Dhillon, Ashley Krupski, Elizabeth Shriberg, and Andreas Stolcke. Proceedings of International Conference Spoken Language Processing (ICSLP). 2002.
  • 2000 Communicator Evaluation
  • 2000 Communicator Dialogue Act Tagged

  • Kappa

    • Annotation Schemes

  • A two dimensional annotation scheme for emotion in dialogue. Richard Craggs and Mary McGee Wood. Working Notes AAAI Spring Symposium: Exploring Attitude and Affect in Text. 2004.
  • A categorical annotation scheme for emotion in the linguistic content of dialogue. Richard Craggs and Mary McGee Wood. Proceedings of Affective Dialogue Systems. 2004.

  • Annotating Student Emotional States in Spoken Tutoring Dialogues. Diane Litman and Kate Forbes-Riley. Proceedings of 5th SIGdial Workshop on Discourse and Dialogue. 2004.
  • Computer Tutoring Excerpt
  • Human Tutoring Excerpt

  • OPTIONAL:"Of all things the measure is man" - automatic classification of emotions and inter-labeler consistency. Stefan Steidl, Michael Levit, Anton Batliner, Elmar Noth, and Heinrich Niemann. Proceedings International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2005.

    • Speech Processing Tools

  • Sections 1-4 of Recognizing Student Emotions and Attitudes on the Basis of Utterances in Spoken Tutoring Dialogues with both Human and Computer Tutors. Diane J. Litman and Kate Forbes-Riley. Speech Communication. In press.

  • SEMI-OPTIONAL: Toward an Affect-Sensitive Multimodal Human-Computer Interaction (particularly III B; skip III A, C; IV A, C, D, E). Maja Pantic and Leon J. M. Rothkrantz. Proceedings of the IEEE 91(9). 2003.

  • LINKS: Praat
  • OPTIONAL: Praat Tutorial and Resources

  • LINKS: Wavesurfer

    • Spoken Dialogue Systems

    Introduction

  • Where do we go from here? Research and Commercial Spoken Dialog Systems. Roberto Pieraccini and Juan Huerta. Proceedings of 6th SIGdial Workshop on Discourse and Dialogue. 2005.
  • LINKS: This page contains lots of pointers to tools/advice on building your own dialogue system!

  • Dialogue and Conversational Agents. Chapter 19 of Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Daniel Jurafsky and James H. Martin. Draft of May 18, 2005.

    • Automatic Speech Recognition

  • Emotions, speech and the ASR framework . Louis ten Bosch. Speech Communication 40(1-2). 2003.

  • Compensating for Hyperarticulation by Modeling Articulatory Properties. Hagen Soltau, Florian Metze, and Alex Waibel. Proceedings of ICSLP. 2002.

  • Interactions between Speech Recognition Problems and User Emotions. Mihai Rotaru, Diane J. Litman, and Katherine Forbes-Riley. Proceedings 9th European Conference on Speech Communication and Technology (Interspeech-2005/Eurospeech). 2005.

  • OPTIONAL: Automatic Speech Recognition. Chapter 8 of Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Daniel Jurafsky and James H. Martin. Draft of November 27, 2005.
  • OPTIONAL: Chapter 1 of Survey of the State of the Art in Human Language Technology. Ron Cole (ed.). 1996.

    • Emotion Detection in "Realistic" Settings

  • Remaining sections of Prosody-Based Automatic Detection of Annoyance and Frustration in Human-Computer Dialog. Jeremy Ang, Rajdip Dhillon, Ashley Krupski, Elizabeth Shriberg, and Andreas Stolcke. Proceedings of International Conference Spoken Language Processing (ICSLP). 2002.

  • How to find trouble in communication. A. Batliner, K. Fischer, R. Huber, J. Spilker and E. Noth. Speech Communication 40(1-2). 2003.

  • Emotion detection in task-oriented spoken dialogs. Laurence Devillers, Lori Lamel, and Ioana Vasilescu. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). 2003.

  • Voice signatures. Izhak Shafran, Michael Riley, and Mehryar Mohri. Proceedings IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). 2003.

  • Toward detecting emotions in spoken dialogs. Chul Min Lee and Shrikanth S. Narayanan. IEEE Transactions on Speech and Audio Processing 13(2). 2005.

    • Dialogue Management and Dialogue Act Recognition

  • Emotion-Sensitive Human-Computer Interfaces. Thomas S. Polzin and Alexander Waibel. Proceedings SpeechEmotion. 2000.
  • Integrating Emotional Cues Into A Framework For Dialogue Management. Hartwig Holzapfel, Christian Fuegen, Matthias Denecke, and Alex Waibel. Proceedings of ICMI. 2002.

  • Affective Advice Giving Dialogs. Addolorata Cavalluzzi, Valeria Carofiglio and Fiorella de Rosis. Proceedings Affective Dialogue Systems. 2004.

  • Emotion and Dialogue in the MRE Virtual Humans. David Traum, Stacy Marsella, and Jonathan Gratch. Proceedings Affective Dialogue Systems. 2004.

  • Endowing Spoken Language Dialogue Systems with Emotional Intelligence. Elisabeth Andre, Matthias Rehm, Wolfgang Minker, and Dirk Buhler. Proceedings Affective Dialogue Systems. 2004.
  • Exploiting emotions to disambiguate dialogue acts. Wauter Bosma and Elisabeth Andre. Proceedings International Conference on Intelligent User Interfaces. 2004.

  • Using Bigrams to Identify Relationships Between Student Certainness States and Tutor Responses in a Spoken Dialogue Corpus. Kate Forbes-Riley and Diane J. Litman. Proceedings of 6th SIGdial Workshop on Discourse and Dialogue. 2005.

    • Text Generation

  • Towards Emotional Variation in Speech-Based Natural Language Generation. Michael Fleischman and Eduard Hovy. Proceedings International Natural Language Generation Conference (INLG). 2002.

  • Generating Tutorial Feedback with Affect. Johanna D. Moore, Kaska Porayska-Pomsta, Sebastian Varges, and Claus Zinn. Proceedings of the Seventeenth International Florida Artificial Intelligence Research Symposium Conference (FLAIRS). 2004.

  • Emotions from text: machine learning for text-based emotion prediction. Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP). 2005.

  • Coloring Multi-Character Conversations through the Expression of Emotions. Patrick Gebhard, Martin Klesen, and Thomas Rist. Proceedings Affective Dialogue Systems. 2004.

  • Modelling alignment for affective dialogue. Carsten Brockmann, Amy Isard, Jon Oberlander and Michael White. Proceedings of User Modeling Workshop on Adapting the Interaction Style to Affective Factors. 2005.
  • Re-Creating Dialogues from a Corpus. Amy Isard, Carsten Brockmann, and Jon Oberlander. Proceedings of the Corpus Linguistics 2005 Workshop on Using Corpora for Natural Language Generation. 2005.

  • OPTIONAL: Chapter 20 (Natural Language Generation) of Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Daniel Jurafsky and James H. Martin. 2000.
  • OPTIONAL: An Annotated Bibliography of Affective Natural Language Generation. Paul Piwek. 2003.

    • Speech Synthesis

  • Emotional Speech Synthesis: A Review. Marc Schroder. Proceedings of Eurospeech. 2001.

  • The IBM Expressive Speech Synthesis System. Wael Hamza, Raimo Bakis, Ellen M. Eide, Michael A. Picheny, and John F. Pitrelli. Proceedings of ICSLP. 2004.

  • Verification of Acoustical Correlates of Emotional Speech using Formant-Synthesis. Felix Burkhardt and Walter F. Sendlmeier. 2000.

  • Input Requirements for the Synthesis of Conversational Dialogue Speech. Nick Campbell.
  • OPTIONAL?:A corpus-based speech synthesis system with emotion. Akemi Iida, Nick Campbell, Fumito Higuchi and Michiaki Yasumura. Speech Communication 40(1-2). 2003.

  • OPTIONAL: Chapter 4 (Computational Phonology) of Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Daniel Jurafsky and James H. Martin. 2000.
  • OPTIONAL: Chapter 5 of Survey of the State of the Art in Human Language Technology. Ron Cole (ed.). 1996.
  • LINKS: Open Source Emotional Text-to-Speech Synthesis
  • LINKS: Text-to-Speech Links and more...

    • Applications: Tutoring Systems

  • Affect and learning: an exploratory look into the role affect in learning with AutoTutor. Scotty D. Craig, Arthur C. Graesser, Jeremiah Sullins, and Barry Gholson. Journal of Educational Media (29). 2004.
  • Integrating Affect Sensors in an Intelligent Tutoring System. Sidney K. D'Mello, Scotty D. Craig, Barry Gholson, Stan Franklin, Rosalind Picard, and Arthur C. Graesser. Proceedings Affective Interactions: The Computer in the Affective Loop Workshop at International conference on Intelligent User Interfaces. 2005.
  • Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor's Mixed-Initiative Dialogue. Sidney K. D'Mello, Scotty D. Craig, Jeremiah Sullins, and Arthur C. Graesser. International Journal Artificial Intelligence in Education (IJAIED). In press.
  • Emotive AutoTutor

  • Hedged responses and expressions of affect in human/human and human/computer tutorial interactions. Khelan Bhatt, Martha Evens, and Shlomo Argamon. Proceedings of the Cognitive Science Society (COGSCI). 2004.

  • Generating Socially Appropriate Tutorial Dialog. W. Lewis Johnson, Paola Rizzo, Wauter Bosma, Sander Kole, Mattijs Ghijsen, and Herwin van Welbergen. Proceedings Affective Dialogue Systems. 2004.

  • Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems. Heather Pon-Barry, Karl Schultz, Elizabeth Owen Bratt, Brady Clark, Stanley Peters. International Journal of Artificial Intelligence in Education. To appear.
  • SCoT

  • OPTIONAL: Remaining sections of Recognizing Student Emotions and Attitudes on the Basis of Utterances in Spoken Tutoring Dialogues with both Human and Computer Tutors. Diane J. Litman and Kate Forbes-Riley. Speech Communication. In press.
  • ITSPOKE

  • OPTIONAL?: Cognitive state classification in a spoken tutorial dialogue system. Tong Zhang, Mark Hasegawa-Johnson and Stephen E. Levinson. Speech Communication. In press.

    • Evaluation

  • Experimentally Augmenting an Intelligent Tutoring System with Human-Supplied Capabilities: Adding Human-Provided Emotional Scaffolding to an Automated Reading Tutor that Listens. Gregory Aist, Barry Kort, Rob Reilly, Jack Mostow, and Rosalind Picard. Proceedings of Intelligent Tutoring Systems Workshop on Empirical Methods for Tutorial Dialogue Systems. 2002.

  • The Role of Affect and Sociality in the Agent-based Collaborative Learning System. Yasunori Morishima, Hiroshi Nakajima, Scott Brave, Ryota Yamada, Heidy Maldonado, Clifford Nass, and Shigeyasu Kawaji. Proceedings Affective Dialogue Systems. 2004.

  • Data-driven Refinement of a Probabilistic Model of User Affect. Cristina Conati and H. Mclaren. Proceedings of the Tenth International Conference of User Modeling (UM). 2005.

  • Negotiated Collusion: Modeling Social Language and its Relationship Effects in Intelligent Agents. Justine Cassell and Timothy Bickmore. User Modeling and User-Adapted Interaction 13. 2003.