O921-Prelims.qxd 5/19/06 11:15 AM Page i This page intentionally left blank CHAPTER Seven The Voice 7.1 What Is Covered and Why Chapters 5 and 6 were about social cues that engage the eyes. This chapter is devoted to the ear. Chapter 7 completes the discussion of characters’ basic social equipment with discussion of the voice—the rich messages that people convey through how they say things. 1 The chapter includes an overview of the kinds of social cues that the voice conveys, with many listenable examples from games (including Warcraft III: Reign of Chaos, Final Fantasy X, The Sims™, Grim Fandango, and Curse of Monkey Island), and offers design tips for considering the aural side of social signals when crafting character voices. Chapter 7 also includes discussion of some future-facing voice technology and an interview with two pioneers in using emotion detection from voice cues to adjust interfaces. 7.2 The Psychological Principles Before reading this section, take a moment to listen to the first two voice samples on the DVD (Clips 7.1 and 7.2). While listening to each person, try to form a mental picture: How old are they? What gender? Is this person of high or low status? Are they in a good or bad mood? Then see Section 7.9 for photos of the speakers. Most likely you correctly identified the majority of these visible traits from voice alone. Listening to a person’s voice on the telephone, you can often make a good guess about age, gender, social status, mood, and other characteristics without any visual cues to help. Even if the person is speaking another language and you cannot understand the meaning of the speech, you can still get pretty far in assessing these qualities. How is this possible? Researchers point to the evolutionary roots of speech in the grunts and calls of our primate ancestors. There are striking similarities in the vocal characteristics of fright, anger, and dominance, among other social cues, when one compares primate 183 1 Analyzing the social meaning of what characters say moves into the territory of linguistics, which would require a book in and of itself. For further reading on this topic, see (Clark 1996). O921-Ch07.qxd 5/10/06 10:36 AM Page 183 CHAPTER SEVEN • THE VOICE 184 and human voices. Researchers who asked participants in a study to listen to male macaque monkeys found that more than 80% of the listeners could accurately identify what the dominance calls meant (Tusing and Dillard 2000, 149). Social scientists refer to the information that is not conveyed by the words in speech as paralinguistic cues. A large proportion of the meaning in everyday conversation emerges through paralinguistic cues—shifts in voice quality while speaking, pauses, grunts, and other nonlinguistic utterances. Paralinguistic cues play an even bigger role in communication between people who already know each other well—a well-placed sigh or lack of a heartfelt tone conveys volumes. To make characters seem richly human in their communication, then, a designer should have a solid understanding of what they are conveying with how they say things. 7.2.1 The Mechanics of Speech To speak, a person pushes air from the lungs through the larynx, mouth, and nose. The pitch and the qualities of sounds are affected in two different ways: phonation and articulation. Phonation is the way a person moves the larynx itself to make the initial sound. When the shape of the muscle folds in the larynx (which used to be called the vocal cords) is altered, it produces different sound pitches (called fundamental frequencies) and also different sound qualities, such as breathiness or harshness. These qualities can shift due to a person’s emotional state—tenseness, tiredness, depression, and excitement all can have effects on phonation. Articula- tion is when a person uses the natural resonance of the mouth, nose, and even of the chest cavity, as well as moving the tongue and lips and palate, to alter the sound as it comes out. People are very sensitive to shifts in articulation—for example, a person can “hear” a smile in another’s voice, in part because the shift in lip shape when speaking affects the articulation of the sound (see Figure 7.1). Listen to audio Clips 7.3 and 7.4 on the DVD. Can you tell which recording was made while smiling? See Section 7.9 for the answer. As with facial expression and gesture, some of what people hear in others’ voices comes from their physical qualities and their body’s involuntary reactions to circumstances. Some comes from learned strategies and responses to social circum- stances. For example, gender and age come across in voice because of physical qualities of the person’s vocal equipment itself (which can be a problem for people whose voices fall outside the usual range for their gender or age group). Mood and emotion are signalled involuntarily (at least in part) because of changes in vocal production as the person’s nervous system reacts—for example, the dry mouth and speedier heart rate of anxiety also have effects on the muscles in the larynx and on breathing itself. However, a person can also mold the tone of his or her voice in some ways, adopting a pacifying, pleading, arrogant, or neutral tone of voice using intonation and rhythm (referred to as prosody by researchers). Failing to adopt the proper social tone of voice is a communication in and of itself. O921-Ch07.qxd 5/10/06 10:36 AM Page 184 7.2 THE PSYCHOLOGICAL PRINCIPLES 7.2.2 The Social Signals in Voice Emotion in Voice Emotions underpin decision-making, including social action and reaction (see [Damasio 1994] for a fascinating account of the role of emotions in thinking). Know- ing that another person is angry is crucial to understanding how they are interpreting social actions and the world at large, thus helping to predict what they might do next. Failure to recognize emotional expression is a serious liability in human inter- action—it is in fact a symptom of some disorders in the autism spectrum. Chapter 5 touched upon Ekman and colleagues’ work on recognizing facial expressions of emotion. There has also been extensive work on the expression of emotion in the voice. Voice researchers have found, when they look for con- sistent signatures of emotions, that there are clear patterns (see [Kappas, Hess, and 185 Tension and movements Lips Tongue Pharynx Height of larynx Tension and shape of folds Regularity of fold vibration Subglottal air pressure Regularity, depth Respiration Respiration Articulation Phonation Cortex Limbic system Limbic system Jaw Pharynx Both phonation— the action of the larynx (vocal cords) and articulation— the shaping of the mouth, tongue, and lips—create the subtle alterations in tone that carry social and emotional information (based on Kappas,Hess, and Scherer 1991). FIGURE 7.1 O921-Ch07.qxd 5/10/06 10:36 AM Page 185 CHAPTER SEVEN • THE VOICE Scherer 1991; Cahn 1990; and Burkhardt and Sendlmeier 2000] for more detail on the taxonomy that follows): • Anger (hot). Tense voice, faster speech rate, higher pitch, broader pitch range • Anger (cold). Tense voice, faster speech rate, higher fundamental frequency and intensity, tendency toward downward-directed intonation contours • Joy. Faster speech rate, raised pitch, broader pitch range, rising pitch pattern • Fear. Raised pitch, faster speech rate, broadened range, high-frequency energy • Boredom. Slower speech rate, additional lengthening of stressed syllables, lowered pitch, reduced pitch range and variability • Sadness (crying despair). Slower speech rate, raised pitch, narrowed pitch range, narrowed variability • Sadness (quiet sorrow). Slower speech rate, lowered pitch, narrower pitch range, narrower variability, downward-directed contours, lower mean intensity, less precision of articulation • Depression. Lower intensity and dynamic range, downward contours Notice the similarities among emotions—fear, anger, and joy all seem to be signalled by faster speech, higher pitch, and more range. In contrast, quiet sadness, depression, and boredom share slowing of pace, lower pitch, and less variability. These effects can be traced back to what is going on in the person’s nervous system. The arousal of a person’s sympathetic nervous system, which causes things like increased heart rate and sweating, also causes these changes in the voice. When a person’s parasympathetic system, which decreases blood pressure and slows heartrate, moves into action, it also shifts what happens in the voice itself. A glance back at the Laban movement graphs in Chapter 6 (Figures 6.12 and 6.13) shows that body movement style also seems to be modulated in this way. So how do people learn to tell apart the high-energy or low-energy emotions in the voice? Certainly they use context contributed by the words themselves, but people are also able to tell what position the mouth is in, based upon sound. As mentioned above, a person can “hear” a smile. Voice researchers have detected different patterns of intonation as well—such as the characteristic rising pitch pattern of joy. It is also the case that people acclimate to one another’s vocal patterns—knowing someone well includes knowing how they, in particular, signal sadness or joy with their voice. One game that takes full advantage of the power of paralinguistic cues in con- veying emotion is The Sims™. Sim characters speak to one another, but their words are entirely incomprehensible. Simlish may be gibberish, but it is laden with emo- tional signals, and it allows the player to draw conclusions about how his or her Sim is feeling in general, and in relation to other Sim characters (see Figure 7.2). For example, listen to Clip 7.5. As the Sim characters move from joy to jealousy, it is easy to follow along despite the lack of words. 186 O921-Ch07.qxd 5/10/06 10:36 AM Page 186 7.2 THE PSYCHOLOGICAL PRINCIPLES Interestingly, researchers have found connections between the expression of emotion in the human voice and strategies for evoking emotion through music. Two researchers performed a meta-analysis of research that had been done on evoking specific emotions with music, with work on emotions in speech. Data “strongly sug- gest . . . that there are emotion-specific patterns of acoustic cues that can be used to communicate discrete emotions in both vocal and musical expressions of emotion” (Juslin and Laukka 2003, 799). This makes sense if one considers that the playing of musical instruments tends to reflect the muscular tension and general arousal state of the performer—creating a bridge to the listener into a particular emotional state. Some games, for example, Grim Fandango, make use of this connection between music and emotion to heighten the player’s experience of a character’s emotional reactions. See Figure 7.3 and Clip 7.6, in which Manny’s boss berates him for a 187 The Sim language—“Simlish”—uses paralinguistic cues of emotion (listen to Clip 7.5).The Sims™ Unleashed image ©2005 Electronic Arts Inc.The Sims is a registered trademark of Electronic Arts Inc. in the U.S. and other countries.All rights reserved. Grim Fandago uses music to heighten the player’s reaction to an NPC’s tirade (listen to Clip 7.6). ©1998 Lucasfilm Entertainment Company Ltd.All rights reserved. FIGURE 7.2 FIGURE 7.3 O921-Ch07.qxd 5/10/06 10:36 AM Page 187 CHAPTER SEVEN • THE VOICE mistake. Notice the music in the background, which displays some of the same aural qualities as the boss’s tirade. Social Context and Identity As researchers begin to assemble a more detailed picture of how the voice con- tributes to social interaction, one thing they are realizing is that emotion is not necessarily the predominant message communicated. Researchers in Japan who gathered a large body of recorded speech by asking people to wear headsets around in everyday life, found few examples of strong emotion in voices. Day to day, peo- ple tended to keep their emotional reactions mostly to themselves. What did show up were big differences in patterns of voice depending upon who the person was speaking to—adjustment based on social roles and relationships (Campbell 2004) and, of course, individual differences in vocal style that emerged from each person’s own personality and physical qualities. Some traces of social roles and relationships in voices can be broken down along the dimensions first discussed in Chapter 2: cues of dominance and of friendliness. People demonstrating dominance tend to lower their voice somewhat and to construct shorter utterances in general. They may sometimes speak more loudly, depending upon the situation. Showing submission with voice involves using a softer, more highly pitched voice, and subordinates tend to say more. As was men- tioned earlier in this chapter, these general vocal contours of dominance are true of other primates as well as people. Clips 7.7 and 7.8 demonstrate the difference between dominant and submissive voices. Although the butler (Raoul) is initially very dominant, he moves to submissive obsequiousness in the second clip once Manny has a pass to the VIP lounge (see Figure 7.4). In general, Grim Fandango 188 Manny tries to gain entrance to the VIP lounge (from Grim Fandango).Listen to Clips 7.7 and 7.8 for the shift in the butler’s voice once he knows Manny will be admitted. ©1998 Lucasfilm Entertainment Company Ltd. All rights reserved. FIGURE 7.4 O921-Ch07.qxd 5/10/06 10:36 AM Page 188 7.2 THE PSYCHOLOGICAL PRINCIPLES makes brilliant use of vocal dominance cues to heighten comic effect. Other exam- ples of games that use dominance cues in similar ways are The Curse of Monkey Island (Figure 7.5, Clip 7.9) and Warcraft III (Figure 7.6, Clip 7.10). Friendliness is shown in various ways. Meeting with a friend usually leads to warmth and energy in the voice, the signals of joy. Conversation among close friends includes more range of emotion than between more distant acquaintances— more revelation of personal emotional state and empathizing with the other’s state 189 The Curse of Monkey Island also makes use of dominance cues to heighten comic effect (listen to Clip 7.9) ©1997 Lucasfilm Entertainment Company Ltd.All rights reserved. FIGURE 7.5 FIGURE 7.6 The peons in Warcraft III are charmingly submissive in their voices and responses to player commands (listen to Clip 7.10).Warcraft III: Reign of chaos provided courtesy of Blizzard Entertainment, Inc. O921-Ch07.qxd 5/10/06 10:36 AM Page 189 CHAPTER SEVEN • THE VOICE through modulation of your own voice. Intimacy with someone is often reflected with a more breathy quality in the voice. For an example of the breathiness of inti- macy, contrast the clips from Grim Fandango above, with Clip 7.11—a conversation between Manny and his love interest, Meche. Both Manny and Meche have a great deal of breathiness in their voices. Individual personality can come through in the voice as characteristic patterns of emotion and energy. For example, in Grim Fandango, the hat-check girl is a high-energy character who makes rapid turns from enthusiasm to anger (see Figure 7.7, Clip 7.12). Social Interaction Logistics Vocal modulations during an interaction show that a person is listening and comprehending and also help to orchestrate turn-taking in conversation. “Back- channel” responses such as “uh hunh” make the speaker feel the listener is engaged with what is happening. People also use such noises to indicate that they are still thinking or to express a range of emotions in response to a statement before they can put them into words. Games with elaborate and extensive cut scenes, such as Final Fantasy X, make artful use of these sorts of cues to reveal the nuances of relationships among characters (see Figure 7.8). Back-channel responses may be one reason that people enjoy using voice- enabled multiplayer online games as well—players can hear the triumph or despair in one another’s voices as they play, heightening the experience itself, and can use vocal cues (e.g., “whoa!” or “uhhhh . . .”) to help guide one another’s actions. 190 The hat-check girl in Grim Fandango has a distinctive way of speaking (listen to Clip 7.12). ©1998 Lucasfilm Entertainment Company Ltd. All rights reserved. FIGURE 7.7 O921-Ch07.qxd 5/10/06 10:36 AM Page 190 7.3 DESIGN POINTERS Missed Opportunity: Real-Time Vocal Adaptation Currently, NPCs rarely offer real-time back-channel sounds and comments during game play. Revealing more complex awareness of a player as well as reactions to the player through emotion- and information-laden audio cues as play situations unfold could greatly increase the sense of social presence and connection a player feels toward an NPC. Imagine a sidekick or a just-rescued character gasping as the player executes a tricky move or making a subtle noise of doubt and hesitation as the player starts to move in a fruitless direction. This will become increasingly practical as voice synthesis becomes more and more realistic, eliminating the need for a huge body of prerecorded audio files (see Section 7.8, for more information about speech synthesis). 191 The cut-scenes in Final Fantasy X use vocal cues to heighten the player’s experience of the NPCs’ emotions and unfolding relationships ©2001 Square Enix Co.,Ltd.Character Design:Tetsuya Nomura. FIGURE 7.8 7.3 Design Pointers Here are some suggestions for heightening character impressions for players using vocal cues. 7.3.1 Focus on Relationships As with body language (Chapter 6), it is helpful in audio design to think about how voice cues will reveal relationships among characters—between NPCs as well as between the player-character and NPCs (see Figure 7.9). When craft- ing audio, think about the ongoing relationships between characters and their social roles in relation to one another, as well as the moment-to-moment game-play state of the player and how the character can respond to what the player is experiencing. O921-Ch07.qxd 5/10/06 10:36 AM Page 191 [...]... Four Characters in Action What Is Covered and Why Part IV asks you to step back from the details of people’s social equipment (faces, bodies, voices) to form an integrated understanding of how characters function as a game unfolds Chapters 8 and 9 introduce psychological concepts relevant to the two major functions that characters play in games: acting as the player’s vessel in the game world (player -characters) ,... discussion in Chapter 5) Player -characters also model the social role and emotions the player should be inhabiting in a game through their actions and reactions toward NPCs in cut-scenes And finally, player -characters offer the player a chance to don a social mask when interacting with other people in a multiplayer game or when manipulating several characters in a party-based game An engaging player-character... sight and sound, sometimes touch) For example, sports games usually give player -characters a much greater level of athletic ability—strength, speed, and reaction time—than the player likely has in everyday life Platform-based games often give player -characters much greater leaping capabilities than they would have in real life Text-based adventure games provide the player with options for witty repartee... thorny problems, or to create a foundation for making an important identity shift Games advocates such as Gerard Jones describe the ways in which games can provide this sort of outlet for exploration and processing (Jones 2003) Player -characters are an important component of the fantasy experience of a game Powerful player -characters are often those that speak to many players’ real-life hopes, fears,... controller constraints of the core game- play dynamics that the team has in mind Following are some stellar examples of well-integrated player characters in games from various genres—action, adventure, and simulation Two Well-Motivated Killers: Max Payne and Gordon Freeman Max Payne and Half-Life are both games that primarily involve shooting The player’s physical control in the game is limited (for the most... in the game world for forming relationships Halfway through the game, he learns that the Marines moving into the facility are actually not on his side, but want to silence him to keep the mess that has occurred in this facility a secret, and so he must begin to fight them as well In both games, the “shoot first, ask questions later” tactics of shooter games have been extremely well-motivated by the... see Sim characters conversing) Part of the dynamic of playing the game is making sure that Sims are getting along (or creating conflict between them) The Sims’ flexibility extends from social styles into the fantasy aspect of the game Sim characters also act as tarot cards for teasing out the fantasies of players, who create stories that arise from the visceral and cognitive activities in the game Players... experienced—that should be considered in an integrated way when designing player -characters Games that balance these levels well, such as Half-Life, Max Payne, Grim Fandango, and The Sims™, as well as games which emphasize particular layers with positive effects, are discussed The chapter includes design tips for tuning player -characters based upon this four-layer model and finishes with an interview with... functions that characters have in games and how these should affect design thinking 7.7 Exercise: Developing a Social “Ear” Each person should capture a brief segment from a movie or television show in which two characters are speaking Take turns listening to (not watching) these brief snippets of dialogue in a group and have everyone try to identify the relative social status and relationship of the characters. .. and 205 O921-Ch08.qxd 5/10/06 10:37 AM Page 206 CHAPTER EIGHT • PLAYER -CHARACTERS other variables, which are learned over time Players will be trying to make sense of the game world in part through the lens of the player-character’s social capabilities and context Player -characters act as an interesting bridge for the player into a game s social world First of all, the player-character (when viewed from . understanding of how characters function as a game unfolds. Chapters 8 and 9 introduce psychological concepts relevant to the two major functions that characters play in games: acting as the. games: acting as the player’s vessel in the game world (player -characters) , and providing social companionship and assistance or resistance (nonplayer -characters, or NPCs). Both chapters argue. to character design, working from game- play mechanics upward to develop social and emotional roles for characters that are truly supported in moment-to-moment game play. As you proceed through