Augmented Cognition Mitigation Strategies Next Generation Concepts

Augmented Cognition Mitigation Strategies Next Generation Concepts Summary Report _ October 14, 2005 Kelly S Hale, Sven Fuchs, Kay Stanney Design Interactive, Inc 897 Kensington Gardens Court Oviedo, Florida 32765 Joseph Juhnke Tanagram Partners 125 North Halsted, Suite 400 Chicago, IL 60661 Prepared For CDR Dylan Schmorrow Augmented Cognition Mitigation Strategies Next Generation Concepts Table Of Contents 1Introduction 2Current Mitigation Strategies 3Conceptual Framework for Mitigation Strategy Selection 18 4Innovative Mitigation Strategies 24 5Mitigation Strategies Summary .30 6Operational Example: Real-Time Mitigation Strategies for Tactical Action Officer 31 7Conclusions and Future Directions 35 8References 36 Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts Introduction DARPA’s Improving Warfighter Information Intake Under Stress (i.e., Augmented Cognition [AugCog]) program aims to extend the information management capacity of human-system dyads in operational environments through real-time system mitigation strategies driven by physiological sensors monitoring operator cognitive state A general premise of the current AugCog effort is that human information processing (HIP) capabilities are fundamentally the weak link in the symbiotic relationship between humans and computers Within human information processing there are several ‘bottlenecks’ or points of limited processing capacity, including sensory memory, working memory (WM), attention, and executive function It has been suggested that through mitigation strategies, the costs of HIP bottlenecks (e.g., degraded human performance due to overload, underload, stress) can be overcome As physiological sensors become more robust and capable of characterizing cognitive states beyond bottlenecks (e.g., situation awareness metrics), the focus of real-time mitigation may expand beyond alleviating overload to optimizing operator state (i.e., predictive mitigation to avoid overload) to measurably improve performance of the human-system dyad The design of real-time mitigation strategies must thus be carefully addressed To date, system mitigation has been applied in a brute force manner and thus tends to cause context switching (Baldonado, Woodruff & Kuchinsky, 2000), as well as loss in situational awareness (Boiney, 2005); there has generally been an observed cost associated with getting context back Thus, there appears to be a need to identify more effective mitigation strategies This effort focused on summarizing current mitigation strategies used in human-systems integration (HSI) as well as techniques utilized in other media (e.g., Fine Arts, Photography, Film, Theatre) that may be adapted to enhance human-system performance within dynamic, information-rich, stressful environments The goal was to identify theory-driven mitigation strategies from HSI, and leverage the Arts to achieve more effective and innovative, next-generation concepts for mitigation strategies, i.e., think outside the box that WIMPs (Windows, Icons, Menus, Pointing devices) have us bounded by Through the current effort, a framework was developed to aid real-time mitigation strategy selection In addition, the repertoire of mitigation strategies was expanded in terms of both the breadth of strategies available and the manner in which each is designed and implemented within varying operational settings When the project commenced and the scientific literature review was underway, a critical gap in the literature associated with how mitigation strategies are conceptualized was uncovered Thus in this effort we undertook filling this gap which involved devising a theoretical foundation from which to conceptualize mitigation strategies; specifically a conceptual framework based on Norman’s (1988) ”Seven Stages of Action” has been developed which relates adaptive mitigation activity to a model of the human action cycle of intent, execution and evaluation The importance of this conceptual model is that it structures and systematically organizes the space of mitigation strategies Thus the current effort focused more on building a theoretical foundation for mitigation strategies rather than the proposed empirical studies which will be pursued in the follow-on efforts Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts Current Mitigation Strategies The current effort focused on identifying mitigation strategies that may be applied to (1) reduce overload/bottlenecks as identified via psycho-physiological sensors or (2) enhance situation awareness (SA) Bottlenecks that have been the focus of AugCog research thus far include sensory, attention, working memory and executive function bottlenecks Future research is planned to develop psycho-physiological metrics for SA, as optimization of SA is critical within complex, information-rich environments such as military Command, Control, Communications, Computers, Intelligence, Surveillance and Recognizance (C4ISR) operations This section describes mitigation strategies that have been developed and are currently used in various interactive systems Each strategy described may be selectively implemented in real-time (following theoretical mitigation selection plans as outlined in this report) to either alleviate processing bottlenecks and/or enhance SA, thereby enhancing performance for an individual operator within a complex, information rich environment 2.1 Context-Sensitive Help By changing the physical layout, useful affordances of both a physical and cognitive nature can be brought closer to where users need them and at the time they need them – providing context-sensitive help Strengths The simple fact that physical and cognitive affordances are available at the right moment can help users notice possibilities they might otherwise overlook (Kirsh, 2000) Contextsensitive help systems take away the task of locating a desired information snippet inside today’s extensive help systems This is important, as we are apt to lose the thread of our composition (Kirsh was commenting on a word editing task) the deeper we have to go outside of our current environment of activity (Kirsh, 2000) Weaknesses Automated help systems, however, can be perceived as helpful and non-intrusive (e.g., Microsoft® Word’s auto-correction feature) or highly annoying (“Clippy,” the Microsoft® Office Assistant) Careful consideration of interruption strategies is therefore needed However, offering help may be generally inappropriate in high-load environments, as it causes distraction from the core task and may disturb the operator’s pace If external resources are available and can be accurately identified, it may be better to delegate the task Specific Execution Methods In today’s computer systems, context-sensitive help is implemented in many different ways, mostly in close distance to where it is needed The right-click context menu, which can be considered a quasi-standard for Windows® environments, adheres to the Gestalt principle of proximity and Fitts’ Law (describes the relationship between distance and size of a pointing target, Fitts, 1954), making the interface more efficient Context-aware environments (e.g Microsoft® Visual Basic®) automatically provide help for the Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts function at the current cursor position, e.g code reference for a highlighted keyword, or they provide a list of available options (e.g attributes in an HTML tag) Software agents can automatically provide assistance in the background They monitor and analyze user activity and trigger events if particular user action is detected 2.2 Cueing Cueing is a common way to compensate for weaknesses in information rich environments, where people receive data at such a rapid rate that it may not be assimilated (Sheridan and Ferrell, 1974) In such cases, users develop and apply prioritizing strategies However, these strategies are not always accurate Limited attentional resources also make users vulnerable to the attentional spotlight effect (Posner, Snyder & Davidson, 1980; Townsend, 1971) which decreases their event awareness (signal detection) capabilities outside of their attentional focus, and may leave a high priority event undetected (“miss”) Cues provided by the system notify users of events by directing/redirection attention Strengths Research has been conducted on the design of cues, e.g which modality provides the fastest response or what properties make a cue more distractive Information can be interpreted with minimal cognitive effort if presented in the appropriate modality (Wickens, 2002), and response time to signals differs among modalities, with auditory and haptic cues resulting in faster response times compared to visual cues (ETSI, 2002) The more intrusive a cue is, the more awareness it will create for the cued event, and the more effective it will be in shifting attention Arroyo, Selker and Stouffs (2002) found that the least used modalities in computer interfaces (i.e smell and vibration) have bigger disruptive effects, probably because of their novelty Weaknesses Arroyo, et al (2002) findings suggest that people’s backgrounds and worldview determine the effectiveness of various modalities for interruption Also, if a cue is given at an inopportune moment, slower task performance (e.g., Czerwinski, Curtell & Horvitz, 2000; McFarlane, 1999), more errors (Kreifeldt and McCarthy, 1981), and worse decisions (Speier, Valacich & Vessey, 1999) may be observed All of these factors may be critical to safety and mission success in C4ISR environments To mitigate the disruptive effects of interruption, researchers are investigating systems that reason about when (Horvitz and Apaciple, 2003; Horvitz, Jacobs & Hovel, 1999; Hudson, Fogarty, Atkenson, et al., 2003) and how to interrupt users McFarlane (2002) evaluated four strategies for coordinating interruption; immediate (with no respect to the user’s current task), negotiated (people have choices about whether to allow interruptions and how and when to handle them), mediated (an attempt to predict people’s interruptibility), and scheduled (provide a degree of reliable Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts expectation about when interruptions will occur) These approaches depend on some model of sensory attention when deciding how and when to interrupt According to Norman’s (1988) task-flow model, each task consists of a planning, evaluation and execution phase Interrupting these phases may be costly (Zijlstra, Roe, Leonova & Krediet, 1999; Monk, Boehm-Davis & Trafton, 2002) Since every subtask is a task itself, high-level tasks (i.e., containing subtasks) have a recursive structure that imposes more cognitive load as more levels of subtasks are present Therefore, task completion is desirable in order to release these cognitive resources If this cannot be accomplished, Trafton, Altmann, Brock, and Mintz (2003) suggest a rehearsal strategy to help resume a task after an interruption Zacks and colleagues conducted studies that provide insights into the ways in which tasks are decomposed hierarchically in the mind (Zacks, Braver, Sheridan, et.al., 2001; Zacks & Tversky, 2001; Zacks, Tversky and Iyer, 2001) These decompositions are linked to distinct patterns of brain activity Based on this research, Adamczyk and Bailey (2004) suggest a task model that reveals the best timing for an interruption They also suggest that, besides task status, an interruption manager should dynamically factor in notification relevance, i.e apply strategies to prioritize subtasks Specific Execution Methods Cueing techniques are in use for all modalities The appropriate modality for a cueing event may or may not be chosen to interfere with the sensory channels of the current task, based on priority assessment and task status The level of intrusion should relate to the priority of the cued information, however, as cues may interrupt users in their current task causing loss of SA and disorientation when resuming from processing the cue For example, in a computer application, an auditory signal does not intrude visual workspace but provides a cue that something happened in the background (see Bailey et al., 2000 for more on balancing information awareness and intrusion) Visual cues may rely on the visual pop-out effect which is implemented by a difference in color, shape, or intensity Intrusiveness of visual cues is also highly dependent on position: For high levels of intrusion (i.e cueing of high-priority events), cues should be placed within foveal vision, low priority events may be cued by changes in the peripheral or ambient environment Auditory cues vary by their nature They come as generic sounds of different complexity (with more complex sounds being more disruptive), earcons (i.e., real-world sounds that carry a metaphorical meaning), or speech, and can vary in volume or pitch In more advanced systems, audio output may also be spatialized in order to distinguish information channels or carry spatial information Haptic cueing is commonly implemented through vibration devices, such as those implemented in pointing devices (e.g., haptic mouse) and cell phones (e.g., meeting setting that gives vibration cue for incoming call) serving as the most common examples Haptic cues may vary in intensity and pattern, and can be used to provide spatial data (if more than one vibration source is present) Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts Sometimes, different modalities are combined to create a system of cues One common application is alarm systems that increase the alarm level step by step, starting with a visual cue (e.g a flashing light) If this cue is not responded to, the system increases its intrusiveness by adding auditory signals (bells and whistles) Advanced car alarm systems even address suspected intruders by speech before setting off a public alarm (e.g entreprix SWAT II, http://entreprix.com/swat2.html) 2.3 Decluttering (Reduce amount of info) Clutter is created when screen real estate or channel capacity is occupied by numerous unrelated entities, or if these entities are not well-arranged Decluttering refers to reducing the amount of information to be displayed Systems may follow design rules like the Gestalt Principles (i.e., the sum of the whole is greater than its parts; Koffka, 1935) that, correctly applied, can declutter displays and these can facilitate the user’s interpretation of interface state by increasing interface consistency Some examples are: - The Gestalt Law of Proximity: Elements that are closer together will be perceived as a coherent object The Gestalt Law of Similarity: Elements that look similar will be perceived as part of the same form The Gestalt Law of Good Continuation: Humans tend to continue contours whenever the elements of the pattern establish an implied direction The Gestalt Law of Closure: Humans tend to enclose a space by completing a contour and ignoring gaps in the figure A second approach towards decluttering is level-of-detail manipulation, where layers of content may be hidden to simplify the current display to highlight critical information Strategies include adaptive menus (where options that are rarely used by the operator are hidden), zoom (where users are able to zoom in on specific context area), fisheye views (where a very wide-angle lens that shows an area of interest quite large and with detail with the remainder of the graph/image successively smaller and in less detail; Sakar & Brown, 1992) and preview techniques (where an incomplete, yet representative subset of an environment’s content is presented) Strengths Interface consistency is helpful to avoid clutter, as it increases learnability and reduces errors (Nielsen, 1989) Consistency should support human perception and cognitive processes such as visual scanning, learning, and remembering (Mahjan & Shneiderman, 1997) Once an interface is learned, cognitive load – necessary to locate and identify elements – is decreased Consistent color schemes and positions also facilitate orientation and help to reduce clutter However, as operators deal with increasing amounts of data, unified appearance caused by consistency efforts, may inhibit the richness of information (Gentner & Nielsen, 1996) and therefore reduce the number of orientation cues Level-of-detail manipulations can be implemented to take away non-relevant information or advanced features that may not be utilized on a regular basis These strategies can be Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts used to simplify displays for novice operators, where advanced features would distract from learning basic system features Preview techniques reduce efforts of assessment and selection of data Schweiger (2001) used link comments in hypertext structures (mousesensitive, popped up next to the mouse cursor) that contained a summary of one or two sentences about the content of the linked page Similar previews enhanced knowledge acquisition and supported intentional and incidental learning (Cress and Knabel, 2003) by improving the link selection and reducing serendipity effects Users with link comments opened significant fewer pages than those without the added information Link comments also reduced the frequency of backward navigation Weaknesses Zooming incorporates a trade-off between detail level and awareness of the global picture One attempt to solve the problem is a second-view approach, where the global view is presented in a second display, highlighting the zoom area (e.g Adobe Photoshop) But this solution is still non-optimal, as it requires additional screen real-estate and forces the user to switch between views and mentally integrate the information of both (Sakar & Brown, 1992) Non-linear magnification implemented with ‘fisheye’ lenses is used to achieve a balance between expansion and compression of the data, depending on the user’s focus point (Gutwin & Skopik, 2003) This non-linearity, however, may cause distortion in proportions of objects it is applied to which may lead to decreased speed and an increased need for accuracy in pointing tasks (Gutwin, 2002) When implementing preview techniques, careful decisions are needed about how to reduce the data, as taking away context also eliminates cues that may be helpful to the user Specific Execution Methods Today’s interfaces use various approaches of visual level-of-detail manipulation Adaptive menus are used in Microsoft® Office, where options that are rarely needed by the user are hidden In graphic software, layers of content can be shown and hidden as appropriate to not obscure the elements that are currently worked on Another classic approach to level-of-detail manipulation is zooming – where “zooming in” provides local detail while “zooming out” allows for a high-level overview This technique is common in electronic maps, such as MapQuest™ (www.mapquest.com) A fisheye view (Furnas, 1986) has been used to balance scope and detail of information presentation Gutwin & Skopik (2003) compared a set of magnification techniques and stated that, although the fisheye lens performed best overall, some tasks may be better off with a ‘flat lens’ (maintains relative sizes and positions in a certain area; e.g on maps, where distance and directions have a meaning) or a panning view (performed best on pointing tasks in high magnification/high accuracy environments; e.g., point-and-click tasks) Preview techniques such as those implemented in Mini Player mode in Apple®’s iTunes® aim at reducing detail by limiting the amounts of data to be processed by presenting an incomplete, yet representative subset of an environment’s content Only upon request a detailed view or extended functionality is displayed Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts Currently, level-of-detail manipulations are not implemented for auditory and haptic modalities Although one could imagine systems that increase or decrease context based on situation (tone/earcon vs spoken word vs spoken sentence), to date most auditory content is statically implemented The same is true for haptics, except for advanced gaming or force feedback devices 2.4 Delegation Delegation involves offloading some aspect of current tasking to another operator Mixed Initiative refers to tasks that are shared between an operator and an intelligent system, where the system takes complete control over certain aspects of a task The term initiative implies an intelligent system that can actively adapt its status to environmental factors This technique has been used extensively to decrease operator overload by dynamically appointing tasks to other, less taxed, users or systems Strengths Delegation utilizes the capabilities of operators that are better suited for this particular task or parts thereof and can be used when misinterpretation inhibits task performance The real strength of the delegation is its dynamic workflow optimization schema When an operator is overloaded, delegation to system or another operator may reduce bottlenecks by reducing the amount of tasking a single operator must complete Weaknesses Delegation should be applied with caution, particularly if the global goal is to increase situation awareness By delegating tasking to outside resources, the current operator reduces his/her awareness of offloaded tasks This may result in decreased awareness of current state In addition, automated delegation is difficult as the delegating agent must not only accurately understand the workload of the primary (the person the task was originally sent to) and potential delegates, it must also know which of its potential delegates is best equipped to address this particular situation with the skill sets and situational awareness required In addition, delegation eliminates the operator’s need for learning If a task is taken away from an operator as soon as he/she does not perform optimally, he/she will never have the opportunity to improve individual performance on the given task Specific Execution Methods Manual delegation is a common practice in the human work place and would seem an intuitive addition for an information flow system One example is the auto-pilot system in aviation that is turned on by the operator when no events are expected which would require human involvement However, these systems are mostly passive: The operator initiates the system to take control of certain aspects of the task but is still in charge, as the system only maintains the current state – sometimes capable of limited adaptive actions supported by sensory technology The user is still required to monitor and make adjustments as needed Cruise control is another example for delegating parts of a task to a system Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts Mixed Initiative is a specific form of delegation, where intelligent systems are assigned responsibility for a task or task component, are already common in current computing environments, car navigation serving as an example: The system detects the current position of the vehicle and adjusts its output accordingly, offloading the task of wayfinding from the driver Also, some car manufacturers have rain sensors which control the wipers and their speed as needed, thus taking away this responsibility from the driver completely 2.5 Modality Augmentation Modality Augmentation can occur in two forms: modality redundancy and modality substitution/switching Modality redundancy may provide the same information in multiple modalities (e.g., flashing light and auditory beep as a warning), and/or may provide complimentary information (i.e., additional information; e.g., flashing light with auditory speech message stating the problem status) in a second modality Modality substitution/switching replaces one sensory modality with another (e.g., change flashing light to auditory beep as warning signal) Strengths According to Wicken’s (2002) Multiple Resource Theory (MRT), separate sensory and cognitive resources are available to process information from different modalities Therefore, simultaneous processing of competing tasks can be supported by strategically allocating data streams to various multimodal sensory systems If resources are depleted for one modality, there may still be cognitive capacity left if the information is presented through other sensory channels Weaknesses Caution is advised when it comes to modality substitution/switching, as researchers increasingly report evidence that knowledge is grounded in modality-specific systems (Barsalou, Simmons, Barbey & Wilson, 2003), and there is no direct empirical evidence for amodal symbols Thus, the presentation modality is proposed to influence encoding which may cause association problems when attempting to display the same information in different modalities (as modality switching does) if previous knowledge or interpretation is required for the task For example, if operators are presented with a visual indication of status, and later presented with an auditory indication of status for the same system, interpreting this second cue as status information may require more processing load due to switched modality In addition, Pecher, Zeelenberg and Barsalou (2003) report a study supporting the hypothesis that perceptual simulation underlies conceptual processing and that switching from one modality to another during perceptual processing incurs a processing cost, implying effects on operator performance Also, Arnell and Larson (2002) showed that, under particular circumstances, multimodal stimuli presentation is subject to the attentional blink effect, where the second of two successive stimuli is likely to be missed when falling into a timeframe of about 500ms after the first Design Interactive, Inc 10 Augmented Cognition Mitigation Strategies Next Generation Concepts objective of decreasing operator overload The challenge to designers is to identify innovative ways to mitigation system display in real-time that efficiently optimizes operator state and measurably improves performance Table lists innovative approaches and techniques for relevant mitigation strategies that may be applied within a complex C4ISR environment to alleviate bottlenecks and enhance HSI within complex, information rich environments Innovative strategies for pacing and delegation were not found, and thus these strategies are not included in table Table 2: Innovative Approaches/Techniques for Real-Time Mitigation of Bottlenecks Mitigation Plan Optimize Individual Processing Capacity Mitigation Strategy Modality Augmentation Innovative Approaches/Techniques Aural-Spatial Representation Ambient Sound Transposition Design Interactive, Inc High-Key/LowKey Lighting Based on echo and reverberations, people can ‘hear’ how big a room is This way of encoding spatial data into auditory information may open new implementation strategies for multimodality Background sounds added in postproduction phase of a movie are perceived as incidental but in fact function to enhance the drama Embedding information into the natural environment so that it is not noticed as such (e.g one notices the weather and acts appropriately without actively attending to it) may allow delivery of information without distraction Lighting style where the scene is evenly lit, suggests a familiar world containing few surprises or mysteries, whereas strongly contrasted areas of light and shadow create a sense of mystery This principle may be transformed into dynamic color schemes that contain information about system status A scheme of harmonic colors might indicate a stable operating environment, whereas strongly contrasted colors may indicate that special attention is needed 25 Augmented Cognition Mitigation Strategies Next Generation Concepts Table (Con’t) Mitigation Plan Direct Attention Mitigation Strategy Cueing Innovative Approaches/Techniques Environmental Cues Induction of Physical Response GalvanicVestibular Stimulation (GVS) Tilt Shot Screen Flash Music Style/Tempo Design Interactive, Inc People place cues in their environment (e.g., sticky notes) to increases affordance of information or to serve as a reminder for tasks that will be pursued at a later time This concept of letting the user “park” information where it is needed allows the operator to temporarily decrease cognitive load To date, haptic cues are usually transmitted by vibration devices However, lowfrequency sound produces a tactile sensation that may be used for cueing in certain environments Adding more dimensions to a modality may reduce the bottleneck and increase information throughput This technique simulates the sensation of movement by applying electromagnetic signals directly to the inner ear GVS may open entirely new ways of cueing by exploiting a new channel A tilted camera suggests a reaction to a scene or object, usually involving strangeness, imbalance, tension, or the unexpected E.g., the camera can have an unusual angle after an accident, or shake during an earthquake Shaking or tilting visual content may be used as ambient cue without distorting or manipulating the content A technique implemented in ego-shooter games to indicate hits, where the screen turns red for an instant, but fades to normal rapidly so no detail is lost Since this cue appears in the foveal focus, it does not require eye gazing or attention shifting and is more likely to be noticed than a peripheral cue Music is often used to create atmosphere in a film Similar to lighting, background music may be used as an ambient cueing tool E.g., systems have been developed that indicate stock activities to brokers by changes in music style or tempo Furthermore, music has been found to affect emotional state which may be utilized to affect stress levels or other parameters of operators 26 Augmented Cognition Mitigation Strategies Next Generation Concepts Table (Con’t) Mitigation Plan Direct Attention Mitigation Strategy Cueing Innovative Approaches/Techniques Environmental Sound Shift in constant tone draws attention Experts can qualify sounds or changes of sound, e.g mechanics may hear problems with the engine and even be able to diagnose based on these sounds, or computer experts may listen to the frequency of the cooling fan to detect a system problem Sequencing Transitions In cinematography, a Cut signifies a shorter time lapse or merely a scene change The use of Out-/Inpoints (e.g ending one shot and starting the next shot with the same element) makes the cut less abrupt A Fade can be used to suggest a passage of time, a journey, or a new location A Dissolve is used to suggest a special relationship between the scenes that dissolve into one another (a relationship closer than suggested by a fade) Transition techniques may be adapted for sequencing to alleviate the effects of task switching Decreased ambient light focuses attention, e.g theaters use lighting to focus attention and notify (cue) upcoming events Sequencing may be implemented by fading out irrelevant content, therefore focusing the user’s attention to the current task If two people share a common level of knowledge, this reduces the amount of information needed to convey a message The same applies to human-computer interaction, allowing the use of codes, abbreviations, shortcuts, etc., thus reducing the amount of information to be processed and therefore alleviating bottlenecks Looking at macro events (pattern recognition) may reduce the need to attend to details, thus reducing the amount of information to be processed and therefore alleviating bottlenecks Ambient Lighting Optimize Communication Requirements Decluttering Learned Efficiency Pattern Monitoring Design Interactive, Inc 27 Augmented Cognition Mitigation Strategies Next Generation Concepts 4.2 Innovative Strategies to Enhance Situation Awareness (SA) The framework for mitigation strategy selection suggests that all current mitigation strategies can be implemented to enhance SA with the exception of delegation This strategy involves offloading tasking to another operator/system, thereby limiting access to all available information and potentially limiting SA The remaining mitigation strategies identified across the five mitigation plans (Figure 2) can be effective in meeting the objective of decreasing operator overload The challenge to designers is to identify innovative ways to mitigation system display in real-time that efficiently optimizes operator SA and measurably improves performance Table lists innovative approaches and techniques for each identified mitigation strategy that may be applied within a complex C4ISR environment to enhance SA within complex, information rich environments Innovative strategies for pacing and task sharing were not found, and thus these strategies are not included in Table Table 3: Innovative Approaches/Techniques for Real-Time Mitigation to Enhance SA Mitigation Plan Optimize Individual Processing Capacity Direct Attention Mitigation Strategy Modality Augmentation Innovative Approaches/Techniques Transposition Cueing See Table Control of Information Stream Validation Feedback Cutaway Sequencing Optimize Communication Requirements See Table Decluttering Establishing Shot ContextSensitive Help Error Correction Change-inState Indicator Transmission Failure Reconciliation Design Interactive, Inc People use fill words to maintain attention during an interruption of the information stream (“well”, “uhm”, …) When communicating, people expect cues for transmission success (“uh huh”) A shot briefly interrupting one action to provide a glimpse of another also taking place, and then returning to the first action Can provide additional knowledge without permanently losing focus on the primary task A long shot giving an overview of a scene so the audience is not confused about what is happening and where Overviews/different perspectives may enhance Situation Awareness Even if signal is improperly transmitted, people can often “fill the gaps” with assumptions based on context If the system can identify these signal gaps it could provide resolution options to the user Animators use “onion-skin” technique to track changes from previous frames Display that/how an object has been manipulated If a communication signal is not responded to as expected, the sender will verify proper transmission 28 Augmented Cognition Mitigation Strategies Next Generation Concepts Table (Con’t) Mitigation Plan Optimize Communication Rate Mitigation Strategy Pacing Innovative Approaches/Techniques Cross Cutting Slow/Fast Editing Design Interactive, Inc Rapid sequence of shots between two different locations, used to create tension The sequence builds to a climax and ends with two things coming together Applied to pacing, cross cutting can be used to create arousal if fatigue or underload is detected Utilize cutting speed (i.e shot length) to generate excitement and anticipation (e.g chase sequence) with short shots, or appear calming and relaxing (e.g love scenes) with long shots Relevance for Mitigation: Could be applied to influence the user’s emotional state through sequencing and pacing 29 Augmented Cognition Mitigation Strategies Next Generation Concepts Mitigation Strategies Summary Through review of current mitigation strategies, a conceptual framework outlining how mitigation plans and associated strategies may be implemented in an ordered fashion was developed as a first step in creating a real-time human-system closed loop system where physiological metrics can drive innovative, seamless, adaptable system displays Building from the framework created in this report which outlines how mitigation strategies should be implemented to achieve one of two global goals (alleviate bottlenecks, enhance SA), current and innovative techniques have been described that may be considered based on system and operator state and constraints (i.e., information being presented at any given time, operator load across various channels) Our review of the Arts and other media resulted in various innovative approaches/techniques that may be applied to C4ISR environments to measurably improve operator performance These approaches were categorized into current mitigation strategies from Figure 3, and may be implemented as alternative mitigation techniques To implement the conceptual mitigation selection framework presented here in real-time, each specific execution method must be assigned given IF-THEN constraints to allow automatic evaluation of effectiveness under specific conditions These constraints must consider all factors that may influence applicability of the specific strategy which may include system state constraints (e.g., what information is currently being presented across the system and how), environment constraints (e.g., environmental noise), user state constraints (e.g., current processing load), and global user constraints (e.g., individual abilities/preferences) The next section provides an example highlighting how the mitigation selection framework would be implemented in an operationally relevant context Design Interactive, Inc 30 Augmented Cognition Mitigation Strategies Next Generation Concepts Operational Example: Real-Time Mitigation Strategies for Tactical Action Officer During high intensity operations, the Tactical Flag Command Center (TFCC) provides intelligence, communications and modifications to the operational plan for the Carrier Strike Group (CSG) The Battle Watch Captain (BWC) is present in the TFCC during high tempo; high alert times and is the center of the shared situational awareness In times when the BWC is not present in the TFCC, the Force Tactical Action Officer (FTAO) assumes the responsibilities of the BWC; in fact, often the FTAO is the BWC The BWC/FTAO is one of the primary roles within the TFCC and has direct authority from the (CSG) Commander in the decision making process (Strausser, Kollmorgen & Juhnke, 2005) Thus, the FTAO’s tasks involve monitoring the ship’s area of responsibility for potential threats, and deciding upon an appropriate course of action should a potential threat be identified In a complex, stressful environment such as this, physiological measures applied to an FTAO may detect operator processing bottlenecks Building from the framework developed in this report, a processing bottleneck would trigger the mitigation selection framework (Figure 2) Figure demonstrates how this framework may be applied in an operational context by specifying specific IF-THEN conditions that lead to implementation of specific mitigation strategies 6.1 Operational Scenario This scenario illustrates a “typical” day in the service of Flag Tactical Action Officer (FTAO), Phil Stevenson References have been placed throughout the story to illustrate relevance to the IF-THEN mitigation selection model (Figure 4) While this scenario is focused on bottleneck resolution, there are also examples of SA improvement mitigations at work “Another day, another tussle.” Those words were becoming Phil Stevenson’s mantra lately It seemed like this conflict was never going to end Phil and his crew were ranked among the highest performing teams in the fleet; a feat Phil knew had much to with his augie rig Phil’s crew had won the “lottery” and been outfitted with some new, just being tested, gear about six months ago He smirks when he recalls the joke he and his crew made about the hassle it was going to be That’s all water under the bridge; he’s a convert now His watch begins like any other day He reviews the last watch summary on his personal viewing system while he sips his morning coffee A process that used to take several hours is compressed into a matter of 15 minutes using the multimodal presentation that is paced to optimize his current state of retention Displaying the last watch’s progress through task based mission plan by crosscutting information and key decision points from the last watch gives Phil a high level view of occurrences and a complete understanding of mission objectives he has yet to achieve He even chuckles when he recognizes one of Randy’s signature moves Randy’s crew is right behind Phil’s on metrics, but it looks like Phil has to a little catch-up work for him again Design Interactive, Inc 31 Augmented Cognition Mitigation Strategies Next Generation Concepts 1a) Modality Augmentation KEY Visual resources are not overloaded IMPROVED SA Augment high priority visual communications with additional BOTTLENECK RESOLUTION visual information If auditory saturation > Substitute auditory communication to visual If auditory saturation > Substitute auditory status representations to visual If auditory saturation > Substitute auditory cues to visual If haptic saturation > Substitute haptic communications to visual If haptic saturation > Substitute haptic status representations to visual If haptic saturation > Substitute haptic cues to visual Auditory resources are not overloaded Augment high priority visual communications with additional auditory information If visual saturation > Substitute visual communication to auditory If visual saturation > Substitute visual status representations to auditory If visual saturation > Substitute visual cues to auditory If visual saturation > Substitute haptic communication to auditory If haptic saturation > Substitute haptic status representations to auditory If haptic saturation > Substitute haptic cues to auditory Haptic resources are not overloaded Augment high priority visual communications with additional haptic information If visual saturation > Substitute visual status representations to haptic If visual saturation > Substitute visual cues to haptic If auditory saturation > Substitute auditory status representations to haptic If auditory saturation > Substitute auditory cues to haptic 1b) Transposition Verbal resources are not overloaded Augment verbal resources with additional verbal detail If spatial saturation > Substitute spatial communication to verbal If spatial saturation > Substitute spatial status representations to verbal If spatial saturation > Substitute spatial cues to verbal Spatial resources are not overloaded Augment spatial resources with additional spatial detail If verbal saturation > Substitute verbal communication to spatial If verbal saturation > Substitute verbal status representations to spatial If verbal saturation > Substitute verbal cues to spatial 2a) Cueing Critical communications are not being acknowledged If no response to critical status > Cue status in same mode If no response to critical status > Cue status in on different least intrusive unsaturated mode If no response to critical status > Cue status in on different next least intrusive unsaturated mode Repeat until acknowledgment 2b) Intelligent Sequencing Executive function is overloaded Present communications in order of priority (Defer lower priority communications) 3a) Declutter Executive function is overloaded Reduce information density to minimal amount required to maintain decision-making (all modes) 4a) Pacing Executive function is overloaded Reduce the rate all communications are presented 5a) Mixed Initiative Executive function is overloaded Automate tasks to assist decision-making Design Interactive, Inc 32 Augmented Cognition Mitigation Strategies Next Generation Concepts 5b) Delegate Executive function is overloaded Redirect communications to available (not overloaded) resources Figure 4: Operationally Relevant Mitigation Strategy Selection Process “Let’s heat ‘em up team!” Phil announces over group comm Projection screens are already displaying summaries of team activities and status [1a] The task for today is to secure a sector that is heavily trafficked by commercial vessels There’s a good chance that some of the commercial vessels are terrorist platforms hiding amongst the civilians in their typical cowardly manner Dynamically assigned their Areas of Interest (AOI)[1a] Phil’s track team begins scanning, and validating traffic The 5-man team is strong except for Rudy; he’s still pretty green and isn’t adjusting to the predictive load models as well as the others [2b] Phil notes that Randy’s AOI has already been reduced because of overload [3a] He stores the view of Randy’s reduced environment into his memory log for later consideration Ironically it’s Rudy who finds the first “pheasant,” a carrier class freighter that has a questionable ownership history “Immediately the rest of the team responds to Rudy’s find That’s when things get interesting Sam, one of Phil’s brightest exclaims, “Oh dear, those are dots I didn’t want to connect!” over group comm A radio transmission comes in from the Nemesis but Phil doesn’t hear it because his attention is focused rightly on Sam’s concerns “If she’s concerned, I’m concerned,” he thinks He notes the radio transmission has been added to his text log for later review [2b] It must not have been important The computer cues [2a] Phil that it is transferring of the lower priority radio channels to text logs [1a], converting a radio based status report to a graphic representation on his dashboard [1b], and redistributing the high priority channels in different places “in his head.” [1a] Phil really loves spatialized radio comms They are so much easier to separate Before he can speak, the computer has already requested further details regarding Sam’s concern [5a] Her analysis of traffic patterns, port summaries, and ship logs has shown that of the maybe 200 ships in the AOI have been traveling the same route over the past 20 days Traffic logs show their actual progress to be normal except for the same 40-mile diversion in the middle of ocean and then back on course There were no records of any platforms or vessels in that location, so it would be easy to consider the diversion a course deviation to avoid an obstacle, except they all stopped at the same location for hours and then continued back to their planned course [3a] Phil notes the room has dimmed quite a bit; group cog-load must be high [3a] A spot over Rudy flashes and catches Phil’s attention [2a]“Rudy, what’s your status?” “I need auth for a boarding party sir” Rudy responds The computer whispers in Phil’s head that there is a 10% probability of concern for Rudy’s track “Rudy, let’s defer for a few minutes I think we’re going to need our resources.” Phil points at the big screen Confirmed “civie” vessels have been reduced to mere dots on the screen while the ships Sam has flagged have their routes, and PIMs plotted [3a] “We need to run these numbers people” Phil announces The computer has already offloaded two of the vessels to James for analysis Design Interactive, Inc 33 Augmented Cognition Mitigation Strategies Next Generation Concepts and is summarizing determinations on the big screen [5a][5b] It becomes evident that these four ships, even though they are on different timetables will be converging on the fleet at the same time The computer whispers statistics in Phil’s head and it becomes evident he needs to react now They are an hour away from an attack Phil orders the weapons teams on alert and cues the rest of the crews on watch of his findings Communications jump exponentially The entire fleet is hot now Comms are coming in so fast Phil has to rely heavily on the computer’s automated prioritizing and response tools [5a] At one point a commander sends Phil an updated mission plan and the computer generously offers to allow Phil to dictate his responses and queries During a few points of very intense communication Phil notices the world seems to slow down He knows it’s the computer pacing his comms.[4a] to help him concentrate He looks forward to the cue that he is back on real-time [3a] Time-late is not Phil’s favorite place Sometime during the conflict the computer assumed weapon allocation and assignment and Lloyd a “heckovaguy” stationed in Ohio was brought online to assume portions of Phil’s load [5b] Lloyd’s performance rating was exemplary Phil knew he had the best help he could get Eventually the conflict resolved, no lives were lost and all four of the enemy vessels were captured It was no surprise that their cargo included an assortment of “dirty” torpedoes A mess had definitely been averted today Design Interactive, Inc 34 Augmented Cognition Mitigation Strategies Next Generation Concepts Conclusions and Future Directions This paper reviewed mitigation strategies that are currently implemented to enhance operator performance within traditional human computer interaction systems Each of these strategies may be implemented within a complex, information rich environment such as military C4ISR environments to measurably improve a number of objectives, including alleviating human processing bottlenecks and enhancing situation awareness Although this review uncovered a number of mitigation strategies that are available at present, there was no conceptual model to aid designers in determining which mitigation strategy is best under a given set of conditions To address this void, a conceptual mitigation selection framework was developed that outlines five mitigation selection plans (optimize information processing capacity, direct attention, optimize communication requirements, optimize communication rate, increase external resources) that may be considered in order to address various global mitigation goals, including alleviating operator processing bottlenecks and enhancing operator situation awareness In addition, media, communication, and the Arts were embraced in order to identify innovative ways of mitigation that have yet been undiscovered Although no innovative strategies were found, numerous techniques that may considerably enhance the way mitigation strategies are implemented were extracted and added to the mitigation selection framework An example of how this conceptual framework may be applied in an operational setting was provided In follow-on efforts, empirical studies will be conducted to validate the mitigation selection framework within an operational environment The conceptual model and associated mitigation strategies (current and innovative approaches to enhancing system design) outlined in this report provide a starting point for realizing the goal of real-time mitigation within complex C4ISR environments by offering a structured approach for strategy selection Other considerations that need to be addressed in future work include the development of exit policies for each mitigation strategy (i.e., when is mitigation no longer required, e.g when to return from down-paced presentation to regular speed), how to coordinate the mitigation cycles for different objectives (i.e reduce overload and enhance situation awareness), and how to apply mitigation framework to team environments In addition, the conceptual model outlined here may be expanded to mitigate various human state conditions beyond processing bottlenecks and situation awareness If such conditions can be captured in real-time via psycho-physiological sensors, there may be an opportunity to enhance operator performance, e.g maximize focus and increase motivation Design Interactive, Inc 35 Augmented Cognition Mitigation Strategies Next Generation Concepts References Adamczyk, P, Bailey B.P (2004) If not now, when?: the effects of interruption at different moments within task execution, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 271 – 278 Arnell, K.M., Larson, J.M (2002) Cross-modality attentional blinks without preparatory task-set switching Psychonomic Bulletin & Review, 9(3), 497-506 Arroyo, E., Selker, T and Stouffs, A (2002) Interruptions as multimodal outputs: Which are the less disruptive? ICMI'02, 479-483 Bailey, B.P., Konstan, J.A and Carlis, J.V (2000) Adjusting Windows - Balancing Information Awareness with Intrusion Proceedings of the 6th Conference on Human Factors and the Web Baldonado, M.Q.W., Woodruff, A., & Kuchinsky, A (2000) Guidelines for using multiple views in information visualization In Proceedings of AVI2000, Palermo, Italy (pp 110-119) Retrieved, July 2005, from http://www2.parc.com/istl/projects/uir/pubs/items/UIR-2000-05Baldonado-AVI2000-MultipleViews.pdf Barsalou, L.W., Simmons W.K., Barbey, A.K., Wilson, C.D (2003) Grounding conceptual knowledge in modality-specific systems TRENDS in Cognitive Sciences, 7(2), 84-91 Bell, L., Boye, J, Gustafson, J and Wirén, M Modality Convergence in a Multimodal Dialogue System Proceedings of Götalog 2000, Fourth Workshop on the Semantics and Pragmatics of Dialogue, 29-34 Berka, C., Levendowski, D.J., Ramsey, C.K., Davis, G., Lumicao, M.N., Stanney, K., Reeves, L Harkness Regli, S., Tremoulet, P.D., Stibler, K (2005) Evaluation of an EEG-Workload model in an Aegis simulation: Biomonitoring for physiological and cognitive performance during Military Operations Proceedings of the International Society for Optical Engineering 5797, 90 - 99 Berz, W L (1995) Working memory in music: A theoretical model Music Perception, (12), 353364 Billinghurst, M, Kato, H., & Poupyrev, I (2001) The Magic Book: Moving seamlessly between reality and virtuality IEEE Computer Graphics and Applications, 21(3), 2-4 Boiney, L (2005) Team decision making in time-sensitive environments Presented at the 10th International Command and Control Research and Technology Symposium: The Future of C2 Retrieved, July 12, 2005, from http://www.dodccrp.org/events/2005/10th/CD/papers/175.pdfRetrieved, July 12, 2005, from http://www.dodccrp.org/events/2005/10th/CD/papers/175.pdf Casini, L & Macar, F (1999) Multiple Approaches to investigate the Existence of an Internal Clock Using Attentional Resources, Behavioral Processes 45, 73-85 Cress, U & Knabel, O.B (2003) Previews in hypertexts: effects on navigation and knowledge acquisition Journal of Computer Assisted Learning, 19, 517-527 Czerwinski, M., E., Cutrell & Horvitz, E (2000) Instant Messaging and Interruption: Influence of Task Type on Performance Proceedings of OZCHI, 356-361 Donmez, B., Boyle, L., Lee, J.D (2003) Taxonomy of Mitigation Strategies for Driver Distraction Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting, 1865-1869 Dorneich, M., Whitlow, S., Ververs, P.M., Mathan, S., Raj, A., Muth, E., Hoover, A., DuRousseau, D., Parra, L., & Sajda, P (2004) The role of peripheral feedback in emotional experience with music Music Perception, 22(1), 79-115) DARPA improving warfighter information intake under stress – Augmented Cognition: concept validation experiment Design Interactive, Inc 36 Augmented Cognition Mitigation Strategies Next Generation Concepts (CVE) analysis report for the Honeywell team Prepared under contract: DAAD16-03-C0054 ETSI [European Telecommunications Standards Institute] (2002) Human factors (HF); Guidelines on the multimodality of icons, symbols, and pictograms (Report No ETSI EG 202 048 v 1.1.1 (2002-08) Sophia Antipolis, France: ETSI Furnas, G (1986) Generalized Fisheye Views ACM SIGCHI Bulletin, 17(4), 16-23 Gentner D and Nielsen, J (1996) The Anti-Mac Interface Communications of the ACM, 30(8), 70-82 Gutwin C and Skopik, A (2003) Fisheye Views are Good for Large Steeering Tasks Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 201-208 Gutwin, C (2002) Improving Focus Targeting in Interactive Fisheye Views, Proceedings of CHI 2002, 267-274 Hildebrandt, M.A., Dix, A.J., Meyer, H.A (2004) Time Design CHI '04 extended abstracts on Human Factors in Computing Systems, 1737-1738 Horvitz, E & Apacible, J (2003) Learning and Reasoning About Interruption Proceedings of the Fifth ACM International Conference on Multimodal Interfaces, 20-27 Horvitz, E., Jacobs, A & Hovel, D (1999) Attention-Sensitive Alerting Proc Uncertainty in Artificial Intelligence, 305-313 Howley, K (2004) Breaking, Making, and Killing Time in Pulp Fiction, Scope – an online journal of film studies May 2004 Retrieved July 13, 2005, from http://www.nottingham.ac.uk/film/journal/articles/making-breaking-and-killing.htm Hudson, S.E., Fogarty, J., Atkeson, C.G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J.C., & Yang, J (2003) Predicting Human Interruptibility with Sensors: A Wizard of Oz Feasibility Study Proceedings of the SIGCHI conference on Human factors in computing systems , 257264 Kirsh, D (1995) The Intelligent Use of Space Artificial Intelligence, 73, 31-68 Kirsh, D (2000) A Few Thoughts on Cognitive Overload Intellectica, 30, 19-51 Koffka, K (1935) Principles of Gestalt Psychology Lund Humphries: London Retrieved October 13, 2005, from http://www.marxists.org/reference/subject/philosophy/works/ge/koffka.htm Kreifeldt, J.G & McCarthy, M.E (1981) Interruption as a Test of the User-Computer Interface Proceedings of the 17th Annual Conference on Manual Control, Jet Propulsion Laboratory, California Institute of Technology, JPL Publication 81-95, 655-667 Langley, P (1999) User modeling in adaptive interfaces In Proceedings of the Seventh International Conference on User Modeling, (p 357-370) Banff, Alberta: Springer Retrieved, July 12, 2005, from http://www.cs.utah.edu/classes/cs5350/handouts/adapt.um99.pdf Lyons, K., Gandy, M., & Starner, T (2000) Guided by Voices: An Audio Augmented Reality System Proceedings of International Conference on Auditory Display (ICAD) 2000 April 2000 Mahjan, R & Shneiderman, B (1997) Visual and Text Consistency Checking Tools for Graphical User Interfaces IEEE Transactions on Software Engineering, 23(11), 722-735 Mamykina, L., Mynatt, E.,Terry, M.A (2001) Time Aura: interfaces for pacing, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, 144-151 Mancini, C & Buckingham, S.S (2001) Cognitive coherence relations and hypertext: from cinematic patterns to scholarly discourse Proceedings of the twelfth ACM conference on Hypertext and Hypermedia, 165-174 McFarlane, D.C (1999) Coordinating the Interruption of People in Human-Computer Interaction INTERACT, 295-303 McFarlane, D C (2002) Comparison of four primary methods for coordinating the interruption of people in human-computer interaction, Human-Computer Interaction, 17(1), 63-139 Design Interactive, Inc 37 Augmented Cognition Mitigation Strategies Next Generation Concepts Meyer, H.A & Hildebrandt, M (2002) Towards Time Design: Pacing of hypertext navigation by system response times Ext Abstracts CHI2002, ACM Press, 824-825 Miyata, Y & Norman, D (1986) Psychological issues in support of multiple activities In: Norman, D.A & Draper, S.W (Eds.), User Centered Systems Design: New Perspectives on Human-Computer Interaction Hillsdale: Lawrence Erlbaum Associates, 265-284 Monk, C A., Boehm-Davis, D A & Trafton, J G (2002) The attentional costs of interrupting task performance at various stages Proceedings of the Human Factors and Ergonomics Society 2002, 1824-1828 Nielsen, J (1989) Coordinating user interfaces for consistency ACM SIGCHI Bulletin, 20(3), 63-65 Norman, D.A (1988) The Psychology of Everyday Things Basic Books, New York Pecher, D., Zeelenberg, R., Barsalou, L.W (2003) Verifying different-modality properties for concepts produces switching costs Psychological Science, 14(2), 119 Posner, M I., Snyder, C R R & Davidson, B J (1980) Attention and the detection of signals Journal of Experimental Psychology: General, (109), 160-174 Predebon J (1999) Time Judgments as a Function of Clock Duration: Effects of temporal paradigm and an attention-demanding nontemporal task, Perceptual and Motor Skills, 88, 1251-1254 Sakar, M & Brown M.H (1992) Graphical Fisheye Views of Graphs Proceedings of ACM on Human Factors in Computing Systems, 83-91 Schmorrow, D., Stanney, K.M., Wilson, G., & Young, P (2005) Augmented cognition in humansystem interaction In G Salvendy (Ed.), Handbook of human factors and ergonomics (3rd edition) New York: John Wiley Schweiger, W (2001) Hypermedien im Internet Verlag Reinhard Fischer, München Sheridan T and Ferrell W (1974) Man-Machine Systems: Information, Control, and Decision Models of Human Performance MIT Press, Cambridge, Massachusetts, USA Shneiderman, B (1984) Response time and display rate in human performance with computers Computing Surveys, 16, 3, 265-285 Speier, C., Valacich, J.S & Vessey I (1999) The Influence of Task Interruption on Individual Decision Making: An Information Overload Perspective Decision Sciences, 30(2), 337-360 St John, M and Osga, G (1999) Supervision of Concurrent Tasks Using a Dynamic Task Status Display Proceedings of the 43rd Human Factors & Ergonomics Society Annual Meeting, 168172 Starner, T., Mann, S., Rhodes, B., & Levine, J (1997) Augmented Reality Through Wearable Computing M.I.T Media Laboratory Perceptual Computing Section Technical Report No 397 Strausser, J., Kollmorgen, G., and Juhnke, J (2005) DARPA Small Business Innovation Research (SBIR) Program Phase I Report for the Period July 14, 2004 to January 14, 2005 Contract ID W31P4Q-04-C-R253 Sulzen, J (2001) Modality based working memory School of Education, Stanford University Retrieved, February 5, 2003 from http://ldt.stanford.edu/~jsulzen/jamessulzen-portfolio/classes/PSY205/modality-project/paper/modality-expt-paper.PDF Szalavári, Z., Eckstein, E., Gervautz, M (1998) Collaborative gaming in augmented reality Proceedings of the ACM symposium on Virtual reality software and technology, 195-204 Teal S.L., Rudnicky, A.I (1992) A performance model of system delay and user strategy selection Proceedings of the SIGCHI conference on Human factors in computing systems, 295-305 Townsend, J.T (1971) A note on the identifiability of parallel and serial processes Perception and Psychophysics, 10, 161-163 Design Interactive, Inc 38 Augmented Cognition Mitigation Strategies Next Generation Concepts Trafton, J G., Altmann, E M., Brock, D P and Mintz, F E (2003).Preparing to resume an interrupted task: Effects of prospective goal encoding and retrospective rehearsal, International Journal of Human-ComputerStudies, 58, 583-603 Wickens, C D (1984) Processing resources in attention In R Parasurraman & R Davies (eds.), Varieties of Attention (pp 63-101) New York: Academic Press Wickens, C D (1992) Engineering Psychology and Human Performance (2nd Edition) New York, NY: Harper Collins Wickens, C.D (2002) Multiple resources and performance prediction Theoretical Issues in Ergonomics Science, 3(2), 159-177 Zacks, J., & Tversky, B (2001) Event structure in perception and cognition Psychological Bulletin, 127(1), 3-21 Zacks, J.M., Braver, T.S., Sheridan, M.A., Donaldson, D.I., Snyder, A.Z., Ollinger, J.M., Buckner, R.L & Raichle, M.E (2001) Human brain activity time-locked to perceptual event boundaries Nature Neuroscience 4, 651 – 655 Zacks, J., Tversky, B., & Iyer, G (2001) Perceiving, remembering, and communicating structure in events, Journal of Experimental Psychology: General, 130(1), 29-58 Zakay, D & Block, R.A (1997) Temporal Cognition, Current Directions in Psychological Science, 6,(1), 12-16 Zijlstra, F R H., Roe R A., Leonova, A B & Krediet, I (1999) Temporal factors in mental work: Effects of interrupted activities, Journal of Occupational and Organizational Psychology, 72, 163-185 Design Interactive, Inc 39 ... Design Interactive, Inc Augmented Cognition Mitigation Strategies Next Generation Concepts Current Mitigation Strategies The current effort focused on identifying mitigation strategies that may be.. .Augmented Cognition Mitigation Strategies Next Generation Concepts Table Of Contents 1Introduction 2Current Mitigation Strategies 3Conceptual Framework for Mitigation. .. text [written directions]) 17 Augmented Cognition Mitigation Strategies Next Generation Concepts Conceptual Framework for Mitigation Strategy Selection Mitigation strategies to date have been