This page intentionally left blank Development of Professional Expertise Professionals such as medical doctors, airplane pilots, lawyers, and technical specialists find that some of their peers have reached high levels of achievement that are difficult to measure objectively In order to understand to what extent it is possible to learn from these expert performers for the purpose of helping others improve their performance, we first need to reproduce and measure this performance This book is designed to provide the first comprehensive overview of research on the acquisition and training of professional performance as measured by objective methods rather than by subjective ratings by supervisors In this collection of articles, the world’s foremost experts discuss methods for assessing the expert’s knowledge and review how we measure professional performance and design-training environments that permit beginning and experienced professionals to develop and maintain their high levels of performance, using examples from a wide range of professional domains K Anders Ericsson, PhD, is presently Conradi Eminent Scholar and Professor of Psychology at Florida State University For the last 30 years he has studied the development of expert performance in domains such as music, chess, medicine, business, and sports and how expert performers attain their superior performance by acquiring complex cognitive mechanisms and physiological adaptations through extended deliberate practice He has edited several books on expertise, including Toward a General Theory of Expertise (1991), The Road to Excellence: The Acquisition of Expert Performance in the Arts and Sciences, Sports, and Games (1996), and the influential Cambridge Handbook of Expertise and Expert Performance (2006) His research has been recently featured in The New York Times, Scientific American, Fortune magazine, New Scientist, and Time magazine He is a Fellow of the American Psychological Association, the Association of Psychological Science, and the Center for Advanced Study in the Behavioral Sciences Development of Professional Expertise Toward Measurement of Expert Performance and Design of Optimal Learning Environments Edited by K Anders Ericsson Florida State University CAMBRIDGE UNIVERSITY PRESS Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Dubai, Tokyo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521518468 © K Anders Ericsson 2009 This publication is in copyright Subject to statutory exception and to the provision of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press First published in print format 2009 ISBN-13 978-0-511-59493-9 eBook (EBL) ISBN-13 978-0-521-51846-8 Hardback ISBN-13 978-0-521-74008-1 Paperback Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate Information regarding prices, travel timetables, and other factual information given in this work are correct at the time of first printing, but Cambridge University Press does not guarantee the accuracy of such information thereafter Contents List of Figures page viii List of Tables xi List of Contributors xiii The Measurement and Development of Professional Performance: An Introduction to the Topic and a Background to the Design and Origin of This Book K Anders Ericsson, Ray S Perez, David W Eccles, Laura Lang, Eva L Baker, John D Bransford, Kurt VanLehn, and Paul Ward Section 1: Challenges in Past and Contemporary Efforts to Measure and Train the Objective Performance of Professionals The 20th-Century Revolution in Military Training Ralph E Chatham Developing Professional Expertise with a Cognitive Apprenticeship Model: Examples from Avionics and Medicine Susanne P Lajoie Leadership Development and Assessment: Describing and Rethinking the State of the Art Michael D Mumford, Tamara L Friedrich, Jay J Caughron, and Alison L Antes Revolutions, Leaders, and Diagnosticians: Reflections on the Themes in Chapters 2–4 Earl B Hunt v 27 61 84 108 vi Contents Section 2: Past and Contemporary Efforts to Design Instruction, Train, and Maintain Professional Performance Research on Past and Current Training in Professional Domains: The Emerging Need for a Paradigm Shift Jeroen J G van Merriënboer and Eddy W Boot 131 Designing Training for Professionals Based on Subject Matter Experts and Cognitive Task Analysis Jan Maarten Schraagen 157 How to Help Professionals Maintain and Improve Their Knowledge and Skills: Triangulating Best Practices in Medicine Dave A Davis Advances in Specifying What Is to Be Learned: Reflections on the Themes in Chapters 6–8 Richard E Mayer 180 203 Section 3: The Assessment and Training of Skilled and Expert Performers in the Military 10 Toward a Second Training Revolution: Promise and Pitfalls of Digital Experiential Learning Ralph E Chatham 11 Evaluating Pilot Performance Brian T Schreiber, Winston Bennett, Jr., Charles M Colegrove, Antoinette M Portrey, David A Greschke, and Herbert H Bell 12 Contrasting Submarine Specialty Training: Sonar and Fire Control Susan S Kirschenbaum, Shelley L McInnis, and Kevin P Correll 13 Training Complex Cognitive Skills: A Theme-Based Approach to the Development of Battlefield Skills Scott B Shadrick and James W Lussier 14 Structuring the Conditions of Training to Achieve Elite Performance: Reflections on Elite Training Programs and Related Themes in Chapters 10–13 Robert A Bjork 215 247 271 286 312 Contents vii Section 4: The Development of Expertise and Expert Performance 15 The Influence of Learning Research on the Design and Use of Assessment Eva L Baker 333 16 Acquiring Conceptual Expertise from Modeling: The Case of Elementary Physics Kurt VanLehn and Brett van de Sande 356 17 Teaching for Expertise: Problem-Based Methods in Medicine and Other Professional Domains Henny P A Boshuizen 379 18 Enhancing the Development of Professional Performance: Implications from the Study of Deliberate Practice K Anders Ericsson 405 19 It Takes Expertise to Make Expertise: Some Thoughts About Why and How and Reflections on the Themes in Chapters 15–18 John D Bransford and Daniel L Schwartz 432 20 The Value of Expertise and Expert Performance: A Review of Evidence from the Military J D Fletcher 449 21 Expertise in the Management of People: A New Frontier for Research on Expert Performance Susan E F Chipman 470 Name Index 495 Subject Index 512 figures 2.1 A notional learning curve 2.2 Training at the knee of the curve 38 2.3 Skills atrophy 40 2.4 Two levels of a training curve family 44 2.5 A hierarchy of learning curves 45 2.6 Pilot survivability in air-to-air combat 48 2.7 The influence upon combat proficiency of training by “simulated engagements” 50 Readiness of the Navy in relation to its anticipated deployments 56 3.1 The Sherlock interface 68 3.2 The BioWorld interface 76 4.1 Model of leader cognition 91 5.1 The power law relation between memory strength to occasions of learning (both in arbitrary units) 114 The power law relation between memory strength and the number of time periods between cessation of training and testing 115 The ratio of memory strengths during the retention period 116 8.1 Triangulating the competence of professionals 192 8.2 A × table outlining effect of CME interventions to current and proposed scenarios in CME delivery 196 10.1 Three parts of the Tactical Language Trainers 223 10.2 Student learning basic Iraqi language and culture 223 2.8 5.2 5.3 page 36 viii 538 Subject Index PC (personal computer) exercises, 240 gaming simulations, 220 interface, 241 simulations, 237–240, 242 training, 218, 220, 221, 241 training on, 326 virtual reality training, 238 pedagogical agents, intelligent, 143 peer nomination, identifying professionals, pejorative terms, absent in Myers-Briggs, 486 people, assessing opinions and beliefs of, 452 people engines, 243 perceptual skills, required for STs, 322 performance acceptable level of, acquisition of experts’ superior reproducible, 18 acquisition of skilled, 426 apparatus, assumed to be stable, 442 arenas, relying on well-specified, 442 assessing improvement in, 452 assessment, 117 government’s official system for, 487 attaining an acceptable standard of, 417 carefully monitoring, 294 choosing conditions enhancing trainees’, 315 conditions knowledge of, 441–442 letting students experience, 441 simulating during practice sessions, 421 stable and variable, 442 continued improvements in, 425 criteria needed for excellent, 350 declining without practice, 41 decreases as a result of decreased flying hours, 37 distinguishing learning from, 313–314 domain-related activities correlated with increases in, enhancing different aspects of, failure, identifying sources of, 158 feedback by peers, 190 after unit training exercises, 464 feedback on to physicians, 195 focusing on measurable, gains as the goal for PC training, 226 versus learning gains, 327 at the highest possible level, 325 identifying aspects to improve, 209, 415 improvement associated with restricted types of experience, 411 critical variable for, 413 measuring during DMO, 253 without learning, 314 improving mechanisms mediating, 417 keys to higher levels of, 45 learning without improvement in, 313 making inferences about future, 342 measurement, 251 measuring and quantifying for DMO exercises, 253 measuring during training, 323 model, of professional development, 181 objectives, for an ISD model, 150 outcomes, analyzing, 343 of Patriot air defense system operators, 453 of PBL graduates, 399 practice, 421 premature automation of, 418 versus process in assessing expertise, 450 reaching an acceptable level of, 412, 417 stable plateau of, 417 stretching beyond current capabilities, 419 superior levels, 426 during a test versus actual on the job, tracking system, 263 during training as an unreliable index of learning, 313, 314 in war, assigning reasons for, 46 at work, 78 work environment, 193 Performance Improvement Modules, 191 per-seat model, of development, 228 personal, normative knowledge, 387, 388 personal computer See PC (personal computer) personality characteristics immutability of, 484 of leaders, 100 personality information, using in managing people, 487 personality measurement, 486 personality tests, for pilot candidates, 265 personality types given by Myers-Briggs results, 484 of leaders, 99 personnel, continuing turnover of military, 29 Personnel Management required course, at Harvard Business School, 482 personnel turbulence, in the military, 55 persons, attributing success or failure to, 111 Peter the Great, 110 physical landscape, in the mindscape, 244 physicians Subject Index accuracy of self-assessments as poor, 13 assessing own weaknesses and level of professional performance, 413 assumption on self-assessment, 188 experience not associated with continued improvements, 413 keeping themselves up to date, 180 need for enforced monitoring of performance, 191 recognized need to undertake CME activities, 190 reminding about specific actions, 194 tests to low-performing, 186 Physician’s Recognition Award, 188, 191 physics, 16 physics experts classifying problems, 344 misconceptions held by, 365 physics knowledge, 360 physics novices, classifying problems, 360 physics problems solvers, developing, 358 physics professors, on introductory physics problems, 408 physiological adaptation, 425 physiological measures, in cockpit design, 265 physiological systems, strained by ongoing activity, 425 pianistic techniques, 416 pianists, acquiring playing techniques, 414 pilots allowing experienced to improve performance, 11 BLUEFOR becoming squadron training officers, 31 certification of, 384 cost to produce military, 266 evaluating performance, 247 Navy performing six times better after Top Gun, 461 selecting, 264, 265 with simulator practice as more successful, 425 survivability of, 48 training against simulated enemy, 424 plan(s) formulating for exercising influence, 100 providing frameworks for exercise of influence, 100 planning different aspects of, 95 permitting formation of a range of intentions, 92 skills, 95 planning confluences, in experts’ schemas, 364 539 plateaus, avoiding, 433 Plato, 406 platoons, performance correlated to AFQT scores, 454 players, as coachable, 433 playful engagement, compared to deliberate practice, 417 “Please understand me” booklet, 487 plunking method, 365 political issues, in carrying out a task analysis, 176 polyrhythms, 415 portability, of learning objects, 140 Positional Play, 290 Positions Assets/Weapons Systems A-10 MEC skill, 259 positive feedback systems as inherently unstable, 110 by Tissu expert, 434 post-training contexts, performance in, 315 power functions, as improvement over notional curves, 113 power law of learning, 113 POWERSOURCE, initiative in pre-algebra, 348 p-prims, 371 practice barriers to adherence of guidelines, 184 dedicated and focused, making permanent, 432 massing on a given subtask, 315 necessary for reaching highest levels of performance, 411 practice activities, 17, 406, 425 practice gap, 183 practice lessons, supported by theoretical lessons, 169 practice of medicine, variation in, 186 practice performance assessment, for specialty boards, 188 practice performance measures, of each physician, 193 practice-related activities, in addition to paid time, 422 predictions, from experts, 408 pre-novices, in elementary physics, 357 preparation for future learning, assessing, 443 prerequisite knowledge, for an ISD model, 150 prescriptive mental models, 94, 98 preventive guidelines, 193 principals, feedback from, 440 principle schemas versus algebraic chaining (pre-novices), 366 defined, 362 as more general, 372 540 Subject Index principles., See also specific principles confluences per, 372 deep understanding of, 336 differences between experts and novices in use of, 350 Pring, Command Sergeant Major Ricky, 30 problem(s) confronting complex, ill-defined, 86 dealing with infrequent and urgent, 425 describing basic approach or plan for solving, 363 experts generating plans for solving, 364 identifying for practice sessions, 421 picking similar, 364 sequence of for Sherlock trainees, 70 sharpening skills in the context of solving, 67 sorting into clusters of similar, 363 students handling representative, 421 turning into opportunities, 465 problem behavior graph, 69 problem construction, in a problem solving scenario, 95 problem description, goals for establishing, 171 problem description step, in troubleshooting, 167 problem schema, 372 problem solvers, 472 problem solving difference between experts and weaker problem solvers, 393 as a family of achievement, 344 impossible to isolate distinct processes of, 426 knowledge constructed by the trainee in Sherlock, 70 as recursive, 65 problem solving strategies, strategies, forwards versus backwards, 366 problem spaces, 69, 75 problematic student, with relevant technical expertise, 480 problem-based curricula, medical schools with, 421 problem-based learning (PBL) See PBL (problem-based learning) problem-solving skills, creative and evaluative, 86 procedural information, 150, 151 procedural knowledge, 75, 204, 205, 387 procedural requirements, 347 procedural skills, 289 procedures, 204, 339–340, 352 process measures, applying to expert performance, processes eliciting in a problem scenario, 95 of individuals, 340 used by leaders, 90 processing activities, with impact on leader performance, 99 processing operations, of the model of leader thought, 101 productions, in cognitive science, 116 professional(s) acting as lifelong learners, 132 keeping themselves up to date, 180 keeping up-to-date, 180 leadership of, 84 methods for training skilled, 14 not receiving constant feedback or pressure, 422 practicing in an organizational setting, 84 relying on knowledge developed in other domains, 426 renewed descriptions and prescriptions of the role of, 384 steps on the road to the status of, 180 task and role of, 383 triangulating the competence of, 192 professional academic education, 381–385 professional achievement, large individual differences in, 405 professional careers, length of, professional competence addressing two domains, 62 assessing, 62–64 developing, 65 professional development, 406–408 professional discussion, in an Army AAR, 464 professional domains complexity of, 79 difficult and challenging situations occuring infrequently, 424 extending cognitive apprenticeship approach to, 78 measuring levels of attained performance objectively, 407 professional education, 17, 381 professional experience, 3, 413 professional mistakes, 383 professional performance acquisition and development of superior, capturing and measuring, research on objective measurement of, professional practice, bridging traditional learning to, 421 professional schools, success in, 407 professional settings, personnel involved in, professional singers, 418 Subject Index professional success, factors leading to, 407 professional training changing goals of, 131–133 consequences of the changing goals of, 133 dealing with unexpected situations, 133 paradigm shift needed in, 149–152 traditional compartmentalization of, 133 transformation of, 132 professional trust, abuse of, 191 professions dealing with mistakes and errors, 384 differing in inherent risks and dangers, 383 toward a new best practice model for, 192–196 professors characterizing effective, 477 grading severely, 112 inflating the egos of students at HBS, 489 proficiency enhanced by flying, 457 examining within a specific context, 62 flying hours as most important determinant of, 459 maintaining continuously at a high level, 217 training, 56 proficiency gains, DARWARS projects and, 245 proficiency levels, for medical training, 424 proficiency of personnel, as a variable of battle capability, 35 progress, of an individual, 340 progress test, at Maastricht University, 391 Project A, of the JPM study, propensities, capturing, 347 Protocol analysis: Verbal reports as data, 162 prototypic case exemplars, 90 prototypical cases, 92, 99 Provides Mutual Support A-10 MEC skill, 259 proximity effect, 218, 235 psychological fidelity, versus engineering fidelity, 321 psychological test development, 486 psychological tests, leaders completing a battery of standard, 88 psychological traits, not sufficient for predicting expertise, 432 The psychology of human-computer interaction, 162 psychometric analyses of achievement, 338 psychometric tests, psychomotor ability, 455 public education, development of, 407 purposes, high stakes, 341 QOS (Quality of Service), 252 qualification process, guided by a qual card, 280 541 qualitative accounts of learning, 358 qualitative confluences, 363 qualitative ideas, in physics, 441 qualitative physics problems, 359 qualitative problems See conceptual problems qualitative reasoning (QR), 365, 374 qualitative relationships, 356 qualitative understanding, 356, 357 qualitative version, of an equation, 373 quality, of training, 58 quality and validity studies, assessment goals for, 335 quality chasm See clinical care Quality of Service (QOS), 252 quality outcomes, 352 quantitative elementary physics problem, 358 quantitative evaluation, of Sherlock, 71 quantitative exercises, 374 quantitative measurements, of improvements in performance, 320 quantitative problems assessments showing, 363 emphasizing feedback from, 437 novice behavior similar to expert behavior, 366 solving, 358, 366–367, 370, 373 quantitative relationship, 356 quasi-experimental manipulation, 175 radar, mental models of, 164 radio, educational use of, 137 radiologists, 422, 423 radiology, 422 Ramification stage, of modeling, 362 rank, performance improving as a function of, 302 rapid prototyping, 134 rapidly changing conditions, 442 rate of advancement, 87 raters, evaluating, 255 rating system, 256 ratio, of North Vietnamese to US aircraft lost, 460 raw data issues, 262 raw network data, 261 RBL (Resource-Based Learning), 139 readiness, components of, 35 realism introducing into training, 44 required at highest levels of unit training, 58 real-life situations, instructional problems in, 157 reasoning, distinct processes of, 426 re-certification, at regular intervals, 180 recognition-primed decision making, 475 Red Flag exercises, of the Air Force, 31, 50 542 Subject Index Red or Opposing Force (REDFOR or OPFOR), 30 See also opposing forces (OPFOR) reduced teacher control, 397 refresher training, 39 regulatory body, acceptance by, 180 regulatory objectives, of CME credits, 188 relationship-oriented behaviors, 87 relative motion, 278 relearning design of, 319 enhancing potency, 319 forgetting as a necessary condition for, 318 importance of, 318 versus learning, 321 role of forgetting in, 317 relicensure in New Mexico, 188 at regular intervals, 180 remarks, connections among, 476 reminding, research on, 473 repeated practice with feedback, 466 required for medical simulation, 423 repeated-study-condition, 318 repetitions, spacing or interleaving versus massing or blocking, 320 reporting techniques, 338 representations, for monitoring performance, 209 representative situations, 410 representative tasks, 410, 421, 423 Republican Guard 18th Brigade, 462, 463 Republican Guard commander, training of, 462 research base, on expertise in the management of people, 470 research questions, outside an area of expertise, 167 researchers, distant and loose management of, 483 Resource-Based Learning (RBL), 139 response strength, 313 results, attributing to a person, 111 retention, effect of overtraining on, 115 retention interval, increasing, 318 retention period, for PBL students, 397 re-testing physicians, 191 retrieval strength, of memory representation, 313 reviews, similar to AARs in civilian life, 110 Revolution in Training initiative, of the US Navy, 15 rigging, of characters, 243 right answer, framed by an expert, 350 risk reduction, tendency toward more, 384 Rohm, Captain Bill, 52 routine expertise, 285, 289, 443 routine experts, 63 routine performance, 417 Royal College of Physicians and Surgeons of Canada, 384 Royal Navy courts-martial, limitations on, 110 Royal Netherlands Navy, Weapon Engineering Service, 165 Rule (or Law) of Least Effort, 425 rules, defining domains, 335 SABA, 139 safe environment, of computer-based simulations, 146 safety technologies, development of aviation, 246 sales performance, relationship to deliberate practice, 424 sandstorm, opening 73 Easting, 462 Savage Entertainment, 231 Saxborg, Dr Bror, 444 SBIR (Small Business Innovation Research) topic, 474 scaffolding, 65, 135, 150 scalability of MECs, 258 of the NTC, 217 of training, 218 scanning activities, 90, 92 scanning interventions, in training programs, 99 scanning skills, viable assessments of, 96 scenarios creating targeted, 237 for crisis response training, 306 ordinary users changing, 231 schemas developing, 16 knowledge driving stages of modeling, 362 schematic knowledge, 89 schematized knowledge, 443 Schmidt, Richard, 324 SCHOLAR, 142, 143 school curricula, as a source of teaching expertise, 434 school learning, transferring to professional environments, 65 school system, assuming stability of abilities, 339 school teachers, classroom management behavior of, 488 schoolhouses, costs of military, 29 schools, comparison with courses, 395 science(s) dynamics and development of, 382 international level success in, 412 science educators, indices of mastery, 366 Subject Index scientific knowledge, new not replacing old knowledge, 383 scoring, used to judge secondary school performance, 350 scoring criteria, in work with complicated tasks, 350 sea-going proficiencies, exercising, 44 SECF (Submarine Electronics Computer Field) course, 281 Second Armored Cavalry Regiment, 462, 466 second training revolution, 219–225 secretaries, differences in quality of, 483 Secrets of Chess Training, 290 segmentation, of dynamic visualization, 145 selection of an individual, 340 of students affecting PBL studies, 395 selection tests, predicting training completion, self esteem, culture of, 112 self-assessment abilities layering of, 195 triangulated model focusing on, 193 self-assessments, 449 along with independent measures of competence, 185 defined, 184, 208 distinct comparisons of clinicians’, 185 impact on 73 Easting, 464–465 by medical doctors, 13 potential for, 184–185 problems with in residents and physicians, 185 self-directed learning, 449 developing a design for, 451 implementing, 151 maintaining and improving professional performance, 14 non-measurable qualities of, 187 self-assessment as the cornerstone of, 184 solid skills in required for PBL, 397 self-directed learning activities, 181 self-reporting, 187 self-selection, of students, 395 semantic equations learning, 370 mastering first, 373 students acquiring, 370 semantic understanding, 357, 374 semantic version, of an equation, 373 Senior Executive Service (SES), 484, 485 sensemaking, linked to crisis resolution, 93 sensitivity training experience, 483 sensor operators, rapidly acquiring complex skills, 455 543 sequestered problems solving (SPS) assessments, 443, 444 services, supporting people engines, 243 SES position, 485 73 Easting, Battle of, 459, 462–463 Seward, William, 124 sex differences, in the rankings of children, 488 sex discrimination, during job hunts, 482 shared situational awareness, for new military tasks, 207 Sherlock (computer program), 11 assessment(s), 71 characteristics of, 118 computer simulation of the job environment, 67 critique of chapter about, 118 design of, 61, 66 effectiveness of, 71 evaluations of, 61 as an example of cognitive apprenticeship, 66 extending success to new domains, 74 interface of, 68 reasons for success of, 72 reducing training by years, not days, 78 resulting in trainees becoming more expert-like, 72 SHERLOCK class programs, adoption of, 119 SHERLOCK class tutors, 118 Shipboard, 283 shipboard assignment, new sailor’s first, 280 Shipman, Harold, 191 shooting, fixed distance, 347 short cases tests, 400 short-range vision, in real life, 241 short-term performance, 458 shots (in racket sports), trajectory of, signal processing technology, 274 sim-munitions, shooting, 238 simple cases, distinguishing from complex skills, 16 simulated engagements, influence upon combat proficiency of training, 50 simulated environment, reasons for using, 148 simulated task environments, 147 simulated work environment, 72 simulation(s) See also engagement simulation; flight simulation expertise and professional performance through, 460 grand-scale high-fidelity, 219 lighter-weight running on common PC hardware, 220 maximizing performance during training, 320 types of, 146 simulation engine, adopting an existing, 229 544 Subject Index simulation exercises, in assessment centers, 88 simulation strategy, 88 simulation studies, 423, 424 simulation training, MEC-based syllabi, 264 simulator training, early studies on, 254 simulators across different geographical locations, 262 anti-armor gunnery, 454 improvements in unit performance in, 46 low-tech compared to high-tech, 321 measuring performance in, 423 objective performance measures from, 246 replicating the characteristics of specific aircraft, 245 of various medical procedures, 193 singers, concentration and effort during lessons, 418 single simulators, 250 situation awareness, 451 situational judgment strategy, 88 situational judgment tests, 88 situational tests, of leadership, 12 situations, mathematical models of, 374 skill(s) acquisition, 406–408, 412 atrophy, 39–41 decay of competence in, 40 mastery, 388 measures, 400 obsolescence, 388 required for tasks, 158 requiring knowledge to be carried out, 170 sharpening in the context of procedural tasks, 67 short-term honing of, 457 training, 41 Skill Builder, in Tactical Iraqi, 222 skill objects building, 281 validated specification of, 322 skill performances, 253 skill related tests, 398 Skill-Builder screen shot, 223 skilled performances, 294, 441 SkillNet catalog, 283 slot fillers, 368 slots, of a schema, 362 SMEs (Subject Matter Experts), 253 analyzing rating data, 255 employed by the DMO organization, 255 falling victim to limitations of subjective evaluations, 255 identifying difficult and significant problems, 75 in the military, 351 observing warfighters throughout training sessions, 253 structured interviews with, 260 SOBT (Submarine On Board Training), 283 soccer players, elite-level youth, 420 social component, to the learning process, 433 social congruence, 389 social connections, among monkeys, 489 social constraints, on the task analyst, 158 social contexts, experts operating in, 126 social dart activities, 419 social dominance, 489 social implications, of actions, 98 social influence, 85 social networks of expertise, 443 social setting, organizations and leaders in, 86 social skills, of leaders, 86 social system, leaders working in, 98 social-constructivist design, 134–136 social-educational context, medicine within, 196 socially constructed expertise, development of, 406 socio-emotional aspects, of a case, 92 socio-emotional issues, 98 soft technology, ISD characterized as, 133 software code, exercising all portions of, 260 software developments, 140 software engineering approach, 160 methods, 160 work in, 391 software engineers, 160 software usability, 279 soldiers, authoring scenarios, 228 solitary activities, as deliberate practice, solitary chess study, as best predictor of chess skill, 419 solitary practice by musicians, time for as a function of age, 414 sonar as a diagnostic tool in medicine, 272 as a perceptual skill, 271 as a submarine’s primary sensor, 272 Sonar Basic Operations Trainer (SBOT), 283 sonar detection, effect of IMAT training, 456 sonar operators, 14, 40 sonar signal, impact of the environment on, 283 sonar system, operating, 283 Sonar Technicians (STs), 271 learning factual information and recognizing sounds, 282 task description, 273–274 training of, 321 using auditory (and visual) cues, 273 Subject Index sonar waterfall display, 274 SOPs (standing operating procedures), 305 Soviet chess training manuals, 290 spacing, study sessions, 314 spacing effect, 318 Spanish Armada, 109 spatial ability, 455 spatial transformations, complicated by bearing references, 277 special cases, applicability conditions for recognizing, 362 Special Force soldiers, directing B52 strikes, 132 special forces, as well trained in many countries, 54 specification desired characteristics of, 342 shortfall, 342 speed, pushing beyond one’s reliable, 421 split attention principle, 145 sponsoring organization, assessing benefits to, 452 sports, international level success in, 412 sports teams, correlating success with abilities of individuals, 454 SPS (feedback free) assessments, 443, 444 SSBNs (ballistic missle subs), 42 SSNs (attack subs), 42 ST See Structured Troubleshooting (ST) stable constructs, conflict with learning, 338–339 stable normal distribution, of psychological characteristics, 338 standardized measures, for adaptive training, 263 standards of expert performance, 441–445 facilitating reuse of learning objects, 140 for pilot selection, 265 Standards for Educational and Psychological Testing, 336 standing operating procedures (SOPs), 305 start-up costs, for PC-based training, 241 state accountability systems, in the U.S., 341 state medical boards, CME participation mandated by, 188 static visualizations, 144 statistical meta-analysis, 396 steel, electricity, and heavy engineering (1895–1940), age of, 158 STEM disciplines, 445 storage strength, 313 strategic knowledge, 204, 205 Strategic Operations, 238, 239 strategic or metacognitive knowledge, 167 strategies, 204, 207 strength of memory, 114 545 Strike University, 38 structural models, 150 structured domain knowledge, 167 Structured Troubleshooting (ST) group results attained by, 173 modifying existing courses based on principles, 173–174 as a one-week add-on course, 168–173 STs See Sonar Technicians (STs) student competence model, in Sherlock, 69 student evaluations, 389, 390 student learning, technology not improving, 210 student models, in a cognitive apprenticeship, 64 student or learner model, in SCHOLAR, 143 student performance, increasing, 300 students analyzing authentic problems, 385 applying new knowledge, 136 as coachable, 433 in a cognitive apprenticeship, 64 discussion in a class of 90, 480 of effective professors, 477 evaluating effects of outstanding teachers on, 480 experiencing specific tasks, 217 focusing on, 477 identifying the most similar, 478 information literacy of, 139 learning better using PBL, 395 learning with Acuitus’ tutor, 235 teachers’ mental models of, 478–479 student-tutor interaction, 234 study of war, as study of history, 46 subject matter applying research knowledge about cognition to, 345 coordinating with free flowing group discussions, 480 presented by CBT, 142 Subject Matter Experts (SMEs) See SMEs (Subject Matter Experts) subject matter relations, 350 subjective assessment, limitations of, 255–257 subjective criteria, of expertise issues, 422 subjective evaluations, 320 subjective metrics, in DMO training environments, 256 subjective perceptions, of leader behaviors, 87 subjective rating techniques, behaviorallyanchored, 260 subjective ratings, by superiors or subordinates, 86 46 Subject Index submarine(s) as a dangerous environment, 273 major classes by primary mission, 42 non-static situation outside, 271 training in specialist positions on, 14 Submarine Electronics Computer Field (SECF) course, 281 submarine engagements, during World War II, 49 Submarine Learning Center, 281 Submarine On Board Training (SOBT) system, 283 submarine warfare, necessity of sea time, 44 submarine world described, 272–278 development of expertise in, 284 subordinates, subjective ratings by, 86 superficial understanding, 357, 369, 373 superior leaders, 85 superior performance capturing reproducible under standardized conditions, 408–412 demonstrating, 466 domain-specific experience necessary for attaining, 411 exhibiting on tasks, 410 focusing on objectively measurable, focusing on reproducible, 408 not explained by traditional study of experts, 408 only attained after extensive experience, 411 recent research on reproducible, superiors, subjective ratings by, 86 superior-subordinate relationship, problems in, 483 supervisors, increases in knowledge and perceived expertise by, 85 supervisory ratings, correlating poorly with hands-on work-sample tests, supportive information designing for an ISD model, 150 learners studying, 151 surgeons helping to improve during first procedures, 424 using advanced technical tools, 423 surgical simulators, 423 survivability, of pilots, 48 survival of the fittest, versus learning, 49 symbolic forms, 371 synchronous communication, in education, 139 system failures, intertwined with personal, 111 system monitoring, as a form of comparative assessment, 341 systematic approach, to troubleshooting, 167 Systematic Approaches to Problem solving (SAPs), 150 systematic changes, to practice methods and training environments, 426 systematic versus apprenticeship-based dimensions, of medical education development, 380 Tactical Action Coordinators (TACCO), 459 tactical experts, 292 Tactical Iraqi Arcade Games, 223 Tactical Iraqi Language Training system, 222 Tactical Iraqi program, 119 Tactical Language and Culture Trainers, 233 tactical language, defining, 222 Tactical Language, license for, 225 Tactical Language Trainers, 223 Tactical Language Training, LLC, 233 tactical military aircraft, 457 Tactical Pashto, new language tutor for, 224 tactical thinking expertise, cognitive model of, 292 tactician, combat with an engineer, 52 tactics something a soldier or a Marine knows, 233 as a variable of battle capability, 35 Talk-On A-10 MEC skill, 259 tangible results, of task analysis, 161 tank commanders and gunners, cognitive ability of, 453 tank crews, 437, 454 tank miles driven, 456, 459 Target Motion Analysis (TMA), 275 target performance, measuring, 342 targeted changes, mechanical notions of, 339 task(s) assumptions about structure of, 343 classification of, 343 designing to reproduce superior expert performance, embedding in natural context, 417 inventorying or describing, 158 performing discrete critical, 451 performing under optimal conditions, 416 specific invented to overcome weaknesses, 294 Task, Condition, and Standard (TCS) paradigm, 131, 133, 149, 207 task analysis advantages of, 158–159 as an applied activity, 160 complaints about, 156 defined, 158 defining itself, 160 Subject Index developing instruction based on, 168 differing over the years, 159 disadvantages of, 159–161 in every instructional design, 156 functions of, 158–159 in instructional design, 161 in instructional systems design (ISD), 13, 205 inventorying or describing tasks, 158 less successful examples, 175 limitation of, 205 methods as cost effective, or just effective, 176 outputs directed by the end users, 175 perceived lack or rigor of methods, 160 required for actual content of a course, 175 sorting needed skills and knowledge, 319 as time-consuming and expensive, 160 uncertainty and ambiguity in, 160 usability of the products of, 159 viewed as complex, 160 of what needs to be trained, 281 task classes, 150, 151 task description, compared to task analysis, 158 task domain, of elementary physics, 358 task environments, simulated, 146 task performance, measured, 453 task training curves, 41 task-level proficiency, 35 task-oriented behaviors, 87 TCS (Task, Condition, and Standard) paradigm, 131, 133, 149, 207 teacher(s) See also expert teachers; mathematics teachers; school teachers in cognitive apprenticeships, 64 effective in an imperfect world, 440 effects of errors, 384 evaluating effects on accomplishments of students, 480 experts interacting with others, 434 guiding continued attempts for mastery, 419 knowledge of performance conditions, 441 learning from learners, 436 mental models of students, 478–479 sense of performance conditions, 442 status and effectiveness of, 341 teacher control, guidelines for reduced, 397 teacher preparation institutions, researchers in, 440 teacher-free instruction, 138, 142 teacher-led instruction, facilitating, 348 teaching, as an initial focus for research, 490 547 Teaching and the Case Method, 471, 477 teaching expertise, 434 teaching machines, in World War II, 133 teaching ratings, 112 team activity, medicine as, 272 team building, effectiveness of, 475 team domain expert, 70 team member assessments, of leaders, 87 team performance in measurement setting, 344 as more than a simple sum of its parts, 454 not emphasized in this book, team training, 147 team-oriented, multiplayer environment, 223 team-related deliberate practice, 419 teamwork, as a family of achievement, 344 teamwork skills, in a multiplayer role-playing game/simulation, 237 technical courses, at Harvard Business School, 480 technical documentation, 168 technical expertise case library, 125 technical school training, 67 “Techniques for the Enhancement of Human Performance”, 312 technological age, technological change, requiring continual training, 284 technology new, 194 new recruits bringing knowledge of, 279 rapid improvements in, rapidly increasing capabilities of, 467 technology driven assessments, 444 technology-based learning environments, 209 telecommunications, age of, 159 TELEs (Technology Enhanced Learning Environments) developments in, 131 emphasizing in professional training, 148 in general education compared to professional training, 147 holistic approach to development of, 131 integrating for complex learning, 149 integration of, 151 learners practicing within, 207 more and more technology driven, 149 in professional training, 141–149 television associating with relaxation, 141 educational use of, 137 10-year rule, 412 tennis, 417 terrain, considering effects of, 293 548 Subject Index terrain databases, improving, 252 test(s) as adaptations of old tests, 335 of competency, 186 designed for particular purposes, 335 justified by stated purpose, 335 as only a sample, 334 versus presentations as learning events, 314 purposes for, 340 required by the profession for certification, 180 test content validation, 306 test design drawing on the literature of learning, 16 inverting the usual process of, 345 involving general abilities, 335 test designer, 350 test developers, 337 test questions, on tasks eliciting common errors, 337 test station in Sherlock, 66, 67 testing, models of learning informing, 344 texts accompanying dynamic visualizations, 145 containing hyperlinks, 145 theme(s) of battlefield thinking, 293 development of, 291 level of generality of, 292 strengths of, 292 in this book, 449–452, 463–466 theme-based method, examining the generalizability of, 305 theme-based training, 286, 289, 304–306, 323 Themes of the Battle Thinking, 292 theoretical knowledge, gap with application in real-life situations, 166 theoretical learning, 407 theory knowledge of, 174 tuning to practice, 169 theory of mind research, 490 thermal layer, 273 Think Like a Commander Training (TLAC) program, 286, 296, 297, 300–304, 324 think-aloud problem-solving tasks, 162 think-aloud protocols, analyzing chess players, 409 think-aloud study, family physician participating in, 392 thinking aloud while troubleshooting, 172 teaching, 478 training as a behavior, 290 thinking behaviors, training, 290–291 thinking enemy, modeling, 293 thinking performance, conforming to that of an expert, 291 ThinQ, 139 Third World armies, acquiring expensive technology, 34 thought habits See cognitive behaviors thought processes of battle, 51 of world-class chess players, 409 360-degree performance rating method, 87 three-block war, 132 time characteristics, of different workplaces and professions, 385 timeline, requiring more individuals aboard ship, 280 timing, considering, 293 Tissu (aerial dance form), 434, 435 TMA (Target Motion Analysis), 275 tool(s) computer as, 138 in the form of construction kits, 135 for local authoring, editing, and navigation, 450 for the production and realization phase, 134 supporting people engines, 243 Top Gun See Navy Fighter Weapons School torpedo attack scores, relationship to flying hours, 459 torpedo training, for SSBN crews, 42 Total Immersion Software, 231 toughness, ranking individuals by perceived, 488 tournament ratings in chess, 409 trace strength, decay of, 114 traditional educational methods, 189 traditional learning, bridging with professional practice, 421 train as you fight maxim, 294 trainability, documented, 406 trained personnel, stress on the importance of, 109 trained skills, retention of, 112 trainees involving in authoring, 80 learning by doing the whole activity, 68 throughput, 266 troubleshooting on their own, 171 using evaluations of ongoing training, 315 trainer-less trainer, 233 trainer-less training, 232–240 training activities, 417 assessment of, 33 beyond the knee of a learning curve, 44 breakthroughs by potential adversaries, 54 Subject Index built into a product, 230 capabilities as transportable, 323 as career-long process, 323 changing for both submarine specialties, 271 coming from user authoring, 232 conditions made artificially easy, 315 conducting in simulators, 425 connecting to military proficiency, 32–49 continuing required, continuum model resulting in a shortening of training, 322 revolution in naval training, 281–284 courses, component versus functionally oriented, 166 curriculum, re-engineering of, 322 deficiencies of current, 337 developer using game technology, 221 differences in effectiveness of, 33 directed toward specific tasks as quite effective, 43 distinguishing from performance, 319 distributing across days, 424 effect on proficiency, 43 environments decreasing time spent in expensive, 240 designing with challenging relevant situations, 425 not sufficient to produce best performers, 413 equal to purchase of military hardware, 58 essential to the U.S military, 210 evaluation of, 174 facilitating performance during, 15 generalized measures of, 38 giving to complete combat elements, 51 as hard to emphasize, 56 hierarchy of, 57–58 hours of, making enormous or little difference, 57 increasing capability per hour, 44 innovations in, 190 interaction with experience, 324 interventions, 19 interweaving with performance, 16 introducing realism into, 44 investment, 36 in promoting learning, 112 in yielding unequal performance changes, 57 issues, 45 keys to higher levels of, 45 levels, illustrated hierarchy of, 45 limitations in tests and assessments for, 337 manuals on the disk with DARWARS Ambush!, 231 549 maximizing performance when it matters, 319 methods, 289 in the military, 452 military value of, 466 necessary for highest levels of performance, 411 needs, identifying, 158 objective assessment of the effect of, 172 objectives, attaining, 452 old model of, 279 outcome measures, 452 paradox of, 45, 451, 467 performance, 46 pipeline in both medicine and submarining, 272 most radical change in, 282 similar to medical training, 280 potential for “negative” transfer of, 233 power of, 316 preparing for war by, 58 principles, 324 programs effect on performance of trainees in the field, 117 improving air-to-air exchange ratio, 30 on leader development, 85 for missions appearing quickly, 119 using electronic records to evaluate, 117 rates for individual skills, 57 representing levels of, 44 required to attain improvement, 413 requirements, 466 results, 218 revealing the student’s thoughts during, 291 revolution in, 29 schedules of senior athletes, 420 sessions, 253 sites, moving students to and from fixed, 29 situations, collections of, 423 software, 233 with Structured Troubleshooting, 174 structuring to achieve elite performance, 312–319 surprises, 53 tailoring for each unit’s needs, 53 for task-level proficiency, 35 tasks, 415, 426 thinking behaviors, 290–291 time allotted for submarine, 279 tools, 226, 326 transferring from lightweight PC simulations, 237–240 trials, 114 training curve family, 44 50 Subject Index training-derived proficiency, 28 traits, applied to general situations, 338 trajectory for change, 63 tranquilizers, prescribing to the elderly, 182 transfer assessing, 337, 342 demonstrating, 350 genuine, 336 of learning, 133 measures of, 336 of training, 303 transition, from student to expert professional, 63 transition operations, 243 trialability, of successful innovations, 183 triangulated credit system or model, 192 trigger events for cognitive behaviors, 291 out of the ordinary, 260 troubleshooting cognitive processes supporting effective, 165 expert-novice differences in, 164 functional representation as a necessary condition, 167 of planted faults in a radar system, 166 problems, 67, 69 skills, 66, 67 steps constituting a systematic approach, 166 structured, systematic approach to, 167 structured approach to, 169 teaching structured in parts, 171 teaching the strategy for, 167 troubleshooting form, 172, 175 troubleshooting gap, 71 troubleshooting proficiency in avionics, 67 improved by Sherlock, 72 trucks, searching for bombs, 217 trust meter, in Tactical Iraqi, 224 trusted mentors, 195 trust-worthier research designs, 397 Turing Tutors, 234 Turtle, drawing geometric figures, 138 tutee, computer as, 138 tutor(s) computer as, 138 never enough, 217 pushing students to solve sub-problems, 236 role and required competencies in PBL, 388 role of, 389 working in the context of the computer environment, 478 tutor achievement, in a law school, 388 tutorial systems, in training and education, 118 tutoring best training from one-on-one, 217 moderating behavior based on student characteristics, 479 as a promising method of training, 11 teaching novices expert reasoning, 164 tutor-student interaction, as observable, 234 typing capturing superior performance of domain experts, 410 speed increasing, 421 typists presenting the same text, 411 targeting problems with special practice, 421 U.K soldiers, training and persistence of, 34 undergraduates, training for practice, 384 undersea environment, 273 understanding, of other people, 490 underuse, clinical care gap as, 182 unexpected, preparation for, 450 Universal Modeling Language, 160 university level teaching expertise, as people management, 471 University of Bologna (Italy), 381 UNIX troubleshooting, 234, 235 unlearning curve, 39–41 Unreal game engine, 221, 224 Unreal Tournament PC game, 239 update model, for professional development, 181 update models of learning, 195 U.S Air Force Aggressor Squadrons created by, 50 AIMVAL/ACEVAL (air-to-air engagement exercises), 38 air-to-air loss ratios (1965–1973), 461 evaluation of Sherlock, 71 loss-exchange ratios, compared to the Navy, 461 performance measurement needs, 264–266 Red Flag exercises, 31 U.S Army culture transformed by engagement simulation, 30 interdeployment readiness bathtub curve, 55 National Training Center, 31 presenting leadership tasks to officers, 94 U.S Army Research Institute for the Behavioral and Social Science, 286 U.S Marine Corps clearing a room in a combat simulation, 240 motivated by getting a score on language progress, 242 U.S Marine Corps, “Officers eat last”, 121 Subject Index U.S Navy air-to-air loss ratios (1965–1973), 461 experiment in warfare, 30, 49 flight trainers assessing student pilot performance, 248 readiness in relation to anticipated deployments, 56 simulated force-on-force combat engagements, 460 Strike University, 38 US Top Gun training program See Navy Fighter Weapons School (Top Gun school) user authoring changing field authoring to, 228 in DARWARS Ambush!, 327 power of, 232 provision for, 231 virtue of, 228 users improving experience, 252 listening to during development, 232 shifting from mission mode to skill builder, 224 testing, 231 U-shaped relation, between levels of expertise and recall measures, 393 validity assessment, 342 validity evidence, for desired assessments, 336 validity studies, 351 value system, 364 variable environments, 443 variables of interest, 250 verbal and conceptual information, optimizing learning of, 324 verbal protocol analyses, 79 Vessey, General Jack, 31 victory, designation of cause of, 46 videoconferencing, in education, 139 videos, as a source of teaching expertise, 434 Viet Nam Navy’s unacceptable combat losses over, 49 U.S defeated in, 109 vignettes, based on tactical situations, 296 violinists time for solitary practice, 414 weekly diary on time spent, 413 Virtual Reality (VR), 146 Virtual Reality Medical Center, 238, 239 virtual simulation, 14 virtual trained pilots, as substantially better, 264 virtual training, 264 vision statements, 93 Visual Flight Rules flying, 39 551 visualization, role of, 451 volleys, mastery of tennis, 417 VR goggles, 147 warfare commanders See battlefield commanders warfare mission, skills required to complete, 41 warfare performance, 46 warfare proficiency, 27, 43 warfare success, requiring interdependent skills, 44 warfare training, revolution in, 27 warfighter feedback, 257 Warfighter Readiness and Performance Measurement Tracking System (WRAPMTS), 260 Warfighter Readiness Research Division, 258 warfighter training, 253 warfighter-defined skills, 258 wars, won by people, 109 waterfall display, 274 weaknesses correcting, 419 focusing on areas of, 295 identifying for practice sessions, 421 Weapon Engineering School, 166 weapon engineers, in the operational Dutch fleet, 165 weapon-counting measure, 34 Weaponeering A-10 MEC skill, 259 weapons assessments measuring military capability by, 33–41 costs as the major variable in military budgets, 33 as a variable of battle capability, 35 weapons systems complicated taking longer to master, 38 counting, 33–34 humans as limiting factor, 59 military potential attained through, 28 superior beaten handily by inferior technology, 28 weather-forecasting knowledge, 176 What a mess! problem, 386, 387 what is learned, 204 what is to be learned, 204 What the Best College Teachers Do, 477 whole tasks managing knowledge within, 208 students working on, 136 whole-task design models, 136–137 whole-task training, for a naval vessel, 44 Will you defend me? problem, 387, 388 52 Subject Index win faces expressions of confidence, 489 win predicting facial expression, 487 women, at Harvard Business School, 481 Woods, Tiger, 433 word-processor, 138 work conditions, work environments, supervisor giving feedback, 422 work versus play characteristic, of deliberate practice, 295 working memory, keeping information elements active in, 144 working memory overload, 167 work-sample methodology, World War II instructional films, 133 submarine commanders in, 455, 456 submarine engagements, 49 teaching machines, 133 trainers for pilot selection and training, 249 World Wide Web (WWW), 139, 145 world-class chess players, 409 See also chess WRAPMTS (Warfighter Readiness and Performance Measurement Tracking System), 260 XAIDA (Experimental Advanced Instructional Design Advisor), 134 years of experience, as a poor indicator of performance, zone of proximal development, 71 ... Feltovich, & R R Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp 685–706) Cambridge, UK: Cambridge University Press Hoffman, R R (Ed.) (2007) Expertise out of context:... (1946/1978) and Simon and Chase (1973) and emphasized the importance Measurement and Development of Professional Performance of extended professional experience for the attainment of the expert level of. .. measurements and training of professional performance in some of the major domains, and a review of the progress toward objective measurement Measurement and Development of Professional Performance 11