Field operation and preparation (FO & P) processes in the outages of nuclear power plants (NPPs) involve tedious team coordination processes. This study proposed a predictive NPP outage control method through computer vision and data-driven simulation. The proposed approach aims at automatically detecting abnormal human/ team behaviors and predicting delays during outages.
Progress in Nuclear Energy 127 (2020) 103448 Contents lists available at ScienceDirect Progress in Nuclear Energy journal homepage: http://www.elsevier.com/locate/pnucene Predictive nuclear power plant outage control through computer vision and data-driven simulation Zhe Sun a, Cheng Zhang b, Jiawei Chen a, Pingbo Tang c, *, Alper Yilmaz d a School of Sustainable Engineering and the Built Environment, Arizona State University, 660 S College Avenue, Tempe, AZ, 85281, USA The Zachry Department of Civil Engineering, Texas A&M University, 201 Dwight Look Engineering Building, College Station, TX, 77843, USA c Department of Civil and Environmental Engineering, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA d Department of Civil, Environmental and Geodetic Engineering, The Ohio State University, 470 Hitchcock Hall, 2070 Neil Avenue, Columbus, OH, 43210, USA b A R T I C L E I N F O A B S T R A C T Keywords: Nuclear power plant outage Computer vision Simulation Field operation and preparation (FO & P) processes in the outages of nuclear power plants (NPPs) involve tedious team coordination processes This study proposed a predictive NPP outage control method through computer vision and data-driven simulation The proposed approach aims at automatically detecting abnormal human/ team behaviors and predicting delays during outages Abnormal human/team behaviors, such as prolonged task completion and long waiting time, could induce delays Timely capturing these field anomalies and precisely predicting delays is critical for guiding schedule updates during outages Current outage control relies heavily on manual observations and experience-based field adjustments, which require extensive management efforts Realtime field videos that capture abnormal human/team behaviors could provide information for supporting the prognosis of abnormal FO & P processes However, manual video analysis could hardly provide timely infor mation for diagnosing delays Previous studies show the potentials of using real-time videos for capturing field anomalies These studies fell short in examining automatic video analysis in compact work environments with significant occlusions Besides, limited studies revealed how the captured field anomalies trigger delays during outages Computer vision techniques have the potential for automating field video analysis and detections of prolonged task completions and long waiting times This paper aims at automating the integrated use of 1) real-time computer vision and spatial analysis algorithms, and 2) data-driven simulations of FO & P processes for sup porting predictive outage control The authors first use the video-based human tracking algorithm to detect human/team behaviors from field videos Then, the authors formalized detailed human-task-workspace in teractions for establishing a simulation model of FO & P processes during outages The simulation model takes the field anomalies captured from videos as inputs to adjust model parameters for achieving reliable predictions of workflow delays Major observations show that 1) task delays often occur at the initial stage of the workflow, and 2) waiting line accumulates due to excessive resource sharing during handoffs at the middle stage of the workflow The simulation results show that tasks on the critical-path are more sensitive to these anomalies and cause up to 5.53% delays against the as-planned schedule Introduction Aging nuclear power plants (NPPs) in the United States require routine maintenance shutdowns (known as “outages”) to ensure continuous power supplies (Lloyd, 2003) Outages are necessary to ensure efficient NPP operations by refueling the reactor and executing essential repairs Abnormal human/team behaviors (e.g., prolonged task completion, long waiting) during field operation and preparation (FO & P) processes bring significant challenges in controlling delays during outages Real-time monitoring of FO & P processes for capturing abnormal human/team behaviors and estimating potential delays is vital to ensure resilient outage control Such a monitoring system aims to support the outage control center (OCC) for making appropriate schedule updates and reducing delays (Fig 1) A typical NPP outage requires hundreds of contract workers to complete thousands of refu eling and maintenance activities within 30 days (B.N Spring 2009) * Corresponding author E-mail address: ptang@andrew.cmu.edu (P Tang) https://doi.org/10.1016/j.pnucene.2020.103448 Received November 2019; Received in revised form 25 May 2020; Accepted 10 July 2020 Available online 27 July 2020 0149-1970/© 2020 The Authors Published by Elsevier Ltd This is an (http://creativecommons.org/licenses/by-nc-nd/4.0/) open access article under the CC BY-NC-ND license Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 However, workers with diverse backgrounds and prior outage experi ences require extensive training to fulfill the productivity requirements during NPP outages (Sun et al., 2018a; Zhang et al., 2018a) Poor human/team performance that causes prolonged task completion and long waiting time could jeopardize NPP outages Timely capturing such poor human/team behaviors that cause performance bottlenecks during FO &P processes is thus crucial for predictive control of NPP outages Integrated uses of automatic video analysis algorithms and compu tational simulation models for automated monitoring, diagnosis, and control of teamwork processes during outages is becoming possible with the development of computer vision techniques and computational simulation methods (Luo et al., 2014, 2019; Sun et al., 2020) Field videos of outage workspaces are becoming widely available for FO &P process monitoring with the increasing use of cameras in workspaces (Bolton, 2015) Current outage control practices rely heavily on tedious and error-prone manual inspections and experience-based judgments of outage professionals (Germain et al., 2014) Video data collection has the advantage of capturing rich spatiotemporal details of human mo tions without having to install any contact sensors on workers A video-based approach for human/team behavior analysis in outages is thus of great potential for supporting human factors analysis and pre dictive NPP outage control (Fang et al., 2018) Such a video-based approach can help to capture the prolonged task completion and long waiting time during FO & P processes The captured anomalies could become rich data sources for OCC to decide schedule updates for miti gating delays Formal models and simulations have the potential to resolve the difficulties of assessing the captured abnormal human/team behaviors and predicting delays caused by those anomalies (Bolton, 2015; Pan and Bolton, 2015) Specifically, agent-based simulation models could represent 1) human agents – representing behaviors of workers and the outage supervisors (Zhang et al., 2019; Bonabeau, 2002); and 2) work flows – representing the maintenance workflow based on given sched ules (Wang et al., 2014; Chen et al., 2012) Some simulation models show the potential of being able to take the captured anomalies from the real-time videos as inputs for predicting workflow delays (Chen et al., 2012; Mohamed et al., 2017; Han et al., 2013; Alzraiee et al., 2015) For example, Chen et al established an intelligent scheduling system for optimizing resource sharing during construction projects through sim ulations (Chen et al., 2012) This study established a simulation model that includes representations of a construction schedule and relation ships between available resources (e.g., workforce, equipment) and workspaces However, these simulation models fell short in providing detailed attributes when modeling human agents in the simulation Such attributes, such as traveling activities, communication behaviors, and waiting at a station, are pivotal for capturing abnormal human/team behaviors A data-driven simulation model that represents detailed human/team behaviors during outages is vital for predicting delays using the captured field anomalies from videos Previous studies have examined the use of computer vision and machine learning techniques to detect abnormal human/team behaviors using real-time videos (Fang et al., 2018; Wang et al., 2018; Wang and Liu, 2018) However, limited studies have examined the use of real-time field videos in a compact workspace with significant occlusions for capturing abnormal human/team behaviors Challenges remain in automatically capturing field teamwork anomalies and related produc tivity losses for predicting delays Such challenges include 1) significant occlusions in the field videos recorded in a compact indoor workspace during outages, and 2) modeling of interwoven human-task-workspace interactions during FO & P processes for predicting delays due to the captured anomalies This study proposed an integrated use of computer vision and simulation for capturing abnormal human/team behaviors and predicting delays caused by the captured anomalies (Fig 2) The proposed method includes using a single-camera based trajectory anal ysis approach for increasing the coverage of monitoring with the same number of cameras during indoor monitoring Specifically, this study 1) examined real-time human tracking and spatiotemporal analysis methods for automatically diagnosing abnormal human interactions and unexpected trajectories of workers, and 2) developed a data-driven agent-based simulation model for using the detected abnormal human/team behaviors as input for predicting delays during NPP outages The proposed framework integrates knowledge from human factors, computer vision, and simulations that enable engineers to fully discover the interactions between humans, resources, and workflow that influ ence outage productivities The goal is to advancing disciplines of human systems integration and computer vision in the construction management domain First, the authors conducted an extensive litera ture review about human factors in NPP outages The review aims to identify abnormal human/team behaviors in FO & P processes during NPP outages The authors then developed and tested state-of-the-art computer vision algorithms that help to monitor and capture field anomalies in compact workspaces during FO & P processes of NPP outages The developed computer vision algorithms enable timely and detailed monitoring of the FO & P processes for supporting real-time schedule adjustments and resource allocations during NPP outages Fig Real-time field information acquisitions for capturing waiting time and task duration variations Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 Fig The proposed predictive NPP outage control framework Last, the presented research study established a data-driven agent-based simulation model based on typical FO & P processes during NPP outages The model integrates the synthesized findings of human errors in outage control to model “difficult” FO & P processes during NPP outages that challenge teamwork performance Running simulations could help to examine the occurrence and propagation of abnormal human/team behaviors during NPP outages The organization of the remaining parts of the paper is: Section provides a detailed literature review about three practical problems in NPP outages Section illustrates the developed agent-based model of human/team behaviors within outage workflows Section introduces the developed computer vision algorithms for automatic human behavior data acquisition and analysis Section describes the frame work of the data-driven simulation model for assessing the impact of abnormal human/team behaviors on workflow delays Section sum marizes the major research findings and technical challenges Section concludes and synthesizes the future research directions window (Tang et al., 2016) Effective FO & P processes at both indi vidual and team levels are increasingly necessary for accomplishing complex tasks and avoid delays (Sun et al., 2020) At the individual level, schedule deviations due to workers’ operational errors could also bring significant risks of delays Such deviations often cause prolonged task completion against the as-planned schedule (Jang et al., 2013) Delays on these individual tasks could propagate to delays to larger workflows or even the complete outage, especially when the scheduled task has limited floats At the team level, NPP outage control is one of those tasks that need multiple professionals to work together for col lective decision-making Workers need to go through Radiation Pro tection Island (RPI), where the space connecting the containment and outside environment for getting prepared for the scheduled task Coor dinating workers from different teams during handoffs require precise estimations of delays inside the RPI Poor coordination can cause un necessary waiting and cause significant delays in RPI Previous studies and practices tried to examine the impact of waiting in such task prep aration processes on workflow delays (Zhang et al., 2018a; Sun et al., 2020) However, most of these studies are Monte Carlo simulations with limited use of real-time data (waiting time) collected in field operations as inputs for reliable delay diagnosis based on real data Literature review This section provides a systematic literature review about 1) prac tical problems during NPP outages and 2) challenges in capturing abnormal human/team behaviors and predicting outage delays Abnormal human/team behaviors during NPP outages could induce severe delays and cause significant financial losses How to capture these anomalies and assess the impacts on outage delays is thus crucial for supporting the outage management efforts Three significant aspects need to be solved to achieve such goal: 1) a better understanding of the interactions between human/team behaviors and workflows of NPP outages; 2) a technology that can effectively capture abnormal human/ team behaviors, and 3) a method that can use the captured human/team anomalies for predicting delays during outages The following sections thus provide reviews on 1) human and team behaviors in NPP outages; 2) computer vision for real-time human/team behavior monitoring; and 3) effective use of field information for resilient outage control 2.2 Practical problem two: computer vision for real-time human/team behavior monitoring FO & P surveillance important for determining whether a project can complete on time and avoiding budget overruns (Ghanem and Abdel Razig, 2006; Cheng et al., 2013; Girardeau-Montaut et al., 2005) Existing sensing techniques show the potential for tracking real-time locations of construction entities during FO & P processes Examples of such technologies include Radio Frequency Identification (RFID), Global Positioning Systems (GPS), and Ultra-Wideband (UWB) (Ghanem and AbdelRazig, 2006) However, all these sensors are usually not applicable for NPP outages due to confidentiality issues (Zhang et al., 2017) Some studies proposed the use of closed-circuit television (CCTV) installed at job sites to monitor construction workers’ behaviors and locations of equipment to ensure safe operation (Shrestha et al., 2015; Hinze and Teizer, 2011) However, such monitoring methods risk exposing workers’ faces and cause confidentiality issues A confidential-protective progress monitoring method is thus necessary to 2.1 Practical problem one: human and team behaviors in NPP outages NPP outages require coordinating hundreds of contract workers with diverse backgrounds to complete thousands of tasks within a tight Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 capture field workers’ behaviors and avoid exposing sensitive produc tivity information Such a monitoring method could target monitoring areas that are mostly preparation activities where workers are mostly waiting and preparing to help the manager to understand the overall productivity and bottlenecks without directly measure the workers’ task performance The confidential-protective monitoring method could target at detecting and tracking workers’ body joints without exposing their faces and other identity information Besides, such a technique could also capture anomalies during the task preparation processes (e.g., extended duration, waiting time) without leaking extensive human privacy At the same time, such preparation and waiting time informa tion could still be useful for inferring possible delays, identifying poor process arrangements, and suggesting schedule updates Monitoring FO & P processes are critical for ensuring a better team situation awareness during outages However, human gestures during refueling and maintenance activities vary significantly On the other hand, human/team behaviors during handoff processes have relatively fewer uncertainties and variations from the predefined procedures in the controlled indoor environments For example, the Radiation Protection Island (RPI), which is the space connecting the containment and outside environment, contains a lot of handoff activities Such handoffs usually involve dosimetry checking, technical debriefing, and tool pick-up/ drop-off Workers have limited options while deciding how to go through their handoffs These limited options can be as waiting (standing still or sitting), walking between stations in the RPI, or talking to each other Monitoring such simple behaviors in an RPI could still be useful for inferring workflow delays during outages An effective and efficient method for preparation and waiting behavior monitoring in indoor environments could thus bring benefits to NPP outage control frequently cause delays during outages The critical path method (CPM) has widely been adopted by the construction industry to control the schedule and estimate delays by identifying the longest path of depen dent activities and measuring the time 3.1 Modeling of detailed spatiotemporal human-task-workspace interactions in a valve maintenance workflow during NPP outage Modeling detailed human-task-workspace interactions in a valve maintenance workflow during NPP outages requires setting up numerous constraints to specify the relationships between humans, task, and workspace The developed model contains a workflow model and a human activity model The workflow model captures the spatiotemporal relationship between tasks The human activity model specifies the re sponsibilities of workers on the scheduled tasks Fig visualizes the developed human-task-workspace model Valves are the critical mechanical component for nuclear reactors The authors model the valve maintenance activities at two job sites (Site A, and B, respectively) during a typical NPP outage (Fig 3) Three workers (insulator, electrician, and mechanic) need to complete five tasks on each site with the information from the supervisor All workers need to go through RPI for 1) checking available work packages, 2) technical briefing, and 3) picking up tools (e.g., earplugs) before the workers start their work at Site A Besides, once the workers complete their tasks, they need to 1) get back to RPI for dosimetry checking; 2) dropping off tools, and 3) check other available work packages Work spaces and workers are all shared resources, which means workers cannot work at two sites at the same time If a worker occupies a workstation for a prolonged duration, other workers have to wait in line for using the workstation Extended task durations and long waiting due to resource sharing could be indicators of delays in the valve mainte nance workflow 2.3 Practical problem three: effective use of field information for resilient outage control Continuously monitoring of FO & P processes and use the captured field anomalies for predicting workflow delays is critical to ensure a resilient outage control (Sun et al., 2020; Yoo et al., 2016) A detailed human-task-workspace model that captures interwoven relationships between human/team behaviors and workspaces can support decision-makers for making prompt field adjustments (e.g., schedule updates) (Sun et al., 2018b) However, the lack of representations of detailed human/team behaviors in current outage scheduling processes causes challenges in risk assessments with full considerations of human factors Such a situation impedes engineers and researchers from using computer algorithms for assessing problematic outage scenarios and mitigation strategies Besides, tedious communications and traveling activities during NPP outages raised additional challenges for precise risk assessments For example, the talking behaviors and the perception processes of the information received are challenging to model in any mathematical model Therefore, project managers chose not to consider complex human/team behaviors as a factor for analytical modeling in the current practice of project management Instead, the use of buffering approaches is favored in the current project management processes to mitigate delays due to abnormal human/team behaviors Besides, the uncertainty of the task durations greatly influences the performance of scheduling techniques In brief, most of the current scheduling tech niques fell short in reducing delays and achieving NPP outage resilience 3.2 Modeling of human/team behaviors In a valve maintenance workflow, the supervisor needs to coordinate several workers to work on several tasks at the same time through tedious communications In this study, the authors have created two types of agents (the worker agent and the supervisor agent) for modeling the human behaviors during NPP outages The supervisor agent collects field information from the worker agent through communications Field information is critical for the supervisor to make appropriate decisions, Agent-based modeling of human/team behaviors within outage workflows This section presents lab experiments for modeling and capturing human/team behaviors during a FO & P process in a typical valve maintenance workflow during NPP outage The developed agent-based simulation model helps to formulate a basis for understanding the im pacts of abnormal human/team behaviors on workflow delays The valve maintenance workflow contains critical-path activities that Fig The spatial and temporal relationship between tasks in the outage workflow Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 such as add additional tasks if discoveries found in the field Field conditions change frequently during outages due to various un certainties (e.g., variations of the task duration, field discoveries) Effective coordination between the supervisor and workers is thus necessary to keep each other informed The authors model such infor mation exchanging processes as the communication behaviors between workers and the supervisor (Fig 4) The communication behaviors in the model considered two communication aspects, 1) communication network patterns (e.g., network structure); and 2) characterizations of communication links (e.g., channel and timing) Specifically, the au thors use a centralized communication network to model the commu nication behaviors that allows 1) the supervisor to assigned work packages to workers when there are available tasks, and 2) the workers to report the task completion status to the supervisor when the current task has completed The authors define four behaviors for the “worker” agents, 1) working on scheduled tasks according to the as-planned schedule, 2) reporting to the “supervisor” agent when the current task has completed, 3) traveling to the next job site for the successor task, and 4) waiting for task availability information from the supervisor The “worker” agents will only enter into the “working” status to execute the scheduled task only when the “supervisor” agent has assigned a task to them When in working status, the “worker” agents will work on the tasks planned by following the as-planned task duration specified in the schedule How ever, uncertainties (e.g., task duration variation) in the “working” status may cause prolonged task completion due to poor human behaviors After the “worker” agents finish the task, they will enter the communi cation status The communication status requires the “worker” agents to report the task completeness to the “supervisor” agent In the meantime, the “supervisor” agent will be in the communication status as well Based on the field information reported by the “worker” agents, the “supervisor” agent can mark the task as complete and identify new available tasks Once the “supervisor” agent identified the available tasks, he/she can call the “worker” agent about available tasks The “worker” agents can travel to a different job site to work on the available task informed by the “supervisor” agent during communications human/team behaviors in an indoor workspace and capturing abnormal poses during NPP outages The authors then introduced the basic graphical user interface (GUI) design to highlight the developed system 4.1 Human joint detection The human joint detection algorithm uses the layout of an indoor workspace (e.g., RPI) for mapping the locations on video frames to the layout (Fig 5) The algorithm first maps the image spaces of raw images to the 2D trajectory on the RPI room layout The authors then use a topdown method for tracking the workers’ poses (Cao et al., 2017) and apply a two-branch Convolutional Neural Network (CNN) network for detecting body joints of multiple workers in the indoor workspace (Cao et al., 2017) The algorithm then outputs the detected body joints of multiple workers in the scene through a refining process (Cao et al., 2017) As shown in Figs and 6, the detected skeletons consist of body joints of workers in the indoor workspace In the human joint detection process, the authors use a graphmatching algorithm for matching the body joints of a worker in the scene The graph-matching algorithm recognizes all body joints of a worker by using the orientation of the worker’s body and the workers’ limbs as the edge weights of the k-partite graph (Cao et al., 2017) The detection randomly chooses an ID for a worker in the video per frame when the worker first appears in the scene However, keeping the same IDs on workers during the tracking process is still a challenge due to significant occlusions that might cause ID switches The outputs (Fig 6) of the human joint detection process grouped the labeled body joints of multiple workers into skeletons These skeletons then serve as inputs to the virtual planes generated based on the anthropometric measures of a typical worker (Zhang et al., 2018b) 4.2 Video projection to layout map Using a single-camera-based approach for detecting and tracking body joints of multiple workers in a compact indoor workspace is challenging due to significant occlusions (Zhang et al., 2018a) An effective tracking algorithm using human body joints is thus necessary to track workers’ movements precisely in a compact workspace How ever, the displacements in the image space become larger when workers getting closer to a camera and result in a higher velocity in the image space Besides, multiple workers might walk across the room, running, and standstill Such multi-worker scenarios bring challenges for effec tive human tracking Using a single-camera-based approach causes dif ficulties in reliably tracking multiple workers that are moving inside a Computer vision algorithms for automatic human behavior data acquisition and analysis In this section, the authors introduce the developed automatic video surveillance system The developed system uses state-of-the-art com puter vision algorithms to achieve effective situation awareness of the outage progress Besides, the surveillance system aims at monitoring Fig Agent-based communication behavior modeling (worker agent; supervisor agent) Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 Fig Human joint detection Fig Body joint detection of workers compact workspace The algorithms transform the detected joints from the image space to the joint space on virtual planes to mitigate the risks of losing depth information by using one single camera Luo et al pro posed a method for generating virtual “Anthropometric Planes” through homograph transformation (Luo et al., 2019) All these virtual planes are in parallel with the horizontal plane of the ground of the indoor work space Then the algorithms generate multiple virtual planes at levels of workers’ body joints The authors then applied Kalman Filter to track those detected joints of multiple workers in the indoor workspace on these “Anthropometric Planes” preparation processes in the indoor workspace (e.g., technical debrief ing, tool pick-up/drop-off) The GUI allows the outage manager to use layout maps of any indoor workspace and select the area of interest for monitoring bases on the layout In this study, the authors use the layout map of an RPI for testing the developed human-tracking algorithms and the developed GUI The GUI shows that the average waiting time will start counting when a worker enters a station and starts the handoff process (e.g., technical briefing) The counting will stop until the worker finishes the handoff and moves on to the next station In this GUI, all stations have different thresholds (alarming and alert times) with the time unit due to the nature of the different tasks during NPP outages The management team can modify the alarming and alert thresholds based on the urgent levels of specific tasks when coordinating activities during NPP outages Besides, the GUI displays the overall waiting time inside the RPI as well With such information, the man agement team will be able to monitor the waiting time inside the RPI and make proper decisions on when to send workers to the RPI for preparing the successor tasks This visualization of the computer vision system enables outage manager to quickly identify the status of multiple stations and spot the field anomalies 4.3 Design of graphical user interface (GUI) In this section, the authors develop a graphical user interface GUI (Fig 7) that displays multiple simultaneously tracked workers in the RPI and identifies field anomalies (e.g., extended task duration, long waiting time) in the workflow The GUI visualizes the human/team behaviors in an indoor environment by showing 1) the moving patterns between stations, 2) the duration each worker spends at each workstation, and 3) wait time for each station Such visualization enables engineers to use the developed tracking algorithm for real-time visualizing of the tracking results The developed GUI allows outage managers to visualize the FO & P processes in the indoor workspace and make proper field adjustments for mitigating delays caused by field anomalies Fig shows the detailed GUI design for visualizing the task Data-driven simulation framework for assessing the impact of human/team behaviors on workflow delays The proposed data-driven agent-based simulation framework Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 Fig Real-time monitoring and statistics output (Red cell indicates the time the worker spent in the station exceeded the alert limits) contains 1) an agent-based simulation platform consists of a process model of an NPP outage workflow, and 2) a human activity model The data-driven simulation platform uses the captured anomalies from the computer vision algorithm (e.g., prolonged task completion, long waiting time) as input to simulate the impact of these detected anom alies on workflow delays (Fig 8) The authors conducted a series of lab experiments by using the valve maintenance workflow to validate the proposed framework During the experiments, the authors implemented the developed computer simulations to examine 1) the developed computer vision algorithm in capturing task duration variations and waiting for lines in RPI; and 2) the proposed simulation framework in predicting potential workflow delays 5.1 Simulation for assessing the impact of task duration variance on workflow delays The authors carried out a series of lab experiments with participants recruited from the construction engineering program at Arizona State University All participants have profound knowledge about the con struction schedule and were provided with extensive training sessions to get familiar with the experiments The experiment was set up based on the established spatial and temporal relationship between tasks in the outage workflow (Fig 3) and the human behavior models (Fig 4) The supervisor will coordinate with three workers (insulator, electrician, and mechanic) to complete five tasks at two job sites (Table listed all the task information) Besides, all workers have to go through RPI for completing the handoff activities (Fig 3) Each task will have an asplanned task duration that requires the participant to follow (partici pants will use the timer provided to count the time for their tasks) However, participants might have different behaviors (e.g., prolonged task completion, waiting) during handoffs and cause delays to the asplanned task duration (Fig 9) Besides, such delays could be critical if the successor tasks are on the critical path of the schedule According to the as-planned schedule, the authors derived the critical path of this workflow to better interpret the delays captured during the experiments According to the observations during experiments, the authors found that some participants could not strictly follow the as-planned schedule and complete the task in time Besides, delays on one task can easily propagate to other tasks and jeopardize the workflow The authors have recorded all such delays during the experiments and incorporated the Fig The data-driven agent-based simulation framework Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 5.2 Simulation for assessing the impact of waiting line during handoff on workflow delays Table Delays captured during lab experiments Site A Task Task (A) Task (A) Task (A) Task (A) Task (A) B Task (B) Task (B) Task (B) Task (B) Task (B) Total Duration Delay Worker Team As-planed Duration (minutes) Avg Delay (minutes) Delays in Simulation (minutes) Insulator 0:25 4:10 Electrician 4.5 0:20 3:20 Mechanic 0 Electrician 4.5 0 Insulator 0 Insulator 0:37 6:10 Electrician 4.5 0:21 3:30 Mechanic 0 Electrician 4.5 0 Insulator 0:20 3:20 The waiting time during RPI is essential for estimating the delays to the valve maintenance workflow During the lab experiments, the au thors observed that the waiting line in the RPI could be up to 30 min, which causes a late-start of the successor task Such delays could prop agate to all following tasks As shown in Table 2, the last column in dicates the percentage of delays to the overall workflow due to waiting time during handoffs For example, the authors added a 30-min delay after the insulator finished Task (A) due to the late-start of Task (A) Such added delay causes a 4.32% delay (30 min) against the as-planned schedule since Task (A) is on the critical path The simulation results suggest that tasks on the critical-path are more vulnerable to delays during handoffs For example, Table shows that a 30-min waiting during the handoff of Task (A) contribute to 5.53% delays against the as-planned schedule Delays on Task (A) had the least impact on the as-planned workflow duration Additionally, such added delays not only affect individual tasks but also affect the prepa ration processes in the RPI If specific tasks had delays, the probability of having conflicts between different workers while in the briefing process would increase Moreover, the waiting time in the RPI would increase due to the resource sharing between multiple workers during handoffs in the RPI Such resource sharing issues could occur when multiple workers traveled to the same station in the briefing process However, one station can only be occupied by one worker at a time Additional delays to the workflow could arise due to such resource sharing during handoffs 11.86 (hour) 0.29 (hour) 2.5% recorded delays into the simulation model By running the simulation, the authors tried to understand the impact of individual task delays on workflow delays The last columns of Table indicate the average delays captured during the lab experiments and the delays during the simula tion (duration in the lab experiments are scaled) The average total duration of is 11.57 h after 1000 runs of the simulation model Compare to the as-planed workflow duration; the delay is 0.29 h (2.5%) Discussion NPP outages are accelerated construction projects that pose signifi cant challenges to the limits and requirements of both human and physical environments A more resilient outage control should imple ment robust control methods to fully assess the reliability of humanphysical interactions and propose mitigating strategies accordingly Fig Images captured during lab experiment (a: experiment scene; b: the insulator is traveling between stations; c: the insulator is working at Station #1; d: a waiting line occurs at Station #3) Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 preparation processes are critical to ensure the smooth transitions be tween tasks and poor handoffs could result in substantial delays to NPP outages Non-value added activates during handoffs demand a more accurate and flexible schedule updating methods for sending the worker team to the RPI when workstations are available Such demand thus requires effective real-time monitoring of the handoff processes in RPI and accurate estimation of the waiting time The proposed single-camera based trajectory analysis approach can significantly increase the coverage of monitoring with the same number of cameras during indoor monitoring In the past, researchers can only analyze 3D trajectories in regions covered by both two cameras; such areas are much smaller than areas that only need to be visible to one camera Several technical challenges need further investigations to quantify the costs and benefits of using a single-camera-based approach for locating and tracking multiple workers, and how such techniques could enhance the performance of a network of cameras in monitoring numerous workers First, implementing the computer vision algorithms with one camera for detecting and tracking human/team behaviors in compact indoor space with severe occlusion issues is still challenging Besides, using multiple cameras could solve the occlusion problems if such resource is available in a packed indoor workspace like RPI, but would raise additional challenges Specifically, challenges of using one single camera mainly lie in 1) the use of one single camera for 3D localization of human individuals moving in the indoor workspace; 2) potential information losses (e.g., depth information) due to the use of one single camera for tracking moving patterns of individuals; 3) the real-time monitoring of moving patterns of multiple individuals in the compact indoor workspace with significant occlusions; and 4) frequent ID switch of multiple individuals for accurate estimating the task duration and waiting time in the indoor workspace Besides, using a dual-camera system to get the 3D locations still has some challenges In certain areas, which are out of the over lapped field of view of multiple cameras, getting 3D locations becomes difficult Another reason for using a layout map to assist the human tracking is that a layout map of RPI is quite accessible and reliable Using multiple cameras for human tracking could raise additional problems, such as 1) calibration of multiple cameras, 2) coordinating the field-ofviews and locations of multiple cameras for ensuring the coverage of the team processes with enough spatiotemporal details Table Delays while considering 30-min waiting-line in RPI for each task Site Task Worker As-planed Duration (minutes) Percentage of Delays A Task (A) Task (A) Task (A) Task (A) Task (A) Task (B) Task (B) Task (B) Task (B) Task (B) Insulator 30 4.32% Electrician 45 4.49% Mechanic 60 5.53% Electrician 45 4.32% Insulator 30 1.38% Insulator 30 4.32% Electrician 45 3.46% Mechanic 60 4.06% Electrician 45 4.41% Insulator 30 4.32% B This study developed a predictive outage control method, which in tegrates knowledge and strength from three domains, 1) human factors, 2) computer vision, and 3) computational simulation The proposed system aims at capturing abnormal human/team behaviors and predict delays during outages However, the authors have discovered challenges while integrating human factors, computer vision, and simulation for effective control of NPP outages This section illustrates specific tech nical challenges associate with every element in the proposed predictive control method 6.1 Challenges of human factors analysis for predictive NPP outage control Human factors in NPP outages are critical for ensuring the safety and productivity of the operation and maintenance activities Human/team behaviors, such as cognitive behaviors, communications, and travel activities, could significantly affect NPP outages Previous human fac tors studies focused more on conducting tedious lab experiments for discovering human/team behavior deviations with limited consider ation of the impact of such deviations to the physical environment and cause severe safety and productivity concerns to the NPP industries Besides, such experiments always tailored to fit into a certain scenario, and the data collected cannot be used for other cases Also, quantita tively defining “normal” interactions among individuals is challenging The OCC has established a detailed procedure for individuals to follow during NPP outages with limited details of the expected behaviors at the individual and team level Specifically, challenges mainly fall into four categories, 1) lack of formalized representations for modeling human/ team behaviors during NPP operation and maintenance activities; 2) lack of comprehensive categorization of normal/abnormal human/team behaviors in the NPP industry; 3) lack of a detailed reasoning method to fully understand the arising and propagation processes of human/team errors in NPP operation and maintenance activities; 4) lack of specific human-task-workspace models for an accurate estimate the impact of abnormal human/team behaviors on the productivity of NPP operation and maintenance activities 6.3 Challenges of data-driven simulation for predictive NPP outage control Data-driven simulation is a powerful tool for assessing the impacts of abnormal human/team behaviors (e.g., prolonged task completion, long waiting time) on NPP outage productivity issues Such simulation plat forms can also examine possible mitigation strategies proposed by NPP professionals in mitigating the risks of delays caused by abnormal human/team behaviors However, frequent schedule updates and changes in work packages require effective outage team coordination for ensuring proper executions of new work packages without elevating the risks For example, discoveries of new tasks due to maintenance failures, defects on mechanical parts, or delays that occur while ordering new parts for maintenance can cause severe delays Unfortunately, current outage control heavily rely on tedious manual inspection and adjust ment based on the control manager’s experience and knowledge A more resilient outage control system through data-driven simulation is thus necessary to automatically 1) propose contingency plans to reduce risks of delays in real-time; 2) and evaluate the performance of the proposed contingency plan in terms of resource allocation, schedule delays and cost overrun Challenges still exist and lie in 1) lack of capabilities for achieving real-time data-driven simulation that can use the captured abnormal human/team behaviors as input to update the simulation model rapidly; 2) lack of formalized methods to consider all extreme events and assess the impacts on outage safety and productivity; 3) lack of systematic approaches for examining mitigation strategies and 6.2 Challenges of computer vision techniques for predictive NPP outage control The developed computer vision algorithms in this study enabled the real-time monitoring of human/team behaviors during FO & P processes in a compact indoor workspace with significant occlusions The Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 optimizing parameters in the simulation models for reducing the safety and productivity risks during outages Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper Conclusion and future research Abnormal human/team behaviors that lead to frequent schedule updates bring significant difficulties to achieve resilient outage control Even NPP outage professionals with extensive field experience could hardly discover such abnormal human/team behaviors and assess the impacts on outage delays The developed computer vision algorithms proved to be able to detect and track multiple individuals with a single camera in a compact workspace with severe occlusions The algorithm then achieves a precision of 70% and a recall of 38% The developed data-driven simulation platform proved to be able to predict delays using the anomalies captured by the computer vision algorithm during handoffs Major observations show that 1) task delays often occur at the initial stage of the workflow, and 2) waiting line accumulates due to excessive resource sharing during handoffs at the middle stage of the workflow The simulation results show that tasks on the critical-path are more sensitive to these anomalies and cause up to 5.53% delays against the as-planned schedule The authors envision the extension of the proposed computer vision algorithms that could also be useful for 1) psychological assessment (e g., situation awareness, workload) of workers; 2) physical capability (e g., fatigue) evaluation of workers Overall, the contributions of the proposed method could be at two levels First, the proposed method could detect abnormal human/team behaviors by using body joints for inferring delays without exposing the identity information Second, the proposed method could capture detailed human/team behaviors for evaluating human/team performances on safety-critical tasks if no confidentiality concerns For example, heavy crane-lifting activities during NPP outages usually require a lot of moving spaces in a compact job site Effective coordination to separate workers and cranes in a safe spatial distance is critical to avoid crane-related accidents Besides, vi sual verification and maintenance on pumps and valves are crucial for ensuring enough coolant and adequate pressure to prevent a core meltdown during NPP outages However, a great number of valves and pumps are in high proximity at certain locations become obstacles for workers to recognize the correct valves Performing scheduled tasks on the wrong valves could be fatal for NPP outages Enhancing the proposed computer vision algorithm is thus necessary to be able to 1) locate workers’ locations in real-time; 2) provide safe navigation routes for workers to stay away from the crane; 3) auto matically match the worker’s location and the physical location of the scheduled task based on the as-planned schedule and the site layout, and 4) generate alarms if a worker presented at the wrong location and performed the task on the wrong object Moreover, the developed computer vision algorithms could also serve as powerful tools for the post-hoc assessments of NPP outage efficiency The captured human and task-related anomalies are thus necessary for determining sections of the outage that have low efficiency when multiple sections are included at one job site Overall, this paper summarizes the challenges and future research directions that could benefit more broad research communities composed of construction engineering and management and computer science researchers Acknowledgment This material is based upon work supported by the U.S Department of Energy (DOE), Nuclear Energy University Program (NEUP) under Award No DE-NE0008403 DOE’s support is acknowledged Any opin ions and findings presented are those of authors and not necessarily reflect the views of DOE References Alzraiee, H., Zayed, T., Moselhi, O., 2015 Dynamic planning of construction activities using hybrid simulation Autom ConStruct 49, 176–192 https://doi.org/10.1016/j autcon.2014.08.011 B N Spring, S (Ed.), 2009 Nuclear Outage Operational Excellence 08/01/2009 Bolton, M.L., 2015 Model checking human-human communication protocols using task models and miscommunication generation J Aero Inf Syst 12, 476–489 https:// doi.org/10.2514/1.I010276 Bonabeau, E., 2002 Agent-based modeling: methods and techniques for simulating human systems Proc Natl Acad Sci U.S.A https://doi.org/10.1073/ pnas.082080899 Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y., 2017 Realtime multi-person 2D pose estimation using Part Affinity fields IEEE Conf Comput Vis Pattern Recognit https://doi.org/ 10.1109/CVPR.2017.143 Chen, S.M., Griffis, F.H., Chen, P.H., Chang, L.M., 2012 Simulation, and analytical techniques for construction resource planning and scheduling Autom ConStruct https://doi.org/10.1016/j.autcon.2011.05.018 Cheng, T., Teizer, J., Migliaccio, G.C., Gatti, U.C., 2013 Automated task-level activity analysis through fusion of real-time location sensors and worker’s thoracic posture data Autom ConStruct 29, 24–39 https://doi.org/10.1016/j.autcon.2012.08.003 Fang, W., Ding, L., Luo, H., Love, P.E.D., 2018 Falls from heights: a computer visionbased approach for safety harness detection Autom ConStruct 91, 53–61 https:// doi.org/10.1016/j.autcon.2018.02.018 St Germain, S.W., Farris, R.K., Whaley, A.M., Medema, H.D., Gertman, D.I., 2014 Guidelines for implementation of an advanced outage control center to improve outage coordination, problem resolution, and outage risk management (No INL/ EXT-14-33182) Idaho National Lab.(INL), Idaho Falls, ID (United States) Ghanem, A.G., AbdelRazig, Y.A., 2006 A framework for real-time construction project progress tracking Earth Space 1–8 Girardeau-Montaut, D., Roux, M., Marc, R., Thibault, G., 2005 Change detection on points cloud data acquired with a ground laser scanner Int Arch Photogram Rem Sens Spatial Inf Sci 36, W19, 10.1.1.221.8313 Han, S., Love, P., Pe~ na-Mora, F., 2013 A system dynamics model for assessing the impacts of design errors in construction projects Math Comput Model 57, 2044–2053 https://doi.org/10.1016/j.mcm.2011.06.039 Hinze, J.W., Teizer, J., 2011 Visibility-related fatalities related to construction equipment Saf Sci 49, 709–718 https://doi.org/10.1016/j.ssci.2011.01.007 Jang, I., Ryum, A., Ali, M., Al, S., Jun, S., Gook, H., Hyun, P., 2013 An empirical study on the basic human error probabilities for NPP advanced main control room operation using soft control Nucl Eng Des 257, 79–87 https://doi.org/10.1016/j nucengdes.2013.01.003 Lloyd, R.L., 2003 A survey of crane operating experience at US nuclear power plants from 1968 through 2002 Division of Systems Analysis and Regulatory Effectiveness, Office of Nuclear Regulatory Research, US Nuclear Regulatory Commission Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Zhao, X., Kim, T.-K., 2014 Multiple Object Tracking: A Literature Review Luo, X., Li, H., Wang, H., Wu, Z., Dai, F., Cao, D., 2019 Vision-based detection, and visualization of dynamic workspaces Autom ConStruct 104, 1–13 https://doi.org/ 10.1016/j.autcon.2019.04.001 Mohamed, E., Jafari, P., Siu, M.-F., Francis, AbouRizk, S., 2017 Data-driven simulation model for planning roadway operation and maintenance projects In: Proc 2017 Winter Simul Conf., pp 3323–3334 Pan, D., Bolton, M.L., 2015 Properties for formally assessing the performance level of human-human collaborative procedures with miscommunications and erroneous human behavior Int J Ind Ergon 1–14 https://doi.org/10.1016/j ergon.2016.04.001 Shrestha, K., Shrestha, P.P., Bajracharya, D., Yfantis, E.A., 2015 Hard-Hat Detection for Construction Safety Visualization, 2015 https://doi.org/10.1155/2015/721380 Sun, Z., Zhang, C., Tang, P., 2018a Simulation-Based Optimization of Communication Protocols for Reducing Delays during Nuclear Power Plant Outages, pp 455–464 https://doi.org/10.1061/9780784481295.046 Sun, Z., Zhang, C., Tang, P., 2018b Simulation-based optimization of communication protocols for reducing delays during nuclear power plant outages In: Constr Res Congr 2018 Infrastruct Facil Manag - Sel Pap From Constr Res Congr 2018 https://doi.org/10.1061/9780784481295.046 Credit author statement Zhe Sun: Conceptualization, Methodology, Writing - original draft Cheng Zhang: Writing - review & editing Jiawei Chen: Writing - review & editing, Computer Vision algorithm development and testing Pingbo Tang: Conceptualization, Methodology, Writing - original draft, Writing - review & editing Alper Yilmaz: Computer Vision algorithm develop ment and testing, Writing - review & editing 10 Z Sun et al Progress in Nuclear Energy 127 (2020) 103448 Sun, Z., Zhang, C., Tang, P., 2020 Modeling and simulating the impact of forgetting and communication errors on delays in civil infrastructure shutdowns Front Eng Manag 1–13 Tang, P., Zhang, C., Yilmaz, A., Cooke, N., 2016 Automatic imagery data analysis for diagnosing human factors in the outage of a nuclear plant Lect Notes Comput Sci Digit Hum Model Appl Heal Safety, Ergon Risk Manag 9745 Wang, Y., Liu, Y., 2018 A novel Bayesian Entropy Network for probabilistic damage detection and classification In: In 2018 AIAA Non-Deterministic Approaches Conference (p 1407) Wang, W.C., Weng, S.W., Wang, S.H., Chen, C.Y., 2014 Integrating building information models with construction process simulations for project scheduling support Autom ConStruct https://doi.org/10.1016/j.autcon.2013.10.009 Wang, Y., Liu, Y., Sun, Z., Tang, P., 2018 A Bayesian-entropy network for information fusion and reliability assessment of national airspace systems Proc Annu Conf Progn Heal Manag Soc PHM Yoo, W.S., Yang, J., Kang, S., Lee, S., 2016 Development of a computerized risk management system for international NPP EPC projects KSCE J Civ Eng 21, 1–16 https://doi.org/10.1007/s12205-016-0784-y Zhang, C., Tang, P., Cooke, N., Buchanan, V., Yilmaz, A., Germain, S., Boring, R., AkcaHobbins, S., Gupta, A., 2017 Human-centered automation for resilient nuclear power plant outage control Autom ConStruct 82, 179–192 https://doi.org/ 10.1016/j.autcon.2017.05.001 Zhang, C., Sun, Z., Tang, P., Germain, S.W St, Boring, R., 2018a Simulation-based optimization of resilient communication protocol for nuclear power plant outages Adv Intell Syst Comput 20–29 https://doi.org/10.1007/978-3-319-60645-3_3 Zhang, B., Zhu, Z., Hammad, A., Aly, W., 2018b Automatic matching of construction onsite resources under camera views Autom ConStruct 91, 206–215 https://doi org/10.1016/j.autcon.2018.03.011 Zhang, P., Li, N., Jiang, Z., Fang, D., Anumba, C.J., 2019 An agent-based modeling approach for understanding the effect of worker-management interactions on construction workers’ safety-related behaviors Autom ConStruct https://doi.org/ 10.1016/j.autcon.2018.10.015 11 ... current outage control heavily rely on tedious manual inspection and adjust ment based on the control manager’s experience and knowledge A more resilient outage control system through data-driven simulation. .. productivity of NPP operation and maintenance activities 6.3 Challenges of data-driven simulation for predictive NPP outage control Data-driven simulation is a powerful tool for assessing the... study developed a predictive outage control method, which in tegrates knowledge and strength from three domains, 1) human factors, 2) computer vision, and 3) computational simulation The proposed