Trang 1 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Doan Thanh Xuan RESEARCH ON DEVELOPING DIGITAL TWINS-BASED APPLICATION FOR INDUSTRIAL ROBOTS Major:
MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Doan Thanh Xuan RESEARCH ON DEVELOPING DIGITAL TWINS-BASED APPLICATION FOR INDUSTRIAL ROBOTS Major: Mechanical Engineering Code: 9520103 DOCTORAL ABSTRACT IN MECHANICAL ENGINEERING Hanoi – 2024 The work was completed at Hanoi University of Science and Technology Supervisors: Prof Dr Vu Toan Thang Assoc Prof Dr Nguyen Thanh Hung Reviewer 1: Reviewer 2: Reviewer 3: This dissertation will be defended before approval committee at Hanoi University of Science and Technology Time …… o’clock, date … month … year ……… The dissertation can be found at: Ta Quang Buu library - Hanoi University of Science and Technology Vietnam National Library LIST OF PUBLICATIONS Đoàn Thanh Xuân, Lê Giang Nam, (2021) “Sự phát triển công nghệ đôi số ứng dụng lĩnh vực” Tạp chí Cơ Khí Việt Nam, số 9, tr 36-45 Đoàn Thanh Xuân, Vũ Toàn Thắng, Đặng Thái Việt, Vũ Tiến Dũng, (2022) “Xây dựng đôi số cho rô bốt UR3 hệ thống lắp ráp bóng đèn” Tạp chí Cơ Việt Nam, số 291, tr 14-17 Doan Thanh Xuan, Le Giang Nam, Dang Thai Viet, Vu Toan Thang “A-star Algorithm for Robot Path Planning Based on Digital Twin” (2022) In: Le, AT., Pham, VS., Le, MQ., Pham, HL (eds) The AUN/SEED-Net Joint Regional Conference in Transportation, Energy, and Mechanical Manufacturing Engineering RCTEMME 2021 Lecture Notes in Mechanical Engineering Springer, Singapore https://doi.org/10.1007/978- 981-19-1968-8_8 Doan Thanh Xuan, Vu Toan Thang, (2022) “Investigation on the influence of obstacle size in path planning by a hybrid model combining an improved A-star algorithm and digital twin” The Third International Conference on Material, Machines, and Methods for Sustainable Development 2022, pp 2195-4364 (accepted for publication) Doan Thanh Xuan, Tran Van Huynh, Nguyen Thanh Hung, Vu Toan Thang “Applying Digital Twin and Multi-Adaptive Genetic Algorithms in Human–Robot Cooperative Assembly Optimization” Applied Sciences 2023; 13(7):4229 https://doi.org/10.3390/app13074229 INTRODUCTION The Reasons for Choosing The Dissertation Topic Simulating a process or a system helps humans gain a deeper understanding of the system, allowing for flexibility to adjust and change device parameters as well as to perform optimization testing during the initial planning phase Due to the advantages mentioned above, simulation has been widely researched and applied However, it has mostly been limited to static models and one-way information exchange from physical objects to digital entities The concept of the digital twin, aiming to create a digital counterpart that faithfully describes a physical device and can exchange data with it, has been gaining increasing attention from researchers Along with the strong development of digital technology and new-generation information, the ability to collect more data and process it efficiently has provided a strong foundation for the development of digital twins The digital factory has become an inevitable trend in various industries, with the digital twin being a fundamental unit; synthesizing digital twins forms a digital factory Research on building digital twins for robots is a relatively new field, meeting practical industrial demands and promising strong development prospects in the coming years, both domestically and internationally This research is based on theoretical studies and experiments to construct a digital model that faithfully describes a robot and establishes communication connections between the physical entity and the digital model The experiments are conducted on the UR3 robot within a lightbulb assembly system The digital twin of the UR3 robot, once created, continues to be applied in tasks such as pathfinding and optimizing collaboration between humans and robots The application of robots is becoming increasingly popular, in line with the development of the automation industry Current research in robotics continues to grow due to high market demand, a wide range of applications, and technical development potential Particularly, robot path planning has been extensively studied and applied across various generations of algorithms Optimizing path planning for robots helps reduce time and costs in robot operation processes One of the widely applied algorithms for robot path planning is the A* algorithm The A* algorithm was invented, and many improvements have been made to enhance its efficiency in the original path planning algorithm In industrial production lines, we often encounter many collaborative work scenarios between humans and robots In these collaborative systems, the advantages of robots, such as speed and precision, can be combined with human qualities like dexterity and adaptability in different situations Through an overall analysis of the digital twin and industrial robotics problem, the postgraduate has recognized that the research and development of digital twins for industrial robots is a promising research direction that can still be explored in various aspects; in particular, applying digital twins to industrial robots to enhance performance in the operations of digital factories This is the reason why the postgraduate chose the topic "Research on Developing Digital Twins-Based Application for Industrial Robots" In this Dissertation, the postgraduate focuses on researching methods for developing digital twins-based application for industrial robots with reference to solving problems related to robot path planning and human-robot collaboration Goal, Objectives, Scope, and Research Methodology Goal The goal of the Dissertation is to gradually master the construction of industrial robots, develop software to enable the application of digital twins in optimizing robot paths and coordinating activities between humans and robots Specific objectives include: - Developing digital twins-based application for industrial robots - Proposing improvements to some digital twins-based applications for industrial robots with reference to solving problems related to pathfinding and human-robot collaboration Objectives The research subjects of the Dissertation are: Digital twin technology, industrial robots, methods for developing digital twins- based application for industrial robots, methods for robot pathfinding and human-robot collaboration Scope The scope of the Dissertation research is focused on developing digital twins and their applications for the UR3 robot within the lightbulb assembly system at the Digital Factory Laboratory of the Hanoi University of Science and Technology Research Methodology The research methodology involves a combination of theoretical study and practical experimentation Based on the theoretical research, including literature and previous works conducted both nationally and internationally on digital twins, industrial robotics, robot pathfinding methods, and human-robot collaboration, the study aims to understand the development of digital twins-based application for the UR3 robot Using the findings from the theoretical research, the study proceeds to implement digital twins in a real-world setting It continues to explore proposed improvements in applications related to robot pathfinding and human-robot collaboration The research involves conducting experiments, comparing theoretical and practical results, and publishing research outcomes in scientific journals and conferences, both nationally and internationally The Scientific and Practical Significance of the Research Topic In this research, a combination of the improved A* algorithm and digital twin technology has been used to plan the robot's path in the lightbulb assembly system The postgraduate highlights the advantages of the A* algorithm and the application of digital twin technology in mapping out the motion stages of the robot, tailored to propose an effective solution for the entire operational process Furthermore, the impact of obstacle sizes is evaluated in terms of the efficiency of one of the two methods to enhance path planning for the robot when applied to a real-world system with obstacles of varying sizes Furthermore, digital twins are applied to optimize the collaboration between humans and robots in the lightbulb assembly production line Firstly, digital twin technology is used to find the motion trajectory of the robot The digital twin of the robot and the human is created by combining cameras to track the position and activities of the human workers in the workspace This helps prevent collisions between humans and robots in the shared workspace Subsequently, an adaptive genetic algorithm is applied to calculate the optimal kinematics and movement schedule for the human workers To ensure uninterrupted operations and avoid material shortages, human workers need to observe and move to the material input conveyor and pallet input conveyor, supplying materials to the assembly system This is done to provide a continuous input of raw materials to the assembly line while allowing workers to perform their tasks in parallel with robot assembly operations The algorithm is designed to minimize the number of movements required for material retrieval, ensuring that the robot always has enough materials to follow the defined trajectory This results in labor savings and process optimization The combination of digital twin technology and the adaptive genetic algorithm optimizes the robotic movement and reduces the number of movements performed by human operators on the system The new findings The new findings achieved in the Dissertation are as follows: - Development of Digital Twins-based application for the UR3 Robot in the lightbulb assembly system - Application of digital twins in conjunction with the improved A* algorithm for robot pathfinding, comparing the cases with obstacles of constant height and varying height; consequently, proposing the use of a combination of digital twins and the improved A* algorithm suitable for each case - Application of digital twins to optimize the coordination of human and robot activities in the lightbulb assembly system The results show a reduction in the number of human movements to supply input materials to the system, ensuring that the robot can operate continuously without interruption due to a lack of materials such as lamp cap and bulb socket at the input of the assembly line Dissertation structure The Dissertation is divided into chapters: Chapter 1: An Overview of Digitial Twins and Their Applications; Chapter 2: Developing Digital Twins-based Application for Industrial Robots in the Lightbulb Assembly System; Chapter 3: Application of Digital Twins in Pathfinding for Robots; Chapter 4: Application of Digital Twins in Human-Robot Collaboration; Conclusion; Future work Chapter 1: OVERVIEW OF DIGITIAL TWINS AND THEIR APPLICATIONS 1.1 Introduction The technology of Digital Twins has been in existence since the 2000s In recent years, the rapid development of information technology (such as big data, cloud computing, artificial intelligence, etc.) has created favorable conditions for the robust growth of Digital Twin technology, attracting significant attention from researchers with a substantial increase in the number of research papers Digital Twins are not only used in a single phase but also have the ability to be used throughout the lifecycle of a system: from (i) planning, designing, optimizing parameters, testing, (ii) creating the actual system, (iii) operation, fault prediction, maintenance planning, lifespan prediction, to (iv) ceasing operation, eliminating from the operational system Digital Twins have wide-ranging applications across various industries (information technology, transportation, aviation, mechanical engineering, construction, healthcare, etc.), helping optimize system operations and reduce costs Alongside this, there is an increasingly widespread application of robotics in industrial production systems The problem of robot pathfinding and the coordination between humans and robots are topics of significant interest with numerous practical applications 1.2 The History of Digital Twins Development and Milestones Figure 1.1 summarizes the important milestones in the development process of Digital Twins according to the report [13] Figure 1 Key Milestones in the development of digital twins technology 1.3 Digital twins concept and classification On July 27, 2020, ISO standards for digital twins were published under the standard number ISO 23247, marking the emergence of standardized documentation for digital twins This standard builds upon all previous research However, this standard is still a framework and not a detailed protocol standard, focusing primarily on the production phase in factories Based on the level of integration of digital twins, Kritzinger and colleagues in their research [46] have categorized digital twins into three types: • Digital Model: In this type of digital twin, data between the physical object and the digital object is manually exchanged Therefore, any changes in the state of the physical object are not directly reflected in the digital one, and vice versa • Digital Shadow: Data from the physical object is automatically transferred to the digital twin, but this is still done manually in the reverse direction As a result, any changes in the physical object can be seen in its digital counterpart, but not automatically in the other direction • Digital Twin: In this type of digital twin, there is a bidirectional automatic data stream between the physical object and the digital object Therefore, real-time changes in either the physical or digital object can lead to changes in the other object 1.4 Digital twin application Digital twins can be applied in various industries and fields Manufacturing is the most common research area for digital twins and represents a significant portion of digital twin research Most studies focus on optimizing production planning, simulating manufacturing, monitoring and predicting product performance, and aiming for sustainable production Another popular field is smart buildings and cities, which concentrate on monitoring the sustainability of structures, building management and control, optimizing project planning, and predicting maintenance Information and Communication Technology (ICT) is also a prominent focus in digital twin research and has wide applications in edge computing systems, communication security, and cloud service monitoring Next, energy-related digital twin research focuses on power systems, fault diagnosis, and optimizing power plant operations Research topics related to automotive engineering, aerospace, healthcare, and medical care have a similar proportion, mainly focusing on product monitoring, prediction, testing, and simulation Finally, educational applications of digital twins have been on the rise in recent years due to the trend of online teaching and learning Various research fields are grouped together, including mining, agriculture, chemistry, etc [47] Among these applications, there are two applications that have attracted significant attention from researchers and have many Tessellation) format Building a digital model of the Robot UR3 (to simulate robotic geometry and dynamics) includes the following steps as shown in Figure 2.10 Figure 10 Steps for building a digital model of the UR3 robot TCP/IP protocol is used to connect real-time communication connection between a real robot and a virtual robot via Ethernet port The “Live Mode” in Tecnomatix Process Simulate software is used to connect real and virtual robots as shown in Figure 2.12 Figure 12 Real time connection between a real robot and a virtual robot Chapter 3: APPLICATION OF DIGITAL TWINS IN PATH FINDING FOR ROBOTS 3.1 Robot path planning problem The role of appropriate path planning in manufacturing is crucial When robot path planning is done correctly, industrial robots can efficiently perform their subsequent tasks Path planning for robots plays a significant role in the following aspects: The accuracy of robots, Task repeatability, Product quality 3.2 The improved A* algorithm The first improvement of the enhanced method is the local path between the current node and the target node, pre-planned in advance, 10 and the subsequent search in the neighborhood of the current node The local path will be traversed directly if it is safe and collision-free The second advantage of this method is that it leverages post- processing to optimize the resulting path by making the local path straight to reduce the number of local paths as well as the path's length In this algorithm, the query phase of the probabilistic motion planning involves path planning based on the improved A* algorithm [65, 66] There are two stages in probabilistic motion planning: the first stage is the preprocessing stage, and the second stage is the query stage In the first stage, a random generation of collision-free sample points is carried out within the robot's workspace These points can also be referred to as nodes in the subsequent stage In this algorithm, the local path planning then constructs a safe and collision-free local path between these points To validate the path's feasibility and collision avoidance, plans are mapped to the robot's configuration space by the local path planning, and this is done through shared space constraints (such as velocity and acceleration, energy optimization) Therefore, collision-free sample points and safe local paths are components of the probabilistic roadmap In the subsequent stage, with the application of the improved A* algorithm, the probabilistic motion planning generates a path that searches and obtains a safe path for the robot's motion connecting the initial node S (starting point) and the target node G (ending point) In this research, configuration space constraints are defined by parameters (such as angular range, velocity range, acceleration, velocity deformation, and optimal inverse kinematics solution) to construct the probability map Figure A simple example of a robot task 11 Figure A* algorithm-based path planning Figure 3 Optimal result of the first step on the left, the second step on the right Figure The optimal result of the final step 3.3 Application combining digital twins and A* algorithm in robot pathfinding Figure Robotic path found by the digital twin method Figure Robotic path found by the original A* algorithm (the left) and by the improved A* algorithm (the right) 12 Figure Robotic path found by the improved A* algorithm when a 30-mm obstacle added Figure Robotic path found by the improved A* algorithm when a 50-mm obstacle added Figure Real image of the pairs of obstacles Consider Case when the A* algorithm (1A) and improved A* algorithm (1B) were used It is shown that the robotic moving time could be shorter or longer when A* algorithm used in place of digital twin The smallest difference is about 3.544 s (-11.93%) corresponding to a velocity of 250 mm/s and an acceleration of 1200 mm/s2; the largest difference is about 4.456 s (5.24%) corresponding to a velocity of 450 mm/s and an acceleration of 500 mm/s2 It obviously means that the A* algorithm is not always better than the digital twin We see that in the case of a few obstacles, the duration for robotic movement obtained by the improved A* algorithm is smaller than that obtained by digital twin method In contrast the improved A* algorithm is always better than digital twin as the average robotic moving time reduced by -20.43% ÷ -22.07% 13 Consider Case when obstacles with constant height (case 2A: an obstacle with 30-mm larger in width and case 2B: an obstacle with 50- mm larger in width) were added the improved A* algorithm helped reduce the average robotic moving time by -18.07% and -16.44%, respectively Consider Case when obstacles with varying dimensions were added in pair It is shown that for obstacles with < 50 mm in height the improved A* algorithm increased the robotic moving time in most cases In case 3, when the number of obstacles increases with varying heights, leading to an increase in the number of passing points We find that starting at point D4, the duration for robotic movement obtained by digital twin method is getting shorter with speeds of 350 mm/s or more When the obstacles 3-4 with < 50 mm in height were added the A* algorithm increased the robotic moving time over the range of velocity and acceleration under study When the obstacles 5-6 with < 50 mm in height were also added the improved A* algorithm further increased the average robotic moving time by 23.9% and 41.36% respectively It implies that the digital twin is better than the improved A* algorithm when obstacles with varying heights used; whereas the reverse is true when obstacles with constant height used Chapter 4: APPLICATION OF DIGITAL TWINS IN HUMAN- ROBOT COLLABORATION 4.1 Theoretical basis There have been numerous methods developed to optimize production planning in human-robot collaboration These methods often fall under the category of evolutionary computation The authors have used algorithms such as Bee Colony Optimization to address the problem of minimizing human labor energy consumption during collaboration in assembly-disassembly lines [76] Feasible solutions for the path-finding problem have been explored using swarm intelligence algorithms [77] Two common types of human-robot collaboration in production are assembly and disassembly, with a 14 focus on assembly lines in this research The assembly problem on production lines is a common and complex issue that is frequently addressed in manufacturing plants 4.1 Genetic algorithm The Genetic Algorithm (GA) was initially proposed by D.E Goldberg and later developed by L Davis and Z Michalewicz Genetic Algorithm is a computer science technique used to find suitable solutions for combinatorial optimization problems [78] It is a branch of evolutionary algorithms that apply principles of evolution, such as genetic operations, mutation, natural selection, and crossover Genetic Algorithms (GAs) utilize the process of evolution found in nature to solve real-world optimization problems Starting from a population of initial candidate solutions, GAs iteratively evolve and apply mutation operators to generate a new set of solutions with improved fitness The final solution is an approximate optimal solution The algorithm's flowchart is shown in Figure 4.1 No Yes Figure Genetic algorithm diagram 4.2 Applying a combination of the digital twin method and genetic algorithm in the problem of human-robot collaboration Initialization of the initial population Based on the analysis of a robot's completion time for assembling one bulb, which is 30 seconds, the work plan for humans is divided into corresponding 30-second steps In total, for a production process of 100 bulbs, a human will execute 100 steps Each step contains the 15 corresponding working state of the human The chromosome representation is in the form of a binary vector with a length equal to the number of tasks in the job, which has shown good results [92] Task allocation in the problem [93] uses a chromosome with halves corresponding to defining tasks This is suitable as actions are grouped A chromosome with double the number of steps in the plan is used, and they are encoded in binary The first half of the chromosome represents the human's moving or standing state, while the second half represents the human's moving direction: either moving to the conveyor or to the pallet The state of each step is determined based on the two corresponding genes in the two halves of the chromosome The gene encoding details on the chromosome are presented in Table 4.1 Table Gene encoding on chromosomes Genes on the Genes on the Status first half second half - Static Moving to supply caps Moving to supply sockets Objective evaluation function Analyzing objective issues: Based on the structural characteristics of the chromosome that has been constructed, we calculate the positions of the genes on the chromosome to provide evaluation criteria for the objectives of the problem Ec: Evaluation index for optimizing the movement of placing lightbulb caps on the conveyor Eb: Evaluation index for optimizing the movement of placing lightbulb sockets into the pallet M: Evaluation index for optimizing the movement 16 The formula for determining the Ec index: 𝑥𝑘 − 𝑥𝑘−1 , 𝑥𝑘 − 𝑥𝑘−1 ≤ 𝑥𝑘 − 𝑥𝑘−1 > 𝑟𝑐𝑖 = { (4.5) , 𝑥𝑘 − 𝑥𝑘−1 𝐸𝑐 = ∑ 𝑟𝑐𝑖 The production constraint is that a person must not leave the conveyor belt without a cover This is equivalent to the time it takes for a person to move between two visits to the light bulb installation position (rc) cannot be less than the time it takes for the robot to produce the same number of light bulbs as the maximum number the conveyor belt can hold According to the chromosome construction model, at the beginning of the movement for cap placement, rc is equal to the time it takes to complete the assembly of one lamp Therefore, rc equals the distance between two genes that encode movement to the conveyor The conveyor can hold a maximum of caps for one feeding cycle If there are more than genes, the robot will run into a situation of material shortage and stop working This is an undesirable scenario, and the waiting time is inversely proportional to optimization The index for cap placement (Eb) plays a role equivalent to Ec, so it is constructed in a similar way The formula for determining the Eb index is: 𝑥𝑘 − 𝑥𝑘−1 , 𝑥𝑘 − 𝑥𝑘−1 ≤ 𝑥𝑘 − 𝑥𝑘−1 > 𝑟𝑏𝑖 = { (4.6) , 𝑥𝑘 − 𝑥𝑘−1 𝐸𝑏 = ∑ 𝑟𝑏𝑖 The maximum number of sockets that a pallet can hold is pieces And the variable rp is also determined in a way similar to how rc was determined The formula for determining the M index is: M =∑𝑚𝑘=1(1 − 𝑥𝑘) (4.7) If 𝑥𝑘 = 0, it means that the person does not move; m=100 It is calculated as the sum of genes represented by "0" The objective of the problem is to minimize the number of times the person 17