(Đồ án hcmute) research and application of augmented reality technology in cnc machine operation training

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT MACHINE MANUFACTURING TECHNOLOGY RESEARCH AND APPLICATION OF AUGMENTED REALITY TECHNOLOGY IN CNC MACHINE OPERATION TRAINING LECTURER: PhD DANG TRI DUNG STUDENT: LUONG HOANG THANG NGUYEN CHI DUC PHAM CONG DUY SKL010426 Ho Chi Minh City, December 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FALCUTY OF HIGH QUALITY TRAINING GRADUATION PROJECT RESEARCH AND APPLICATION OF AUGMENTED REALITY TECHNOLOGY IN CNC MACHINE OPERATION TRAINING STUDENT’S NAME AND ID: LUONG HOANG THANG – 18146061 NGUYEN CHI DUC – 18146015 PHAM CONG DUY – 18146011 INSTRUCTOR: PhD DANG TRI DUNG HO CHI MINH CITY, 12/2022 Acknowledge After years of study and obtaining valuable new knowledge at Ho Chi Minh City University of Technology and Education, the mission of the group has been fulfilled through the time spent to carry out the project up until this point Although there are certain problems that still need to be fixed, overall, the group still achieves acceptable outcomes We would like to thank the teachers on behalf of the group members This project benefited greatly from the knowledge that every teachers, who introduced us to new sources provided Above all, Mr Dang Tri Dung, the teacher in charge of overseeing our group, deserves the genuine gratitude of our entire group We appreciate your support of the group as it worked to complete the graduation project The last word is a wish for a happy new year, good health from our group to the teachers Table of Contents Abstract Object and scope of the study CHAPTER 1: OVERVIEW CHAPTER 2: THEORETICAL BASIS 2.1 Function of CNC machines in the project 2.2 Fundamentals of AR 2.2.1 Theory of AR 2.2.2 Application of AR in the topic 2.3.1 Reasons to choose Doosan 5700 CNC milling machine for simulation 2.3.2 Reasons to use Unity software 2.4 Modeling facility in Unity Hub 2.5 The process of building CNC AR machine CHAPTER 3: THE PROCESS OF BUILDING A CNC MILLING MACHINE MODEL 3.1 Android simulation 3.2 Doosan CNC milling machine modeling 3.3 Introduction to Blender 10 3.3.1 Blender's uses 10 3.3.2 Blender's application in the project 10 3.3.3 File formats that unity supports 12 3.3.4 Exclusive file formats 12 3.3.5 Unsupported file extension format 13 3.4 Set colors material for CNC machine 13 3.5 Each button's function is specified 16 3.6 Software development process 24 3.7 Programing the animation for the CNC model 30 CHAPTER 4: INSTRUCTIONS 40 4.1 Manual mode 40 4.2 AR mode 47 CHAPTER 5: EXPERIMENT AND RESULT 49 5.1 Experiment 49 Simulation criteria 50 5.2 Result 50 5.2.1 Can be used on both phones and computers 50 5.2.2 Can move the X Y Z axis 51 5.2.3 Can operate a workpiece 52 5.2.4 Can start the app for a long time 53 5.2.5 Can display the 3D CNC machine on the enviroment 53 5.2.6 General Evaluation 54 CHAPTER 6: CONCLUSION AND FUTURE DEVELOPMENT 55 6.1 Lesson learnt 55 6.2 Achievement 55 6.3 Limitation 55 6.4 Suggestion for future development 56 REFERENCES 57 Table of Figures Figure 1: AR example Figure 2: CNC Doosan DNM 5700 machine Figure 3: Procedure Diagram Figure 4: App Mixed Reality Toolkit Figure 5: After opening the App Figure 6: SolidWorks machine 3D model [5] 10 Figure 7: Uploaded 3D files with Blender 11 Figure 8: the machine's outer frame after being separated and concealed 11 Figure 9: After separated 12 Figure 10: Open Blender file with Unity 13 Figure 11: Three main parts 14 Figure 12: Colors’s setting table 15 Figure 13: Color’s show table after finished setting 15 Figure 14: After adding color 16 Figure 15: V Model 25 Figure 16: Iterative and Incremental model 26 Figure 17: Agile Methodologies 27 Figure 18: Waterfall model 27 Figure 19: Flowchart for programming X Y Z axis moving coordinates – Part A 31 Figure 20: Flowchart for programming Spinning – Part B 32 Figure 21: Flowchart for programming Spin percentage – Part C 33 Figure 22: Flowchart for programming keyboard – Part D 34 Figure 23: Localize 36 Figure 24: ViewManager – CreateView 38 Figure 25: ViewManager - BackView 39 Figure 26: Main screen 40 Figure 27: Language setting in Vietnamese 40 Figure 28: Language setting in English 41 Figure 29: After finished choosing 42 Figure 30: Manual and AR mode 42 Figure 31: After choosing the manual mode 42 Figure 32: After clicking the "action of the workpiece" button 43 Figure 33: The tool moves to the set coordinates 44 Figure 34: Axis table X Y Z 44 Figure 35: Rotary control 45 Figure 36: Tool rotation speed control 45 Figure 37: Command keyboard 46 Figure 38: Adjust the camera's angle 46 Figure 39: Notifying errors once the tool makes contact with the workpiece 47 Figure 40: The machine displays on the environment 47 Figure 41: AR mode 49 Figure 42: Manual mode 50 Figure 43: Build Setting 51 Figure 44: Display coordinates 52 Figure 45: Workpiece after performing drilling 52 Figure 46: Workpiece after performing cutting 53 Figure 47: Scan the environment through the camera 54 Figure 48: Not enough space 54 Abstract Owing to the swift advancement of software technology in particular and information technology in general Real-world software applications are becoming more and more common, usually in the form of virtual reality simulation technology Before, augmented reality simulation technology was only used for video games, but now it can be used to correctly model the operation of vehicles, electronics, and other machines If students simply learn through theory and subsequently practice, accidents will be reduced A high level of simulation programming knowledge and a thorough grasp of the electrical equipment to be replicated are required for the deployment of augmented reality simulation technology to model electronic devices In order to enhance the training of future CNC workers, this project focuses on the use of virtual reality technology to imitate the operation of CNC machines in industry Vietnamese CNC companies are growing, however it is hard to find experienced CNC workers, and training CNC workers takes time and may be dangerous after training The capacity to experiment with the CNC machine depicted in a virtual environment will expand, but without the risk of machine failure or user danger Object and scope of the study Learn how mechanical structure of the CNC machines such as: X Y and Z axis movement, control the blade spinning speed and make contact with the workpiece through a research project before using virtual reality simulation technology to create virtual models Although there are many other kinds of CNC machines, the DOOSAN 5700 CNC milling machine is the subject of this project Greater types can be simulated with more time Unity Hub is the program utilized in this project to facilitate virtual modeling Research Techniques Overview of the DOOSAN 5700 CNC milling machine, including a look at the model (Blade movement, X Y and Z axis, machine frame) and how to use it (set tool, center, execute program) while learning about Unity In order to achieve this, the following chapters of the project's key research materials and findings are presented: Chapter 1: Overview Chapter 2: Theoretical Basis Chapter 3: The process of building a CNC milling machine model Chapter 4: Instructions Chapter 5: Experiments and results Chapter 6: Conclusion and feature development The conclusion section provides an evaluation of the outcomes obtained and the challenges faced during the project's study and execution Errors cannot be prevented during the project implementation process due to the time and qualification constraints For further development for this project, we anticipate receiving the contributions of professors and students Sincere appreciation goes out to PhD Dang Tri Dung and the department of high quality English lecturers for their tireless assistance with both the project implementation and learning processes CHAPTER 1: OVERVIEW The most obvious thing, in the intern's opinion, is that you have to stand about and watch other people work the machine for up to two months before you can interact with it, and even then, only for straightforward tasks as straightforward as cleaning the machine or putting the workpiece inside of it, but excluding real operation The student will subsequently be able to use the machine to basic tasks like installing the workpiece, configuring the tool, obtaining the coordinates, and finally running the available program created by an expert From that, it can be observed that training CNC workers takes a lot of time and that even after the process of theoretical training and instructor observation, mistakes can still be made frequently Therefore, practicing first on a computer is the best course of action at this time Numerous CNC training programs exist, including MasterCam, Cimco, Cimatron, etc However, why are employees still having trouble using CNC machines Because this software only directs CNC machine programming, practically all coordinate entry is to the correct position and the workpiece also has available coordinates, making it simple to accomplish on the software, but the workpiece is actually placed in the CNC machine To program and run the machine, you must locate the corner board to obtain the coordinates, and then compute the coordinates of the center of the workpiece surface Thus, utilizing virtual reality simulation software, workers can experience operating a genuine CNC machine in a manner similar to that Then students will interact with the machine model, watch it in action, and use it almost exactly like a real machine This is what inspired the creation of this project Figure 37: Command keyboard Figure 38: Adjust the camera's angle Because this is a simulation of a CNC milling machine, there may be some error cases while drilling, milling, and adjusting the X, Y, and Z axis Examples of these mistake instances include: When the tool strikes the workpiece The blade will vanish, and an error message will show when the image is used Press the reset button, which is next to the 46 camera adjustment button, to display the blade once more This button will restore everything to its previous state (Fig.39) Figure 39: Notifying errors once the tool makes contact with the workpiece 4.2 AR mode At the mode selection screen, press the other mode in instead of manual to start AR mode The screen will use the camera on the mobile device the user is using to scan the area for a flat area before commencing to load the 3D CNC milling machine there (Fig.40) Figure 40: The machine displays on the environment 47 After the software has successfully launched the CNC machine model, you can click the done button If there is a problem, click Reset and try again until everything operates The machine control mechanism is the same whether the user selects manual or AR mode; users will still have a table of buttons displayed on the left side of the screen to carry out fundamental actions like controlling the machine X, Y, and Z axis Alternately, you can adjust the speed of the tool rotation and turn the tool rotation on and off Of course, there will still be a button in the lower left corner of the screen in manual mode to enable drilling or cutting operations By pressing this button, the user can perform drilling and cutting operations on the workpiece in a similar way to manual mode 48 CHAPTER 5: EXPERIMENT AND RESULT 5.1 Experiment In this first part of chapter our group are going to launch an experiment to to test about some basic functionalities of CNC AR, by which can guarantee if it is able to meet some standard requirements to operate properly in environment (Fig.41, Fig.42) Figure 41: AR mode 49 Figure 42: Manual mode Simulation criteria Number of criteria Objectives of CNC AR The app is available for usage on mobile devices and desktop computers Able to the basic movement of X Y Z axis Able to make contact with the workpiece such as: drilling and cutting Able to run the app for a long time without crashing Able to scan the environment to load the CNC machine 5.2 Result 5.2.1 Can be used on both phones and computers It is simple to obtain the files and easily install them on the phone due to Unity's support for exporting files specifically for phones The Build Setting panel can be viewed in two different ways: by selecting File → Build Setting or by pressing the Ctrl + Shift + B key combination, which will display the table as shown below (Fig.43) 50 Figure 43: Build Setting Users only need to click Build and Run on this panel's display for Unity to start generating an apk file for Android mobile devices 5.2.2 Can move the X Y Z axis As was explained in the preceding chapters, the group was successful in programming the machine so that users could move and control the X, Y, and Z axis using buttons To make it simpler for users to move the axes precisely, the group also programmed a table that shows coordinates when moving the axes (Fig.44) 51 Figure 44: Display coordinates 5.2.3 Can operate a workpiece Learn more from internet resources after doing some study Eventually, the group was able to program the drilling and cutting animations on the workpiece's surface When the blade moves across the workpiece's surface, the animation appears to be flawless and without of glitches or bugs Additionally, if the user touches the workpiece incorrectly, an error case has been written by the group will display on the screen (Fig.45, Fig.46) Figure 45: Workpiece after performing drilling 52 Figure 46: Workpiece after performing cutting 5.2.4 Can start the app for a long time There are several apps available, which let users simulate their surroundings, and after using them for a while, the majority of them raise the temperature of the user's mobile device, which causes slowness Might possibly be forced to crash from the app and because most users not save data while working, which lead to lose data Therefore, each member of the group tried to install the group's app on the smartphone, choosing AR mode because manual mode not generate much heat for the smartphone After placing the CNC machine in its environment in AR mode, the team tested running the software for 15 to 30 minutes to check for lag or smartphone being overheated The results are encouraging because the group app is still functioning normally after 30 minutes and there is no indication that the phone's temperature is rising 5.2.5 Can display the 3D CNC machine on the enviroment The camera on mobile devices is the most important factor of this project since it allows users to scan the environment and identify where a flat, suitable area for a CNC milling machine should be placed The app is utilizing the camera on the mobile device to scan the surrounding area to make sure it is flat and has enough area, which is why there are yellow dots in the image The other yellow components called foundation that is where the CNC machine will be displayed will start to emerge from there However, there must be sufficient space to display this background (Fig.47) 53 The user must then tap the reset button so that the app can scan the environment once more in search of a more acceptable spot when they receive a red alert since there is not enough area to load enough of this foundation (Fig.48) Figure 47: Scan the environment through the camera Figure 48: Not enough space 5.2.6 General Evaluation After doing numerous tests on the application and exchanging ideas between each member, any errors were fixed The team succeeded in getting the outcomes it has now Although some criteria still have not met with the standards Because the group lacks the necessary knowledge and time, these conditions are nonetheless acceptable The application has so far continued to function in accordance with the requirements 54 CHAPTER 6: CONCLUSION AND FUTURE DEVELOPMENT 6.1 Lesson learnt The team has gathered a lot of new information after more than four months of study and preparation for the graduation project The group had the chance to gain a greater understanding of augmented reality (AR), about how to use and program it The team was given information about the new programming library DOTween and be able to program to create motion animations of the X, Y, and Z axis The group was also able to adjust how quickly the tool rotated thanks to this library The team was capable of gaining a deeper understanding of the singleton design pattern as a result of working on this project With the help of this design, the team was able to program the tool's drilling and cutting operations on the surface of the workpiece Through this project, the team's programming skills have significantly increased When discussing the lessons that each team member has learned while carrying out this project, teamwork is the main element that must not be ignored This is what, after four years of university study, helped the group achieve success with the graduation project 6.2 Achievement The group was successful in developing a whole CNC AR application Simple CNC machine operations can be carried out with this app Each scan of an environment like this generally takes to 10 minutes for the app to scan, successfully build the virtual environment, and simulate the machine This application has two modes: AR and Manual In AR mode, it is possible to scan the surrounding environment through the camera of mobile devices, simulating a 3D image of the DOONSAN DNM 5700 CNC machine Both Computers and mobile devices can use the application The app is quite lightweight and does not overheat when used repeatedly Additionally, there is an animated interaction between the tool and the workpiece 6.3 Limitation After finishing this significant project, the group gained new experiences and knowledge, but there are still certain problems that need to be solved The first problem is that this app cannot presently be used on iOS-powered devices The app's inability to execute complicated commands is the second problem; the group has only coded the blade to execute simple programming The third problem the team ran into was that while some mobile devices running the Android operating system can simulate CNC machines in augmented reality, other times the simulation of the machine failed or takes a very long time In addition, when the user causes the tool to rotate too quickly to touch the workpiece, the group has not yet been able to display the broken tool The group will work to address these three issues in the future so that the application can be made even better than it is now Currently these common problems appear when using the app 55 6.4 Suggestion for future development Include some roaming features for CNC machines:  Add some buttons to control modes like turning on and off the lights and releasing the cooling water  Assistance with the iOS operating system  Reduce the amount of time needed to construct a CNC machine model by scanning the environment  Virtual reality goggles can be used to use the app  Add a feature that stores data automatically after cutting 56 REFERENCES [1] Introduction to AR (Augmented Reality) (September 30, 2021) Available: https://fptshop.com.vn/tin-tuc/danh-gia/ar-va-vr-la-gi-so-sanh-diem-giong-va-khac-nhaucua-ar-va-vr57045#:~:text=AR%20(Augmented%20Reality)%20hay%20c%C3%B2n,x%C3%BAc% 20gi%C3%A1c%2C%20v%C3%A0%20kh%E1%BB%A9u%20gi%C3%A1c [2] Git introduction Available: https://topdev.vn/blog/git-la-gi/ [3] What is Unity programming, and what are the foundational skills to learn (October 14, 2020) Available: https://itnavi.com.vn/blog/lap-trinh-unity [4] DOONSAN 5700 CNC machine model (June 4, 2022) Available: https://grabcad.com/library/doosan-dnm-5700-1 [5] Getting Started with Blender (January 28, 2023) Available: https://docs.blender.org/manual/vi/dev/getting_started/about/introduction.html#:~:text=B lender%20l%C3%A0%20b%E1%BB%99%20c%C3%B4ng%20c%E1%BB%A5,%2C% 20T%E1%BA%A1o%20Ch%E1%BA%A5t%20Li%E1%BB%87u%20%E2%80%94%2 0Texturing%2C [6] Instructions for using and downloading Unity (March 2021) Available: https://docs.unity3d.com/Manual/index.html [7] An overview of Shaders in Unity3D (December 21, 2016) Available: https://viblo.asia/p/gioi-thieu-khai-quat-ve-shader-trong-unity3d-roavrJxjvRM [8] Software development process and software development models (June 2, 2020) Available: https://viblo.asia/p/quy-trinh-phat-trien-phan-mem-va-cac-mo-hinh-phat-trien-phanmem-gAm5ybP8Kdb [9] Overview of DOTween (since 2014) Available: http://dotween.demigiant.com/ [10] Explaining Facemills Available: https://hutscom.vn/phay-mat-face-milling-la-gi-cac-yeu-to-de-gia-cong-phay-mat-hieuqua.html#:~:text=d%E1%BB%91c%20(ramping%20milling),Phay%20m%E1%BA%B7t%20(Face%20Milling)%20l%C3%A0%20g%C3%AC%20% 57 3F%3F,ra%20c%C3%A1c%20b%E1%BB%81%20m%E1%BA%B7t%20ph%E1%BA% B3ng [11] Explaining Endmills Available: https://cncanhkim.com/dao-phay-end-mill-la-gi/ [12] Explaining Chamfers (Side Face Cutter) Available: https://maycongcuthanhloi.com/san-pham/dao-phay-chamfer-canh-nhatban/#1_Dao_Phay_Chamfer_Canh_La_Gi [13] Get started with Sourcetree Available: https://confluence.atlassian.com/get-started-with-sourcetree [14] Destructible Terrain: Simple destructible terrain in Unity based on bitmaps and Quadtree Pretty efficent and works with Unity Colliders Very reusable and customizable for your own needs (February 25, 2020) Available: https://www.youtube.com/watch?v=EygJSTL-B-k [15] AR foundation: AR Foundation enables you to create multi-platform augmented reality (AR) apps with Unity In an AR Foundation project, you choose which AR features to enable by adding the corresponding manager components to your scene When you build and run your app on an AR device, AR Foundation enables these features using the platform's native AR SDK, so you can create once and deploy to the world's leading AR platforms (November 15, 2022) Available: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@5.0/manual/index.html [16] AR core: The Android devices support ARCore via Google Play Services for AR, which enables augmented reality (AR) experiences built with an ARCore SDK You can access to this link for more information and see the listed of every Android devices that supported for AR (December 9, 2022) Available: https://developers.google.com/ar/devices?hl=en [17] Localize: Read file CSV in Unity (Nov 13, 2018) Available: https://www.youtube.com/watch?v=xwnL4meq-j8 [18] DesignPattern: A singleton design guarantees that a class has just one instance and offers a universal method of accessing that instance (December 22, 2017) Available: https://viblo.asia/p/hoc-singleton-pattern-trong-5-phut-4P856goOKY3 In addition, our group will also add a youtube link to help people have a closer look at Singleton (March 25, 2017) Available: https://www.youtube.com/watch?v=CPKAgyp8cno 58 [19] Parse explaination: Means to examine a certain object In computer science, it is frequently used to refer to the code that a software reads Available: https://tudienso.com/thuat-ngu/index.php/parse-la-gi/ [20] Reference icon Our group get icons from these two link below Available: Link 1: https://www.flaticon.com/ Link 2: https://assetstore.unity.com/ [21] CNC information (January 1, 2023) Available: https://vi.wikipedia.org/wiki/CNC#:~:text=CNC%20%E2%80%93%20vi%E1%BA%BF t%20t%E1%BA%AFt%20cho%20Computer,hi%E1%BB%87u%20chuy%C3%AAn%20 bi%E1%BB%87t%20theo%20ti%C3%AAu 59 S K L 0