Object detection is computer vision and image processing technology base on ability of computer systems and software to locate objects and recognize each object in animage or video.. Obj
Building and deploying real-time detection and monitoring PPE system: 37 1 Install Jetson nano OS and necessary lIbrarles
Install Yolov4 from Darknet framewWOrK: s s ô<< xcsss++e+ 38 3.3.3 Implement and run detecting and monitoring system
Open terminal (CTRL+SHIFT+T) e Update libraries:
$ sudo apt-get update e Export CUDA path:
38 e Download YOLOV4 and Darknet framework: e Edit Makefile to enable GPU, OpenCV by change value in Makefile
3.3.3 Implement and run detecting and monitoring system:
We use our trained PPE model (yolov4-tiny.weights) for detection The system will run to detect PPE by capturing video from worker are coming closer entry of construction site Frame captured by camera will compare with PPE model which was collected after training The results of detection PPE will be shown on screen or laptop screen if using VNC.
F = = "H | ® | nee Sed netllcadiug oo yereee
Figure 3.20: Real-time detecting and monitoring system diagram.
For the user interface At the beginning, system run the camera The system will show NOT PASS state on screen If camera detects full of 4 type PPE (hard hat, goggles, safety vest, gloves) at the same time, it will change to PASS state After that, the system will capture images and results of detection (including bounding boxes of the objects and % accuracy rate of them) at that time and save them to storage Also, it will record time of detecting into log file and loop again to begin The system will end if user press end key.
Detect full of PPE*s object? Show “NOT PASS” on screen
Figure 3.21: Flow chart of detecting and monitoring system.
Packaging camera box for detection system? - ô5< ô+++ 42
Camera box is packaged suitably for on site or carriable device For on site, it is placed at the entry of construction site and use adapter connected with power source to run system For carriable device, it connects with 3 cell battery:
- Camera used for camera box is Raspberry Pi v2 Camera.
- We choose 2 power source for Camera box: o Adapter Philips 5V 4A
42 © 3 Cell battery 75C 7.4V: we connect it with low voltage converter circuit to collect 5 - 5.5V suitable for Jeson Nano’s power.
EXPERIMENTS - LG LH ST tk ky 44 4.1 Evaluate trained PPE model .- - 5 << * E2 ** 3 £**E£+vEeeeEeskeeseeskrre 44 4.2 Testing in real CAS©: .c- 2c 2c HH HH HH He 45 4.2.1 Full of light condition accuracy r€SuÌ(S: ô ô=<sô+sc+sx++ 46 4.2.2 Lack of light condition accuracy resuẽ(S: . ô<< ô+<<+ 50 4.2.3 R0
The metrics we use to evaluate the trained model’s accuracy are Precision, Recall and
F1-score First, we define meaning of TP (true positive), FP (false positive), FN (false negative). e True positive results when an object is correctly identified with IOU
(Intersection Over Union) between ground truth and bounding box predicted to be greater than a threshold. e False positive is the result of the wrong identity, which means that the wrong class can be identified (IOU < 0.5). e False negative is the result of miss identification, which means that the object appears but is not recognized.
“Precision” measures the level of accuracy which is the prediction of the model ie the predicted percentage of the model is correct:
"Recall" measures how well the pattern finds all positive patterns:
The Fl-score is the harmonic mean of Precision and Recall (assuming these two quantities are nonzero).
We use results evaluation model from Figure 4.1 after complete training:
44 detections_count = 5183, unique_truth_count = 1525 class id = 9, name = hard hat, ap = 94.76% (TP = 718, FP = 86) class_id = 1, name = goggles, ap = 84.73% (TP = 158, FP = 52) class_id = 2, name = safety vest, ap = 98.15% (TP = 438, FP = 15) class_id = 3, name = gloves, ap = 83.66% (TP = 111, FP = 42) for conf_thresh for conf_thresh
8.25, precision = 8.88, recall = 8.93, Fl-score = 8.98 8.25, TP = 1417, FP = 195, FN = 108, average IoU = 71.94 %
IoU threshold = 58 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@@.5@) = 8.983238, or 98.32 %
“-points 11° for PascalVOC 2887 (uncomment “difficult” in voc.data)
"-points 8` (AUC) for ImageNet, PascalVOC 2818-2812, your custom dataset
Figure 4.1: Evaluate of mode using testing set
Hard hat objects and safety vests both have highest average precision because they have largest images for training (3951 for hard hat and 1972 for safety vest).
We set up environment for placing camera box We placed camera box at the entry to test the ability of detecting PPE of the worker when worker come closer to the camera box The camera box was set with 1.6m of height We tested it in different light conditions consist of: full of light, lack of light The person wearing PPE was standing at each distance: 1.5m, 2.5m, 3.5m We tested 10 times each distance with only one person The resolution of the camera was set at 1280 x 720 and the frame rate: 30 FPS.
Figure 4.2: Test scenario of detecting ability at different distances.
4.2.1 Full of light condition accuracy results:
After testing in full of light condition, we collected charts of accuracy rate of 4 PPE’s object for each distance with each light condition:
==———Hard hat ====Goggles ====Safety vest ====Gloves
Figure 4.3: Chart of accuracy rate of PPE’s object in full of light condition.
- At distance 1.5m the system almost detects all of 4 PPE’s object with high
Figure 4.4: Accuracy rate at 1m5 (full of light).
- At distance 2.5m the accuracy rate of 4 PPE’s object changes too much The accuracy of gloves reduces a little bit.
- At distance 3.5m the accuracy rate of gloves reduces from 75% to — 55% The hard hat, safety vest and goggles have still kept highest accuracy rate.
Figure 4.6: Accuracy rate at 3.5m (full of light).
- Atdistance 4.5m, gloves and goggles became difficult to detect The accuracy rate of goggles reduced a lot to under 50% while gloves was unable to detect.
— 1.5m — 3.5m is the best distance that the system can detect 4 PPE’s object When the distance is getting further and further away, gloves become more difficult to detect because gloves are getting smaller and smaller Besides, we have to raise our hands and show gloves so the system can easily detect This is the disadvantage of our system.
Furthermore, we also test how far does our system still can detect 2 object safety vest and hard hat:
- At 7m, the accuracy of hard hat reduced a lot from 99% to 79%, safety vest
- The limitation of distance that the system can detect object is 25m At 25m, hard hat becomes unable to detect The system still can detect safety vest with
~82% accuracy rate Over that distance, the system can no longer detect anything.
Overall: Hard hat and safety vest give good result for detecting but not good with gloves when they are not in right position and far distance We have to raise hand so that camera can detect gloves Sometimes we have some false positive of we not config threshold to 0.4, the system will detect wrong objects into hard hat which have similar feature with PPE like color (matches with safety vest) or shape.
4.2.2 Lack of light condition accuracy results:
In lack of light condition, the results of detecting are very bad:
Figure 4.10: : Chart of accuracy rate of PPE’s object in lack of light condition.
- Atdistance 1.5m, the system can detect hard hat and safety vest with accuracy rate: 45% - 55% for safety vest and 84% for hard hat The detecting ability of the system becomes more difficult to detect.
Figure 4.11: Accuracy rate at distance 1.5m (lack of light)
- At distance 2.5m, the system can only detect hard hat but accuracy rate only
Figure 4.12: Accuracy rate at distance 2.5m (lack of light)
- Out of 2.5m the system no longer detects anything.
— The system gives bad results in lack of light condition Gloves and goggles can not be detected in this condition The system can detect safety vest and hard hat but the accuracy rate is not high.
Speed of the system when we run detection is stable at 10 - 12 FPS and it is not
Figure 4.13: Speed of detection when 6 objects are detected.
The speed will reduce if we change the resolution of camera and frame rate Besides, changing network resolution 416 x 416 to bigger resolution will reduce the speed too.
CONCLUSION AND FUTURE WORK .sĂcSecseeersee 53 5.1 COnCẽUSIOT 2G TH HH gi 53 bố ha e
Disadvantages escccesccesseceseecescecsseeeseecseecsaeecsaeceseeceaeecsaeecsaeeeseeseaeereneeees 53 3.4 Future Work: oo eee cscssecssecseessecesecseesseesseessecseessecssecssecseeseesaeessesseeeaeens 54 3581545105070
- The system shows bad results in lack of light condition In full of light condition, the system can detect gloves only if they are placed right position (raise hand) The accurate rate of gloves is still not high (60-70%) and reduced
5.4. fast when we move them far away The limitation of distance that the system can detect gloves and goggles is another disadvantage.
The system can only detect specified gloves and goggles, there are still a lot of type of gloves and goggles that system could not detect Besides, it could not detect between glasses and goggles so we accept it as exception.
We have not still developed GUI to interact with user.
Basically, our system still has not had tracking feature and it could not check if PPE’s objects are being worn by worker or not.
Moreover, the speed is a little bit slow although it is acceptable for real-time detection.
First, we will make current classes (4 kind of PPE) more diverse by collecting more dataset to increase accuracy Next, we will expand the dataset with other personal protective equipment and conditions for developing better system Furthermore, we will build addition function or even combine with face recognition for construction complete attendance system.
[1] Ministry of labor-invalids and social affairs (2019) “Notification notice of the situation of labor accidents in 2019”, 2019
[2] Z Zhenhua, P Man-Woo, E Nehad, “Automated monitoring of hardhat wearing for onsite safety enhancement”, International Construction Specialty Conference of the Canadian Society for Civil Engineering (ICSC), 2015.
[3] B E Mneymneh, M Abbas, and H Khoury, “Automated hardhat detection for construction safety applications,”, Primosten, Croatia, http://www.sciencedirect.com/science/article/pii/S 18777058 17331430, 19-22 June 2017.
[4] Wikipedia Team, “Object detection”, https://en wikipedia.org/wiki/Object_detection, Accessed 20 — 10 -2020.
[5] ODSC Team, "Overview of the YOLO Object Detection Algorithm," https://medium.com/@ ODSC/overview-of-the-yolo-object-detection-algorithm-
[6] Z Jiang, L Zhao, S Li, Y Jia, “Real time object detection method based on improved YOLOv4-tiny”, arXiv preprint arXiv:201 1.04244, 2019.
[7] OpenCV, “About OpenCV”, https://opencv.org/about/, [Accessed] 22 — 11 - 2020.
[8] Wikipedia Team, “CUDA”, https://en.wikipedia.org/wiki/CUDA, Accessed: 22 —
[9] Pjreddie, "YOLO: Real-Time Object Detection, https://pjreddie.com/darknet/yolo/, [Accessed] 02 — 11 - 2020.