Technical model for management and interaction between teachers and students in smart ClaSSTOOMS.. Technical model for management and interaction between teachers and students in smart C
Webcam - 5-5 Sen OO
Image sensor: 1/6" CMOS Resolution: 640 x 480P Wide angle 60 degrees Frame rate: 30 fps
KIT Development Environment . ¿-¿- 5< s5 +++x+xexererrkeeerrre 36 3.2.2 Programming languages - - ¿525 St ekekerrrkrkrkrrree 37 3.3 Technical model for management and interaction between teachers and
OpenCV is the leading open source for image processing and computer vision Written in C++ so the calculation speed is very fast and also supports the interface for python so it is very convenient to use Supports a lot of handy features such as cropped photos, rotate photos, convert photos, read and write photos,
CUDA is a parallel computing architecture developed by Nvidia Use common languages and through CUDA to perform calculations on the GPU.
In terms of algorithms and image processing programming languages, my team uses C++ as the main programming language for the the treate C++ is an easy- to-use language and is supported by powerful libraries in terms of processing speed and adaptability.
In terms of algorithms and programming languages used in backends, my team uses C# as their primary language C# is a NET Framework-supported language and C# is also easily the same language as C++ used in image processing. For database management, groups that use SQL languages are a common language in data queries Any relational database system must meet such as Oracle Database, SQL Server, MySQL
3.3 Technical model for management and interaction between teachers and students in smart classrooms.
Frontend Backend Teacher’s Device en _=—-
Figure 3.4: Technical model for management and interaction
In this thesis, my team has successfully built a technical model to serve management and interaction between students and teachers in smart classrooms In order to have a connection between teachers and students, there needs to be a management system that makes interaction easy.
In fugure 3.4, we build components including: Database management system - this is where all information of the system is stored including teacher information, student information, time schedule, check-in information, all are stored in the cloud Database of all students of the class Algorithms for obtaining information from the backend Interactive student interface (frontend) Interaction between students and teachers through the internet and IOT Communication devices include nano jetson kit, mice, keyboards, and monitors And finally, the indispensable component is the server - used to store all the information of the database, frontend and backend management system
Teachers interact with the system by successfully logging in each time, the system will send the necessary information about the teacher's working application such as: lesson information, automatic list of students’ check-in, the list of all students in the lesson, images/videos of students during the learning process and take the test on the device, teachers can also interact with the system such as adding learning materials, creating online tests From there, teachers can be more proactive in managing their students.
Student side oo eseescsesesesssneseseesesesesesessevesssesssessesesesesesseeeneses OS, 3.3.3 Admin side
Students interact with the system by successfully logging in each time, the system will send the necessary information about the student's studying application such as: notification of successful check-in, the list of all subjects that students participate in, the schedule of the student's week, personal information from the student, the list of downloadable learning documents From here, students can easily access learning resources from teachers without too much paper, as well as access to modern technology through learning and online testing.
Admin has tasks such as, adding, deleting, editing objects maintain a stable operation and maintenance system,
In order to get a smart table, it is necessary to have a management system, interaction between teachers and students My team has created a basic management and interaction system The system has advanced and novel functions such as face recognition, automatic check-in, live streaming, downloading learning materials, viewing personal information, identifying throughout the learning process and the exam process,
3.4.1 Facial Recognition and login method
In figure 3.5, using the algorithm Multi-task Cascaded Convolutional Networks with input is image containing face to be identified and output is image with bounding boxes consisting of 5 landmarks (eyes, mouth and nose) Using the algorithm facenet to extract facial characteristic After identifying the face that matches the face present in the database, log in to the main interface of the application.
When and the system, students need to enter a username and choose how to unlock the system with facial recognition or enter a password.
In figure 3.6, if student choose to unlock the system with facial recognition, the camera will be started and captured the student's image, then processed through the facial recognition program If the student chooses to unlock with a password, the student will have to enter the password of the account he is granted, if successful will log in to the system, and if entered wrong, it will still be on the login method options page.
Image from webcam|[——*|Algorithm MTCNN [——®| Algorithm Facenet [——>| Output
Figure 3.5: A facial recognition overview diagram for logging into the system
Enter your sign-in account name and tap sign in button
Login using identification method ogin by entering password
Open the main Interface of the application
Figure 3.6: A diagram of login system
3.4.2 Main functions and flow of smart table websites
In figure 3.7, after successfully logging in with your face, the system redirects to the student's interactive interface page, which has key functions such as:
View personal information: Each student has their own personal information including the name and account of the system login system as a quiz, full name, phone number, date of birth,
Automatic or self-tinged admission: After successful login, the system will automatically make a list, in case of unsymed or the system has malfunctioned, it is possible to re-enroll by self-sy entry.
View course schedules and subject lists: Students can view the schedule for their current lesson and self-study there (if necessary) and can see the full schedule of their course, and students can see the list of current subjects of the student.
Download learning materials: In each subject, there will be separate sections of learning materials for that subject, students can download the machine to serve the viewing, exchanging and learning of students. View lectures directly: In each lesson, students can view the teacher's live lecture (if any) by clicking on the button to view the lecture, the system will redirect to the teacher's live lecture page.
[Use Axios to in-order
W hen the login is APIs from sever [Use Axios to in-orde Vhen logging into the)
APIs from sever web running scripts for automatic entry
Use Axios to in-order) APIs from sever successful, go to URL:
T Create a roves [create a table to contain dat ik - The | Create a router o save save the call data b ia the section: The
Get URL from the api hen the website renders schedule will show the | [te call data from the api
Use Axios to call Get data Ta the table create 2 butions or] eee coal api from sever from router students to click on the check-in} | me: ao Meera’ |
I and watch the teacher lecture | | rudents will manually | score
Fillin the Name in | [Create a table to contaia| Create ob he API link to get the} — | data when the website eee ee correct student's data renders : *
When clicking the | [When clicking the button ] button, you will senda | | to watch the lecture, go When clicking on ticket to sever to set || - to the teacher’ live Display student the subject card active stream website information including
|basic contact information| Display details of the subject] | Displays a dialog including: Subject box informing information, teacher name, students of document list, doing | kueeessilcheck-in exercises
‘When clicking on any | | When you click the document, it will be button to do your work, downloaded to the | | you will go to the online
‘machine work siteFigure 3.7: A diagram of the main functions in the smart table system
3.4.3 Main functions and flow of the website take the online
In figure 3.8, in each student's subject, there is a button to navigate the next axis to the online test site, where students can take the tests that the teacher creates.
At the start of the test, the camera will be turned on to show the student's presence and capture the student's face while taking the test to combat fraud.
(—~< ] < Do the exam questions bank
Add, delete, edt i Make the u Multiple choice teachers, students, exam Exam papers by
Figure 3.8: A diagram of the main functions in the exam system 3.4.4 Automatic student attendance
In figure 3.9, each student after successfully logging in with their face, the system will automatically take attendance of the student's presence Students will receive a successful attendance notification, the data will be sent to the database management system for management Then, the teacher gets the attendance list from the database management system to display on the teacher's podium system In case the teacher has not received the student's attendance, the teacher can ask the student to take attendance by clicking the "Attendance" button in the current subject section without exiting the system After successful attendance, students continue to study on the system.
Take attendance manually at the
"Attendance" button in the current subject section Yes ,
Successful attendance message appears in the system interface
Figure 3.9: The system's automatic attendance process
3.4.5 Identify the right students throughout their studies and take tests on the system.
Learning process The process of exams tivity identification procé (student photography)
3 minutes/1 image save the image in the device's folder Ỷ send data to the server
| teachers can view this image data
Figure 3.10: Student identity process diagrams throughout the learning process
In figure 3.10, the student's identity process will begin as soon as it is successfully logged into the system every 3 minutes, the system will take a photo from the student's webcam, the process of repeating until the lesson ends this image will be saved automatically to the folder of the device, after which this image will be sent to the server and the teacher can use this image to check the presence of students in the lesson.
Similar to when doing online assignments, during of the test, the webcam will always be on to monitor the student's work progress After the process of completing the test, students press the "Submit" button, all student identification will be sent to the folder for management.
The strength of this identity algorithm is that all images/videos will be transferred to the server as soon as the system captures students during learning as well as doing online testing Webcams will be turned on continuously to obtain student images continuously every 3 minutes Teachers when viewing or grading the test will also apply this identity algorithm to ensure the right teacher marks the test.
My team will conduct experiments on Jetson Nano.
4.1.1 Train model process (Test case 1)
- Dataset details: The total number of photos is 11498 photos of 5749 people My team conducted test on 2000 photos.
- Test image specifications: Tested photos are between 10 and 20KB in size, the image format is JPG with a resolution of 250x250
- Image source: Labeled Faces in the Wild (LFW)
- Experimental process: All 2000 images are identified with the following ratio: the number of successful identification shots is 1657 reaching approximately 83%, the number of incorrectly identified images is 343 reaching approximately 17% as figure 4.1 below.
Through the train model in section 4.1.1, the recognition rate is quite low at only about 83% So my team tested an additional data set 2 with higher image quality and was tested directly by camera with a data set that includes images of team members Testing is conducted in 4 cases The assessment will also be based on facial recognition speed.
Figure 4.2: Face recognition of students Identity name Avatar banhngocninh tử, daoduykhang bl vuminhhoang ie | nguyenkhanhminhtan @
Table 4.1: Set of photos representing training data for facial recognition
* With the tests tested below, if the system does not recognize the face in the data set, it will display as “New Person” If more than 5s for one identification will display as “New Person”.
EXPERIMENTS AND EVALUATIONG .c:csccscssessesssseeseseeseessees 47 4.1 Face recognition experiment 0.0.0.0 esseseeseseseeseseseessneseaeseeeeeeneeeae 47 4.1.1 Train model process - 1S TT HH 47 4.1.2 Actual r€COETIfOH - St HT H1 H1HHu HH 48 K6 " 51 4.2 Interactive experiments with databaSe€s + Street 51 4.2.1 Student admissions procesS cccccsceesesseseeseseesesseseseseeseseeseseseeseeness 51 4.2.2 Get information from the database - ¿+ + sssterkreketerrkrke 53
The process of exams with the method of identity
_ banhngocni banhngocni banhagocni banhngocni banhngocni banhagocni banhngocni banhngocni nh2319- nh2#19- nh23201 nh23204 nh23-207 nh23Zi nh2321- nh2321- Desktop 53jpg 56jpg ipa jpg ipa 30jpg 32jpg 34jp9 banhngocnl banhngocni banhngocni banhngocni banhngocni banhagocni banhngocni banhngocni nh2321- nhZl8- nhz513 nhzs19- nh2519 nhZ519 nh?5-19- nhữ519-
35jpg 35jpg 38jpg 42jpg 44jpg 45jpg 46jpg 47jp min 2 die fs Bs i banhngocni banhngocni banhngocni banhngocni banhngocni banhngocni banhngocni banhngocni nh2519- —nh25-19- nh2519- nh2521 nh252i- nh-25-21- nh2521- nh2521-
49,09 SLjPg 53jpg 27jpg 28jpg 29jpg 30jpg 31jpg
Rewer ea anhngocni.banhngocni banhngocni banhngocni banhngocni banhngocni banhngocni banhngocni nh25Z1- nh25222 nh25223 nh27l4 nh2712 nh71# nh2714- - nh2714- 32ip9 ipo ipa 13jpg 16jpg 28jpg 3tjpg 34jpg
KT Bs E1Et ie E?ứ đaoduykha daoduykha daoduwkha daoduykha daoduwkha daoduykha đaoduykha - ninh Ng2521- ng?521: ng252I- ng22I-° ng2521- ng252i- ng2521- - 27061428
4ãjpg 44jpg 45jpg 46jpg — 47jpg 50jpg 53jpg png Figure 4.9: Images of students when study or make test captured from the system
To ensure that students are participating in the lesson, the system will automatically take photos of students during the course of the lesson every 3 minutes. All these images will be transferred to the server for management by teachers and administrators In Figure 4.8, the interface when students do their work online, images or videos will be saved and sent to the device folder In Figure 4.9, show us all the image data stored in a separate folder to transfer to the server.
Figure 4.10: The main interaction interface with students
Features Team's smart table In the world
Facial recognition system v x Automatic attendance v x Interact with students v
Watch lectures directly from teachers v x
Identify students throughout their v x studies and take online tests Table 4.6: Comparison between smart team table and smart table in the world
Through the comparison table, we see that in the smart table that my team implements there are many outstanding and new features compared to the types of tables in the world today My team's smart table fully meets the entertainment needs as well as has additional advanced functions such as face recognition, automatic attendance, online identity throughout the learning process and online testing,watching the lectures directly from the teacher, downloading learning materials,
CONCLUSION AND ORIENTED OF DEVELOPMENT
Conclusions 5 57 S.1.1 What we've dON€ 5-5552 t2 223 1 1212 1212111 11112121111 111gr 57 2 Things that haven't been done ¿5-5 25+ 5++x++cxexeresrrerrrre 57 5.2 Oriented developIm€ni( -¿- - - + +11 110101 1 1111111 58
Over 5 months of researching and evaluatin the method from searching for algorithms on facial detection and recognition to methods of connecting the database to the backend to display on the frontend My team integrated functions such as facial recognition, creating a database to manage students, creating an interactive interface between smart tables with students, student check-in after successful identification, viewing online lectures, and especially the ability to detect fraud in the learning process using the identification method is a new point in this thesis.
- Having background knowledge of CNN, MTCNN, Facenet, NET Framework, SQL technology
- Learn more about new facial recognition algorithms and methods.
- Use more proficient in managing, accessing databases as well as having basic knowledge about database management system.
- Get the foundational knowledge of backend to manage and communicate with databases and frontends.
- Can be applied to practice if further developed
- Facenet algorithm is the foundation algorithm so there are many sources of reference
5.1.2 Things that haven't been done
- In this thesis, my team have not applied much and expanded machine learning [3] and deep learning [2] applications.
- The database of a person's characteristics in facial recognition is limit.
- Have not solved the problem of facial recognition at the national level.
- There is no function to prevent attacks in cyberspace.
- The data set of students is too large, there will be unwanted errors and difficult to link the data together if the group is not sufficiently manpowered.
- The network connection is oriented towards access.
Through data training, next development orientation will be to collect more images to extract a person's characteristics data to increase the accuracy of the facial recognition system.
Next, my team will handle the face recognition problem on the cloud computing platform then apply more and expand the application of machine learning and deep learning to make the facial recognition system more accurate.
Finally, my team will embeded the web platform on the app, study integrating security and safety functions in cyberspace so that it can enhance device security to the highest level and bring project to the internet platform without running local My team will continue research to be able to transfer student identification video data directly to the server without saving it to the device's folder