Khóa luận tốt nghiệp Kỹ thuật máy tính: Bục giảng thông minh trên nền tảng IoT ứng dụng trong lớp học thông minh

THANK YOUTo successfully implement the graduation thesis section with the topic “SmartPulpit with IoT for smart classroom “in addition to the efforts of the group - wealways receive dedi

NATIONAL UNIVERSITY OF HO CHI MINH CITY

UNIVERSITY OF INFORMATION TECHNOLOGY

FACULTY OF COMPUTER ENGINEERING

VU MINH HOANG

NGUYEN KHANH MINH TAN

GRADUATION THESIS

Smart Pulpit with IoT for smart classroom

Buc giáng thông mình trên nền tang IoT ứng dụng trong lớp hoc

thông minh

ENGINEER IN COMPUTER ENGINEERING

HO CHI MINH CITY, 2021

NATIONAL UNIVERSITY OF HO CHI MINH CITY

UNIVERSITY OF INFORMATION TECHNOLOGY

FACULTY OF COMPUTER ENGINEERING

VU MINH HOANG - 17520523

NGUYEN KHANH MINH TAN - 17521015

GRADUATION THESIS

Smart Pulpit with IoT for smart classroom

Buc giảng thông mình trên nền tang IoT ứng dụng trong lớp học

thông minh

ENGINEER IN COMPUTER ENGINEERING

INSTRUCTOR

PhD NGUYEN MINH SON

HO CHI MINH CITY, 2021

INFORMATION OF THE GRADUATION COURSE GRADING BOARD

The board marks the graduation course, established under Decision No

day of the President of the University of Information Technology.

THANK YOU

To successfully implement the graduation thesis section with the topic “SmartPulpit with IoT for smart classroom “in addition to the efforts of the group - wealways receive dedicated help from teachers, colleagues, friends, and many

agencies and organizations

First, we would like to express our sincere thanks to the University of

Information Technology — VNU Ho Chi Minh City, the board of directions, and

teachers at the Faculty of Computer Engineering for creating the best conditionsfor us to have a dynamic and effective learning environment throughout thelearning process at the school Thanks to the guidance of our teachers, we havelearned and accumulated the foundational knowledge of Computer Engineering

We especially thank Ph.D Nguyen Minh Son and PhD Do Tri Nhut for always

supporting and guided us throughout the implementation of the topic We learnedmany useful and necessary knowledge, as well as having the opportunity to

practice and cultivate the above knowledge

Finally, we sincerely thank our families, parents, who are always there tosupport us, help us have time to study the topic and wholeheartedly support us.During graduation, despite trying to complete the topic, it is inevitable that there

are some faults We are very welcome and sincerely thank the comments of

teachers and readers for the thesis to be improved We hope this thesis will be auseful reference for those who are interested in relevant areas of the subject

Ho Chi Minh City, July 2021

Student Vu Minh HoangStudent Nguyen Khanh Minh Tan

Sincerely thank you!

TABLE OF CONTENTS

Chapter 1 INTRODUCTION ssessssssssssesssssusssssssnsssssssusssssnsssessssssssssssseessssusssssnsetsesssussnsssts 2Chapter 2 LITERATURE OVERVIEW .sesssssssssssssssssusssssnsssssssssssssssssesssssssssssssstessssisnssssssD2.1 Related Research eecerrriiiiiiiriiiiiiiiiiiiririiiiiiiiirirneeD2.1.1 International research -eeeecrrrerreseeertrrrrrrrrirrreeD2.1.2 Domestic research - e.-cccerreeeeeeeeeeeerrreeeesure LA2.2 Smart Pulpit :-:-ceessitirtrtririrrie LZ2.2.1 Smart classfOOm OVeTVi€W em LD2.2.1.1 What is Smart ClassroOm? eececeeeeeererrreeseerrrreeee L 72.2.1.2 Benefits of smart ClassrOOms -cc-ececcceeeeeeseeereeeeeeee L 72.2.1.3 Benefits for teachers eeeeeeeeeererrrrrsrrrereeee L 7

2.2.1.4 Benefits for Student -ccccceeeerereeeeeesersre LO

2.2.2.1 What is a smart pulpit? cece LO2.2.2.2 Teacher Recognition SyS(em -. -c-e-eeccvecveseerrerreeee ee 20.2.2.2.3 Automatic attendance ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssses LO,

2.2.2.4 IOT System

2.2.2.5 Aniti-fraud SYStM sssssssssssssssssssssssssssssssssssssssnsssssssssnsssssnsssssssssssnssssssssessssssss 2D

2.2.2.6 Lecture presentation system

Chapter 3 SYSTEM DESIGN eeeeerrririirrrrrrrrriiirirrrirririu OD3.1 System Architecture

3.2 Hardware ArchiteCture eeerarrrrrrrrar OA

3.3 Software Architecture ws OD3.3.1 Facial recOgnition sssssssssssssssssssssssssssssesssssssssssssssssssessssssssssssssssssssssssssssssssssssssss OD

4.2 Experimental result e- eeeeerrrreeerrrrrrrrrmoereooo 4

4.2.1 Facial reCOgnition e.cceeccccceccceeeeeeseeeeeo 40)

4.2.3 Streamig ccsssssesescssssssnseesesssesssssssseessessssssnssesesesessssssssmessessssssssssaseessessssssnsees 494.2.4 Auto attendance servers D2

4.3 Experimental conc]uSÏOH ccccccsstccsevexvvseeerietrirtkettrrrrrirrririirsrrioe DDChapter 5 CONCLUSION AND ORIENTED DEVELOPMENT D75.1 COMCTUSION ecscsesssssssssssssssssssssssssssssesssssesssesssesesssesessecsesesesseesesesnesseessesessssssssssensssessssensssesees OZ5.2 Oriented development

PICTURE CATEGORY

Figure 2 1 Picture about Smart class of ANU_V[12] , -c‹«eveseseeecrercee

Figure 2 2 Smart class of Blingnetworks[ 13] ssscccssssssssssssssssssssssssssssssssssssssssssssssssssssssssssesstete

Figure 2 3 Basis setup for Smart CÍdsS[ Ï^|] eccceee«eeecerkekikeerrsetriiiiiiririrrierFigure 2 4 Smart class with level Personal Devices[14 ] css

Figure 2 5 Smart class with level Virtual Reality[ 14]

-Figure 2 7 Model smart class of Alo360 (tt)[ 15] e e.-cccceeseseeeeeeecervevveseeererrrerFigure 2 8 NVIDIA Jetson NANO[2 J crscsssssssssssssesesessnsaiesssesessssnsuesssesesessnsnseesseesessnnnnneeseeteFigure 2 9 Graphic about how TensorRT WOrk[2] -.‹eeesecccccccsevveeserserrer

Figure 2 10 mtONN SITFIICEUF€[ Ố -ccceeceeeeeccecveeeeeeeetirtrtrtrteeeeiriirrtirtkrirsrrrirrrer

Figure 2 11 Three step of model mICiNN[Õ] cccccvvceeetserrerkekeeeesrrsrrterFigure 2 12 Fully Connected Layer | Š | eeec-ccccvvveeeeteeecerkekvvveeesetrerkkkkteraeisrrter

.26.27

Figure 2 13 Triplet-loss and learning[5]

15

1619

21

2223

24

30

31

32

Figure 3.

Figure 3

Figure 3

Figure 3

Figure 3

Figure 3

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

Figure 4

3 Facial jack recognition ALgOrithm vss OD

4 Algorithm diagram for facial recognition

S TOT algorithm đỉA9TGHH cài D7 6 điagram [OT SIY€QIHÌH eeeeec-cccccccceeeeetrererteeteeeerrirrrrrrrirrirrrssoooe OO 1 Successful identifying image (name Will be the SAME) 40

2 Images when the recognition fails (name will be different) +1

3 Images in good resolution, good condition and near distance +3

4 Images in good resolution, good condition and far distance +3

5 Images in good resolution, bad condition and near distance swe +5 6 Images show identification after 200 times 7 Lights off when transmitting signal 0 from leVÌC€ ee ee-eeee.e A] 8 The light on when transmitting signal 1 from the deviCe 4B 9 Streaming lecture with 8 devices participating at the same time 49

10 Streaming lecture with 12 devices participating at the same time 50

11 Streaming video achieves HD with multiple conneCtiOns 5 1 12 If there are no students in the class, the are no image

13 If there are students in the class, the image Will appeqF Ð 2

14 Image of teacher subjects in the feature CXAM sissies DS

15 Teacher can create an exam during the lecture through feature exam

a 4

16 Teacher can edit questions after they've created them crus 5

TABLE CATEGORY

Table 4 1 Data table of dataset Ï, -ccccccceeeeeeeerkekexveeeterterttrterirrrsrrrrrirkeieerree 2Table 4 2 Data table of dataset 2 srsssssscsseresescsssesesssesssssesssesssssesssssssssssssssssssssssssssssesees OOTable 4 3 Facial recognition result of use dataset] for test case 1 +1Table 4 4 Recognition speed with 1000 times eee-c-ecc5ccSe+ -Ö42Table 4 5 Facial recognition result of use dataset2 for test case 5 with distance 44

Table 4 6 Recognition speed with 200 times in good condifiOH 44

Table 4 7 Facial recognition result in bad CONItION wissen 40Table 4 8 Recognition speed with 200 times in bad CONGItION 46Table 4 9 Data table results after 100 smart pulpit lights on and off via Wi-Fi 48Table 4 10 Data table results streaming 8 devices participating at the same time 49Table 4 11 Data table results streaming 12 devices participating at the same time

50

Table 5 1 Overview about the function of the smart PUIPit css D7Table 5 2 Overview of the hardware used in the smart pHÏDÍI - 2Ö

Convolutional Neural Network

Multi-Task Cascaded Convolutional Neural Networks

Web Real-Time CommunicationGraphics Processing Unit

Central Processing Unit

With the start of the 4.0 industrial revolution, the loT platform is being used in all

areas around us In the field of education, there is one such device that has been

integrated with this technology and that is the smart pulpit Smart pulpits have beenused for a long time in many famous universities around the world such as Oxford,

MIT with functions such as lectures presentations, devices control in the classroom

as well as being able to do all of that on the device

In Vietnam, there are a few products with similar features but not manyimprovements and the disadvantage are that the price is very high and cumbersome

to use, it does not bring high flexibility and has not been able to apply many

technologies in it According to our findings, there is no research published on

applying artificial intelligence to the pulpit to be able to easily identify teachers inclass as well as adding an improved function of combating academic fraud The

application of artificial intelligence can help prevent cheating in exams as well aslearning

At the same time, the implementation of the system on a compact device can help

teachers connect with students wherever they are expressing the flexibility of the

product and adapting to many different learning conditions, especially in the situation

of online learning is growing today

Based on the results of the survey, our team proposed to carry out the project study

" Smart Pulpits applying IOT platform to smart classrooms " The results of theproject will contribute to the creation of a domestic product with many improvedfunctions such as teacher facial recognition as well as validate students to prevent

fraud The project can be applied to all schools in the future

Chapter 1 INTRODUCTION

- Reasons for implementing the topic

To help teachers apply technology to teaching with simple and intuitive operations, applying 4.0 technology as well as AI Our team decided to research and develop a smart pulpit device with functions such as facial recognition, using IoT to

be able to connect devices in the classroom and especially to be able to stream lectures remotely in the current epidemic situation.

- Goal

Create a physical product in a smart classroom to be able to present a few functions such as facial recognition, device control in the classroom, automatic attendance, streaming,

To accomplish the set goal, the following tasks should be performed:

e Build a facial recognition model.

e Build an IoT system with inter-linked nodes.

e Build an image streaming system using WebRTC.

e Create a product interface.

e Integrating the above functions into the product.

e Building experimental environments and experimental products.

- Subjects of study

e Method of identifying the face in the photo.

e Facial recognition methods.

e MQTT protocol for IOT device control.

e WebRTC connection for lecture presentations.

e QT platform for product interface building.

- Scope of research

Content 1: Document research

e Study relevant documents and develop a plan to implement the topic.

e Study image processing algorithms related to the scope of topics.

e Evaluate the results of research works and the applicability to the problem

of the topic.

Expected results: Overview report and results of evaluation of related topics.

Content 2: Build a facial detection program in an image

e Pre-processing of photos before identification processing.

e Partition objects on photos.

e Recognize face area.

Expected result: zoning the face in the image.

Content 3: Build a facial recognition program

e Extract face embedding.

e Compare data with know faces in database.

Expected result: Facial recognition.

Content 4: Building a system that connects devices in class

e Create sever MQTT so that nodes can be public and subscribe.

e Client sends signals through broker to be able to transmit signals to other

clients.

e Client receives the signal sent and conducts control of the device through

the received signal.

Expected result: Remote control of devices via Wi-Fi.

Content 4: Build a slideshow system using WebRTC

e Create a broadcast sever.

e Presentation pulpit through sever created.

e Clients can connect to sever to access the streaming system through the

tables

Expected results: lectures are shown and students can watch live

- Topic limitations

e Achieve at least 90% Recognize success rate

e Voice recognition not yet been implemented

e Experimental lice with 2 devices and classroom items on a small scale

e Not able to stream sound yet

e Streaming with max connection is 15

e Build a demo program with a simple interface

Chapter 2 LITERATURE OVERVIEW

2.1 Related Research

2.1.1 International research

Smart classrooms have become increasingly popular in the world with the greatdevelopment of technology as well as being aware of the importance of educationtoday With the possible future trend, there has been a lot of research on smartclassrooms in the world, so we compare the pros and cons of this type of class Some

Smart classrooms are modern methods that provide quality education to the

students for better learning outcomes with the help of technology Traditionalclassrooms are losing their effectiveness as all schools are adapting smart classrooms

in their educational institutions Smart classrooms are equipped with the latesttechnology such as visualizers, smartboards, laptops/computers, Internet

connectivity, projectors, amplifiers, speakers, pulpits, microphones and many more

As you know, nowadays students are inclined towards gadgets and technology

such as mobiles, tablets, computers, etc They prefer to use technological tools forsharing and transferring study materials for their learning With this, learning is

becoming more fun and interesting for the students along with complete attention in

the classroom Smart classrooms are rapidly changing the way teachers and studentsare experiencing learning activities with innovative technology Technology is very

important today in the world and students have plenty of tools to gain knowledge

from

Based on the above ANU_V divided into 3 types of Smart classrooms

e Basic smart classrooms: equipped with gadgets like computers, laptops,

projectors, viewing screen, etc

e Intermediate smart classrooms: little advanced then the basic smart

classrooms with the addition of small pulpit and a control panel

e Advanced smart classrooms: have all the equipment from basic to

intermediate but with the advanced features

Impact of Smart classrooms on Students

Learning is fun when the subjects are interesting Smart classrooms are doing thesame thing by imparting knowledge alongside fun elements Quality education iswhat we look for in students Today's world is very competitive, and one needs tokeep up It does not take away the concept of traditional education but addsopportunities to the existing one Smart classrooms are opportunities that provide

students with quality education and help them in understanding concepts better,improving their reading and writing skills to achieve academic success They have

helped students in such areas as responsibility, time management, communication,

interaction, collaboration, retention power, online accessing resources, learning newtechnological tools, good academic results, etc

Impact of Smart classrooms on Teachers

As you know, with advanced technology teaching methods are changing andevolving for better For this, smart classrooms are the best option for the teachers.Teachers have developed an impact on smart classrooms for the following reasonssuch as interactive sessions, detailed synchronization, east maintenance, interactive

learning environments, accessibility of resources, good teaching methods,

confidence, time management and personal guide for the students

Future of Smart classrooms concept

The concept of smart classrooms is very effective and important for all schools

in improving the teaching and learning environment Everyone is impressed with theidea of developing smart classrooms in every part of the country so that all thestudents enjoy the learning phase in their lives The classroom should be made a

better place for students to learn

- The Future of Education is Here: Smart Classrooms’ on

Blinqnetworks.com [13]

What are Smart Classrooms?

SMARTBOARD FOR TEACH HQ ANIMATED CONTENT

0#

DESKTOP CPU

PROJECTOR

Figure 2 2 Smart class of Blingnetworks[13]

While we are seeing only a small percentage of futuristic education being implemented due to the pandemic, there is so much more potential in this sector According to Statistical, the e-learning market worldwide is forecast to surpass 243 billion U.S dollars by 2022 With evolving technologies such as artificial intelligence and virtual reality, we are looking at the possibility of smart classrooms

as early as two years from now.

A smart classroom is a digitally equipped classroom with a huge variety of teaching and learning methods using technology It could be something as simple as incorporating audio and visual components to a lesson, which many classrooms are doing these days (and have done so in the past as well) It could also be something

as advanced as performing a practice medical surgery using virtual reality Additionally, a smart classroom helps the teacher through technology as well There

are apps that have been designed to assist the teacher in lesson planning or even give them the ability to save an online taught class for future reference.

Advantages of a Smart Classroom?

Having a smart classroom can have its pros and cons The biggest pros is remote learning, which is a big ask for the current world scenario We are already at the threshold of smart education with the online classes we have going on today Pre- COVID, we saw some use such as pre-recorded classes for students with clashing timetables, or some form of interactive learning.

Some universities also use artificial simulations for engineering subjects such as automotive or aerospace Students can use artificial intelligence to simulate their models of various automotive components, allowing them to test out motor parts without actually having to build them Once the simulation passes a test, they can go

ahead and build a prototype This saves them time and resources, making the learning process far more efficient.

Disadvantages of Smart Classrooms?

Many parents and university students may be worried about the collaborative and

social aspect of smart classrooms With most of the communication being dependent

on technology, there could be a barrier in communications However, some experts

suggest that working in an environment that involves interactive learning through technology encourages collaboration through discussion and group projects.

The one major disadvantage of smart classrooms or technology taking over the education sector is limited internet access for students living in rural areas There are many challenges that are present in providing internet to rural areas such as physical

barriers (for example, hills) or limited growth and resources.

Are Smart Classrooms Actually the Future of Education?

We are growing digitally and when we are able to tackle the problem of internet access for remote places, smart classrooms can certainly be the future of education.

Although we are looking at remote education right now, it is not necessarily the future of education Smart classrooms can also be incorporated while students are present in the classroom through augmented reality, artificial intelligence, and virtual reality.

It is important to note that smart classrooms can be implemented without these advanced technologies as well Circling back to the beginning of this discussion, digital transformation happens depending on how we use technology Some players

in the education sector use technology that is easily available to them, such as computers, to create simulation software on desktops and teach students through that, which is also a form of a smart classroom It all comes down to what technological resources we have in hand and how we can use them to make a smart environment With the evolution of technology, different sectors will continue to transform and become "smart" in their own ways, including the education sector.

- What are Smart Classes? Types, Advantages and Disadvantages

onedsis.in [14]

What are Smart classrooms?

Smart classrooms use digital technology to enhance the learning process The various audio and visual aids enhance the learning process and make it more fun.

The possibilities are endless.

Specialized software is designed and optimized to the needs of a specific class,

unlike universal solutions The digital infrastructure enhances the teacher to students

and peer to peer communication because data streams are separated and structured.

An example of this will be the raise hand feature in video conferencing apps now.

Smart classrooms can also educate students about the potentials of using it As a result, they will resort to more innovative ideas after school.

10

A key part of classes in the part was the lectures, this is not the case now Interactive learning is proved to help students better Projectors and computers are used to exchange information Interactive touch screens help the students become immersed.

Types of Smart classrooms

Learning solutions have evolved and are smarter by the say There are various approaches you can take when making the school smarter The budget, the digital literacy of the students all influences it.

1 The Basic Setup

Figure 2 3 Basis setup for Smart class[14]

These are the most common ones used in schools It involves projectors, a computer, and a screen The system will be preloaded with lessons that utilize visual and auditory learning Animations and Videos are the most common mediums used

to convey information The lessons can be easily updated according to changes in the curriculum.

11

Figure 2 4 Smart class with level Personal Devices[14]

At the top of the basic setup, students can possess devices like tablets and laptops Personal tablets have replaced the simple digital slate because the cost of making is very low now Students can take notes, receive material and complete assignments and tests in the same device This also saves paper which is a plus for the environment.

3 Virtual Reality

12

By using a VR headset, students can enter a new world or a sandbox In a sandbox, the students can do experiments, especially physical ones to learn theories and incite the brain.

The contents of the VR can also be controlled by the teacher so that everyone experiences the same From a DNA molecule to the French revolution can be shown

devices E-books, pdf and other study material can be sent to the device eliminating the need for books.

6 Apps

Hardware side is not the only solution that is needed when it comes to smart classrooms The app development should be tailored to fit the wants of your class Kahoot for example has a very simple, uncluttered layout that younger people will like.

7 Communication Devices

You should use quality microphones (preferably wireless), speakers, and other gadgets to address the whole class more effectively.

8 IoT (Internet of Things)

Many textbooks now feature links and QR codes to its internet counterparts to

give more content to the students Attendance and progress can be tracked.

2.1.2 Domestic research

Currently, there are also some places in the country that apply smart classroom models such as VinSchool or RMIT, but there has not really been a specific study related to the benefits and disadvantages of smart classroom models but only focuses

on modeling and trading devices in the classroom We have cited an introduction to smart classrooms in the country as well as websites where you can buy smart classroom devices as below.

- GIOI THIEU MO HINH PHONG HQC THONG MINH VA MAN

HÌNH CAM UNG of alo360.com [16]

What is smart class?

Smart classroom is an ideal learning environment for both teachers and students because they are exposed directly to the most modern and advanced learning

equipment This smart classroom solution includes devices such as computers,

14

laptops, smartphones, Directly connected to the interactive display In which the system will integrate the best supporting software for teaching and learning,

teachers can organize the entire teaching and learning process in one hour of

in-person instruction or remote online teaching.

Benefits of smart classroom in an educational environment

Smart classroom is a solution to increase multi-dimensional interaction between teachers and students, as well as between students through group learning, group discussion with the decentralizing of classroom management software.

Teachers can manage and run the classroom, monitor lessons on the selected student's machine or all the student machines in the class with just a few simple operations on the teacher's machine.

Chat between individual or group of students with teachers through the off/open application controlled by the teacher himself.

Teachers can share images, videos, files directly from the student's screen to the classroom touch screen or to show on the machine other students.

15

e The examination of students' knowledge through tests, accurate time

management and honest results with learning level.

e Improve learning excitement, give teachers and students a lively, interesting

and highly effective learning environment, helping to significantly improve

the learning results of students The student's learning becomes simpler and more effective than his.

In addition, you can buy devices in the smart classroom model in the links as

below:

https://saomaiedu.com/buc-giang-thong-minh/

https://dienmaystar.vn/buc-giang-dien-tu-thong-minh-pk-220sr

16

2.2 Smart Pulpit

2.2.1 Smart classroom overview

2.2.1.1 What is Smart Classroom?

Smart Classroom is a classroom solution that integrates a full range of teaching and learning tools that come from connecting devices such as computers, interactive screens, smart platforms, sound systems, projectors, computers fully integrated with necessary data such as electronic textbooks, e-subjects in accordance with textbook standards of the Ministry of Education and Training And a management software that helps teachers organize the entire process of teaching and learning in an hour — including the interaction between teachers and students through the use of built-in devices.

Each of the above teaching tools is based on powerful, friendly, easy-to-use technology, but all are designed to work in sync to liven up teaching content, enhance learning excitement, bring teachers and students liveliness, interesting and effective

in environmental learning, significantly improving student learning results.

2.2.1.2 Benefits of smart classrooms

The traditional approach of listening to lectures and notes has not worked as expected as education grows Instead, smart classrooms are a solution that allows to enhance multi-dimensional interaction between teachers and students through group

teaching, group learning, group discussion through classroom management software.

2.2.1.3 Benefits for teachers

Teachers can run the classroom and track lessons on the student's computer or all

the students’ computers in the classroom with just a few simple operations on the teacher's computer.

More specifically, the teaching software allows teachers to put the curriculum or

compile E-learning lectures to present in front of the class or teachers can make

requests to each student from the Smart Interactive Screen The app also supports

17

students to interact directly with teachers by sending questions to teachers, tracking learning schedules or lessons anytime, anywhere with a tablet, with a smartphone even outside the classroom.

2.2.1.4 Benefits for students

Students can easily practice their listening, speaking, reading and writing skills

thanks to the high-quality audio system of the classroom, group discussions, topic discussions created through classroom management software The lessons also become more interesting instead of looking at the books as before.

2.2.2 Smart Pulpit

2.2.2.1 What is a smart pulpit?

Unlike regular pulpits that are only attached to one microphone or otherwise,

smart pulpits are even more equipped Not only does it serve as a standing position

for teachers, but it also has a large computer screen, control keyboard, speakers, and microphone Integrated into it is an IOT system for controlling devices in the classroom as well as artificial intelligence to authenticate teachers as well as the important improvement function is to combat academic fraud.

Integrating multiple functions as well as requiring high-speed processing requires

a powerful hardware to meet the requirements such as: Processing with real-time response speed, support in processing on GPUs instead of CPUs only like the Raspberry Pi Handling tasks such as identification requires handling it in parallel is also one of the conditions that makes choosing an embedded computer with a powerful processor to support it.

- To implement this device, we use hardware platform: NVIDIA Jetson

deployment of neural networks from the Computer Vision domain since it provides

472 GFLOPS of FP16 compute performance with 5—I0W of power consumption [Source] There are many tutorials that assure an easy start with the Jetson platform such as the Hello AI-World tutorials or Jetbot, a small DIY robot based on Jetson

Nano.

Jetson Nano Specifications:

e Model: Jetson Nano Developer Kit Carrier Board A02 (1 camera port)

e GPU: 128-core Maxwell

e CPU: Quad-core ARM A57 @ 1.43 GHz

e Memory: 4 GB 64-bit LPDDR4 25.6 GB/s

e Storage: microSD (not included)

19

e Video Encode: 4K @ 30 | 4x 1080p @ 30 | 9x 720p @ 30 (H.264/H.265)

e Video Decode: 4K @ 6012x 4K @ 30 | 8x 1080p @ 30 | 18x 720p @ 30

(H.264/H.265)

e Camera: 1x MIPI CSI-2 DPHY lanes

e Connectivity: Gigabit Ethernet, M.2 Key E

e Display: HDMI 2.0 and eDP 1.4

e USB: 4x USB 3.0, USB 2.0 Micro-B

e Others: GPIO, I2C, I2S, SPI, VART

e Mechanical: 100 mm x 80 mm x 29 mm

Although there is a disadvantage of not having Wi-Fi integrated, it can be easily overcome by using USB Wi-Fi to be able to get Wi-Fi as a regular device.

2.2.2.2 Teacher Recognition System

Building a system that uses artificial intelligence in the authentication of classroom teachers to unlock the pulpit To ensure the data security of teachers when switching teachers between lessons The system will recognize the face of the teacher in the class and compare it with the data warehouse to be able to determine

if the teacher in that subject is correct and will open the system interface if authentication is successful.

To build a teacher identification system we use technologies such as NVIDIA TensorRT, mtCNN and Google Facenet.

- NVIDIA TensorRT

NVIDIA® TensorRTTM is an SDK for high-performance deep learning inference It includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for deep learning inference applications.

20

TensorRT enables the optimization machine learning models trained in one of your favorite ML frameworks (TensorFlow, Keras, PyTorch, ) by merging layers and tensors, picking the best kernels for a specific GPU, and reducing the precision (FP16, INTS) of matrix multiplications while preserving their accuracy Note that for INT8 precision an extra calibration step is needed to preserve accuracy Since this significantly (at least in most cases) reduces the inference time and increases the resource efficiency, this is the ultimate step for the deployment of a machine learning model in robotics, embedded systems (with GPU), autonomous driving, and data

centers.

Layer & Tensor Fusion

Precision Calibration Kernel Auto-Tuning

MTCNN or Multi-Task Cascaded Convolutional Neural Networks is a neural

network which detects faces and facial landmarks on images It was published in

2016 by Zhang et al MTCNN is one of the most popular and most accurate face detection tools today It consists of 3 neural networks connected in a cascade.

The multi-task Cascaded Convolutional Networks (mtCNN) is a deep based approach for face and landmark detection that is invariant to head pose,

learning-21

1lluminations, and occlusions Face and landmark locations are computed by a

three-staged process in a coarse-to-fine manner while keeping real-time capabilities which

is particularly important in the facial recognition scenario.

P-Net R-Net

eek TT TỊ1[[Tta vi Conv: tice: 32 fy on 4

Conv ĐỒ Cony: 3x3 Cony: 3x3 ro 00v: 3x mV: NIC ony: 3x2 fully i

MP: 3a3 classification | | MP: 3x3 MP: 3x3 connevs UNS classiBcatien |

[ Conv: 3x3 Cony: 3x Conv: 343 Conv 2x2 fully | |

MP ì MP: 3 MP Cc onnect ( face classification

(NMS) to merge highly overlapped candidates.

Stage 2: all candidates are fed to another CNN, called Refine Network (R-Net), which further rejects a large number of false candidates, performs calibration with bounding box regression, and NMS candidate merge.

Stage 3: This stage is similar to the second stage, but in this stage, we aim to describe the face in more details In particular, the network will output five facial point positions.

22

They proposed an approach in which it generates a high-quality face mapping from the images using deep learning architectures such as ZF-Net and Inception Then it used a method called triplet loss as a loss function to train this architecture Let’s look at the architecture in more detail.

23

Figure 2 12 Fully Connected Layer[5]

FaceNet employs end to end learning in its architecture It uses ZF-Net or Inception as its underlying architecture It also adds several 1*1 convolutions to decrease the number of parameters These deep learning models outputs an

embedding of the image f(x) with L2 normalization performed on it These

embeddings then passed into the loss function to calculate the loss The goal of this loss function is to make the squared distance between two image embedding is independent of image condition and pose of the same identity is small, whereas the squared distance between two images of different identity is large Therefore, a new loss function called Triplet loss is used The idea of using triplet loss in our architecture is that it makes the model to enforce a margin between faces of different

identities.

+ Triple loss

The embedding of an image is represented by f(x) such as x € R This

embedding is in the form of vector of size 128 and it is normalized such that.

2 (2.1)

5=1

|F(2)

24

We want to make sure that the anchor image (x?) of a person is closer to a

positive image(xƒ ) (image of the same person) as compared to a negative

image(x;') (image of another person) such that:

If (a2) - ƒ øƒ)llŠ + a < iF (z‡) — FeII3 (2.2)

V(ƒ (z‡).ƒ (zP), f (w?)) € T

Where a is the margin that is enforced to differentiate between positive and

negative pairs and T are the image space Therefore the loss function is defined as

the following:

N , “7 py yy2 , , 2

If triplets are easily satisfied above property, then it would not helping training,

SO it is important to have the triplets that violate above equation.

+ Triplet selection

To ensure faster learning, we need to take triplets that violate the equation

above This means for given( x;*) we need to select triplets such that

Generating triplets on every step on the basis of previous checkpoints and

compute minimum and maximum on a subset of data.

Selecting hard positive (x?) and hard negative (x?) by using minimum and

maximum on a mini batch.

25

Teachers can score with the push of a button on the pulpit so that they can send a

signal to the smart table, then the desk will take a picture of the student at the timethe desk opens, then send it to the pulpit to update the number automatically as well

as the teacher can verify that the student is in class or not videos Teachers canauthenticate by looking at images sent from the smart table at the pulpit

2.2.2.4 IOT SystemThe pulpit is connected to classroom devices through Wi-Fi infrastructure thanks

to IOT technology integrated into the pulpit With just 1 tap on the teacher screen

can easily control the devices in the classroom without moving, the teacher can turnoff all devices or turn it all on in the early hours or at the end of the hour with just 1tap And to do so, we use the MQTT protocol to be able to create connections

between nodes in the classroom