sign language recognition system

Project description Building a sign language recognition system is an innovative AI project aimed at facilitating communication between individuals with disabilities and those who wish t

UNIVERSITY OF ECONOMICS AND LAW FACULTY OF INFORMATION SYSTEMS

Al Mini Project

SIGN LANGUAGE RECOGNITION SYSTEM

Lecturer: Nguyen The Dai Nghia

Group: Group 8

Course ID: 225MI15211

No Student ID Name

1 K224111480 Nguyên Thị Thanh An

2 K224111481 Phạm Thuy Thái An

3 K224111484 Châu Ngọc Hân

4 K224111488 Do Khanh Linh

5 K224050741 Chong Sinh Đông

Ho Chi Minh City, July/2023

Table of Contents

Table of Figure 2

L

II Function of hand gesture recognition system

Project description 3

1 PUN CHHON Lee eee cece ceeccee cee ceee see ceeecsseensceeessseneeceessssnesceeesseeuessesssssusseecesesssseeeeeces 4

2 ASS€S Q TQ HQ Q00 0n 1n HH HH HH TY gu KH cu KH HE 9 ch 5

TI Application 6

1 Enable multi-sensory impaired people to communicate with the computer without needing to rely on other D€OÌC - - - Ă +22 + + HS HH HH th He re 6

2 People who desire to study sign language can benefit from the sign language TCCOBTIfIOTI SYSÍCTT 2Á HH ch Tre 6

3 Hearing impairment individuals using the system can be used for the language and

3001121003 2110101:i2v10:19) 0n 7

4 Supporting the impaired In many aspecfs Of ÏIÍ€ + - +5 se nen ee 7

5 The premise for the development of science and technology especially applies to

AI for supporting the lives of handIcapped peopÌe «+ se «sen 8

IV Strengths and Weaknesses of the system 8

1 ð ae 8

“N,/.)°› 9

V, Future development 10

VI Conclusion 11 VII Group Member Evaluation and Trello 12 Ni na ố ẽ e 12 PN.oau hon 13

Table of Figure

Table 1-Overview of Sign Language RecognifIion System for the Deaf - 4

E060 nh hố 12

I0) u00 0 0 13

E0 Suioudgul i0 0n 13

Table 5-Labels v1 14

I Project description

Building a sign language recognition system is an innovative AI project aimed at facilitating communication between individuals with disabilities and those who wish to learn sign language This project is constructed using Teachable Machine, which allows developers to create machine learning models through a simple drag-and-drop interface The device has been trained on various commonly used hand sign languages of individuals with disabilities, and it can accurately identify sign language with high precision The sign language recognition machine utilizes a webcam to capture images of the hand signs used by individuals with hearing impairments to communicate Subsequently, the machine analyses the images and employs teachable machine technology to understand the intended message This information is then displayed on a screen, enabling individuals without hearing impairments to understand and communicate with those who have hearing disabilities

The lack of information and knowledge also contributes to a barrier between these two groups Individuals without hearing impairments often do not know how to effectively interact and communicate with individuals who are deaf or hard of hearing They may be unfamiliar with sign language or lack the ability to read and write captions to convey information to individuals with hearing impairments This creates a communication gap and makes understanding each other challenging As an AI project, this sign language recognition system helps break down the language barrier and reduces the difficulties in communication between individuals without hearing impairments and those who are deaf

or hard of hearing By employing Teachable Machine in sign language recognition, it represents a significant step towards creating equality and opportunities for individuals with hearing impairments to engage in communication and social interaction Simultaneously, it provides individuals without hearing impairments with a tool to understand and participate in the sign language community, contributing to building a more

diverse and inclusive world

Name of project Sign Language Recognition System for the Deaf

Link of project https://teachablemachine withgoo gle.com/models/o4muOzuLu/

Type Pose project

Number of classes 6 classes

Target customers Mainly hearing loss individuals and people urging to

communicate with the hearing loss

Table 1-Overview of Sign Language Recognition System for the Deaf

H Function of hand gesture recognition system

1 Function

Gesture recognition can be used for disabled people Because handicapped people account for a large percentage of our community, we should make an effort to interact with them in order to exchange knowledge, perspectives, and ideas To that aim, we wish to establish a means of contact Individuals who have hearing-impaired can communicate with one another using sign language A handicapped person can communicate without using acoustic noises when they use sign language

The objective of this project is to explain the design and development of a hand gesture- based sign language recognition system The solution is based on a web camera as the major component, which is used to record a live stream video using a proprietary TensorFlow.js algorithm Recognition of hand movements is possible with the technology Recognizing hand gestures is a straightforward technique of providing a meaningful,

highly flexible interaction between robots and their users There is no physical communication between the user and the devices A deep learning system that is efficient

at picture recognition is used to locate the dynamically recorded hand movements Convolutional neural networks are used to optimize performance A static image of a hand gesture is used to train the model Without relying on a pre-trained model, the CNN is

constructed

Gesture recognition can be used to control devices or interfaces, such as a computer or

a smartphone, through movements or actions, such as hand or body movements, facial expressions or even voice commands

Why do many people want to use gestures instead of just touching or tapping a device?

A desire for contactless sensing and hygiene concerns are the top drivers of demand for touchless technology Gesture recognition can also provide better ergonomics for consumer devices Another market driver is the rise of biometric systems in many areas of people’s lives, from cars to homes to shops

During the coronavirus pandemic, it’s not surprising that people are reluctant to use touchscreens in public places Moreover, for drivers, tapping a screen can be dangerous, as

it distracts them from the road In other cases, tapping small icons or accidentally clicking

on the wrong field increases frustration and makes people look for a better customer experience Real-time hand gesture recognition for computer interactions is just the next step in technological evolution, and it’s ideally suited for today’s consumer landscape Besides using gestures when you cannot conveniently touch equipment, hand tracking can

be applied in augmented and virtual reality environments, sign language recognition, gaming, and other use cases

2 Classes

a) Class |: “Xin chao ban”: This class identifies greetings between communicators b) Class 2: “Rat vui duoc gap ban”: When “Nice to meet you” is actioned, the system will recognize

c) Class 3: “Xin 16i”: When an apologize needs transmitting, this class will transfer action into words

d) Class 4: “Cam on”: This class stands for “Thank you”

e) Class 5: “Khong thể nhận dién”: In case none of above classes are identified, this class will be shown

I Application

1 Enable multi-sensory impaired people to communicate with the computer without needing to rely on other people

People with deaf-blindness have both hearing and visual impairments Some dual sensory loss people have profound blindness and deafness, while others can use their hearing and vision to varying extents This condition can make it challenging for affected individuals to communicate with others Deaf-blind people can use a broad range of hearing and vision aids to communicate These devices improve quality of life by allowing people in the sensory impaired community to convey their needs and interact with computers

The deaf-blind person puts his or her hands opposite the camera of the device then starts some movements and location of the signs Some signs and facial expressions may need to be modified People can use one-handed or two-handed tactile sign language

2 People who desire to study sign language can benefit from the sign language recognition system

Sign language can be particularly useful for those working in public facing roles such

as police officers, paramedics, nurses, educators and social workers Learning sign language could also enhance the ability to recognise and interpret body language Therefore, more and more people have the need to learn sign language by taking advantage of the sign language recognition system to make learning faster and more convenient

3 Hearing impairment individuals using the system can be used for the language and eventually daily interaction

Hearing loss persons can utilize this system as a communication tool to help the deaf and the community for daily interaction In other words, this system is the bridge for communication between deaf and normal people It is defined as a mode of interaction for the hard of hearing people through a collection of hand gestures, postures, movements, and facial expressions or movements which correspond to letters and words in our real life To communicate with deaf people, an interpreter is needed to translate real-world words and sentences

4, Supporting the impaired in many aspects of life

Al has the potential to greatly benefit people with disabilities in a number of ways in

the future:

Assistive Technology: AI can be used to develop assistive technologies that can help people with disabilities to perform tasks that would otherwise be difficult or impossible for them For example, Al-powered devices like speech recognition software and smart home devices can help people with mobility or speech impairments to communicate and control

their environment

Improved Accessibility: Al can be used to improve the accessibility of products and services for people with disabilities For example, AI can be used to develop audio descriptions for videos, making them more accessible to people who are visually impaired Enhanced Medical Care: AI can be used to improve medical diagnosis and treatment for people with disabilities For example, Al-powered devices can be used to monitor the health of people with chronic conditions and alert their care providers in the event of any changes or emergencies

Increased Employment Opportunities: AI can help people with disabilities to find

employment and participate in the workforce For example, Al-powered tools can help to match people with disabilities with employers who are looking for their skills and abilities

5 The premise for the development of science and technology especially applies to AI for supporting the lives of handicapped people

Technical helpers are present in the every-day-life of a person with a disability and Artificial Intelligence (AD) is already helping in many ways AI based technology can adapt interfaces to the needs of the person sitting or standing in front of a screen An interface could then switch into a speech or a text-based mode applying different contrast and size

of elements on the screen In that way, an AI based system could learn how to adapt and better present the content of applications in a personalized manner This would not only affect persons with learning or cognitive problems, but also a growing part of our aging society Automated customisation may help a blind person to adapt the system according

to his or her needs, but the system then will know that there is a blind person in front of it

AI in general could bring major improvements for the independent living of persons with disabilities in all parts of the world, not only in the industrialized countries, but in all parts

of the world

IV Strengths and Weaknesses of the system

1 Strengths

a) Accessibility: One of the significant strengths of your project is its focus on accessibility By providing a means of communication for individuals with hearing and speech impairments, the project empowers them to express themselves using sign language This inclusive approach enables those individuals to participate more fully in conversations and interactions, bridging communication barriers b) User-Friendly: Teachable Machine's user-friendly interface simplifies the process

of training machine learning models It eliminates the need for extensive programming knowledge, making it accessible to a broader range of users This ease

c)

d)

a)

b `

c)

of use enables individuals with little to no coding experience to create and deploy their own sign language recognition models quickly

Real-Time Recognition: The potential for real-time sign language recognition is a significant strength of your project With the right setup, the system can accurately identify sign language gestures in real-time, allowing for immediate communication This real-time aspect is crucial in facilitating smooth and interactive conversations, enhancing the user experience

Customization: Teachable Machine's ability to train and customize the model specifically for the sign language gestures you want to recognize is a powerful feature This customization ensures that the model is tailored to the unique characteristics and nuances of the target sign language, resulting in improved accuracy and performance Users can focus on training the model for the specific gestures relevant to their sign language dialect or specific communication needs

2 Weaknesses

Limitations in Recognizing Continuous Actions: A notable weakness lies in the project's ability to recognize continuous actions within sign language phrases Sign language often involves sequences of interconnected or continuous gestures, such

as phrases that convey meaning holistically Training the model to differentiate between similar actions within different phrases can be challenging, potentially leading to confusion or inaccurate recognition

Limited Dataset: The accuracy and reliability of the sign language recognition model heavily rely on the quality and diversity of the training dataset However, acquiring a diverse and comprehensive dataset of sign language samples can be challenging The limited availability of such datasets may result in reduced accuracy and the model's inability to recognize certain gestures or variations

Environmental Constraints: Sign language recognition can be affected by various environmental factors such as lighting conditions, background clutter, and camera