Graduation project chatbot in admission consultant in international school – vnu using rasa framework

Learn language processing techniques in NLU, NLP such as language representation, intent classification or intent detection, information extraction, conversation management.... the knowl

Bùi Tuấn Anh

Hanoi — The year 2022

USING RASA FRAMEWORK

STUDENT: Bui Tuan Anh STUDENT ID: 17071375 COHORT: K16

MAJOR: Informatics and Computer Engineering

SUPERVISOR: Nguyễn Doãn Đông

Hanoi — Year: 2022

THE COMMITMENT

My name is Bui Tuan Anh, a student in the class ICE2017A, majoring in Informatics and Computer Engineering at International School — Vietnam National University

I promise that this project is my research, study, and construction.The project's content has references and uses information and documents from different sources books, journals, and articles listed in the list of references are legal

citations

Article

BUI TUAN ANH

ACKNOWLEDGEMENT First, I would like to express my deep gratitude to Mr Nguyen Doan Dong for wholeheartedly guiding, helping, and motivating me to complete the project on chatbots applied in international school enrollment

I would like to thank the teachers at the Faculty of Sciences and Applications, for helping me gain more useful knowledge and opportunities to develop more in this field of artificial intelligence

Because I just switched to this Artificial Intelligence major, my knowledge is still limited, the project will inevitably have certain shortcomings I respectfully accept the comments of teachers and teachers to improve the project better

I would also like to thank my family, relatives, and friends who have supported, cared for, and encouraged me throughout the process of doing this project

I0 1909) 0 000/057 00a 3 ACKNOWLEDGEMENTT, - - Ăn Họ HH Họ nọ HH tu gà 4 LIST OF SIGNS AND ARRIVALS -G Gv 8 LIST OF FIGURES AND GRAPHS HH He 8 LIST OF TABLES 2.0.0 a 13 6N 49)9)60/9419) 011 14

IS Con cac na ố 15

2 GOal OŸ the DTOJ€C - L3 HT TH Hà TH Tà HT TH 18

3 Thesis structure 19 CHAPTER 1: CHATBƠOT SYSTEM OVERVIEW HH he 20 LoL: Introduction nh 20 1.2: Related work in chatbot hisfOFV . - HH HH nh nơ 22 1.3: Structure of Chatbot COmpOn€TIFS - 2 2 223k TH TH TH HT HH ưy 24 1.4 NLU (Natural Language nderstandinng) - - 2 cành vn HT ky kưy 25

IS 2u án nh 28 1.6: Dialogue Manà€Im€fI - HH HH nh nh nưg 30 1.6.1: Finite state machine model FŠA - - 1n n nh 1H HT TH KH HH nà rớt 31 1.6.2: Erame-based model - - +: + + 1 E1 E3 v1 TH TH ke 32 1.7: NLG (Natural Language CeneratfiOI)) - 2+2 nh kTnHkTnHkHn ky ri 33 II ma 33 17.2: Plan-based -. + nà HH HH HH nh HE 34

l 9 " 34 1.8: 0° co nh 35 CHAPTER 2: DEEP LEARNING APPLTIED TO CHATBOT 38 2.2: Recurrenf neural đ€fWOFK c2 LH HH HH HH 4 2.3: Long-Short i2v 8c ynn ố 44 2.3.1: The problem of long-term depencl€nnC1€S sàn HH nh, 44 P208) 0 46 P00 nh 48 2.3.4: LSTM worling process sfep Dy Sf€P - HH HT TH HH TH Hư 49

PP À ca on cố ố ố 51 2.4.1: WOTC2VGC LH HH HH HT HH HT 51

2.5: Apply RNN to Dialogue ManageIm€TIL - - - +23 t n3 Hư 55

2.5.1: So 06 55

2.5.2: Global-Locally Self-Attentive DST (GLAID) LH «SH khe, 56 CHAPTER 3: LEARN ABOUT RASA FRAMEWORK « 59

3.1: Introduction nh cố 59

3.2: RASA Architecture 0.0.0 ố ố 59

3.3: Install and initialize RASA 00 ee 63

3.3.1: Program structure of RASA 6n ốc cố ố ố 64

K0 65

Ea an <5Aa ẻ 65

3.3.4: SHOES na cece 65

3.3.5: ACTIONS oo eect cece 66

3.3.6: Policies nh 67

=5 69

K02 69

0.7.0.4 1 71

BUILD A CHATBOT IN THE FIELD OF ADMISSION CONSULTANT 71

4.1: Problem sfaf€Imen[ - 1à HH HH HH HH 7I 4.2: SolUtiOns -.- co csrersee 4.3: Applied RASA to build chatbot 4.3.1: Install RASA nh 73

4.3.2: Comfi guration oo ố.ố 74

4.4: System on hố ố ố 75

4.5: Template for clatoOI - . 2 1131133 vn nh nh HH Ha 77 AS.1: Intent 77

Y2 an a dd 77

h9 iu.) an ố ố ố ẻ 79

4.5.4: Script ÍframeworkK and TUÏÌ€§ - s3 *TxkT Tk TH krkp S0 4.6: Using PhoBERT to improve intent and proper name reCOgnifIOn -. - - 81

CTÀAI 22a 82

47.1: Tesfing €IVITOTITTI 2 011 vn HH nhe 82 X2 19( oi 83

"1c on cố 88

(HP \aớổaa cVddANo(dl(OloPPiđiiiiii't-A 88

4.8.2: Participant’s rating ốc ốc ố ố ố e 89

4.9: Demo s0 in 90

4.9.1: Run chatbof on ferTmITiA]L -. - 2 2 s33 x3 E3 TT TH TT net 90 4.9.2: Connect chatbot to EacebOOK + + + 3 HT TH KH HT TH 94 ®9)i90013/90) 0 105

APPENDIX 01777 107 4505350010-117 .- ÔỒÔỒÔÒ 109

LIST OF SIGNS AND ARRIVALS

NLG Natural Language Generation Sinhngônngữtgnhiên

contain a layer

contain four layers

backend

environment

chatbot

majors

the training model

whitespace tokenizer

line

on Ngrok

INTRODUCTORY

Artificial intelligence (AID) has influenced the way we participate in our daily activities by designing and evaluating advanced applications and devices, known as intelligent agents, that can perform different functions Along with the development

of science and technology, Chatbot is being widely and strongly applied in many fields and is expected to develop even more in the future

“Chatbot will fundamentally revolutionize how computing is experienced by everybody”

Satya Nadella, CEO of Microsoft

“Now, to order from 1-800 flowers, you never have to “dial” 1800 again “

Mark Zuckerberg, CEO of Facebook

“ Artificial Intelligence would be the ultimate version of google “

Larry Page

Chatbots or virtual assistants, in general, are getting smarter and more complete It helps us have better interactions and experiences with the software Here are some achievements that chatbot has reached (Jagan Jami, “infographic: The Future of Chatbots Statistics & Trends,2017)

O Currently, more than 100,000 chatbots are being used on Facebook

O 63 percent of individuals would think about contacting an online chatbot to get in touch with a company or brand

O 60 percent of Gen Xers and 59 percent of millennials have used chatbots on

a messaging app

O In the USA, chatbots have a greater retention rate than the best apps

Oo More than half of customers like companies that utilize chat applications

O By 2020, 80% of organizations plan to use chatbots

O By 2020, the average individual will converse with chatbots more frequently than with their spouse

O From 2017 to 2021, the global chatbot market is projected to expand at a CAGR of 37,11%

O By 2018, 62 percent of businesses will have used AI technology, up from the current 38%

O By 2022, it is anticipated that chatbots would save organizations $8 billion

O Chatbots may save service providers an average of a little over 4 minutes for every inquiry

O Over 75% and 90%, respectively, of bot interactions in the banking and

healthcare industries, will be successful

The 4.0 technological revolution has had a huge impact on many facets of life, notably in the domain of natural language processing and comprehension, where it has led to substantial advancements (NLU and NLP) Since 2013, several algorithms have been used to make chatbots even smarter and more precise Even though the age of chatbots is just getting started, we have already reaped several rewards from employing them We may foresee a future in which chatbots can not only assist people in solving all aspects of problems, but also make decisions for them as technology continues to grow, particularly with the developments in artificial intelligence over the past few years

1 Research Motivation Online messages are a common way to respond to customer service inquiries in Vietnam However, this is also done manually and has several issues, including the fact that it takes a lot of time and money to pay personnel to perform the same basic and comparable tasks As a result, the need for an intelligent and autonomous control system to increase productivity is critical, and chatbots are the ideal

solution

Currently, the widely utilized online chat tools are starting to take over as people's preferred channels for getting in touch with businesses and resolving consumer issues It's hardly surprising that chatbots are growing in popularity because instant messaging applications have established themselves as the primary channel for all companies to communicate with customers

We must deal with a lot of inquiries every day, including those from clients who inquire about goods and services, staff members who inquire about rules at

work, and kids who have queries Thus, chatbots are used in a variety of industries,

including:

O Media Publishing Application The content delivery sector is one of the industries where chatbots are used most frequently Chatbots are seen by media publishers as a potential method for interacting with the audience and keeping track of that involvement Chatbots can therefore assist media publishers in getting reliable and beneficial insights about audience interests

On platforms like Facebook Messenger, users may immediately communicate with a brand's chatbot, improving the convenience and responsiveness of communications Numerous media organizations use chatbots that use AI and machine learning to forecast the kind of material that would appeal to their audience As a result, content delivery can become quicker and more personalized

O Food ordering The second saw use of the chatbots is the delivery of food, which is an obvious example Chatbots are used by well-known companies like Pizza Hut and KFC to help customers place orders while conversing Taco Bell's adoption of chatbots with

a twist also illustrates the potential for tailoring chatbot deployment to provide the greatest client experience The Taco Bell chatbot, commonly known as Taco Bot, can joke about and use emoticons in its responses in addition to responding to trivia questions These chatbot characteristics improve a customer's affinity for a certain

brand

O Companionship Application Chatbots may completely alter the way we interact with others The use of chatbots as virtual and digital assistants may benefit those in need, like the elderly and those with Alzheimer's disease, by offering companionship In this instance, the chatbot created by the Russian technology company Endurance is one of the famous instances

Endurance's chatbot is designed primarily to be a friend for older persons and Alzheimer's patients The chatbot converses with older citizens on subjects

including the weather, news, nature, music, and other things to act as a virtual

companion The conversational chatbot may ask questions, get responses, and choose the best topics based on the user's preferences

O Market research applications The use of chatbots may also lead to the establishment of new standards for market research Chatbots can aid in creating individualized interactions with respondents

to market research Chatbots offer personalization and conversational elements to specialist survey programs, which can still be dependable choices for market research Consequently, researchers may increase respondents’ involvement in the study and guarantee greater completion rates

O Healthcare application The variety of applications for chatbots is also demonstrated by the examples from the healthcare industry Medical personnel havebeen helped by chatbots like Super Izzy in offering speedy medical diagnoses and responses to health-related concems Tracking menstrual cycles, fertile windows, and monthly dates may be made easier with the aid of the Super Izzy chatbot Along with asking questions

about menstruation, it also offers useful information about sexual difficulties and

menstrual health

O Hospitality sector The use of chatbots is also widespread in the hotel industry, with a range of

results For instance, Edward, a chatbot, is used by Radisson Blu Edwardian Hotels

to help guests The chatbot may assist visitors with a variety of tasks, including handling customer queries, facilitating online check-in, and confirming the availability of rooms The hotel management staff also uses data from Edward to track general user demands and adjust the quality of their services

O Legal Industry Most importantly, chatbot applications are being approved in the legal industry Due to their capacity for work process improvement, chatbots are suitable for the legal industry since they result in significant time and money savings The Immigration Virtual Assistant (TVA) chatbot from Montreal-based law firm EXEO

is one of the famous instances of the use of chatbots in the legal sector

IVA is a sophisticated virtual assistant that can offer dependable assistance to all prospective immigrants It can make sure consumers finish the immigration procedure promptly and without too much stress The most crucial feature is that customers may use IVA on Facebook Messenger to get trustworthy information on rules governing work permits, immigration programs, and programs for permanent residency

2 Goal of the project

Research the components of chatbots Learn language processing techniques in NLU, NLP such as language representation, intent classification or intent detection, information extraction, conversation management in building chatbots

The thesis focuses on finding ways to solve problems that chatbots apply in closed domains and answer according to the information retrieval model (retrieval based) The chatbot provides pre-written replies or adheres to predetermined patterns in the information retrieval methodology The information retrieval paradigm is largely used by chatbot systems that are now in use, and they are implemented in certain application fields

With this topic, the thesis will focus on building a chatbot system to support users in the field of admissions consulting based on the Rasa framework and apply

the knowledge learned about chatbots to be able to customize this open-source code.Theobject of supporting chatbot users will be students and parents who are wishing to learn detailed information about universities, how to apply, and specific fields of study

3 Thesis structure The thesis has a structure as follows:

INTRODUCTORY: An overview of chatbots, giving directions for chatbot research

CHAPTER 1: Chatbot system overview

Introduction to chatbot system overview

Structure and tasks of chatbot system components

CHAPTER 2: Deep learning applied to chatbot

Understanding and grasping some techniques The basic algorithm or algorithm used in the chatbot so that it can be adapted to the Vietnamese language, helps the chatbot handle smarter

CHAPTER 3: Learn about the RASA framework

An overview of the RASA framework

CHAPTER 4: Build a chatbot in the field of an admission consultant

Building a university chatbot application on the open-source platform Rasa Make conclusions, evaluate andimprove the system

CONCLUSION: Making conclusions, evaluations,and directions for further research

CHAPTER 1: CHATBOT SYSTEM OVERVIEW

1.1: Introduction

A chatbot is computer software that can instantly communicate with people and answer queries or manage issues The algorithms used by chatbot designers influence their breadth and complexity Chatbotsare usually applied in many fields

such as e-commerce, customer service, healthcare, finance and banking,

entertainment services

Chatbots can be classified using different parameters:

O the knowledge domain (open domain / close domain)

task-oriented Non-task oriented

Oo Text-based: frequently combined with messaging applications, social

media, SMS, and other channels to manage normal client interactions However,

there is a catch Customers must write or tap in their questions and commands since these chatbot systems are text-based This will take more time and effort than a voice-based chatbot

oO Voice-based: an AI conversational communication tool that can collect, understand, and analyze the speaker's speech input to answer in natural language Users can use voice commands to communicate with a speech AI chatbot and obtain contextualized relevant replies

O Knowledge domain chatbot The knowledge domain categorization considers the quantity of data a chatbot is educated on as well as the information it can access

oO Chatbots with a closed domain are only capable of responding to questions related to that field of study

oO Open domain chatbots may converse about various issues and respond effectively

O Goals

Oo Task-oriented chatbots are one-purpose systems that accomplish a specific task They provide automated yet conversational replies to user inquiries using rules, NLP, and very little machine learning

oO Non-task-oriented chatbots can be used to characterize interviewing someone and, in most cases, chatting for the aim of amusement in open domains

O Design Approach

Oo Rule-based: This chatbot only executes predefined scenarios If the user enters a question that is not in the original script, this chatbot will not be able to answer it correctly The intelligence of this type of chatbot depends on the degree to which we have pre-programmed it

21

oO Generative based: Thanks to artificial intelligence, the chatbot can learn and understand what the user wants Generative-based models are sequence models that generate appropriate responses in a conversation

O Retrieval-based: chatbot will learn the answers from the archive, then

will give the appropriate answer to the situation

1.2: Related work in chatbot history

An online chat system for college queries that used a comprehensive database to do pattern matching for chatbots' information retrieval was described in (Prashant et al., 2017) With the help of UML and other process diagrams, all the specific working stages in this research are transparent Their bot, however, did not employ machine learning, making it still too strict by regulations for parents and pupils to ask questions Since the patterns were manually created, they do not scale for the actual use case

Eagle Bot is a comparable piece of art (Rana, 2019) Based on the data on the institution's website, the authors created a semantic search strategy to respond to visitors’ questions about a university One of their methods entailed using BERT for locating documents and choosing paragraphs This chatbot system is expandable that uses Dialogflow as its primary architecture and three route selection techniques Due to request limitations in the Dialogflow chatbot platform, Eagle Bot still has certain restrictions, and its accuracy rate was just 56%

The authors (Windiatmoko et al., 2020) developed a chatbot for university inquiries that is combined with a MySQL database and API and provides step-by- step explanations However, this basic system can only respond to a limited number

of client intentions Additionally, the authors failed to disclose how they changed their tokenizer and pipeline because Indonesian differs significantly from other languages, such as Vietnamese or English The research didn't make advantage of the platform's advanced features, instead sticking with the fundamental framework Rasa provided when the project was first started

22

Encoder-decoder models are sequence-to-sequence networks that form the foundation of many studies on generative chatbot models (Vinyals& Le, 2015) For training, these models need a sizable corpus of single-turn speech Except for English and Chinese, with millions of pieces, it is challenging to create a high- quality discussion corpus in practice There aren't many generative chatbots accessible for other languages, especially Vietnamese, as far as we know

A football-themed social chatbot was developed by (Segura et al., 2019) to respond to queries regarding the "La Liga" Spanish football competition The chatbot is set up as a Slack client and only supports text-based communication When responding to user inquiries that are outside of its scope, it performs SPARQL queries to get data from Wikidata and engages in a retrieval-based chat- oriented discourse However, the fact that we have no control over the outputs that are provided to clients is one of the generation-based chatbot's most glaring flaws These products could have delicate information or grammatical mistakes that are inappropriate for official sectors like education

Each Rasa project must have a reasonable process to create an effective AI-

based chatbot Anran Jiao, the author (Jiao, 2020), compared the effectiveness of

the Rasa NLU stack with that of the Entity Extractor and Neural Network Classifier According to the research, the Rasa NLU technique outperforms by extracting every entity and categorizing user intentions Consequently, utilizing the Rasa platform, our data preprocessing, and well-known pre-trained models like BER, we created the individual pipeline for our chatbot Additionally, the Rasa chatbot will slow if

we only connect through the webhook channel when roughly 25 userslogin at once

We constructed a gateway offered by the proxy server to address this issue As a result, the chatbot may address up to 50 individuals at once The chatbot was then made available on Messenger so that parents and students could quickly use the services For the user to better visualize the solution, we also substituted a picture for the lengthy, tedious text Other chatbots in any language can likewise use these

tactics

1.3: Structure of Chatbot components

O NLU: includes natural language processing (NLP) whose task is to identify

O get an intent classification and information extraction (slots filter)

Ol DM: The conversation manager is responsible for determining the next action

O based on previous action state or conversation context These contexts must be matched in the bot's trained histories This component also takes care of getting data from other systems via API calls in action

O NLG: is a language-generating component based on policies and actions defined in the DM through conversation sets NGL can generate answers based on a pre-defined template that has been trained for the bot The diagram below shows how the input messages are processed

aah | NLU component |

‘Hello (extract intent and -——intent and entities

next action —

how about day after?

Generator (use pre-defined <——next action

O The first is the division of the subject matter; examples include banking, health care, and insurance This stage of categorization is unnecessary if there is just one field in the situation

O Next is the intent classification, for example, determining the intent to look

up account information or the intention to check the balance

O The last step is to extract information (slot filter or entity extraction) from the user question For instance, we mustextrapolate the months' duration information in the user question: "how much is the 3-month term interest

rate” From the 3-month information extraction, the chatbot has a basis to

answer the user

The NLU processes user messages through the pipeline that configures sequential processing steps:

25

Figure 1.4: NLU process(Yun-Nung (Vivian) Chen, 2018)

In this pipeline, we can customize components from data preprocessing, language modeling, algorithms used to extract words and extract entity information, etc

We can see the detail of the process in the below diagram:

Lãi suất vay 12 tháng 2 intent: "Interest”

Figure 1.5: Data processing(Tom Bocklisch, 2018)

We must model the language or express the language as a machine- understandable arithmetic vector, to categorize the user's sentence purpose (vectorization) Word embedding is now the most often used technique Word

26

embedding refers to a class of feature learning methods and language models used

in natural language processing (NLP), in which lexical words or phrases are converted into a vector of numerical values From a conceptual standpoint, it entails embedding arithmetic from space with one dimension per word into a continuous vector space with a significantly less dimension In the part that follows, certain well-liked representation techniques like Word2Vec, GloVe, or more recent FastText will be discussed

Identifying the user intent from the user inquiry based on the trained set is the stage of intent classification or text classification after the language modeling incorporates training input data for the bot We may employ a variety of methods in

this stage, including Decision Trees (Random Forest), SVM, Convolutional Neural

Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and Bi-LSTM To categorize user intent, the majority of the existing chatbots employ deep learning models like RNN and LSTM The hardest part of this stage for chatbots is figuring out the different intentions in a user statement For instance, the bot must distinguish between the two meanings of "hello" and "check balance" in the case of a command like "hello, check my account balance."in the user's statement If the bot can understand and answer this type of question, it will make interacting with the bot more natural

The next step is information extraction from user conversations Typically, the information that must be retrieved takes the shape of integers, strings, or timings, and they first need to be declared and trained

Word segmentation (Tokenization or word segmentation): Word separation is a processing process aimed at determining the boundaries of words in a sentence, can also be simply understood that word separation is the process of identifying single words, compound words are in the sentence For language processing, to be possible to determine the grammatical structure of a sentence, and to identify a word's kind in a phrase, it is necessary to determine which word is in the sentence This problem seems simple to humans, but for computers, this is a very difficult

27

problem to solve Normally, languages separate words by spaces, but in Vietnamese, there are many compound words and phrases For example, the compound word "account" is created by two single words "account" and "account" There are several algorithms to help solve this problem such as the longest word matching model, maximum matching model, hidden Markov model (Hidden Markov Models (HMM), or CRF model(conditional random field)

1.5: Determine the user’s intent

Pre-Processing Feature extraction

Input message E> Classification | Result

Model

Figure 1.6: Determination of user’s intent(Tom Bocklisch, 2018)

The user intent classifier has several basic steps:

O Pre-Processing Data

O Feature extraction

O Training model

O Classification

In the stage of "cleaning" the data, superfluous information is removed, the data

is normalized, misspelled words are changed to the right spelling, acronyms are normalized, etc The chatbot system heavily relies on the data preparation stage The accuracy and intelligence of the bot will enhance if the input data is handled at this stage

The next step is to extract features from the cleaned data Before the widespread use of the deep learning model, the feature extraction stage had a significant impact

on the classification model's accuracy It takes careful data analysis and specialist knowledge in each application domain to extract useful characteristics

28

The characteristics that have been retrieved and put through machine learning techniques to develop a classification model are used as input in the model's training phase When decision trees are utilized, classification models can either be weight vectors that correlate to the retrieved features or classification rules (as in logistic regression models, SVMs, or neural networks)

After we have an intent classifier model, we can use it to class a new

conversation.This exchange too goes through the preprocessing and includes extraction steps, at that point the classification demonstrate will decide the "score" for each expectation within the set of bury and grant the expectation with the most noteworthy score

The chatbot must determine the user's purpose to deliver the correct help The following exchange between the user and the chatbot will be handled depending on what the user's goal is The chatbot will therefore provide inaccurate and contextual replies if the user's purpose is incorrectly determined The user can then get repulsed and decide not to use the system again As a result, the issue of figuring out user intent is crucial to the chatbot system

We can be able to limit the sum of clients eagerly inside a limited set of indicated bury that is important to the company that the chatbot can bolster for the closed application space With this restriction, it is possible to simplify the issue of figuring out user intent to a text categorization issue The classifier selects the intent from a predetermined set of intents that corresponds to a user communication statement given as input

To construct an aim classification show, we require a preparing dataset that incorporates distinctive expressions for each aim For illustration, for the same reason of inquiring around confirmations experts, clients can utilize the taking after expressions:

L1 Tôi muốn được tư vấn v các ngành

1 Trường có những ngành nào vậy?

Ngành nào là tốt nhất trong trường vậy?

29

One of the most important tasks in developing a chatbot system, and one that significantly affects how well the system operates in the long term, is the generation

of training data for the intent classification problem.The chatbot developer must invest a significant amount of time and energy into this task

1.6: Dialogue Management

In long conversations between people and chatbots, chatbots will need to remember information approximately setting or overseeing exchange states The issue of conversation management (dialogue management) when is important toensure smooth human-machine communication

The work of the discussion supervisor component is to get input from the NLU component, oversee the discourse state, and the discourse setting, and pass the yield

to the language generator

previous

Action

Under The Hood

Rasa Core: Dialogue Handling

updated after next

T

Figure 1.7: Dialogue handling(Naveen Kumar , 2020)

30

From figure 1.7 above, the dialog state is saved and relies on the dialog policy set to decide the next action for the bot's response in a conversation scenario, or the action just depends on the conversation depends on its previous dialog state

1.6.1: Finite state machine model FSA F——————————>_' Xin quý khách cho biết họ tên

t

Chúng tôi đã kiếm tra và thấy tốc độ Internet thực tế của quý khách chậm hơn tốc độ cam kết Nhân viên chăm sóc khách hàng sẽ liên hệ trực tiếp với quý khách trong

thời gian sớm nhất

Chúng tôi giúp gìđược cho quý khách nữa không ạ? (Có/không)

Có Không

| Cam on quy khach!

Figure 1.8: FSA model(Tecknovice, Aug 13, 2017) The simplest conversation management FSA modelis infigure 1.8 above For illustration, the client care framework of a broadcast communications company serves clients who complain about moderate organizing issues The errand of the chatbot is to inquire about the customer's title, phone number, web bundle title the client is utilizing, and the customer's genuine Web speed The figure outlines a discussion administration show for a client care chatbot The states of the FSA compare to the questions the discourse chief inquires the client The circular

31

segments between states compare to the activities the chatbot will perform These activities depend on the client’s reactions to questions Within the FSA

demonstration, the chatbot is the user-oriented side of the discussion

The FSA approach has the benefit of being straightforward, and the chatbot will already know the expected response from the user The FSA approach, however, is not truly appropriate for sophisticated chatbot systems especially when users provide inconsistent information throughout a single discussion When the user enters theirinformation into the system in the example above, the user can become irritated if the chatbot repeatedly requests their phone number

1.6.2: Frame-based model

The Frame-based demonstration can unravel the issue confronted by the FSA show The Frame-based show depends on predefined outlines to direct the discussion Each outline will incorporate the data (opening) to fill and the comparing questions that the exchange director inquires the client This show permits the client to fill in different spaces within the outline

Số điện thoại Số điện thoại của quý khách là øì ạ?

Tên gói Internet Gói internet mà quý khách đang dùng là gia

Tốc độ Internet thựctế | Tốc độ vào Internet của quý khách hiện thời là bao

nhiéu a?

Table 1.0: Frame-based model example Until the required information is available, the conversation management component will ask the customer questions and fill in the blanks depending on the information they supply This is by the Frame-based approach When a user responds to many questions at once, the system must fill in the appropriate gaps and keep track of which questions have previously been asked and received a response

A discussion might have a variety of frames in complicated application areas When the transition between frames is thus a challenge for chatbot creators Creating production rules is a popular strategy for controlling the handoff of control

32

between frames These laws are dependent on several factors, including the user's most recent query or chat

1.7: NLG (Natural Language Generation)

NLG is the chatbot's answer generator It relies on mapping conversation manager actions into the natural language to respond to users

NLG could be a multi-stage handle, with each step advance refining the information being utilized to create a substance with natural-sounding dialect The six stages of NLG are as takes after:

O Content analysis: To decide what should be included in the material created after the process, data is filtered This step entails figuring out the primary themes and connections between the original document's contents

O Data understanding: The data is analyzed, trends are found, and its context is established The usage of machine learning is widespread at this point

O Document structuring: Based on the type of data being analyzed, a documented plan is produced, and a narrative framework is selected

O Sentence Aggregation: The topic is succinctly and accurately covered through the combination of pertinent phrases or sentence fragments

O Grammatical structuring: Grammar principles are used to create content that sounds natural The sentence's syntactical structure is inferred by the software It then makes use of this knowledge to recreate the statement using the proper grammar

O Language presentation: The ultimate yield is produced based on a format or arrangement the client or software engineer has chosen

There are 4 commonly used mapping methods: Template-Base, Plan-based,

Class-base, RNN-base

1.7.1: Template-based This answer mapping method is to use predefined bot response templates to

generate answers

33

looking for.”

Confirm(food=$V,area=$W) | “Do you want a $V restaurant in the $W?”

Table 2.0: Template-based example(Yun-Nung (Vivian) Chen, et.al, 2018)

O Pros: Simple, easy control Suitable for closed domain problems

O Cons: Time-consuming to define rules, not natural in answer For large systems, it is difficult to control the rules, resulting in the system being difficult to develop and maintain

Sentence

Sentence Plan Plan Surface Generator Reranker Realizer

Figure 1.9: Plan-based example(Yun-Nung (Vivian) Chen, et.al, 2018)

O Pros: Can model complex language structures

O Cons: Heavy design, requires clear knowledge domain

1.7.3: Class-based

34

{depart_date}? input frame

O Pros:easy to implement

O Cons:depends on the response data previously labeled with training.Besides, the inefficient score calculation also leads to the generation of wrong answers

1.8: NLG models

O Markov chain: One of the earliest techniques for generating languages was the Markov chain The next word in a phrase is predicted using a sequence of

35

n words that are now accessible, and the probability of the following word is calculated by considering how all previous words relate to this sequence Markov sequences are used to generate word suggestions for sentences in the auto-suggest section of smartphone keyboards, where we have already

in the dictionary depending on the word before it, picks the phrase with the highest likelihood, and memorizes it This paradigm is perfect for language generation since the RNN's "memory" always recalls the "content" of the conversation However, given the size of the sequence increases, the RNN is unable to store the remote encountered words in the sentence and only makes expectations based on the foremost later word Due to this limitation, RNNs cannot formulate complex, cohesive sentences

O LSTM (Long short-term memory): To solve the long-range dependency problem, a variant of RNN called Short-Long-Term Memory (LSTM) was introduced Short-long-term memory is an artificial regressive neural arrangement utilized within the field of profound learning Unlike standard

feedforward neural networks, LSTMs contain feedback connections The

network not only deals with single data points; but also processes entire data

series

36

Although like RNNs, LSTM models consist of a four-layer neural network The LSTM consists of four parts: the unit, the input port, the output port, and the bypass gate These permit the RNN to keep in mind or disregard words at any time by altering the data stream of the unit When a dab is experienced, the Overlooked Door recognizes that the setting of the sentence may alter and may disregard the current unit state data This permits the arrange to specifically track as it were significant data whereas minimizing the vanishing slope issue, permitting the show to keep in mind the data for a longer period

However, the capacity of LSTM recollections is restricted to several hundred words due to their inalienably complex successive ways from the past unit to the current unit The same complexity leads to tall computational necessities that make LSTMs troublesome to prepare or parallelize

O Transformer: A relatively new model first introduced in a 2017 Google paper, it proposed an unused strategy known as the "self-attention component” The transformer comprises of a square of numerous encoders to prepare input of any length and another decoder to produce yield sentences

In differentiate to LSTM, Transformer takes as it were a little, steady number

of steps, and applies a self-attention component that specifically mimics the relationship between all the words in a sentence Not at all like past models, the Transformer employments the representation of all words in a setting without having to compress all data into a single fixed-length representation permitting the framework to handle sentences longer without the requirement for skyrocketing computation

37

CHAPTER 2:

DEEP LEARNING APPLIEDTO CHATBOT

This chapter introduces some basic knowledge about artificial neural networks, how neural networks work, and some techniques applied in natural language processing in particular or building chatbots in general

2.1: Artificial Neural Network — ANN

An artificial neural network is a model of information processing that simulates the activity of an organism's nervous system and is made up of several neurons wired together to process information Like the human brain, ANN can learn from experience (via training), store knowledge, and use that knowledge to make predictions about future ambiguous data

A neural network is a group of interconnected nodes that simulate the neural network of the human brain The artificial neural network is represented through

three basic components: the model of the neuron, the structure, and the connection

between the neurons In numerous cases, a counterfeit neural arrangement is a versatile framework that changes its structure based on outside or inner data streaming through the organization amid the learning process

38

Input Hidden Output

Input 1

Input 2

Figure 2.1: Structure of ANN(Denny Britz, 2015)

From figure 2.1 above, the general architecture of an ANN consists of 3 components which are Input Layer, Hidden Layer and Output Layer

In which, the hidden layer (Hidden Layer) consists of neurons, that receive input data from the neurons in the previous layer (Layer) and converts these inputs for the next processing layers In an ANN there can be many Hidden Layers

The main benefit of ANNs is their usage as a tool for approximating arbitrary

functions that "learns" from observed data However, some of the characteristics

and experiences when creating an ANN neural network make their use less straightforward

This method calculates the exact ratio of output data (output) from input data (input) by calculating weights for each connection (connection) from iterations during "training" data for Chatbot Each step of "training" data for the Chatbot will modify the weights resulting in output data being output with high accuracy + Algorithm

Each ANN must learn two issues, which is referred to as referential learning,

learning about structures and parameters While structural learning modifies the

39

network structure by altering the number of hidden layers, the number of neurons per layer, and the way the connections are created, parameter learning modifies the weights of the connections between neurons in a network they You may work on these two issues together or individually The resultant ANN can be quite strong and efficient if the models, cost functions, and learning algorithms are selected properly

Weights Inputs

Xj (wm Summations Transfer function

Figure 2.2: algorithm of ANN(Denny Britz, 2015)

O Input: Each Input corresponds to a feature of the data

Oo Output: The result of an ANN is a solution to a problem

O Connection Weights: This is a very important component of an ANN; it shows the importance and quality of the input information to the data preparation process that converts data from one layer to another The learning process of ANN is the method of altering the Weight weights of the input data to induce the specified result

O Summation Function: Sum the weights of all inputs given to each neuron The sum function of a neuron for n inputs is calculated according to the following formula: