Developing a virtual sales assistant using deep learning faculty of high quality training graduations thesis of the information technology

ABSTRACT Today information technology has been developing to a high level, is applied to all sectors and fields of the country's economy, making an important contribution to making these industries develop faster and more efficiently Online shopping now is necessary because the Covid epidemic is still raging When customers shop online, they will also need to be consulted by staff for more product information, and how to order and purchase Therefore, the communication system between customers and employees is improved to always serve customers Through the research, the implementation team found that using a virtual sales assistant to replace real people in customer support brings great benefits This reduces staffing costs, ensures customers are always supported, and improves user experience Therefore, the implementation team built a chat bot system on an e-commerce website The chat bot system is developed by python language, using Deep Learning model and Tensorflow, Keras, NLTK, Under The Sea, Fast API libraries When customers send messages in the chat interface, the chat bot will respond based on the content of those messages and send them back to the customer The chat interface also has customer support functions such as allowing the use of microphones to speak instead of typing, converting text to speech and vice versa In addition, the e-commerce website that we built using MEARN stack also has basic functions such as viewing product details, adding products to the cart, reviewing products, making payments, and placing orders (customer side) and managing product, managing user, managing order, managing question and answer sets (admin side) vii TABLE OF CONTENTS PROJECT ASSIGNMENT i ADVISOR’S EVALUATION SHEET ii PRE-DEFENSE EVALUATION SHEET iii EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER iv ACKNOWLEDGEMENTS v DISCLAIMER vi ABSTRACT vii TABLE OF CONTENTS viii LIST OF ACRONYMS xiii LIST OF TABLES xiv LIST OF FIGURES xv Chapter 1: INTRODUCTION 1.1 The reason for choosing the topic 1.2 Goals 1.3 Technologies Chapter 2: LITERATURE REVIEW 2.1 MERN stack 2.2 Deep Learning 2.3 Chatbot 2.3.1 Definition 2.3.2 Building a Chatbot 2.3.3 Advantages of Chatbot 2.3.4 Disadvantages of chatbot 2.4 Virtual assistant 2.4.1 Definition 2.4.2 The advantages of virtual assistants set up in chatbot 2.5 Text-To-Speech platforms 2.5.1 FPT API Text-To-Speech 2.5.2 ZALO API Text-To-Speech Chapter 3: THEORETICAL BASIS viii 3.1 Deep Learning 3.1.1 Python programming language a Advantages b Disadvantages c Versions 3.1.2 Tensorflow a Advantages b Disadvantages c Version 10 3.1.3 Keras 11 a Advantages 11 b Disadvantages 11 3.1.4 Natural language processing and the NLTK library 11 a Definition 11 b Tokenization of NLTK 12 c Bag-of-words 12 d Stemming 13 e Lemmatization 13 3.1.5 Underthesea – A Vietnamese NLP Toolkit 14 a Installation 14 b Usage: 14 3.1.6 FastAPI 15 3.2 a Definition 15 b Compare Three Python Frameworks Django, Flask and FastAPI 15 Technologies for website building 17 3.2.1 React Library 17 3.2.2 MongoDB 17 3.2.3 Node js 17 3.2.4 Express framework 17 3.2.5 Socket.IO 18 Chapter 4: SYSTEM DESIGN 19 4.1 System Architechture 19 4.2 System design 19 4.2.1 Database Design 19 ix 4.2.2 Sequence Diagram 21 4.2.2.1 Add product to cart 21 4.2.2.2 Place order 21 4.2.2.3 Make payment 22 4.2.2.4 Chat with Virtual bot 23 4.2.2.5 Chat with Admin 24 4.2.2.6 Update Question 25 4.2.3 Use Case Diagram 26 4.2.4 Use Case Description 27 4.2.4.1 Search product 27 4.2.4.2 Sort product 28 4.2.4.3 Chat with bot 30 4.2.4.4 Add product to cart 32 4.2.4.5 Add shipping address 33 4.2.4.6 Choose a payment method 35 4.2.4.7 Place order 37 4.2.4.8 Pay order 38 4.2.4.9 Rate and Review product 40 4.2.4.10 Chat with Admin 42 4.2.4.11 Chat with user 43 4.2.4.12 Train dataset 45 4.2.4.13 Manage question 46 a Edit question 47 b Delete question 48 4.2.4.14 Manage dataset 50 a Add dataset 50 b Delete dataset 52 4.2.5 User Interface Design 54 4.2.5.1 Screens for unauthenticated users 54 a Home Screen 54 b Product Screen 55 c Cart Screen 55 d Search Result Screen 56 e Chat with virtual bot 57 4.2.5.2 Screens for customers 58 x a Chat with virtual bot and admin 58 b Place Order Screen 59 c Order History Screen 59 4.2.5.3 Screens for admins 60 4.3 a Chat with user 60 b Q&A Screen 61 c Dataset Screen 61 Chatbot system 63 4.3.1 General system 63 4.3.2 Details of each part 65 a Speech to text 65 b API Input and Output Processing Center 66 c Natural language processing 67 d Dataset 68 e System deep learning and message predicting results 69 f Text to speech 71 g Develop new data addition functionality for chatbot 74 h Deploy server 75 Chapter 5: SYSTEM IMPLEMENTATION 76 5.1 Software development environments 76 5.2 Source code management 76 5.3 Deployment environment 76 5.3.1 Deploying the website on Heroku 76 5.3.2 Deploying the Chatbot API on Google Cloud Platform 79 5.4 Built screens 89 5.4.1 Home Screen 89 5.4.2 Product Screen 90 5.4.3 Cart Screen 90 5.4.4 Place Order Screen 91 5.4.5 Welcome interface of Chatbot 91 5.4.6 Chatbot 92 5.4.7 List Question Screen (Admin) 92 5.4.8 Edit Question Screen (Admin) 93 5.4.9 List Dataset Screen (Admin) 93 xi 5.4.10 Edit Dataset Screen (Admin) 94 5.4.11 Create New Dataset Screen (Admin) 94 Chapter 6: CONCLUSION 95 6.1 Conclusion 95 6.2 Advantages 97 6.3 Disadvantages 98 6.4 Development directions 98 REFERENCES 99 JOB TABLE OF WORK 102 xii LIST OF ACRONYMS DB: Database TTS: Text to Speech STT: Speech to Text NLTK: Natural Language Toolkit NLP: Natural language processing API: Application Programming Interface UI: User Interface gTTS: Google Text-to-Speech VM: Virtual Machines SSH: Secure Shell xiii LIST OF TABLES Table 3.1 Compare three python frameworks 15 Table 4.1 Actors in usecase diagram 26 Table 4.2 Usecase description for searching product 27 Table 4.3 Usecase description for sorting product 29 Table 4.4 Usecase description for chatting with BOT 30 Table 4.5 Usecase description for adding product to cart 32 Table 4.6 Usecase description for adding shipping address 33 Table 4.7 Usecase description for choosing a payment method 35 Table 4.8 Usecase description for placing order 37 Table 4.9 Usecase description for paying order 38 Table 4.10 Usecase description for rating and reviewing product 40 Table 4.11 Usecase description for chatting with admin 42 Table 4.12 Usecase description for chatting with user 43 Table 4.13 Usecase description for training dataset 45 Table 4.14 Usecase description for editing question 47 Table 4.15 Usecase description for deleting question 49 Table 4.16 Usecase description for adding dataset 50 Table 4.17 Usecase description for deleting dataset 52 xiv LIST OF FIGURES Figure 2.1 The MERN stack architecture Figure 2.2 Introduction of chatbot Figure 4.1 System architechture 19 Figure 4.2 Tables in the database 20 Figure 4.3 Sequence diagram for adding product to cart 21 Figure 4.4 Sequence diagram for placing order 21 Figure 4.5 Sequence diagram for making payment 22 Figure 4.6 Sequence diagram for chatting with virtual bot 23 Figure 4.7 Sequence diagram for chatting with admin 24 Figure 4.8 Sequence diagram for updating question 25 Figure 4.9 Usecase diagram for system 26 Figure 4.10 Usecase diagram for searching product 27 Figure 4.11 Usecase diagram for sorting product 28 Figure 4.12 Usecase diagram for chatting with bot 30 Figure 4.13 Usecase diagram for adding product to cart 32 Figure 4.14 Usecase diagram for adding shipping address 33 Figure 4.15 Usecase diagram for choosing a payment method 35 Figure 4.16 Usecase diagram for placing order 37 Figure 4.17 Usecase diagram for paying order 38 Figure 4.18 Usecase diagram for rating and reviewing product 40 Figure 4.19 Usecase diagram for chatting with admin 42 Figure 4.20 Usecase diagram for chatting with user 43 Figure 4.21 Usecase for training dataset 45 Figure 4.22 Usecase diagram for managin question 46 Figure 4.23 Usecase Diagram for editing question 47 Figure 4.24 Usecase diagram for deleting question 48 Figure 4.25 Usecase diagram for managing dataset 50 Figure 4.26 Usecase diagram for adding dataset 50 Figure 4.27 Usecase diagram for deleting dataset 52 xv Figure 4.28 UI design for Home Screen 54 Figure 4.29 UI design for Product Screen 55 Figure 4.30 UI design for Cart Screen 55 Figure 4.31 UI design for Search Result Screen 56 Figure 4.32 UI design for chatting with virtual bot 57 Figure 4.33 UI design for chatting with virtual bot and admin 58 Figure 4.34 UI design for Place Order Screen 59 Figure 4.35 UI design for Order History Screen 59 Figure 4.36 UI design for chatting with user 60 Figure 4.37 UI design for Q&A Screen 61 Figure 4.38 UI design for Dataset Screen 61 Figure 4.39 Chatbot system model 63 Figure 4.40 Re-trainning function for admin 64 Figure 4.41 Speech to text model 65 Figure 4.42 FastAPI and FlaskAPI 66 Figure 4.43 Natural Language processing model 67 Figure 4.44 The price of Text to speech FPT.ai 72 Figure 4.45 The price of Text to speech vbee.vn 73 Figure 4.46 The first research process model 74 Figure 5.1 Login to Heroku 77 Figure 5.2 Create heroku app 77 Figure 5.3 Create build script in package.json 77 Figure 5.4 Set node and npm version on the Heroku server 77 Figure 5.5 Create Procfile 78 Figure 5.6 Set database connection in Heroku env variables 78 Figure 5.7 Set env variable to skip preflight check 78 Figure 5.8 Commit and push to Heroku 78 Figure 5.9 Deploy changes to Heroku 78 Figure 5.10 Create VM instances 79 Figure 5.11 Installed VM successfully 79 Figure 5.12 Connect Server Ubuntu generated from VM by SSH 80 Figure 5.13 Clone source code by Git 80 xvi { "tag":"current_tag", "patterns":"new_patterns", "responses":"new_responses" } Figure 5.26 API add new dataset (pattern, response) o Execute API delete “Tag”in list dataset POST: https://quocdatit.tk/dataset/delete-dataset/ { "tag":"your_tag" } Figure 5.27 API delete Tag in list dataset o Execute API Test-to-Speech of Zalo.ai POST: https://quocdatit.tk/texttospeech/soundAPI/ 88 { "text": "xin chào, nhân viên bán hàng ảo" } Figure 5.28 API Test-to-Speech of Zalo.ai Return url mp3 sound: https://chunk.lab.zalo.ai/705d27a244f0adaef4e1/705d27a244f0adaef4e1 5.4 Built screens 5.4.1 Home Screen Figure 5.29 Home Sreen 89 5.4.2 Product Screen Figure 5.30 Product Screen 5.4.3 Cart Screen Figure 5.31 Cart Screen 90 5.4.4 Place Order Screen Figure 5.32 Place Order Screen 5.4.5 Welcome interface of Chatbot Figure 5.33 Welcom interface of Chatbot 91 5.4.6 Chatbot Figure 5.34 Main interface of Chatbot 5.4.7 List Question Screen (Admin) Figure 5.35 List Question Screen (Admin) 92 5.4.8 Edit Question Screen (Admin) Figure 5.36 Edit Question Screen (Admin) 5.4.9 List Dataset Screen (Admin) Figure 5.37 List Dataset Screen (Admin) 93 5.4.10 Edit Dataset Screen (Admin) Figure 5.38 Edit Dataset Screen (Admin) 5.4.11 Create New Dataset Screen (Admin) Figure 5.39 Create New Dataset Screen (Admin) 94 Chapter 6: CONCLUSION 6.1 Conclusion The implementation team has completed 90% of the work according to the initial target - Website: Completed main features for an e-commerce website with actors: o Unauthenticated user: This actor can view list of products, view the details of the products, add products to the cart, search for products, chat with BOT o Customer: customers can place order, make payment, rate and review products, update personal information, chat online with BOT and Admin o Admin: admin can manage product (create product, edit product, delete product), manage user (edit user, delete user), manage order (edit order, delete order), manage question (edit question, delete question), manage dataset (add dataset, delete dataset), train dataset, chat with customer - Deep learning service: The system is complete with processing functions: o Automatic message reply function o Training data with good results (over 95%): When training with n_epoch = 500, we get a graph with a significantly reduced loss rate (loss less than 0.1%) Figure 6.1 The loss rate after training 95 o Predict the answer well: when the probability reaches more than 60% will return the prediction result, otherwise send the message "not learned" ▪ Example 1: Message customer: “chào bạn” Figure 6.2 The predict over 60% Red point: location tag highest (more than 60%) Blue point: location of other points Result messages predict: “Xin chào Mình hỗ trợ cho bạn khơng? Bạn hỏi .” ▪ Example 2: Message customer: “có bán giày khơng” 96 Figure 6.3 The predict under 60% Red point: location tag highest (less than 60%) Blue point: location of other points Result messages predict: “Có lẽ tơi chưa học, bạn đợi Admin để giải đáp nhé.” - Experience gained: o Deeply understand about the deep learning, specifically the chatbot o Come up with ideas and design a processing flow that is a data update system for chatbot o API design and system deployment on Ubuntu server of the Google Cloud Platform o Speech-to-Text and Text-to-Speech technology can be applied o Understand how deep learning fits into a website 6.2 Advantages - Frontend and backend are both uploaded to Heroku, this keeps the website ready, anyone can access the website at any time - MERN stack is a modern technology, it helps to build an organized, easy-to97 manage, extensible and upgradeable source code structure, in addition, it also helps to process requests quickly, improve website performance - The chatbot system has been trained using deep learning, which gives the best results With the ability to update training data, new learning will be more convenient and can be applied to many fields - From the Speech-to-Text, Text-to-Speech technology of virtual assistants, we can apply that to chatbots to improve the ability to communicate online with customers more closely 6.3 Disadvantages - The interface is not really eye-catching, some of the layouts are not properly arranged - The source code still has many places that are not well optimized, leading to reduced performance - Although Speech-to-Text has been applied, it has not yet brought the closest approach to users - Processing speed has not been improved well 6.4 Development directions - Using system Sentiment Analysis customer - Optimize code for improved performance and easy extensibility - Redesign the interface and add advanced functions to make it easier to reach users 98 REFERENCES [1] MongoDB Inc, MERN Stack [online] Available at: https://www.mongodb.com/mern-stack [Accessed July 05, 2021] [2] Mau Ha Quang, Nhung gi can biet ve Chatbot [online] Available at: https://viblo.asia/p/tat-ca-nhung-gi-ban-can-biet-ve-chatbot-Az45bnNg5xY [Accessed July 05, 2021] [3] Wikipedia, Chatbot [online] Available at: https://en.wikipedia.org/wiki/Chatbot#Customer_Service [Accessed July 05, 2021] [4] Wikipedia, Virtual assistant [online] Available at: https://en.wikipedia.org/wiki/Virtual_assistant [Accessed July 05, 2021] [5] Zalo, Text to Speech [online] Available at: https://zalo.ai/docs/api/text-to-audio-converter [Accessed July 05, 2021] [6] FPT Corporation, Text to Speech [online] Available at: https://docs.fpt.ai/docs/en/speech/documentation/text-to-speech [Accessed July 05, 2021] [7] Wikipedia, Python [online] Available at: https://en.wikipedia.org/wiki/Python_(programming_language) [Accessed July 06, 2021] [8] Wikipedia, TensorFlow [online] Available at: https://en.wikipedia.org/wiki/TensorFlow [Accessed July 08, 2021] [9] JavaTpoint, Advantage and disadvantage of TensorFlow [online] Available at: https://www.javatpoint.com/advantage-and-disadvantage-of-tensorflow [Accessed July 08, 2021] [10] Trần Ngọc Minh, Cập nhật nhanh TensorFlow 2.0 [online] Available at: https://ngocminhtran.com/2020/03/22/cap-nhat-nhanh-tensorflow-2-0/ 99 [Accessed July 08, 2021] [11] Wikipedia, Keras [online] Available at: https://en.wikipedia.org/wiki/Keras [Accessed July 08, 2021] [12] DataFlair, Keras Advantages and Limitations [online] Avaiable at: https://data-flair.training/blogs/python-keras-advantages-and-limitations/ [Accessed July 08, 2021] [13] Wikipedia, Natural Language Toolkit [online] Available at: https://vi.wikipedia.org/wiki/Natural_Language_Toolkit [Accessed July 12, 2021] [14] Guru99, NLTK Tokenize words sentences [online] Available at: https://www.guru99.com/tokenize-words-sentences-nltk.html [Accessed July 10, 2021] [15] Wikipedia, Bag-of-words [online] Available at: https://en.wikipedia.org/wiki/Bag-of-words_model [Accessed July 12, 2021] [16] Tran Xuan Chien, NLP Stemming and Lemmatazation [online] Available at: https://chienuit.wordpress.com/2016/05/03/nlp-stemming-va-lemmatization/ [Accessed July 12, 2021] [17] Vu Anh, Readme underthesea [online] Available at: https://underthesea.readthedocs.io/en/latest/readme.html [Accessed July 14, 2021] [18] FastAPI, FastAPI [online] Available at: https://fastapi.tiangolo.com/ [Accessed July 14, 2021] [19] Đào Hải Long, Vì chọn FastAPI [online] Available at: https://viblo.asia/p/vi-sao-chon-fastapi-gGJ59M6D5X2 [Accessed July 14, 2021] [20] Wikipedia, ReactJS [online] Available at: https://en.wikipedia.org/wiki/React_(JavaScript_library) [Accessed July 06, 2021] [21] Wikipedia, MongoDB [online] Available at: https://en.wikipedia.org/wiki/MongoDB [Accessed July 06, 2021] 100 [22] Wikipedia, NodeJS [online] Available at: https://en.wikipedia.org/wiki/Node.js [Accessed July 06, 2021] [23] Wikipedia, Express Framework [online] Available at: https://en.wikipedia.org/wiki/Express.js [Accessed July 06, 2021] [24] Socket.IO, Introduction [online] Available at: https://socket.io/docs/v4/ [Accessed July 08, 2021] 101 JOB TABLE OF WORK Job Vo Thanh Ho Quoc Actual start Actual end Van Dat date date x 15-03-2021 15-04-2021 Study and design Chatbot system Design UI for website x 15-03-2021 30-03-2021 Build backend for website x 01-04-2021 10-07-2021 Build frontend for website x 01-04-2021 10-07-2021 x 16-04-2021 01-05-2021 x 02-05-2021 15-05-2021 01-06-2021 03-06-2021 x 16-05-2021 16-06-2021 x 01-07-2021 17-07-2021 Build chatbot run in terminal Study use Text-to-Speech and Speech-to-Text Deploy website on Heroku Research, design x and deploy APIs chatbot on Ubuntu server of Google Cloud Flatform Write report documents x 102 ... large amounts of natural language data • The goal of NLP is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them The technology. .. Virtual assistant 2.4.1 Definition • An intelligent virtual assistant (IVA) or intelligent personal assistant (IPA) is a software agent that can perform tasks or services for an individual based... media playback via voice, and manage other basic tasks such as email, to-do lists, and calendars with verbal (spoken) commands [4] 2.4.2 The advantages of virtual assistants set up in chatbot •