(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning(Đồ án tốt nghiệp) Developing a virtual sales assistant using deep learning
HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING CAPSTONE PROJECT DEVELOPING A VIRTUAL SALES ASSISTANT USING DEEP LEARNING HO QUOC DAT Student ID: 17110117 VO THANH VAN Student ID: 17110252 Major: INFORMATION TECHNOLOGY Supervisor: TRAN NHAT QUANG, PhD Ho Chi Minh, July 2021 ACKNOWLEDGEMENTS First, the implementation team would like to send our sincere thanks to the Faculty for High-Quality Training, Ho Chi Minh City University of Technology and Education for creating favorable conditions for us to study and research The Faculty for High-Quality Training has provided the necessary information and documents for us to acquire and expand our knowledge in the field of Information Technology, thereby helping us to successfully complete the Graduation Thesis The sincerest thanks to the implementation team would like to sincerely send to Mr TRAN NHAT QUANG - who used all the teacher's enthusiasm and knowledge, accompanied, and directly taught, guided, and created all favorable conditions, help the group to develop their best as well as improve their knowledge throughout the learning process, especially in the process of preparing and implementing the Graduation thesis Thank you for your enthusiasm, which is a tremendous motivation to help the group perform, persevere throughout the process of implementing the topic and discover new interesting and useful knowledge related to the topic namely, building a virtual sales assistant on the website with deep learning Student group implementation v DISCLAIMER The group hereby declares that this project is done by the members of the group The group does not copy or use any other people's documents, source code, The group will take full responsibility for plagiarism violations vi ABSTRACT Today information technology has been developing to a high level, is applied to all sectors and fields of the country's economy, making an important contribution to making these industries develop faster and more efficiently Online shopping now is necessary because the Covid epidemic is still raging When customers shop online, they will also need to be consulted by staff for more product information, and how to order and purchase Therefore, the communication system between customers and employees is improved to always serve customers Through the research, the implementation team found that using a virtual sales assistant to replace real people in customer support brings great benefits This reduces staffing costs, ensures customers are always supported, and improves user experience Therefore, the implementation team built a chat bot system on an e-commerce website The chat bot system is developed by python language, using Deep Learning model and Tensorflow, Keras, NLTK, Under The Sea, Fast API libraries When customers send messages in the chat interface, the chat bot will respond based on the content of those messages and send them back to the customer The chat interface also has customer support functions such as allowing the use of microphones to speak instead of typing, converting text to speech and vice versa In addition, the e-commerce website that we built using MEARN stack also has basic functions such as viewing product details, adding products to the cart, reviewing products, making payments, and placing orders (customer side) and managing product, managing user, managing order, managing question and answer sets (admin side) vii TABLE OF CONTENTS PROJECT ASSIGNMENT i ADVISOR’S EVALUATION SHEET ii PRE-DEFENSE EVALUATION SHEET iii EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER iv ACKNOWLEDGEMENTS v DISCLAIMER vi ABSTRACT vii TABLE OF CONTENTS viii LIST OF ACRONYMS xiii LIST OF TABLES xiv LIST OF FIGURES xv Chapter 1: INTRODUCTION 1.1 The reason for choosing the topic 1.2 Goals 1.3 Technologies Chapter 2: LITERATURE REVIEW 2.1 MERN stack 2.2 Deep Learning 2.3 Chatbot 2.3.1 Definition 2.3.2 Building a Chatbot 2.3.3 Advantages of Chatbot 2.3.4 Disadvantages of chatbot 2.4 Virtual assistant 2.4.1 Definition 2.4.2 The advantages of virtual assistants set up in chatbot 2.5 Text-To-Speech platforms 2.5.1 FPT API Text-To-Speech 2.5.2 ZALO API Text-To-Speech Chapter 3: THEORETICAL BASIS viii 3.1 Deep Learning 3.1.1 Python programming language a Advantages b Disadvantages c Versions 3.1.2 Tensorflow a Advantages b Disadvantages c Version 10 3.1.3 Keras 11 a Advantages 11 b Disadvantages 11 3.1.4 Natural language processing and the NLTK library 11 a Definition 11 b Tokenization of NLTK 12 c Bag-of-words 12 d Stemming 13 e Lemmatization 13 3.1.5 Underthesea – A Vietnamese NLP Toolkit 14 a Installation 14 b Usage: 14 3.1.6 FastAPI 15 3.2 a Definition 15 b Compare Three Python Frameworks Django, Flask and FastAPI 15 Technologies for website building 17 3.2.1 React Library 17 3.2.2 MongoDB 17 3.2.3 Node js 17 3.2.4 Express framework 17 3.2.5 Socket.IO 18 Chapter 4: SYSTEM DESIGN 19 4.1 System Architechture 19 4.2 System design 19 4.2.1 Database Design 19 ix 4.2.2 Sequence Diagram 21 4.2.2.1 Add product to cart 21 4.2.2.2 Place order 21 4.2.2.3 Make payment 22 4.2.2.4 Chat with Virtual bot 23 4.2.2.5 Chat with Admin 24 4.2.2.6 Update Question 25 4.2.3 Use Case Diagram 26 4.2.4 Use Case Description 27 4.2.4.1 Search product 27 4.2.4.2 Sort product 28 4.2.4.3 Chat with bot 30 4.2.4.4 Add product to cart 32 4.2.4.5 Add shipping address 33 4.2.4.6 Choose a payment method 35 4.2.4.7 Place order 37 4.2.4.8 Pay order 38 4.2.4.9 Rate and Review product 40 4.2.4.10 Chat with Admin 42 4.2.4.11 Chat with user 43 4.2.4.12 Train dataset 45 4.2.4.13 Manage question 46 a Edit question 47 b Delete question 48 4.2.4.14 Manage dataset 50 a Add dataset 50 b Delete dataset 52 4.2.5 User Interface Design 54 4.2.5.1 Screens for unauthenticated users 54 a Home Screen 54 b Product Screen 55 c Cart Screen 55 d Search Result Screen 56 e Chat with virtual bot 57 4.2.5.2 Screens for customers 58 x a Chat with virtual bot and admin 58 b Place Order Screen 59 c Order History Screen 59 4.2.5.3 Screens for admins 60 4.3 a Chat with user 60 b Q&A Screen 61 c Dataset Screen 61 Chatbot system 63 4.3.1 General system 63 4.3.2 Details of each part 65 a Speech to text 65 b API Input and Output Processing Center 66 c Natural language processing 67 d Dataset 68 e System deep learning and message predicting results 69 f Text to speech 71 g Develop new data addition functionality for chatbot 74 h Deploy server 75 Chapter 5: SYSTEM IMPLEMENTATION 76 5.1 Software development environments 76 5.2 Source code management 76 5.3 Deployment environment 76 5.3.1 Deploying the website on Heroku 76 5.3.2 Deploying the Chatbot API on Google Cloud Platform 79 5.4 Built screens 89 5.4.1 Home Screen 89 5.4.2 Product Screen 90 5.4.3 Cart Screen 90 5.4.4 Place Order Screen 91 5.4.5 Welcome interface of Chatbot 91 5.4.6 Chatbot 92 5.4.7 List Question Screen (Admin) 92 5.4.8 Edit Question Screen (Admin) 93 5.4.9 List Dataset Screen (Admin) 93 xi 5.4.10 Edit Dataset Screen (Admin) 94 5.4.11 Create New Dataset Screen (Admin) 94 Chapter 6: CONCLUSION 95 6.1 Conclusion 95 6.2 Advantages 97 6.3 Disadvantages 98 6.4 Development directions 98 REFERENCES 99 JOB TABLE OF WORK 102 xii LIST OF ACRONYMS DB: Database TTS: Text to Speech STT: Speech to Text NLTK: Natural Language Toolkit NLP: Natural language processing API: Application Programming Interface UI: User Interface gTTS: Google Text-to-Speech VM: Virtual Machines SSH: Secure Shell xiii { "text": "xin chào, nhân viên bán hàng ảo" } Figure 5.28 API Test-to-Speech of Zalo.ai Return url mp3 sound: https://chunk.lab.zalo.ai/705d27a244f0adaef4e1/705d27a244f0adaef4e1 5.4 Built screens 5.4.1 Home Screen Figure 5.29 Home Sreen 89 5.4.2 Product Screen Figure 5.30 Product Screen 5.4.3 Cart Screen Figure 5.31 Cart Screen 90 5.4.4 Place Order Screen Figure 5.32 Place Order Screen 5.4.5 Welcome interface of Chatbot Figure 5.33 Welcom interface of Chatbot 91 5.4.6 Chatbot Figure 5.34 Main interface of Chatbot 5.4.7 List Question Screen (Admin) Figure 5.35 List Question Screen (Admin) 92 5.4.8 Edit Question Screen (Admin) Figure 5.36 Edit Question Screen (Admin) 5.4.9 List Dataset Screen (Admin) Figure 5.37 List Dataset Screen (Admin) 93 5.4.10 Edit Dataset Screen (Admin) Figure 5.38 Edit Dataset Screen (Admin) 5.4.11 Create New Dataset Screen (Admin) Figure 5.39 Create New Dataset Screen (Admin) 94 Chapter 6: CONCLUSION 6.1 Conclusion The implementation team has completed 90% of the work according to the initial target - Website: Completed main features for an e-commerce website with actors: o Unauthenticated user: This actor can view list of products, view the details of the products, add products to the cart, search for products, chat with BOT o Customer: customers can place order, make payment, rate and review products, update personal information, chat online with BOT and Admin o Admin: admin can manage product (create product, edit product, delete product), manage user (edit user, delete user), manage order (edit order, delete order), manage question (edit question, delete question), manage dataset (add dataset, delete dataset), train dataset, chat with customer - Deep learning service: The system is complete with processing functions: o Automatic message reply function o Training data with good results (over 95%): When training with n_epoch = 500, we get a graph with a significantly reduced loss rate (loss less than 0.1%) Figure 6.1 The loss rate after training 95 o Predict the answer well: when the probability reaches more than 60% will return the prediction result, otherwise send the message "not learned" ▪ Example 1: Message customer: “chào bạn” Figure 6.2 The predict over 60% Red point: location tag highest (more than 60%) Blue point: location of other points Result messages predict: “Xin chào Mình hỗ trợ cho bạn khơng? Bạn hỏi .” ▪ Example 2: Message customer: “có bán giày không” 96 Figure 6.3 The predict under 60% Red point: location tag highest (less than 60%) Blue point: location of other points Result messages predict: “Có lẽ tơi chưa học, bạn đợi Admin để giải đáp nhé.” - Experience gained: o Deeply understand about the deep learning, specifically the chatbot o Come up with ideas and design a processing flow that is a data update system for chatbot o API design and system deployment on Ubuntu server of the Google Cloud Platform o Speech-to-Text and Text-to-Speech technology can be applied o Understand how deep learning fits into a website 6.2 Advantages - Frontend and backend are both uploaded to Heroku, this keeps the website ready, anyone can access the website at any time - MERN stack is a modern technology, it helps to build an organized, easy-to97 manage, extensible and upgradeable source code structure, in addition, it also helps to process requests quickly, improve website performance - The chatbot system has been trained using deep learning, which gives the best results With the ability to update training data, new learning will be more convenient and can be applied to many fields - From the Speech-to-Text, Text-to-Speech technology of virtual assistants, we can apply that to chatbots to improve the ability to communicate online with customers more closely 6.3 Disadvantages - The interface is not really eye-catching, some of the layouts are not properly arranged - The source code still has many places that are not well optimized, leading to reduced performance - Although Speech-to-Text has been applied, it has not yet brought the closest approach to users - Processing speed has not been improved well 6.4 Development directions - Using system Sentiment Analysis customer - Optimize code for improved performance and easy extensibility - Redesign the interface and add advanced functions to make it easier to reach users 98 REFERENCES [1] MongoDB Inc, MERN Stack [online] Available at: https://www.mongodb.com/mern-stack [Accessed July 05, 2021] [2] Mau Ha Quang, Nhung gi can biet ve Chatbot [online] Available at: https://viblo.asia/p/tat-ca-nhung-gi-ban-can-biet-ve-chatbot-Az45bnNg5xY [Accessed July 05, 2021] [3] Wikipedia, Chatbot [online] Available at: https://en.wikipedia.org/wiki/Chatbot#Customer_Service [Accessed July 05, 2021] [4] Wikipedia, Virtual assistant [online] Available at: https://en.wikipedia.org/wiki/Virtual_assistant [Accessed July 05, 2021] [5] Zalo, Text to Speech [online] Available at: https://zalo.ai/docs/api/text-to-audio-converter [Accessed July 05, 2021] [6] FPT Corporation, Text to Speech [online] Available at: https://docs.fpt.ai/docs/en/speech/documentation/text-to-speech [Accessed July 05, 2021] [7] Wikipedia, Python [online] Available at: https://en.wikipedia.org/wiki/Python_(programming_language) [Accessed July 06, 2021] [8] Wikipedia, TensorFlow [online] Available at: https://en.wikipedia.org/wiki/TensorFlow [Accessed July 08, 2021] [9] JavaTpoint, Advantage and disadvantage of TensorFlow [online] Available at: https://www.javatpoint.com/advantage-and-disadvantage-of-tensorflow [Accessed July 08, 2021] [10] Trần Ngọc Minh, Cập nhật nhanh TensorFlow 2.0 [online] Available at: https://ngocminhtran.com/2020/03/22/cap-nhat-nhanh-tensorflow-2-0/ 99 [Accessed July 08, 2021] [11] Wikipedia, Keras [online] Available at: https://en.wikipedia.org/wiki/Keras [Accessed July 08, 2021] [12] DataFlair, Keras Advantages and Limitations [online] Avaiable at: https://data-flair.training/blogs/python-keras-advantages-and-limitations/ [Accessed July 08, 2021] [13] Wikipedia, Natural Language Toolkit [online] Available at: https://vi.wikipedia.org/wiki/Natural_Language_Toolkit [Accessed July 12, 2021] [14] Guru99, NLTK Tokenize words sentences [online] Available at: https://www.guru99.com/tokenize-words-sentences-nltk.html [Accessed July 10, 2021] [15] Wikipedia, Bag-of-words [online] Available at: https://en.wikipedia.org/wiki/Bag-of-words_model [Accessed July 12, 2021] [16] Tran Xuan Chien, NLP Stemming and Lemmatazation [online] Available at: https://chienuit.wordpress.com/2016/05/03/nlp-stemming-va-lemmatization/ [Accessed July 12, 2021] [17] Vu Anh, Readme underthesea [online] Available at: https://underthesea.readthedocs.io/en/latest/readme.html [Accessed July 14, 2021] [18] FastAPI, FastAPI [online] Available at: https://fastapi.tiangolo.com/ [Accessed July 14, 2021] [19] Đào Hải Long, Vì chọn FastAPI [online] Available at: https://viblo.asia/p/vi-sao-chon-fastapi-gGJ59M6D5X2 [Accessed July 14, 2021] [20] Wikipedia, ReactJS [online] Available at: https://en.wikipedia.org/wiki/React_(JavaScript_library) [Accessed July 06, 2021] [21] Wikipedia, MongoDB [online] Available at: https://en.wikipedia.org/wiki/MongoDB [Accessed July 06, 2021] 100 [22] Wikipedia, NodeJS [online] Available at: https://en.wikipedia.org/wiki/Node.js [Accessed July 06, 2021] [23] Wikipedia, Express Framework [online] Available at: https://en.wikipedia.org/wiki/Express.js [Accessed July 06, 2021] [24] Socket.IO, Introduction [online] Available at: https://socket.io/docs/v4/ [Accessed July 08, 2021] 101 JOB TABLE OF WORK Job Vo Thanh Ho Quoc Actual start Actual end Van Dat date date x 15-03-2021 15-04-2021 Study and design Chatbot system Design UI for website x 15-03-2021 30-03-2021 Build backend for website x 01-04-2021 10-07-2021 Build frontend for website x 01-04-2021 10-07-2021 x 16-04-2021 01-05-2021 x 02-05-2021 15-05-2021 01-06-2021 03-06-2021 x 16-05-2021 16-06-2021 x 01-07-2021 17-07-2021 Build chatbot run in terminal Study use Text-to-Speech and Speech-to-Text Deploy website on Heroku Research, design x and deploy APIs chatbot on Ubuntu server of Google Cloud Flatform Write report documents x 102 S K L 0 ... Virtual assistant 2.4.1 Definition • An intelligent virtual assistant (IVA) or intelligent personal assistant (IPA) is a software agent that can perform tasks or services for an individual based... 1.2 Goals Create a virtual sales staff to advise products on a website selling clothes using deep learning: • Find out the features and functions required for a virtual salesperson • Learn and... media playback via voice, and manage other basic tasks such as email, to-do lists, and calendars with verbal (spoken) commands [4] 2.4.2 The advantages of virtual assistants set up in chatbot •