Study lambda architecture and build a demo application

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION GRADUATION THESIS INFORMATION TECHNOLOGY STUDY LAMBDA ARCHITECTURE AND BUILD A DEMO APPLICATION LECTURER: M.S QUACH DINH HOANG STUDENTS: NGUYEN THANH PHAT TRAN DUC TUAN SKL 011153 Ho Chi Minh City, August 2023 MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH-QUALITY TRAINING  CAPSTONE PROJECT INFORMATION TECHNOLOGY STUDY LAMBDA ARCHITECTURE AND BUILD A DEMO APPLICATION LECTURER: M.S Quach Dinh Hoang STUDENTS: Nguyen Thanh Phat - 19110101 Tran Duc Tuan - 19110140 Ho Chi Minh City, August 2023 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 11, 2023 GRADUATION PROJECT ASSIGNMENT Student name: Nguyen Thanh Phat Student ID: 19110101 Student name: Tran Duc Tuan Student ID: 19110140 Major: Software Engineering Class: 19110CLA4 Advisor: M.S Quach Dinh Hoang Date of assignment: 06/03/2023 Date of submission: 19/06/2023 Project title: Study Lambda architecture and build a demo application Initial materials provided by the advisor: Advisor suggests me to read about some reference sources or articles about Lambda architecture which will help me to this project Content of the project: Combine Big Data tools together and get advantage of them Then learning about Lambda architecture and how it resolves problems about scalability and complexity of Big Data systems Final product: A model that applies Lambda architecture to stream and process a massive amount of data and a dashboard illustrates these data under charts CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 11, 2023 ADVISOR’S EVALUATION SHEET Student name: Nguyen Thanh Phat Student ID: 19110101 Student name: Tran Duc Tuan Student ID: 19110140 Major: Software Engineering Project title: Study Lambda architecture and build a demo application Advisor: M.S Quach Dinh Hoang EVALUATION Content of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: ……………… (in words: ) Ho Chi Minh City, August 11, 2023 ADVISOR (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 11, 2023 PRE-DEFENSE EVALUATION SHEET Student name: Nguyen Thanh Phat Student ID: 19110101 Student name: Tran Duc Tuan Student ID: 19110140 Major: Software Engineering Project title: Study Lambda architecture and build a demo application Name of Reviewer: PhD Tran Nhat Quang EVALUATION Content and workload of the project Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, August 11, 2023 REVIEWER (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 11, 2023 EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Nguyen Thanh Phat Student ID: 19110101 Student name: Tran Duc Tuan Student ID: 19110140 Major: Software Engineering Project title: Study Lambda architecture and build a demo application Name of defense Committee Member: EVALUATION Content of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: ……………… (in words: ) Ho Chi Minh City, August 11, 2023 COMMITTEE MEMBER (Sign with full name) ACKNOWLEDGEMENTS We would like to offer our heartfelt appreciation to the Department of High-Quality Training, Ho Chi Minh City University of Technology and Education, and all the professors and teachers who have actively taught and aided me during my study and research In particular, we would like to express our heartfelt thanks to Mr Quach Dinh Hoang for personally instructing, leading and establishing all suitable conditions to let me carry out the thesis He helped us a lot in picking a topic and directing us to understand, investigate the theory and practice the issue During the thesis implementation, the knowledge that the teacher provided us was extremely valuable, not only assisting us in completing the thesis but also supplementing us with a significant amount of core knowledge However, due to our limited professional expertise and limitations in our personal experience, the substance of the report is unavoidable We would like to get your thoughts and more direction He guides and criticizes to make this report more thorough Ho Chi Minh City, August 11, 2023 Group student Nguyen Thanh Phat - 19110101 Tran Duc Tuan - 19110140 FOREWORD As the technologies are more and more developing, there is a massive amount of data created every day which leads us to the word “Big Data” When Big Data comes in, there are a lot of problems emerging such as scalability, complexity, high latency and fault tolerance Traditional databases have no ability to deal with Big Data and NoSQL databases are very powerful but it has some trade-offs to process To restrict the limitations of NoSQL and traditional databases, we will combine these tools together to ensure the system runs with low latency, high performance and fault tolerance which allows to continue running even if the system goes down randomly This architecture is called the Lambda architecture which we will discuss and put into practice later Applying that practice, our team would like to research deeper and present a report on the application topic of Lambda architecture which is "Study Lambda architecture and build a demo application" TABLE OF CONTENTS ACKNOWLEDGEMENTS FOREWORD TABLE OF CONTENTS LIST OF FIGURES 13 CHAPTER INTRODUCTION 15 1.1 Motivation for selecting the topic 1.2 Research object 1.3 Project scope 1.4 Aims and objectives CHAPTER OVERVIEW OF LAMBDA ARCHITECTURE 18 2.1 Definition 2.2 Desired properties 2.2.1 Fault tolerance 2.2.2 Low latency 2.2.3 High throughput 2.2.4 Scalability 2.2.5 Extensibility 2.2.6 Ad hoc queries 2.3 Architecture 2.3.1 Batch layer 2.3.2 Speed layer 2.3.3 Serving layer CHAPTER APPLY LAMBDA ARCHITECTURE TO SOLVE THE PROBLEMS OF DATABASE SCALABILITY IN THE REAL WORLD 22 3.1 Some related research 3.1.1 Research on “Big Data: Principles and Best Practices of Scalable Realtime Data Systems” [1] 3.1.1.1 Summary 3.1.1.2 Result of the research 3.2 Design Lambda architecture CHAPTER EXPERIMENTS AND RESULTS 26 4.1 Data for experiments 4.1.1 Using Twitter API 4.1.1.1 Definition 4.1.1.2 Limitation 4.1.2 Using static dataset 4.1.2.1 Reference source for data 4.1.2.2 Description of the dataset 4.2 Environments and tools 4.2.1 Python 4.2.2 Docker 4.2.3 Visual Studio Code 4.2.4 Great Expectations 4.3 Big data tools 4.3.1 Snowflake 4.3.1.1 Definition 4.3.1.2 Architecture 4.3.1.2.1 Database storage 4.3.1.2.2 Query processing 10