Khóa luận tốt nghiệp Hệ thống thông tin: Applying satellite data with ground-based air quality monitoring stations to predict hourly air quality of Ho Chi Minh

LIST OE FIGURESFigure 2.1 AQI table with air quality description...-.--‹¿-+©--+c-+cscccrxer 6 Figure 2.2: Time series data example ...scsssssessesecsecseeseeseeseesesesnseneeseeseeneenee

VIETNAM NATIONAL UNIVERSITY HOCHIMINH CITY

UNIVERSITY OF INFORMATION TECHNOLOGY ADVANCED PROGRAM IN INFORMATION SYSTEMS

PHAN HOANG NAM - 16520776

LÊ THỊ PHỤNG - 16521775

USING GROUND-BASED AIR QUALITY

MONITORING STATIONS TO PREDICT HOURLY

AIR QUALITY OF HO CHI MINH CITY

BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS

HO CHI MINH CITY, 2021

NATIONAL UNIVERSITY HOCHIMINH CITY

UNIVERSITY OF INFORMATION TECHNOLOGY

ADVANCED PROGRAM IN INFORMATION SYSTEMS

PHAN HOANG NAM - 16520776

LÊ THỊ PHUNG - 16521775

USING GROUND-BASED AIR QUALITY

MONITORING STATIONS TO PREDICT HOURLY

AIR QUALITY OF HO CHI MINH CITY

BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS

THESIS ADVISOR

Associate Professor NGUYEN DINH THUAN

HO CHI MINH CITY, 2021

First of all, we would like to express our appreciation to Associate Professor

Nguyễn Dinh Thuan for his time and guidance during the making of this thesis.

His teaching has greatly influenced our works and help us change in positive

ways.

Our top positive receptions also go to all the members of Faculty of

Information Systems as well as everyone of University of Information

Technology for their guidance, supports to us with greatest cares

Not the least, we feel in extremely need of showing our gratitude to our

family, our friends, and our classmates for every support and love that we havereceived on our maturity path

July, 2021.Phan Hoang Nam & Le Thi Phung — students of aep 2016

1i

1.2 Current status of research

1.3 _ The problems and its significanc “ 1.4 Motivation cccchnhìnhnình nhu 3 1.5 _ Contributions ccccccccctcttihn ngờ, 3 Chapter 2 RELATED RESOURCES -.- Ăn nnneerrrrrrrrrrre 5

2.1 Air Quality Index (AQT) cccsesessessesecsesseeseeseeatessesnssseneeneeneeneenteneene 5

21.1 Calculate hour AQI for in Viet Nam

2.1.7 Rolling window for time series regression

2.2 Recurrent neural netWOrK -. sc- sec 15

2.2.1 Overview of recurrent neural network

2.2.2 Long Short-Term Memory (LSTM) 2.2.3 Bidirectional RNN and Bidirectional LSTM 2.2.4 Batch Normalization

2.2.5 Dropout

2.3 TensorFlow

2.3.1 Overview

2.3.2 Reason for choosing TensorFlow

2.4 — Google cloud platform

2.5 Evaluation metrics and result

2.5.1 Mean Absolute Error 2.5.2 Mean Absolute Percentage Error

iii

2.5.3 Root Mean Squared Error

Chapter 3 Deep learning model design and evaluation « 27

3.1 SySt€M OVCTVICW tru 27 3.2 Building dataset Ă eo, 287

3.3.1 Overview of model 3.3.2 Evaluation

3.4 Deploy models to google cloud platform - 45

3.4.1 Deploy models to google cloud platform

3.4.2 Make a Flask API to get new data and get new prediction from models

46 3.4.3 Make a Streamlit Front end to show our models on the internet.

Chapter 4 CONCLUSION AND FUTURE WORKS - 50

AA Conclusion căĂ Street 50 4.2 — Future WOrks chi He he 50 Chapter 5.

REFERENCES - con ceeirirrrrrrrrrrrrrrrrrrrrrrrrrrmrrrnrersrersree 52

LIST OE FIGURES

Figure 2.1 AQI table with air quality description -. ‹¿-+© +c-+cscccrxer 6 Figure 2.2: Time series data example scsssssessesecsecseeseeseeseesesesnseneeseeseeneeneenee 9 Figure 2.3: Time series’ COIDOTIIES - St ‡EEkEEEEEEkEEEEkrkrrkrkerkee 10 Figure 2.4: Stationary time series ©XaITDÏA ¿5c cccc+scsrrrxerxerxrrxrrr 11 Figure 2.5: Time series analysis example cccccssessesesseesseeseessseeseseeneseeneenees Figure 2.6: Time series forecasting example

Figure 2.7: Different method for modeling time series data

Figure 2.8: Rolling window description

Figure 2.9: Recurrent network illustration ccs TÔ, Figure 2.10: One neuron in LSTM layer ccscessessesssessessesseeseeseeseesessesseseeseneesees 17 Figure 2.11:The repeating module in an LSTM contains three interacting layers.

Figure 2.14: Dropout method on deep learning neural networKk 21 Figure 2.15: TensorFlow adVanfage : :-cscsccxererrerrrrrrrrrrrrrrrrrrrrrrrk 22 Figure 2.16: Some google cloud platform services ccscseessesseseseeneenseneenees 23 Figure 3.1: Thesis general processes

Figure 3.2: Raw data file - ch re Figure 3.3: Plot of raw data

Figure 3.4: Remains of raw data.

Figure 3.5: Missing data spot

Figure 3.6: Missing data spot 2 - sec OO Figure 3.7: Cleaning data process - 5: tt TH Hư 31 Figure 3.8: Missing data eXapÏC +-5scxertererterterrrrrrrrrrrrrrerrrrrrrer 31 Figure 3.9: Front fill and back fill example respectiveÌy ‹- + 32 Figure 3.10: Time continuous marking ‹ c-s5xe+xssxsxsrerxerxerxerxerxrrv 32 Figure 3.11: Calcualte AQT pTOC€SS - 6 Street 33 Figure 3.12: Dickey fuller test result

Figure 3.13: Autocorrelation plot.

Figure 3.14: Partial Autocorrelation plot

Figure 3.15: Seasonal decomposed pÏOt - + 5c++cscsseeeseeersexereeereee SO

Overview of model.

Training/ validation loss with model prediction on test data General steps to upload model on google cloud platform 45 Upload model to Google Cloud AI Platform -. - 45 Google cloud storage files esses + tt 46 Cronjob for Flask APP[ 55-55 5cccccxeztcrrcerrrrrrrrrrrrrrrrer 47 The interaction between client and docker host -. -:- 48 Streamlit screennshOL 5-55 5+ccxccxerxerkerkerkrrrrrrrrrrrrrrkrrrer 49

vi

Air quality prediction is one of the problems which has been focused recently

On the side of research and application, many machine learning models and

deep learning models has attempted to forecast quality of the air and achieve

noticeable success Viet Nam is also trying to integrate air quality monitoringwith the inclusion of forecasting system in recent years Due to having just

been noticed in Viet Nam, air quality models suffered from data shortage both

by the length of data and number of monitoring sites currently available By

applying some time-series techniques and bi-directional LSTMs, we have tosome extend improve the forecasting result of models on Ho Chi Minh City In

this thesis, we focus on predicting the next five hours air quality in Ho Chi

Minh City with the help of data from space by NASA

vii

Chapter 1

PROBLEM STATEMENT

1.1 Context

Air affects most aspects of human life and contributes greatly to the development of

the economy Today, air pollution is becoming one of the most significant

environmental problems in the world In recent years, air quality is continuously

decreasing because of human’s activities: urbanization, industrialization, vehicles

emission, and some from natural sources like volcanic eruptions and forest fires All

these activities raise the volume of many pollutants in the atmosphere, such as SO2,

NO;, CO2, NO, CO, NOx, especially particulate matter pollutants (PM; and PMjo)

The volume of PM2; negatively influences on human wellbeing mainly because less

than 2.5 microns’ matter can penetrate deep into the lung and cause various diseases

including heart and respiratory problems

According to World Health Organization (WHO), around 4.2 million people die

every year from exposure to ambient air pollution and as many as 60,000 deaths in

Viet Nam in a report from 2016 [1] In another report for air pollution in the year 2020from IQAir, Viet Nam ranked in the top 25 of the most polluted countries in the worldwith the air quality remains nearly 4 times the WHO target for annual exposure The

annual volume of PM: is more than 55.4 ug/m3 or 110 in Air Quality Index (AQD

As one method for monitoring air quality, outside air pollution forecasting have

shown great result in warning population of incoming polluted air and also in raisingawareness of air quality to the people Viet Nam has taken many actions to reduce the

cause of air pollution in recent years Mainly using greener vehicles, alternative energy

sources, reinforcing urban planning and encouraging efficient agriculture practices

1.2 Current status of research

In order to solve increasingly serious environmental pollution problems, many

countries have attempted many machine learning and deep learning system in order to

forecast the air quality in the future

Earliest systems dated back in 1970s and continued to today Overall, systems can

be separated into two types: numerical and data-driven

Numerical models are based on classical physical and chemical theories which

relied on simulating the transport and conversion of chemical components in the air topredict the concentration [2, 3] These deterministic models strive to comprehend theresult of many factors that make up the pollutant concentration but have not achievedsuccess partly because data for these models is hard to collect and difficult to ascertainthe quality

Due to these limitations in numerical approaches, data-driven approach has become

popular as the method for forecasting time-series data Various methods have been

employed with different results One good reference on the issue of forecasting withANN is Kukkonnen et al [4] In their paper, they have succeeded in evaluating and

comparing forecasting models for hourly concentration and PM jo in Helsinky, Finland

Kurt et al [5] also proposed a simple neural network to predict daily air quality and

some parameter investigation in time series model for Greater Istanbul Area In Kurt’s

and colleagues paper, they find that including day of week, holidays, weekend have a

significant effect on deep learning models Some other novel models like MLP, RBF

have also been implemented and shown good results [6, 7]

In the process of making this thesis, maybe due to being a new subject , this work

can’t find any related research in Viet Nam With our limited knowledge, this is the

first model which is studied in Viet Nam

This thesis focuses on forecasting air quality with deep learning Bidirectional Long

Short-Term Memory inspired by [8]

1.3 The problems and its significance

In many past research, researchers have proven that the effect of data in machine

learning models is of great significance In this work, we tried to solve two most

important problem

The first problem we encountered is choosing the right model that would give greatperformance for our goals Based on our gathered data and the natures of our goal wewent with bidirectional LSTM models after some testing This model is designed to

tackle time-series problems and have had some success according to scientific papers

we have read Building and testing model’s parameters are one of the main parts of this

project We also had to modify our data using several time-series techniques to make

them compatible with our model

The second problem we encounter is to meaningfully utilize our model in

production To qualify for production, models need to be robust, free of latency and are

easy to maintain To tackle this problem, we used cloud services and develop the

system into two parts: API server and front-end web Each part solely handles the

backend and frontend of the production environment

1.4 Motivation

Due to time constraint and data constraint, we have set three goals for this thesis:

1 Build air quality dataset of 1 air monitoring station in Ho Chi Minh city

2 Build a model to predict the air quality of that station for the next five hours

3 Successfully deploy models on cloud services for public use

1.5 Contributions

1 We collected and cleaned data from official sites US embassy

2 We organize data in csv files to train and test model

3 Our group applied Vietnamese method certified by the government to calculate

AQI

4 Additional features were extracted and put into models by applying several

time-series techniques

5 With at least thirty minutes latency, we can output the expected air quality for

Ho Chi Minh City region for the next five hours with average root mean squareerror at 15

Chapter 2

RELATED RESOURCES

2.1 Air Quality Index (AQI)

An air quality index (AQI) is used by government agencies to communicate to the

public how polluted the air currently is or how polluted it is forecast to become Public

health risks increase as the AQI rises Different countries have their own air quality

indices, corresponding to different national air quality standards In this thesis we will

be calculating AQI by Viet Nam official standard as stated in this document [9]

There are 6 major pollutants in the air according to the document:

For each pollutant, an AQI value of 100 generally corresponds to an ambient air

concentration that equals the level of the short-term national ambient air quality

standard for protection of public health AQI values at or below 100 are generally

thought of as satisfactory When AQI values are above 100, air quality is unhealthy: at

first for certain sensitive groups of people, then for everyone as AQI values get higher

AQI Basics for Ozone and Particle Pollution

Dally AQT Values of

Color Levels of Concern Index Description of Air Quality

Orange | Unhealthy for Sensitive | 101 to 150 | Members of sensitive groups may experience health effects The general public Is less likely to be

Groups affected.

Figure 2.1 AQI table with air quality description

The AQI is divided into six categories Each category corresponds to a different

level of health concern Each category also has a specific color The color makes it

easy for people to quickly determine whether air quality is reaching unhealthy levels in

their communities

2.1.1 Calculate hour AQI for in Viet Nam

Hour AQT is calculated separately for each pollutant Then the pollutant with the

highest AQI is picked to be presented as that hour AQI

2.1.1.1 What is NowCast?

For some air pollutant, the current amount of concentration does not portrait the

current air quality situation in that monitoring area For those pollutant, we need to

calculate the average concentration for a specific past time frame For PM2.5 and

PM10 pollutant, that average concentration is called NowCast Concentration and

calculated with time frame of 12

2.1.1.2 Calculate Nowcast for PM2.5 and PM10

Exclusively for PM2.5 and PM10 concentration in the air, we must calculate

Nowcast value to estimate them

We call cl, c2, c3, to c12 is the average pollutant concentration for 1 hour (with

cl as the average concentration for the current hour, c12 as the average concentration

for the 12" past hour).

First, we calculate weights: w* = “mứt

Cmax

With C,,jn as the minimum concentration from the past 12 hours

Cnax as the maximum concentration from the past 12 hours

In case that w = 3 then Nowcast = ;Π+ (:) Cott (3) C2

2.1.1.3 Calculate AQI for each hour:

AQI for each hour for PM2.5 and PM10 concentration is then calculated by

— 1;

= ——_ (N t, — BP,) +];BP a BP, 0WCasty ¡) +i;

e AQI„: AQI value for pollutant x

e BP;: Floor concentration rate for the pollutant x Listed in Table 1 according

Giá trị BP; quy định đối với từng thông số (Don vị: ug/m®

O3(1h) | O;(8h) co SO, NO, | PMi | PM;;

Time Series

2.1.2 Overview

A time series is a sequence of data points that occur in successive order over some

period, or a series of data points indexed in time order This can be contrasted with

cross-sectional data, which captures a point-in-time

—— Visitors per month_=— Temperature (F)

Figure 2.2: Time series data example

Most of the data gathered has a temporal structure There are times when this

structure is concealed or ignored, but there are other instances when it can be used to

extract essential information from the existing data

Time series data usually include:

e Timestamp: a mark of the moment in time when the event was registered Its

accuracy will depend on the measured event

e Value: what is the value that this phenomenon had at that moment? Can be just

one or more values When there is more than one value per timestamp, we have

a multivariable time series

2.1.3 Components of time series

Irregular /Random

Component

There are four components to time series:

Figure 2.3: Time series” components

e Trend: Overall and persistent long-term movement

e Seasonal: Regular periodic fluctuations, usually within a 12-month period

¢ Cyclical: Repeated movement not of fixed period, usually of at least 2 years

e Random: Erratic fluctuations

2.1.4 Stationary

A stationary time series is one whose properties do not depend on the time at which

the series is observed Thus, time series with trends, or with seasonality, are not

stationary the trend and seasonality will affect the value of the time series at different

times It does not matter when you observe the time series, it should look much the

same at any point in time A non-stationary series is made stationary by differencing

techniques

10

t t

Stationary series Non-Stationary series

Figure 2.4: Stationary time series example

Some time series model’s performances are greatly influenced by the stationary of

the time series In our thesis, we have also tested our data for stationarity

2.1.5 Usage of time series

For analysis: Time series analysis comprises methods for analyzing time series data

to extract meaningful statistics and other characteristics of time series data It focuses

on comparing values of a single time series or multiple dependent time series at

different points in time We identify the nature of the phenomenon represented by thesequence of observations in the data

11

Time-Series Analysis

Dependent Variable

Time

Figure 2.5: Time series analysis example

For forecasting: Time series forecasting is the use of a time series model to predict

future values based on previously observed values in the series We use the data to

forecast or predict future values of the time series variable

Forecasting With and Without Outliers

— Actual — Prediction (outlier included) — Prediction (outlier removed)

Active Subscription Volume

Week

Figure 2.6: Time series forecasting example

We use Time Series Analysis and Forecasting for many applications where

pertinent time series data can be collected, such as:

e Marketing and Sales Forecasting

Both objectives necessitate identifying and explicitly describing the pattern of

observed time series data We can evaluate and integrate the data once we have

established the pattern (i.e., use it in our theory of the investigated phenomenon, e.g.,seasonal commodity prices) We can extend the identified pattern to predict future

events regardless of our level of comprehension or the correctness of our interpretation

of the phenomenon, with the proviso that the further out in time we try to predict, theless accurate the forecast becomes

2.1.6 Prediction techniques:

The fitting of time series models can be a difficult but necessary task It necessitates

far more data preparation than the standard statistical models used to analyze

“ordinary” data, such as response models, uplift models, and so on, where trends and

seasonal effects are not always present

13

There are several different methods for modeling time series data including thefollowing:

¢ Box-Jenkins ARIMA models

¢ Box-Jenkins Multivariate Models

e Holt-Winters Exponential Smoothing (single, double, triple)

e¢ Unobserved Components Model

¢ Smoothing methods: Averaging and Exponential Smoothing Methods

14

2.1.7 Rolling window for time series regression

Full Seris

——ễễẽ

: Time Imputation

Figure 2.8: Rolling window description

Rolling window is one time series technique to get the subsample of a time seriesfor prediction models The output for rolling window is a data set and label set for

fitting model Steps to perform rolling window are:

¢ Choose a rolling window size m, or in detail the consecutive observation per

rolling window

e Choose the forecast horizon h, which is the desired label for our model

¢ Choose the time step which is the step incremented between each subsample

2.2 Recurrent neural network

2.2.1 Overview of recurrent neural network

Recurrent neural networks, also known as RNNs, are a class of neural networks that

allow previous outputs to be used as inputs while having hidden states They are

typically as follows:

15

The disadvantage of the classic recurrent neural network being:

e Slow computation speed

e Difficulty of accessing information from a long time ago

e Cannot consider any future input for the current state

e With large models and dataset, the backward learning error in recurrent models

tend to vanish or explode Both of which greatly impact model’s performance

leading to poor result.

To counter those disadvantages, this thesis proposed Long Short-Term Memory

designed to fix the short coming of the classic model.

2.2.2 Long Short-Term Memory (LSTM)

The growth of a typical recurrent neural network can be seen in Long Short-Term

Memory (LSTM) Proposed by Sepp Hochreiter and Jiirgen Schmidhuber in a paper,

LSTM model is designed specifically to deal with the vanishing gradient problem

encountered by traditional RNNs.

In concept, this variants of RNN’s recurrent unit tries to “remember” all the past

knowledge that the network is seen so far and to “forget” irrelevant data This is done

by introducing different activation function layers called “gates” for different purposes.

A forget gate, an input gate, and a cell state are proposed by LSTM to decide whether

16

to preserve information via layers Each gate has a specific purpose which allow

LSTM model to perform well on time series problem:

e Forget Gate(f): It determines to what extent to forget the previous data.

e Input Gate(): It determines the extent of information to be written onto the

Internal Cell State.

e Output Gate(o): It determines what output (next Hidden State) to generate

from the current Internal Cell State.

Figure 2.10: One neuron in LSTM layer

q) (h) (2

&)

Figure 2.11:The repeating module in an LSTM contains three interacting layers.

17

LSTM used tanh function as its activation function This is to overcome the

vanishing gradient problem Tanh is the Hyperbolic tangent function, defined as the ratio between the hyperbolic sine and the cosine functions Fig 2.12 show the tanh representation.

2

- -f 4] ĩ #

x

Figure 2.12: Tanh representation

2.2.3 Bidirectional RNN and Bidirectional LSTM

Bidirectional RNNs were introduced by Schuster & Paliwal, 1997 [10].

Bidirectional RNN expands on the ideas of RNN by introducing another set of RNN

cells for each layer When training, one set is used for forward direction and one for reverse direction This means bidirectional RNNs hidden state retain both direction information This hidden state then goes to a decoder, such as fully a connected

network followed by a SoftMax.

18

RNN OUTPUT

BACKWARD

LAYER

FORWARD LAYER

WORD EMBEDDINGS

INPUTS THE

Dimension of ai = size of hidden state vector h

In our code we define h dimension as 64

Figure 2.13: Bidirectional has forward and backward RNNs Source: MLWhiz 2018.

Bi-LSTM has become a popular architecture for many NLP tasks Its applications

include sentence classification, speech recognition, sentiment analysis, medical event

detection.

One paper that greatly inspired us to take on BLSTM direction is [8] In their paper, they proposed IDW-BLSTM models to predict air quality on a regional scale Their

BLSTM model applied IDW interpolation as a deep learning layer which can forecast

not only air quality in areas with monitoring stations but also in surrounding area

without stations Some other models which have the same direction [11, 12] have been researched on and produced a positive result.

19

2.2.4 Batch Normalization

There are many challenges in training deep neural networks (networks with tens of

hidden layers One aspect of this challenge is that the model is updated layer-by-layer

backward from the output to the input using an estimate of error that assumes the

weights in the layers prior to the current layer are fixed In our situation, this challenge

double when we train in two directions

Batch normalization is proposed as a technique to help coordinate the update of

multiple layers in the model It does this by standardizing the activations of each input

variable per mini-batch This means that the spread and distribution of inputs during

the weight update will not dramatically change This has the effect of stabilizing and

speeding-up the training process of deep neural networks

Batch normalization have been very popular in deep learning models and

extensively used in computer vision and speech recognition

2.2.5 Dropout

Large neural nets trained on relatively small datasets can overfit the training data.Since the information is not large enough, models tend to learn the statistical noise inthe training data, which result in poor performance when evaluated on new data

Dropout is a regularization method designed to counter this During training, some

number of layer outputs are randomly ignored or “dropped out.” This makes each

epochs will be trained on a different models with different architecture This can evenout the attention each nodes/layers have in large networks

20

Input Layer Hidden Layers Output Layer Input Layer Hidden Layers Output Layer

TensorFlow is a free and open-source software library for machine learning

TensorFlow is a symbolic math library based on dataflow and differentiable

programming It is used for both research and production at Google Due to being veryuseful for deep learning development, Google open sourced it Now TensorFlow has

been developed to be used across a range of tasks but has a particular focus on training

and inference of deep neural networks

TensorFlow works based on data flow graphs that have nodes and edges As the

execution mechanism is in the form of graphs, it is much easier to execute TensorFlowcode in a distributed manner across a cluster of computers while using GPUs

21

2.3.2 Reason for choosing TensorFlow

Figure 2.15: TensorFlow advantage

1 Graph: Better graph visualization compared to Torch and Theano

2 Library Management: Being backed by Google, although TensorFlow is open

source it has good quality of performance, and frequently added new features.

3 Debugging: TensorFlow allows user to execute only a subpart of a graph In the

execution we can introduce and retrieve data which helps in finding bugs.

4 Scalability: TensorFlow is highly parallel and designed to use various backends

software (GPU, ASIC).

2.4 Google cloud platform

Google Cloud Platform is a suite of public cloud computing services offered by

Google The platform includes a range of hosted services for compute, storage and

application development that run on Google hardware Google Cloud Platform services can be accessed by software developers, cloud administrators and other enterprise IT

professionals over the public internet or through a dedicated network connection.

22

Google Cloud Platform

©0000 0000

Figure 2.16: Some google cloud platform services

Google Cloud Platform offers services for compute, storage, networking, big data,machine learning and the internet of things (IoT), as well as cloud management,

security and developer tools The core cloud computing products in Google Cloud

Platform include:

1 Google Compute Engine, which is an infrastructure-as-a-service (IaaS) offering

that provides users with virtual machine instances for workload hosting

2 Google App Engine, which is a platform-as-a-service (PaaS) offering that gives

software developers access to Google's scalable hosting Developers can also

use a software developer kit (SDK) to develop software products that run onApp Engine

3 Google Cloud Storage, which is a cloud storage platform designed to store large,

unstructured data sets Google also offers database storage options, including

Cloud Datastore for NoSQL nonrelational storage, Cloud SQL for MySQL fully

relational storage and Google's native Cloud Bigtable database

4 Google Container Engine, which is a management and orchestration system for

Docker containers that runs within Google's public cloud Google Container

Engine is based on the Google Kubernetes container orchestration engine

23