HO CHI MINH NATIONAL UNIVERSITYUNIVERSITY OF INFORMATION TECHNOLOGYDEPARTMENT OF COMPUTER SCIENCE GRADUATION THESIS REPORT Enhancing robustness against adversarial attacks in machine lea
2
Objectives, Subject, and Scope of the Study
The research objectives of this study are focused on addressing the limitations of traditional models in intrusion detection systems (IDS) To achieve this, the study proposes new models that leverage the advantages of Adversarial Training, Multiple Mutated Classifiers (MMC), and Multimodal methods The specific objectives include:
Improving Classification Capabilities: The research aims to develop new mod- els that enhance the classification capabilities of intrusion detection systems (IDS) Traditional models often struggle to effectively detect and classify various types of intrusions due to their limited feature representation and classification techniques By incorporating Adversarial Training, Multiple Mutated Classifiers
(MMC), and Multimodal methods, the proposed models aim to improve the ac- curacy, robustness, and capability of the IDS to identify and classify both known and unknown attacks.
Harnessing the Power Multimodal Methods: The research seeks to explore the benefits of integrating Multimodal methods in IDS By combining information from multiple data modalities, such as network traffic data, system logs, and application-level data, the proposed models aim to improve the IDS’s detection capabilities The research objective is to develop effective techniques for feature extraction, fusion, and classification that leverage the complementary strengths of different modalities, leading to enhanced detection accuracy and reduced false positives.
Assessing Robustness against Transferability of Adversarial Examples: The study aims to evaluate the robustness of the proposed models against the trans- ferability of adversarial examples Adversarial examples are inputs specifically crafted to deceive machine learning models, including IDS The research will
8 investigate whether the enhanced models, incorporating Adversarial Training, MMC, and Multimodal methods, exhibit improved resilience against adversar- ial examples that can transfer across different models This objective involves generating and testing adversarial examples against the models, analyzing their transferability, and evaluating the models’ ability to detect and mitigate such attacks.
The research subject in this thesis includes: e Machine Learning e Deep Learning e Generative Adversarial Networks e Intrusion Detection Systems e Adversarial Training e Ensemble Learning e Multimodal e Multiple Mutated Classifiers
The research focus of this study is to enhance the robustness of Network Intrusion Detection Systems (NIDS) by utilizing multiple mutated models and a multimodal approach The research will involve conducting various experiments to investigate the effectiveness of these techniques Additionally, the study aims to further enhance the performance of the IDS by incorporating adversarial training and conducting additional experiments to evaluate its impact.
We would like to present the content of the Thesis in the following structure: e Chapter 1: Introduction to the topic of the Thesis. e Chapter 2: Presenting the theoretical foundation and related research. e Chapter 3: Presenting an overview of the proposed model. e Chapter 4: Presenting experiments and evaluations. e Chapter 5: Conclusion and future development of the topic.
amp; RELATED WORK 10
Evasion Attacks 2 2 ee 12
Evasion Attacks are a type of adversarial attack that specifically target secu- rity systems, such as Intrusion Detection Systems (IDS) The objective of these attacks is to manipulate the machine learning model during the testing phase in order to cause misclassification of the learned information In the context of an IDS network system, attackers aim to avoid detection and warnings from the IDS by concealing malicious traffic within normal patterns This is typi- cally achieved by exploiting vulnerabilities in the intrusion detection system or altering the characteristics of the malicious data to make it appear as normal data.
An evasion attack is considered successful when the machine learning model is deceived and fails to detect the malicious data, or when the malicious data is misclassified as normal data The success of such attacks depends on various fac- tors, including the complexity of the security system, the detection capabilities and flexibility of the security system, the knowledge and skills of the attacker.
Evasion attacks present a significant risk to IDS systems, as security systems can be easily deceived by making the malicious data difficult to detect To mit- igate the risk of such attacks, it is crucial to design and implement IDS systems with robust and adaptable intrusion detection mechanisms These mechanisms should be capable of adapting to evolving and updated malicious traffic.
A common approach to evasion attacks is modifying the transmitted data to IDS in order to make it appear benign This is done by adding noise to the data, modifying header information, and changing the data encoding By doing so, attackers can evade intrusion detection systems as traditional IDS may not
13 accurately identify the accessed content as malicious.
Another method is intervening in existing protocols by altering their structure or behavior For example, attackers can modify the structure of the TCP/IP protocol to make the attack behavior appear normal, benign, and legitimate, while in reality, it is transmitting malicious data into the system.
Lastly, attackers can also create new protocols specifically designed to evade detection For instance, they can create a new protocol that bypasses firewalls and intrusion detection systems by using encryption, compression, or other tech- niques to conceal malicious content within the attack traffic.
In general, the motivations and methods behind evasion attacks are diverse and evolving in various directions, posing a significant challenge to organizations seeking to secure their network systems and data Figure 2.2 describe a diagram of the model evasion attack against a trained machine learning model during the testing phase.
Generative Adversarial Networks (GAN) 0
Generative Adversarial Networks (GANs) were developed in 2014 by Ian
Goodfellow and his teammates GAN is basically an approach to generative modeling that generates a new set of data based on training data that look like training data GANs have two main blocks(two neural networks) which compete with each other and are able to capture, copy, and analyze the variations in a dataset The two models are usually called Generator and Discriminator.
The generator network takes random input (typically noise) and generates samples, such as images, text, or audio, that resemble the training data it was trained on The goal of the generator is to produce samples that are indistin- guishable from real data. tu
Figure 2.2: Diagram of the model evasion attack
15 The discriminator network, on the other hand, tries to distinguish between real and generated samples It is trained with real samples from the training data and generated samples from the generator The discriminator’s objective is to correctly classify real data as real and generated data as fake.
The training process involves an adversarial game between the generator and the discriminator The generator aims to produce samples that fool the discrim- inator, while the discriminator tries to improve its ability to distinguish between real and generated data This adversarial training pushes both networks to im- prove over time.
Figure 2.3: The block diagram of a classical GAN model
A generative adversarial network system comprises two deep neural networks the generator network and the discriminator network.
The generator introduces random noise or random changes to certain at- tributes of the data, modifying it The modified data is then passed to the dis- criminator, which calculates the probability of the generated output belonging
The discriminator guides the generator by reducing the randomization in the noise vector for the next cycle, allowing the generator to generate more realistic data This adversarial training process between the generator and discriminator drives the GAN to generate increasingly realistic outputs.
By iteratively updating their respective neural networks using techniques like gradient descent, the generator and discriminator aim to reach an equilibrium. This equilibrium corresponds to the generator producing synthetic samples that are indistinguishable from real data, while the discriminator struggles to accu- rately classify between them At which point, the training process is over.
A system called an intrusion detection system (IDS) observes network traffic for malicious transactions and sends immediate alerts when it is observed It is software that checks a network or system for malicious activities or policy viola- tions Each illegal activity or violation is often recorded either centrally using a SIEM system or notified to an administration IDS monitors a network or sys- tem for malicious activity and protects a computer network from unauthorized access from users, including perhaps insiders The intrusion detector learning task is to build a predictive model (i.e a classifier) capable of distinguishing between ‘bad connections’ (intrusion/attacks) and ‘good (normal) connections’. e NIDS: Network Intrusion Detection Systems are placed in a position where they can scan the entire network traffic, but this can potentially slow down the network and affect the user experience. e HIDS: Host Intrusion Detection Systems run directly on individual servers and only monitor inbound and outbound data packets to make determina-
System User Check for Packets
Figure 2.4: The Intrusion Detection and Prevention System (IDPS) model tions if they are malicious network traffic or not. e Signature-Based: Detection is based on signatures similar to how an- tivirus software operates However, it can suffer from false positives if the signatures are not updated. e Anomaly-Based: Detection identifies attacks based on abnormal traffic compared to a predefined baseline.
IDS can be created in different ways In the traditional approach, experts conduct research and analyze attacks to establish new rules that classify attack traffic and normal traffic However, this method encounters challenges in detect- ing new and constantly evolving attack data As a result, machine learning-basedIDS was developed to assist in creating machine learning models that can contin- uously learn from new data This enables more accurate predictions for detecting and alerting against malicious attacks.
An Intrusion Detection System (IDS) functions by continuously monitoring the traffic on a computer network, actively searching for and identifying any sus- picious or unauthorized activity It achieves this by analyzing the data traversing the network, meticulously examining patterns and signs that deviate from nor- mal behavior By comparing the network activity against a predefined set of rules and patterns, the IDS detects any activity that may indicate a potential attack or intrusion Once such activity is detected, the IDS promptly sends an alert to the system administrator, who can then investigate the alert and take appropriate action to prevent further damage or intrusion This proactive and vigilant approach to network security helps to safeguard the integrity and con- fidentiality of the network and its associated resources.
The decision tree algorithm is a popular supervised machine learning algo- rithm for its simple approach to dealing with complex datasets Decision trees get the name from their resemblance to a tree that includes roots, branches and leaves in the form of nodes and edges They are used for decision analysis much like a flowchart of if-else based decisions that lead to the required prediction.The tree learns these if-else decision rules to split the dataset to make a tree-like model Figure 2.5 shows the diagram basic Decision Tree A decision tree starts with a root node, which does not have any incoming branches The outgoing branches from the root node then feed into the internal nodes, also known as decision nodes Based on the available features, both node types conduct evalua- tions to form homogeneous subsets, which are denoted by leaf nodes or terminal
19 nodes The leaf nodes represent all the possible outcomes within the dataset.
Internal node Internal node ô== a ôa ô=
Figure 2.5: Basic diagram of decision tree
Decision tree learning employs a divide and conquer strategy by conducting a greedy search to identify the optimal split points within a tree This process of splitting is then repeated in a top-down, recursive manner until all, or the majority of records have been classified under specific class labels.
While there are multiple way to select the best attribute at each node, two methods, information gain and Gini impurity, act as popular splitting criterion for decision tree models They help to evaluate the quality of each test condition and how well it will be able to classify samples into a class.
Entropy is a concept that stems from information theory, which measures the
20 impurity of the sample values It is defined by the following formula:
Entropy(5) = — $7 p(e) logp((c)) (2.1) ccC where: e S represents the dataset for which entropy is being calculated, e C represents the set of classes in the dataset S, e p(c) represents the proportion of data points belonging to class c relative to the total number of data points in dataset S,
51
GAN-based Model Target Attacks
We have constructed simple baseline models using machine learning algo- rithms such as DT, RF, XGB, ET, LDA and MLP for tabular data and CNN, RNN for both tabular and image data in the process of identifying whether patterns are benign or malicious.
Subsequently, we employed GANs to generate noisy samples based on the original malicious data in order to deceive the target model into identifying them as benign samples The pseudocode for GAN training process is outlined in Algorithm 3.1 The Figure 3.1 illustrates impact of the generated minority sample attack After employing GANs to generate adversarial samples, the entire model was deceived by the GAN, misclassifying malicious data as benign data,result shown in table 4.15.
Algorithm 3.1 Training GAN Input: Malicious class attack samples
Initialization 64 for D (Discriminator), 0, for G (Generator) for i = 1, 2, 3, number of epochs learned do
Step 1: Select n samples from dataset p_ dafa(z) samples #1, #a, , #„
Step 2: Select n vectors from the Gaussian distribution p,(z) samples
Step 3: Put p;(z) n vectors to the generator and generate n data £1, 2, oy Un
Step 4: Back propagation calculates the gradients of D, G
Step 5: D: max(V = Diarlog(De,) +2251 log 1—D(Gs,)) )
Step 7: Optimize the loss of D, G with Adam optimizer end
Step 8: When the discriminator determines that the generated sample is true, the generated data are output
In order to mitigate the impact of adversarial attack samples on machine learning models, we propose and experiment with several defense models againstGANs in the following section 3.3.
Proposed Method 0000000000 53
Intrusion detection plays a crucial role in ensuring the security and integrity of computer networks Traditional intrusion detection systems (IDS) often rely on single classifiers, which may struggle to effectively detect and classify vari- ous types of intrusions accurately To address this limitation, we will conduct experiments on the the approach: Multiple Mutated Classifiers (MMC) for intru-
54 sion detection The MMC method leverages the power of ensemble learning and mutation-based techniques to enhance the accuracy and robustness of intrusion detection systems.
The proposed MMC approach employs ensemble learning, which combines the predictions of multiple base classifiers to make a final decision Ensemble methods have shown significant improvements in classification tasks by reducing bias, increasing generalization, and improving overall accuracy By leveraging the collective intelligence of multiple classifiers, the MMC method aims to enhance the performance of intrusion detection systems.
In addition to ensemble learning, the MMC method incorporates a mutation- based technique to introduce diversity among the base classifiers Mutation plays a vital role in evolutionary algorithms and helps explore new regions of the search space By applying mutation operators to the base classifiers, we aim to create a diverse set of mutated classifiers that can capture different aspects of intrusion patterns and improve the overall detection capabilities of the system.
In addressing attacks from GAN models, there are several key factors to con- sider Firstly, it is important to establish the goal of the defense method This involves clearly defining the objectives we aim to achieve in order to prevent or minimize the impact of GAN attacks Also, the training context plays a crucial role in developing defense methods This involves studying the existing methods for attacking GANs, gaining an understanding of how these attacks operate, and identifying the vulnerabilities in GAN models that attackers can take advantage of By comprehending the training context of these attacks, we can create ef- fective defense methods to safeguard IDSs from malicious data Moreover, the quality of adversarial samples is also crucial in developing defense methods In or- der to create effective defense methods, it is necessary to comprehend how GANs generate adversarial samples and discover techniques to identify and eliminate
To improve resistance against GAN attacks while preserving the training per- formance of the model, the considerations of training data and model architec- ture are essential.
Training data can encounter issues such as data imbalance, imbalanced la- bel quantities, or disparities between majority and minority attributes, making the data generated unavailable or preventing the machine learning model from achieving optimal performance Several data augmentation models have been proposed to address data imbalance [26], [27], [28].
To improve the IDS’s ability to detect anomalies and attack traffic, researchers have synthesized adversarial data using GAN models with data poisoning or eva- sion capabilities [29], [30], [31] However, evasion samples that use random noise spaces or modify small factors in attributes can cause inconsistency between the remaining attributes in the traffic and reduce the performance of the differenti- ation process [32], [33].
Defense methods against adversarial attacks often focus on analyzing a single type of input data, which can limit the learning process of machine learning or deep learning models when it comes to identifying malicious data However, the self-attention-based architecture, with the Self-Attention mechanism as its backbone, has gained popularity in recent years This architecture offers flexible scalability and highly efficient computation, leading to successful applications in various natural language processing fields such as text classification and dialogue tasks.
Building upon this architecture, He et al [34] and Bikmukhamedo et al.
[35] were inspired to apply the Transformer model to attribute classification in network flow analysis Their work achieved impressive performance results, demonstrating the effectiveness of leveraging self-attention in tackling attribute classification tasks.
Based on our research and understanding, we propose a novel approach for
56 network traffic classification to detect adversarial data capable of evasion, which is generated by GANs We named this approach "Multimodal" as it draws inspiration from a paper titled Multimodal Attention-based Deep Learning for
Alzheimer’s Disease Diagnosis (MaDDi) [23], which demonstrates the model’s effectiveness in diagnosing different stages of Alzheimer’s disease We have cho- sen to explore this model architecture in our research and experimentation to detect abnormal network traffic.
The Multimodal approach combines the strengths of two existing models: the
GAN adversarial network model (EVAGAN) [36] and the Transformer archi- tecture (TransGAN, RTIDS) [37], [38] By integrating these models, we lever- age the multi-head self-attention mechanism of the Transformer to enhance the recognition capabilities of the system This enables us to establish meaningful connections between the harmful attributes generated by GANs and improve the performance of the Discriminator as an intrusion detection classifier during the training process.
Inspired by the success of self-attention, we propose a cross-modal self-attention module to jointly exploit attentions over multimodal features In addition, we have utilized various ML algorithms including LR, SVM, NB, DT, RF, KNN, MLP, CNN, RNN to evaluate the evasion and deception capabilities of intru- sion detection systems against adversarial attack samples generated from GANs. Furthermore, we have also employed ML algorithm models to defend against ma- licious attack samples as well as combining them to create a Multimodal fusion model from two types of data, tabular data and images However, a model devel- oped from two ML algorithm models alone cannot fully fulfill the requirements of effectively defending against all forms of adversarial attacks that are difficult to detect We have tested the resistance ability against Multimodal GAN at section 4.2.1
Therefore, we suggest using a combination of Multimodal and AdversarialTraining to improve the detection of new and complex attack patterns By re-
57 training the model with adversarial attack patterns, we can enhance its ability to defend against them more efficiently and effectively You can find more infor- mation about this approach in our paper on adversarial training against GAN attacks In the Multimodal Adversarial Training model, we utilize GAN to gen- erate synthetic data, which is then used to enhance the training data for our proposed model This approach allows us to balance the data distribution and improve the classification performance of the IDS model.
Figure 3.2 depicts the process by which the GAN generates adversarial sam- ples to attack Multimodal In order to train Multimodal using clean data, we exclusively provide the normal malicious and benign samples to the tabular classifier and image classifier To integrate Multimodal with the Adversarial Training approach, we alter the training data for the aforementioned classifiers by incorporating attack samples generated by the GAN and benign samples.
Figure 3.2: GAN’s attacks on Multimodal
Architecture & Operation of the Proposed Model
The architecture of the proposed model is built based on the research, explo- ration, and analysis of relevant scientific publications in the field of ML (MachineLearning) and DL (Deep Learning) In the following sections, we will present the
58 structure and construction process of the model based on the proposed method in Section 3.3.
In this architecture, the algorithm starts by obtaining the training data (train) and test data (test) Then, for a specified number of tries, the algorithm will iteratively performs the following steps: it generates random hyper-parameters
(p) based on the the optimal hyper-parameters of the classifier, fits a mutated model with p on the training data (train), calculate f1 score of the mutated model on the test data (test), and saves the mutated model along with its f1 score.
After generating multiple mutated models, the algorithm selects the top n classifiers with the highest F1 scores, forming the set F These classifiers are considered the most effective models for the given task Finally, the algorithm obtains the predictions of the models in F on the test data and combines them using soft voting, which typically involves aggregating the predicted probabilities or scores from each model to make the final prediction.
Overall, this architecture follows a process of generating and evaluating mu- tated models based on random hyper-parameters, selecting the top performers,and combining their predictions to produce a final prediction using soft voting.
Algorithm 3.2 Generate and prediction with MMC Step 1: Get model training data train
Step 2: Get model test data test for i = 1, 2, 3, number of tries do
Step 3: Generate random hyper-parameter p Step 4: Fit mutated model with p on train
Step 5: Evaluation mutated model on test
Step 6: Save mutated model end
Step 7: Get the top n classifiers with the highest fl scores F
Step 8: Get F predictions on test
Step 8: Combine F’s prediction with soft voting
3.4.2 Multimodal Aversarial Training (MAT) Architecture
The proposed model takes pre-processed tabular and image data as input and labels them Benign, Bot, Brute Force, DDOS, Infiltration, SQL Injection For the tabular data, we employ a Random Forest classifier as the machine learning algorithm backbone We selected this method based on its stability and superior performance compared to other machine learning algorithms, as evaluated on our training data CSE-CIC-IDS2018 (see Table 4.15).
Regarding the image data, we utilize a model architecture consisting of a
Gated Recurrent Unit (GRU) layer followed by two fully connected layers This architecture is chosen due to its efficient performance in image classification tasks [39] The output of these layers is then passed through a multi-headed self-attention layer, enabling the inputs to interact with each other and identify which features require the most attention within each modality.
Subsequently, a cross-modal bi-directional attention layer is applied, which performs a similar calculation to self-attention but across different pairs of modalities The purpose of this cross-modal attention is to model interactions
60 between modalities For instance, tabular features may reinforce the informa- tion provided by image features, thereby enhancing the model’s decision-making process.
Both types of attention mechanisms are rigorously defined in the Neural Net- work Attention section Finally, the outputs of the parallel attention computa- tions are concatenated and fed into a final dense layer for prediction.
Our framework utilizes attention mechanisms in two different ways: self- attention and cross-attention.
Self-Attention: Self-attention allows us to learn the interactions among different parts of an input ("self") and determine which parts are relevant for making predictions ("attention") In our case, the prior neural network layers generate two latent feature matrices for each modality: J for image data and T for tabular data Self-attention refers to the attention computation performed within the same modality Therefore, the self-attention module performs the following operations: self - attention(I > 7) self - attention(T — 7)
Cross-Modal Attention: In each bi-directional cross-modal attention layer
[40], there are two unidirectional cross-modal attention sub-layers: one from modality 1 to modality 2 and one from modality 2 to modality 1 In our case, the cross-modal attention layer takes the output of each self-attention computation: the image self-attention output J, and the tabular self-attention output T, Thus, the cross-modal attention module performs the following operations: concatenate(cross-modal attention)(Ig + Ts), cross-modal attention(Ts — Ts))
61 These attention mechanisms allow our model to capture both intra-modal interactions within each modality and inter-modal interactions between different modalities.
For the combining of multimodal and adversarial training , the input data for the training process of the defense model (blackbox detector) will comprise normal data, actual attack data, and attack data generated by the generator. Through retraining with adversarial examples, the classifier learns the pertur- bation pattern and generalizes its prediction to account for such perturbations.
After updating the input of the two original models with the data that has been trained on adversarial examples generated by the generator, these data will be used to train the two models The methods employed will be similar to the Multimodal architecture.
IU93V-JI9S uonueny IPPOur-sso12.
| eRUu nit vn Dropout Fuly ị =m pone uonuanv
= a f) Classification output b) Random Forest d) Cross-modal (Benign, Bot, Brute Classifier for tabular data, ©) Self Attention attention between e) FC layer Force, DoS, Infittration, RNN for image data every modality pair SQL Injection) a) Input
Figure 3.3: Multimodal Adversarial Training Architecture
(a) Data input - Benign samples, Malicious samples and adversarial exam- ples generated by GAN (b) The input sources are combined and fed into RF classifier for tabular data and a GRU-based RNN network for imaging data.
AND EVALUATION 65
Experimental Design co 65
We carried out our experiment with Google Colaboratory, this is a cloud-based platform that offers a free and interactive environment for writing, running, and sharing Python code It provides a Jupyter notebook interface that allows users to create and execute code cells, write explanatory text, and visualize data Furthermore, Colab provides free GPU and TPU (Tensor Processing Unit) resources, enabling users to accelerate their computations for machine learning and deep learning tasks.
To improve and increase the usage limit of GPUs and optimize time in our experiments, we utilize both the free version of Google Colab (Colab Free) and the Colab Pro version This allows us to have up to 14 working sessions in total.
We use Python3 as the programming language, utilizing notebook format on Google Colaboratory.
In order to support programming machine learning models and visualizing results and data, we rely on several key libraries: e NumPy: This library supports scientific computing and numerical opera-
66 tions, allowing for efficient array manipulation and calculations in Python. e Pandas: It aids in processing tabular data, offering convenient data struc- tures like DataFrames for seamless data transformation and analysis. e matplotlib: It is a powerful library for data visualization, enabling the creation of high-quality charts and images to visualize data and training results. e scikit-learn: This library provides a wide range of tools for training, testing, and evaluating machine learning models. e TensorF low: It provides tools for building and deploying machine learning models, particularly those based on neural networks. e Keras: This library simplifies the construction of neural networks and deep learning models, offering flexibility and ease of use. e smote variants: This library is used specifically for implementing the
SMOTE method, which helps address data imbalance in machine learning tasks
We collect and utilize open-source data sources provided by international scientific publications in the field of cybersecurity, specifically focusing on the CIC-IDS2018 dataset This dataset contains samples of normal access traffic and network intrusions from the Canadian Institute for Cybersecurity (CIC).
It provides a realistic and challenging environment for students and researchers to develop and test their intrusion detection models CIC-IDS2018 encompasses seven different attack scenarios, including Brute force, Heartbleed, Botnet, DoS,DDoS, Web attack, and insider network intrusion.
67 The attack infrastructure consists of 50 machines, and the victim organization is composed of five departments with 420 machines and 30 servers The dataset includes network access traffic and system logs from each machine, along with
80 features extracted from the collected traffic using the CICFlowMeter-V3 tool. The CIC-IDS2018 dataset is a valuable resource for studying network security and developing intrusion detection systems.
Benign Bot Bruteforce Dos Infiltration SQL Injection Total
Number of samples in the majority class Imbal Ratio = —
NaN NEES = Number of samples in the minority class
In this study, the dataset underwent several preprocessing steps to ensure its suitability for machine learning models The following steps were performed, based on the method proposed by Karatas et al [24]:
1 Handling Missing Values: Missing values, also known as NaN values, were converted to 0 to prevent value errors during model training.
2 Handling Infinity Values: Two columns, ’Flow Bytes’ and ’Flow Pkts’, con- tained infinity values To address this issue, the infinity values were replaced with the maximum value in the respective column plus 1.
3 Timestamp Separation: The ‘Timestamp’ column was separated into two new columns, Date’ and ’Time’, to eliminate text values from the dataset.
Both columns were converted into numeric forms, such as YearMonthDay for the Date’ column and HourMinuteSecond for the ’Time’ column.
4 Handling Negative Values: In the columns ‘Init Fwd Win Byts’ and ‘Init
Bwd Win Bytes’, some samples had a value of -1 To handle this, two new columns, ‘Init Fwd Win Byts Neg’ and ‘Init Bwd Win Byts Neg’, were created These new columns contained a value of 1 where the original columns had -1 values and 0 where non-negative values were present.
5 Encoding Attack Labels: The Label’ column contained identified attack names To facilitate analysis, these attack names were transformed into numerical values based on a mapping outlined in Table 4.2.
6 Shuffling the Dataset: As a final step of preprocessing, the combined dataset was shuffled randomly to ensure randomness and eliminate any potential bias.
Table 4.2: CSE-CIC-IDS2018 label mapping
Infitration SQL injection Củ BB] C5| DO] Fe | ©
After the completion of these preprocessing steps, the dataset was expanded to include 83 features from the original 80 features.
By performing these data preprocessing steps, the dataset was prepared for subsequent analysis and model training, ensuring the accuracy and reliability of the results obtained.
Benign Bot Bruteforce Infiltration SQL Injection Total
Table 4.4: CICIDS2018 functional feature groups
Intrinsic Content Time-based Host-based SQL injection X X
To preserve the original functionality of a malicious sample after applying adversarial perturbations, it is important to ensure that the essential functional features of the sample remain intact This can be determined by referring to two key references: Table 4.4: "Functional Groups" and Table ??: "Functional Features."
The table 4.4 informs us about the feature groups that should be kept intact in order to preserve the functionality of specific attack types when applying adversarial perturbations Let’s explain each functional group: e Intrinsic: The Intrinsic feature group consists of features that are essential for maintaining the validity and integrity of network traffic These features capture fundamental properties of the traffic, such as destination ports, packet counts, flow duration, and packet lengths Preserving the Intrinsic feature group ensures that the basic attributes of the network traffic, which are crucial for analyzing and understanding any type of traffic, remain in- tact.
70 e Content: The Content feature group encompasses features that characterize the content or payload of network packets These features include packet lengths, flag counts, average segment sizes, and bulk-related metrics By preserving the Content features, the distinctive content-related patterns and properties of both normal and malicious traffic can be retained This enables the detection and analysis of specific types of attacks that exhibit unique content-based characteristics, such as SQL injection or cross-site scripting. e Time-based: The Time-based feature group comprises features that capture temporal aspects of network traffic These features include flow inter-arrival times, idle times, and active times Preserving the Time-based features al- lows for the preservation of time-based patterns and behaviors in both nor- mal and malicious traffic Certain attacks, such as DDoS attacks or port scans, may exhibit specific temporal patterns that can be detected by ana- lyzing these features. e Host-based: The Host-based feature group involves features that describe host-level characteristics of network traffic These features may include host flags, window sizes, or other attributes specific to individual hosts Preserv- ing the Host-based features allows for the retention of host-level information,which is vital for identifying attacks that target specific vulnerabilities or behaviors at the host level Examples of such attacks include brute force attacks or remote code execution.
Experimental Methodology 00.0 70 1 Experimental Scenarios .2-2004 71
In this section, the experimental methodology is formulated based on address- ing three key questions:
Question 1: How do the MMCs, Multimodals, and MATs perform in terms of their strengths, performance, and achieved results compared to traditional
Destination Port, Flow Duration, Total Fwd Packets, Total Backward Packets, Total Length of Fwd Packets, Total Length of Bwd Packets, Fwd Packet Length Max, Intrinsic Fwd Packet Length Min, Fwd Packet Length Mean, Fwd Packet Length Std,
Bwd Packet Length Max, Bwd Packet Length Min, Bwd Packet Length Mean,
Bwd Packet Length Std, Flow Bytes/s, Flow Packets/s Min Packet Length, Max Packet Length, Packet Length Mean, Packet Length Std, Packet Length Variance, FIN Flag Count, SYN Flag Count, RST Flag Count,
PSH Flag Count, ACK Flag Count, URG Flag Count, CWE Flag Count, ECE Flag Count, Down/Up Ratio, Average Packet Size, Avg Fwd Segment Size, Avg Bwd Segment Size, Fwd Header Length.1, Fwd Avg Bytes/Bulk,
Fwd Avg Packets/Bulk, Fwd Avg Bulk Rate, Bwd Avg Bytes/Bulk, Bwd Avg Packets/Bulk, Bwd Avg Bulk Rate, Subflow Fwd Packets,
Subflow Fwd Bytes, Subflow Bwd Packets, Subflow Bwd Bytes, Init Win_bytes_ forward, Init_Win_bytes_ backward, act_data_pkt_ fwd, min_seg_size_ forward
Flow IAT Mean, Flow IAT Std, Flow IAT Max, Flow IAT Min, Fwd IAT Total, Fwd IAT Mean, Fwd IAT Std, Fwd IAT Max, Fwd IAT Min, Bwd IAT Total, Bwd IAT Mean, Bwd IAT Std, Bwd IAT Max, Bwd IAT Min, Active Mean, Active Std, Active Max, Active Min, Idle Mean, Idle Std, Idle Max, Idle Min Host-based | Fwd PSH Flags, Bwd PSH Flags, Fwd URG Flags, Bwd URG Flags
Time-based machine learning methods, ensemble learning [41], adversarial training [42], and ELAT [22]?
Question 2: How does the proposed MMC, Multimodal, and MAT methods defend against adversarial network traffic from GANs?
Question 3: Are MMCs, Multimodals, and MATs less susceptible to transfer- ability in adversarial samples than tradition method like simple ML, DL model and ensemble learning?
We conducted experiments on the proposed models using the CIC-IDS2018 datasets as mentioned in Section 4.1.2, with four main scenarios as follows: e Scenario 1: Experiment, evaluate, and compare Mutated models with tra- ditional machine learning and deep learning models in terms of training performance, model robustness, and classification accuracy. e Scenario 2: Experiment, evaluate, and compare Multimodal and MAT mod-
Description learning rate 0.0001 Model’s learning rate optimizer Adam Optimization algorithms tram test rate 7:1 Ratio of train set to test set epoch_ tran GAN 10 Number of epoch for training GAN batch_ size 4096 Number of sample in one batch loss_ function MSE Loss function els with traditional machine learning and deep learning models in terms of training performance, model robustness, and classification accuracy, before and after adversarial training. e Scenario 3: Experiment, evaluate, and compare the Multimodal and MAT models with ensemble learning models in terms of training performance, model robustness, and classification accuracy, before and after adversarial training.
In all cases, we utilized a total of 8 machine learning and deep learning models such as DT, RF, ET, XGB, MLP, RNN, CNN, and LDA Specifically, for the RNN and CNN models, we experimented with two types of data: images and tabular data, in preparation for the Multimodal model.
And in scenario 3, our baseline ensemble learning model will consist of 4 tree-based learning algorightms stacked together: DT, XGB, ET, and RF.
To standardize and ensure consistency in the experiments and across the sce- narios as presented in Section 4.2.1, we established parameters for the proposed machine learning model based on the detailed values and parameters presented in the following table.
Table 4.7: Baseline models hyperparameter values
DT criterion = "gini", splitter = "best", min samples_ split = 2, min _samples_ leaf = 1
RF n_ estimators = 100, criterion = "gini", min samples_ split = 2, min _samples_ leaf = 1 XGB n_ estimators = 10
ET random state = 42, criterion = "gini", n_ estimators = 100
MLP hidden_ layer_ sizes=(600,), max_ iter@0, alpha-4, solver=’adam’, verbose=1, random _ stateB, activation=’relw’, learning rate_ init=.0000001, learn- ing rate=’adaptive’, tol = le-8
LDA solver = "svd", store _ covariance = False, tol 0.0001 CNN optimizer= adam', loss=’categorical_ crossentropy’, met- rics=’accuracy’
RNN optimizer= adam', loss= categorical crossentropy”, met- rics=’accuracy’
CNN image optimizer=’adam’, loss=’categorical_ crossentropy’, met- rics=’accuracy’
RNN image optimizer=’adam’, loss=’categorical_ crossentropy’, met- rics=’accuracy’
Hidden_layer_ sizes = (1100 to 1300, ); ivation = ‘relu n T istic’;
Activatio ‘relu’, ‘tanh’, or ‘logistic’;
We will evaluate the Multimodal and MAT models based on two criteria: accuracy and robustness against GAN-generated adversarial attack samples.
To measure accuracy, we will utilize metrics such as Accuracy, Detection Rate (DR), and F1 score These metrics provide insights into the model’s ability to correctly classify and detect attack traffic while minimizing false positives and false negatives.
To assess the models’ robustness against GAN-generated adversarial attacks, we will employ the Effective Input Rate (EIR) metric EIR measures the rate at which the models misclassify or fail to detect adversarial attack samples gener- ated by GANs A lower EIR indicates a higher level of robustness, as the models are more resilient to the perturbations introduced by GAN-based attacks.
By evaluating accuracy and robustness, we aim to assess the models’ overall performance in detecting and classifying evasive attack traffic, while also con- sidering their ability to withstand sophisticated adversarial attacks generated by GANs These evaluation criteria provide a comprehensive understanding of the models’ effectiveness in real-world scenarios and their potential for practical deployment in intrusion detection systems.
Accuracy = 55 TN 4 FP > FN (4.1) TP
The table 4.9 provides experimental metrics for evaluating different types of
Acc Accuracy, representing the proportion of correctly classified sam- ples.
F1 F1 score, a measure of the model’s accuracy that considers both precision and recall.
DR Detection Rate, indicating the proportion of truly identified posi- tive samples out of all positive instances.
O-DR Original Detection Rate, representing the estimated probability of truly identifying botnet or minority samples.
A-DR_ Adversarial Detection Rate, indicating the estimated probability of detecting adversarial attack samples.
O-Acc Ordinary Accuracy, measuring the proportion of correctly classi- fied samples in the absence of adversarial attacks.
A-Acc Adversarial Accuracy, measuring the proportion of correctly classi- fied samples in the presence of adversarial attacks. models, including baseline models, Multimodal models, MATs, MMCs, ensem- ble learning, and Ensemble Learning with Adversarial Training (ELAT) These metrics offer valuable insights into the performance and characteristics of these models.
In this section, we will present the experimental results for each model in the experiments and scenarios outlined in Section 4.2.1.
4.8.1 Scenarios 1: Comparison between MMCs and Baseline mod- els, with and without adversarial retraining, against GAN’s Attacks.
Looking at table 4.10, we can see that despite multiple attempts to find the optimal number of classifiers for the MMC approach, it did not yield signifi-
77 cant performance improvements compared to the baseline Decision Tree (DT) model Interestingly, the retrained DT model, which achieved a perfect accuracy score when tested with DT’s own adversarial samples, outperformed the MMC approach.
Table 4.10: Comparison of Accuracy between Multiple Mutated DTs and a
Simple DT Model, Pre and Post Adversarial Retraining, Using Different Tar- get Models’ Adversarial Samples
The results obtained for the Random Forest (RF) classifier using the MMC approach also indicate a lack of significant improvements in defending against adversarial attacks As observed in Table 4.10, the performance of a single RF classifier does not differ significantly from that of 80 mutated RF classifiers Fur- thermore, the MMC approach’s accuracy and robustness fall short compared to the effectiveness of the adversarial retraining technique, which achieved perfect detection of all adversarial samples specifically designed for the RF classifier.
Table 4.12 further reinforces the observation that the performance of the MMC approach is comparable to that of a single model when applied to the
Extra Trees (ET) classifier The MMC approach does not exhibit significant improvements in accuracy or robustness against adversarial attacks compared to
Model name DT RF XGB ET LDA MLP CNN RNN
RF-retrained 59 100 62 58 99 99 59 58 the single ET model Additionally, the performance of the MMC approach falls short when compared to the effectiveness of the adversarial retrained model, which achieves superior results by correctly detecting all adversarial samples specifically designed for the ET classifier.
Model name DT RF XGB ET LDA MLP CNN RNN