Research and implement a preprocessor for network intrusion detection system NIDS

HCMC NATIONAL UNIVERSITY UNIVERSITY OF TECHNOLOGY TRẦN HUY VŨ RESEARCH AND IMPLEMENT A PREPROCESSOR FOR NETWORK INTRUSION DETECTION SYSEM NIDS Major: Computer Science Major ID: 604801……………… GRADUATE THESIS HO CHI MINH CITY - December, 2011 CƠNG TRÌNH ĐƯỢC HOÀN THÀNH TẠI TRƯỜNG ĐẠI HỌC BÁCH KHOA –ĐHQG -HCM Cán hướng dẫn khoa học : TS Trần Ngọc Thịnh (Ghi rõ họ, tên, học hàm, học vị chữ ký) Cán chấm nhận xét : TS Đinh Đức Anh Vũ (Ghi rõ họ, tên, học hàm, học vị chữ ký) Cán chấm nhận xét : TS Trần Mạnh Hà (Ghi rõ họ, tên, học hàm, học vị chữ ký) Luận văn thạc sĩ bảo vệ Trường Đại học Bách Khoa, ĐHQG Tp HCM ngày tháng năm 2012 Thành phần Hội đồng đánh giá luận văn thạc sĩ gồm: (Ghi rõ họ, tên, học hàm, học vị Hội đồng chấm bảo vệ luận văn thạc sĩ) TS Trần Văn Hoài TS Trần Ngọc Thịnh TS Đinh Đức Anh Vũ TS Trần Mạnh Hà TS Vũ Đức Lung Xác nhận Chủ tịch Hội đồng đánh giá LV Trưởng Khoa quản lý chuyên ngành sau luận văn sửa chữa (nếu có) CHỦ TỊCH HỘI ĐỒNG TS Trần Văn Hoài KHOA QUẢN LÝ CHUYÊN NGÀNH TS Đinh Đức Anh Vũ ĐẠI HỌC QUỐC GIA TP.HCM TRƯỜNG ĐẠI HỌC BÁCH KHOA CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc NHIỆM VỤ LUẬN VĂN THẠC SĨ Họ tên học viên: Trần Huy Vũ MSHV:09070923 Ngày, tháng, năm sinh: 09/12/1896 Nơi sinh: Đồng Nai Chuyên ngành: Khoa Học Máy Tính Mã số : 604801 I TÊN ĐỀ TÀI: Nghiên cứu thực tiền xử lí cho hệ thống phát xâm nhập NIDS II NHIỆM VỤ VÀ NỘI DUNG: - Tìm hiểu tiền xử lí hệ thống phát xâm nhập mạng NIDS có giới …………… - Đề xuất giải pháp cải tiến cho tiền xử lí - Lựa chon platform để thực tiền xử lí, kiểm nghiệm hệ thống III NGÀY GIAO NHIỆM VỤ :…04/07/2011 IV NGÀY HOÀN THÀNH NHIỆM VỤ: …06/01/2012 V CÁN BỘ HƯỚNG DẪN : …TS Trần Ngọc Thịnh Tp HCM, ngày 05 tháng 12 năm 2011 CÁN BỘ HƯỚNG DẪN (Họ tên chữ ký) TS Trần Ngọc Thịnh KHOA QUẢN LÝ CHUYÊN NGÀNH (Họ tên chữ ký) TS Đinh Đức Anh Vũ Ghi chú: Học viên phải đóng tờ nhiệm vụ vào trang tập thuyết minh LV I ACKNOWLEDGEMENT Foremost, I would like to thank my advisor Dr Tran Ngoc Thinh, Head of Department of Computer Engineering, Faculty of Computer Science and Engineering, Ho Chi Minh city University of Technology His encouragement and his guidance is the motivation for me to proceed my thesis I greatly appreciate his comments on this thesis as well as on my conference papers I would also like to thank the Computer Engineering graduate committee for the comments They offer me many ideas to improve my work in the future They also provided me a chance to prove myself capable Finally, I would like to thank my family members and friends for their supports and encouragements Tran Huy Vu Author: Tran Huy Vu II TÓM TẮT LUẬN VĂN Ngày nay, hệ thống mạng đóng vai trị quan trọng lĩnh vực Chính quan trọng hệ thống mạng làm cho trở nên thành phần dễ bị tổn hại Để khắc phục nhược điểm trên, Hệ Thống Phát Hiện Xâm Nhập Mạng giới thiệu Khi hệ thống này, hệ thống NIDS phần cứng, xử lí gói tin TCP, chúng cần Tiền Xử Lí để lắp ghép dịng TCP nhằm làm tăng sức mạnh cho hệ thống NIDS Trong luận văn này, đề xuất phương pháp lai để thực Tiền Xử Lí cho NIDS Hệ thống luận văn không hỗ trợ hang trăm ngàn kết nối đồng thời với kết nối đứt đoạn, mà sử dụng nhớ hiệu hệ thống trước Kết thực nghiệm cho thấy hệ thống hỗ trợ lên tới 256 ngàn kết nối đồng thời, khoảng 46 ngàn kết nối đứt đoạn với 64MB DRAM Hệ thống hỗ trợ NIDS pháp mẫu cơng rải nhiều gói Author: Tran Huy Vu III ABSTRACT Nowadays, networking plays a very important role in every fields of life The importance of the network also makes it a vulnerable part of many organizations To overcome this weak point, Network Intrusion Detection Systems (NIDS) are introduced When these NIDSes, especially hardware NIDS, process packets of Transmission Control Protocol (TCP), they need a preprocessor to reassemble discrete TCP packets in a flow to strengthen the NIDS In this thesis, we propose a hybrid method to implement a preprocessor for an NIDS Our system not only supports thousands of TCP connections with multiple out-of-sequence data segments but also uses memory more efficiently than other systems The experimental results show that our system can hold about 256K connections simultaneously and support up to 46K out-of-sequence connections with only 64MB DRAM This system also supports NIDSs to detect attack patterns which expand over packets Author: Tran Huy Vu IV COMMITMENT I commit that this thesis is made based on my own research I not copy or use any illegal material in this thesis All reference of this thesis are cited from public resource or in permission of the authors/publisher I am directly responsible for any contradiction with the content of this thesis Tran Huy Vu Author: Tran Huy Vu V Table of Contents ACKNOWLEDGEMENT I TÓM TẮT LUẬN VĂN II ABSTRACT III COMMITMENT IV LIST OF FIGURES VIII LIST OF TABLES XI Chapter Introduction - - 1.1 Motivation - - 1.2 Statement of problem - - 1.3 Contribution - - 1.4 Organization - - Chapter 2.1 Background - - Transmission Control Protocol - - 2.1.1 TCP/IP model - - 2.1.2 Flow control mechanism - - 2.1.3 Three way hand shaking - - 2.2 TCP reassembly - - 2.2.1 TCP packets re-ordering - - 2.2.2 TCP flow reassembly - 11 - 2.3 Network intrusion detection system - 11 - 2.3.1 Snort IDS - 13 - Author: Tran Huy Vu VI 2.3.2 NIDS project at Faculty of Computer Science and Engineering HCMUT - 14 2.4 NetFPGA board - 15 - 2.4.1 Board specification - 15 - 2.4.2 Design with the reference design - 17 - Chapter Related works - 18 - 3.1 The TCP processor [4] - 18 - 3.2 Out-of-order TCP stream scanning [3] - 19 - 3.3 Robust TCP reassembly for backbone traffic [1] - 19 - 3.4 TCP reassembly for Sachet IDS [5] - 20 - 3.5 Robust TCP stream reassembly of Sarang Dharmapurikar and Vern Paxson [2] - 20 - Chapter 4.1 Method of TCP reassembly - 22 - Method for re-ordering TCP packets - 22 - 4.1.1 The data structure - 22 - 4.1.2 Operation on reassembly memory - 28 - 4.2 Method for reassembling TCP flow - 31 - Chapter System implementation - 33 - 5.1 The Input Controller module - 33 - 5.2 The Packet Manager module - 35 - 5.3 The Flow Controller module - 35 - 5.4 The Reassembler module - 37 - Author: Tran Huy Vu VII 5.5 The Memory controller module - 38 - 5.6 The Output controller module - 39 - Chapter Evaluation of the TCP Reassemble Engine - 40 - 6.1 Deployment model - 40 - 6.2 Experimental result - 42 - 6.2.1 Concurrent connections - 42 - 6.2.2 Memory utilization - 43 - 6.2.3 Throughput - 44 - 6.2.4 Capability of supporting NIDS - 48 - Chapter Conclusion and future work - 50 - REFERENCE - 51 - Author: Tran Huy Vu - 41 - full system test which will be available after all component of the system has finished However, individually testing the Preprocessor can be done by a simpler model as in Figure 6-2 Ethernet port Ethernet port Preprocessor Ethernet port NetFPGA board Figure 6-2 Individual test for the Preprocessor Using the testing model in Figure 6-2, the system can re-order out-of-sequence packets and store and load the edge data For batch testing, we also simulate our design with many data patterns Figure 6-3 shows the incoming packet has 32 ending bytes of , and the Figure 6-4 shows that the next in-sequence packet is prepended with these 32 bytes This is for the edge storing testing The Post-Route simulation gives the correct result when we simulate our design in many data patterns Figure 6-3 The incoming paket, rx_ll_data holds the data Author: Tran Huy Vu - 42 - Figure 6-4 The payload of the output packet is inserted with the last 32 bytes of the previous packet 6.2 Experimental result Based on experimental we compare our system to other systems In this section, we not compare our system with the system in [3] because that system use the method of out-of-order matching, and thus not need to reorder out-of-sequence packets We assume the network traffic conforms to the CAIDA_10G in [1] These statistics are recorded by the Cooperative Association for Internet Data Analysis in 2009 6.2.1 Concurrent connections Table 6-1 shows that our system can support 96.9% of out-of-sequence connections compared to 89.6% in system in [2] On the other hand, our system can easily scale up to support 4-hole connections; in that case the ratio of supported out-of-sequence connections can exceed 98.8% Theoretically, the fixed length buffer method [1] and simple linked list method [5] can support more than concurrent holes in a single connection However, the number of connections with more than holes is very small, so dropping packets which create more than holes in a connection is more practical Author: Tran Huy Vu - 43 - Table 6-1 Percentage of supported connection types of the TCP Reassembly Engine and other systems Number of holes ≥4 Total percent age Fixed buffer [1] 1-hole linked list [2] Simple linked list [5] 89.6% 7.3% 1.9% 1.2% 100% Support Support Support Support 100% Support Support Support Support Support 100% 89.6% Our system Support Support 96.9% Table 6-2 Memory utilization of the TCP Reassembly Engine and other systems for single-hole connections only No of out of sequence packets in a single hole connection Fixed buffer [1] 1-hole linked list [2] Simple linked list [5] Our system 1024MB 1024MB 1024MB 1024MB 128MB 128MB 128MB 128MB 93.75MB 187.5MB 281.25MB 375MB 66MB 66MB 130MB 130MB 6.2.2 Memory utilization In Table 2, we compare the reassembly memory utilization of our system with other systems Because the design in [2] supports only single-hole connections, we compare the memory utilization for single-hole connections only The first column is the number of out-of-sequence packets in a single-hole connection Based on the statistical data in [1], the mean packet size is 441 bytes, so the memory utilization of each system is calculated corresponding to this mean packet size In fixed length buffer method [1], each out-of-sequence connection has a fixed length buffer Based on the experimental result of the authors, the minimum size of buffer is 16KB, so buffering 64K Author: Tran Huy Vu - 44 - connections requires 1024MB of memory In 1-hole linked list method [2], the page size is 2KB, so the memory requirement is calculated as in Table 6-2 If this system uses the page size of 1KB, the memory requirements is a little smaller than the memory requirement of our system, about 2MB However, this system cannot handle connections with more than hole In simple linked list method [5], the system uses linked list of packets, so the system has to reserve a large space enough to store a biggest packet For example, the maximum packet size is 1500B, so the memory requirement can be calculated as in Table 6-2 The Table 6-2 shows that our system uses memory more efficiently than the others 6.2.3 Throughput 1.40 1.20 Throughput (Gbps) 1.00 0.80 edge length = 0.60 edge length = 16 0.40 edge length = 32 0.20 0.00 500 1000 1500 2000 Packet length (bytes) Figure 6-5 Maximum throughput of the system when percentage of out-ofsequence packets is 0% Author: Tran Huy Vu - 45 - 1.40 Throughput (Gbps) 1.20 1.00 0.80 edge length = 0.60 edge length = 16 0.40 edge length = 32 0.20 0.00 500 1000 1500 2000 Packet length (bytes) Figure 6-6 Throughput of the system with clock rate=125MHz when percentage of out-of-sequence packets is 0% 1.40 Throughput (Gbps) 1.20 1.00 0.80 edge length = 0.60 edge length = 16 edge length = 32 0.40 0.20 0.00 200 400 600 800 1000 1200 1400 1600 Packet length (bytes) Figure 6-7 Maximum throughput of the system when percentage of out-ofsequence packets is 5% Author: Tran Huy Vu - 46 - 1.40 Throughput (Gbps) 1.20 1.00 0.80 edge length = 0.60 edge length = 16 edge length = 32 0.40 0.20 0.00 200 400 600 800 1000 1200 1400 1600 Packet length (bytes) Figure 6-8 Throughput of the system with clock rate = 125MHz when percentage of out-of-sequence packets is 5% 1.40 Throughput (Gbps) 1.20 1.00 0.80 edge length = 0.60 edge length = 16 edge length = 32 0.40 0.20 0.00 200 400 600 800 1000 1200 1400 1600 Packet length (bytes) Figure 6-9 Maximum throughput of the system when percentage of out-ofsequence packets is 10% Author: Tran Huy Vu - 47 - 1.40 Throughput (Gbps) 1.20 1.00 0.80 edge length = 0.60 edge length = 16 edge length = 32 0.40 0.20 0.00 200 400 600 800 1000 1200 1400 1600 Packet length (bytes) Figure 6-10 Throughput of the system with clock rate = 125MHz when percentage of out-of-sequence packets is 10% The throughput of the TCP Reassembly Engine depends on the average length of packets, the length of edge data, the percentage of out-of-sequence packets, as well as the capability of the application circuit to receive multiple bytes in a clock cycle At present, the targeted NIDS application is a system developed by another group at the University of Technology; it can receive only byte in a clock cycle, so it is the bottle neck of the system With short packets, the ratio of the edge length and the packet length is large, it means that the throughput varies in a wider range, so the system are tested with more short packets than long packets The payload length of packets in these charts are 10, 20, 50, 100, 500, 1000 and 1400 bytes The edge length is chosen from 8, 16 and 32 bytes because the current system supports maximum 32 bytes of edge data, we are considering increasing this number when we can change to a new platform with more memory The percentage of out-of-sequence packets (P) is typically smaller than 10% Currently the throughput is measured by implementing two Author: Tran Huy Vu - 48 - counters, the first counter is used to count the number of clock ticks to receive all packets, the second counter counts the number of clock ticks to transmit all packets to network The throughput is calculated by this equation: T = counter0*1000/counter1 This is a rough number; the exact number will be measured by sending packets to the full system, increase the speed until the system start to drops packets The Figure 6-3 to Figure 6-8 displays the maximum throughput and the onboard throughput of the system at P = 0%, 5% and 10% The Figure shows that, the longer the packet is the higher throughput the system can support On the other hand, the shorter the edge data is the higher throughput the system can support 6.2.4 Capability of supporting NIDS 6000 Quantity (rules) 5000 4000 3000 2000 1000