Data deduplication for data optimization for storage and network systems

Daehee Kim Sejun Song Baek-Young Choi Data Deduplication for Data Optimization for Storage and Network Systems Data Deduplication for Data Optimization for Storage and Network Systems Daehee Kim • Sejun Song • Baek-Young Choi Data Deduplication for Data Optimization for Storage and Network Systems 123 Daehee Kim Department of Computing and New Media Technologies University of Wisconsin-Stevens Point Stevens Point, Wisconsin, USA Sejun Song Department of Computer Science and Electrical Engineering University of Missouri-Kansas City Kansas City, Missouri, USA Baek-Young Choi Department of Computer Science and Electrical Engineering University of Missouri-Kansas City Kansas City, Missouri, USA ISBN 978-3-319-42278-7 ISBN 978-3-319-42280-0 (eBook) DOI 10.1007/978-3-319-42280-0 Library of Congress Control Number: 2016949407 © Springer International Publishing Switzerland 2017 This work is subject to copyright All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed The use of general descriptive names, registered names, trademarks, service marks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer International Publishing AG Switzerland Contents Part I Traditional Deduplication Techniques and Solutions Introduction 1.1 Data Explosion 1.2 Redundancies 1.3 Existing Deduplication Solutions to Remove Redundancies 1.4 Issues Related to Existing Solutions 1.5 Deduplication Framework 1.6 Redundant Array of Inexpensive Disks 1.7 Direct-Attached Storage 1.8 Storage Area Network 1.9 Network-Attached Storage 1.10 Comparison of DAS, NAS and SAN 1.11 Storage Virtualization 1.12 In-Memory Storage 1.13 Object-Oriented Storage 1.14 Standards and Efforts to Develop Data Storage Systems 1.15 Summary and Organization References 3 7 10 12 13 13 15 16 16 20 21 Existing Deduplication Techniques 2.1 Deduplication Techniques Classification 2.2 Common Modules 2.2.1 Chunk Index Cache 2.2.2 Bloom Filter 2.3 Deduplication Techniques by Granularity 2.3.1 File-Level Deduplication 2.3.2 Fixed-Size Block Deduplication 2.3.3 Variable-Sized Block Deduplication 2.3.4 Hybrid Deduplication 2.3.5 Object-Level Deduplication 2.3.6 Comparison of Deduplications by Granularity 23 23 25 25 30 34 34 38 44 54 55 55 v vi Contents 2.4 Deduplication Techniques by Place 2.4.1 Server-Based Deduplication 2.4.2 Client-Based Deduplication 2.4.3 End-to-End Redundancy Elimination 2.4.4 Network-Wide Redundancy Elimination 2.5 Deduplication Techniques by Time 2.5.1 Inline Deduplication 2.5.2 Offline Deduplication 2.6 Summary References 56 56 57 58 60 71 71 73 74 75 Part II Storage Data Deduplication HEDS: Hybrid Email Deduplication System 3.1 Large Redundancies in Emails 3.2 Hybrid System Design 3.3 EDMilter 3.4 Metadata Server 3.5 Bloom Filter 3.6 Chunk Index Cache 3.7 Storage Server 3.8 EDA 3.9 Evaluation 3.9.1 Metrics 3.9.2 Data Sets 3.9.3 Deduplication Performance 3.9.4 Memory Overhead 3.9.5 CPU Overhead 3.10 Summary References 79 79 80 80 82 82 82 83 83 85 85 86 89 92 94 94 94 SAFE: Structure-Aware File and Email Deduplication for Cloud-Based Storage Systems 4.1 Large Redundancies in Cloud Storage Systems 4.2 SAFE Modules 4.3 Email Parser 4.4 File Parser 4.5 Object-Level Deduplication and Store Manager 4.6 SAFE in Dropbox 4.7 Evaluation 4.7.1 Metrics 4.7.2 Data Sets 4.7.3 Storage Data Reduction Performance 4.7.4 Data Traffic Reduction Performance 4.7.5 CPU Overhead 4.7.6 Memory Overhead 97 97 98 99 100 103 104 106 107 107 109 109 110 113 Contents vii 4.8 Summary 113 References 115 Part III Network Deduplication SoftDance: Software-Defined Deduplication as a Network and Storage Service 5.1 Large Redundancies in Network 5.2 Software-Defined Network 5.3 Control and Data Flow 5.4 Encoding Algorithms in Middlebox (SDMB) 5.5 Index Distribution Algorithms 5.5.1 SoftDANCE-Full (SD-Full) 5.5.2 SoftDance-Uniform (SD-Uniform) 5.5.3 SoftDANCE-Merge (SD-Merge) 5.5.4 SoftDANCE-Optimize (SD-opt) 5.6 Implementation 5.6.1 Floodlight, REST, JSON 5.6.2 CPLEX Optimizer: Installation 5.6.3 CPLEX Optimizer: Run Simple CPLEX Using Interactive Optimizer 5.6.4 CPLEX Optimizer: Run Simple CPLEX Using Java Application (with CPLEX API) 5.7 Setup 5.7.1 Experiment 5.7.2 Emulation 5.8 Evaluation 5.8.1 Metrics 5.8.2 Data Sets 5.8.3 Storage Space and Network Bandwidth Saving 5.8.4 CPU and Memory Overhead 5.8.5 Performance and Overhead per Topology 5.8.6 SoftDance vs Combined Existing Deduplication Techniques 5.9 Summary References 119 119 121 121 124 125 125 126 127 128 130 130 130 135 137 139 139 140 140 140 142 142 143 145 147 150 151 Part IV Future Directions Mobile De-Duplication 6.1 Large Redundancies in Mobile Devices 6.2 Approaches and Observations 6.3 JPEG and MPEG4 6.4 Evaluation 6.4.1 Setup 155 155 156 156 156 157 viii Contents 6.4.2 Throughput and Running Time per File Type 6.4.3 Throughput and Running Time per File Size 6.5 Summary References 158 161 161 164 Conclusions 165 Part V Appendixes Appendices 169 A Index Creation with SHA1 A.1 sha1Wrapper.h A.2 sha1Wrapper.cc A.3 sha1.h A.4 sha1.cc 171 171 172 173 177 B Index Table Implementation using Unordered Map B.1 cacheInterface.h B.2 cache.h B.3 cache.cc 193 193 195 198 C Bloom Filter Implementation 201 C.1 bf.h 201 C.2 bf.c 202 D Rabin Fingerprinting Implementation D.1 rabinpoly.h D.2 rabinpoly.cc D.3 rabinpoly_main.cc 209 209 211 216 E Chunking Core Implementation E.1 chunk.h E.2 chunk_main.cc E.3 chunk_sub.cc E.4 common.h E.5 util.cc 219 219 221 223 226 227 F Chunking Wrapper Implementation F.1 chunkInterface.h F.2 chunkWrapper.h F.3 chunkWrapper.cc F.4 chunkWrapperTest 231 231 233 233 237 Contents G Sample Programs Using libnetfilter_queue Library G.1 ndedup.h G.2 ndedup.cc G.3 ndedup_main.cc References ix 239 239 243 255 260 Glossary 261 248 G Sample Programs Using libnetfilter_queue Library } unsigned short NDedup : : t c p _ s y n ( ) { return tcp_syn_ ; } unsigned short NDedup : : t c p _ f i n ( ) { return t c p _ f i n _ ; } unsigned i n t NDedup : : tcp_window ( ) { r e t u r n tcp_window_ ; } unsigned i n t NDedup : : t c p _ c h e c k s u m ( ) { return tcp_checksum_ ; } void NDedup : : showTCPInfo ( ) { / / show TCP i n f o r m a t i o n printf (" TCP h e a d e r printf ( " sport : p r i n t f ( " dport : p r i n t f ( " seq : p r i n t f ( " ack seq : p r i n t f ( " header length : p r i n t f ( " flag ( urgent ) : p r i n t f ( " f l a g ( ack ) : p r i n t f ( " f l a g ( push ) : printf ( " flag ( rst ) : p r i n t f ( " f l a g ( syn ) : printf ( " flag ( fin ) : p r i n t f ( " window s i z e : p r i n t f ( " checksum : \ n" ) ; %u %u %l u %l u %u ( b y t e ) %u %u %u %u %u %u %u ( b y t e ) x%0x \ n" , \ n" , \ n" \ n" \ n" , \ n" , \ n" , \ n" , \ n" , \ n" , \ n" , \ n" , \ n" , tcp_sport ( ) ) ; tcp_dport ( ) ) ; , tcp_seq ( ) ) ; , tcp_ack_seq ( ) ) ; tcp_hdr_len ( ) ) ; tcp_urg ( ) ) ; tcp_ack ( ) ) ; tcp_psh ( ) ) ; tcp_rst ()); tcp_syn ( ) ) ; tcp_fin ( ) ) ; tcp_window ( ) ) ; tcp_checksum ( ) ) ; } // // // // // // // // // // // Compute c h e c k s u m o f TCP Checksum f i e l d i s c l e a r e d b e f o r e c h e c k s u m i s c o m p u t e d TCP c h e c k s u m i s c o m p u t e d w i t h p s e u d o h e a d e r and TCP s e g m e n t TCP s e g m e n t c o n s i s t s o f TCP h e a d e r and TCP d a t a Pseudo h e a d e r c o n s i s t s o f i p s o u r c e a d d r e s s ( b i t s ) , ip d e s t i n a t i o n address (32 b i t s ) , reserved (8 b i t s ) , p r o t o c o l ( b i t s ) , and l e n g t h o f c o m b i n e d TCP h e a d e r and TCP d a t a ( b i t s ) G.2 ndedup.cc 249 // IMPORTANT / / I f TCP d a t a i s n o t a l i g n e d t o 16 b i t s , i s padded / / E g x0A i s l a s t b i t s i n a TCP d a t a , / / i t i s c h a n g e d t o x0A00 (8 b i t s a r e padded ) // / / For more i n f o r m a t i o n , r e f e r t o f o l l o w i n g URLs / / h t t p : / / www t c p i p g u i d e com / f r e e / / / t _ T C P C h e c k s u m C a l c u l a t i o n a n d t h e T C P P s e u d o H e a d e r htm / / h t t p : / / www p e r s o n a l u n i j e n a de / ~ p l o s t /DC/ s o f t w a r e // / t u t o r i a l s / TCP_IP_checksum p d f // / / r e t u r n v a l u e : TCP c h e c k s u m unsigned i n t NDedup : : t c p _ c o m p u t e C h e c k s u m ( u n s i g n e d char n f _ p a c k e t _ p a y l o a d ) { int i ; int index = 0; / / TCP d a t a l e n g t h i n b y t e s int tcp_data_len = ip_tot_len () ( ip_ihl () + tcp_hdr_len ( ) ) ; i n t num_inputs = / Pseudo Header / + / / i p s o u r c e a d d r e s s ( b y t e s > 16 b i t words ) + / / ip dest address + / / reserved (8 b i t s ) + protocol (8 b i t s ) + / / l e n g t h o f TCP s e g m e n t i n b y t e s / TCP Header / ( tcp_hdr_len ( ) / 2) + / TCP Data / ( ( t c p _ d a t a _ l e n + 1) / ) ; / / i s u s e d t o r o u n d up s i z e f o r p a d d i n g u n s i g n e d i n t words [ n u m _ i n p u t s ] ; / / 16 b i t words // / / make i n p u t s // / Pseudo Header words [ i n d e x ++] = words [ i n d e x ++] = words [ i n d e x ++] = words [ i n d e x ++] = words [ i n d e x ++] = words [ i n d e x ++] = / ( ip_saddr () ( ip_saddr () ( ip_daddr () ( ip_daddr ( ) ip_protocol tcp_hdr_len >> ) ; / / l e f t 16 b i t s & 5 ) ; / / r i g h t 16 b i t s >> ) ; & 65535); (); / / " ip reserved " is () + tcp_data_len ; / TCP Header / for ( i = i p _ i h l ( ) ; i < ( i p _ i h l ( ) + t c p _ h d r _ l e n ( ) ) ; i = i +2) { words [ i n d e x ] = ( n f _ p a c k e t _ p a y l o a d [ i ] ) & 5 ; n f _ p a c k e t _ p a y l o a d [ TCP_CHECKSUM_SECOND_OFFSET] = checksum & 65535; / / to object t c p _ c h e c k s u m _ = checksum ; } / / ###################################################### / / UDP / / ###################################################### / / i t i s n o t shown f o r UDP / / ###################################################### / / Block / / ###################################################### unsigned i n t NDedup : : b l o c k _ b e g i n _ o f f s e t ( ) { return b l o c k _ b e g i n _ o f f s e t _ ; } G.2 ndedup.cc 251 unsigned i n t NDedup : : b l o c k _ e n d _ o f f s e t ( ) { return block_end_offset_ ; } / / in bytes / / IP t o t a l l e n g t h ( IP h e a d e r s i z e + TCP h e a d e r s i z e ) unsigned i n t NDedup : : b l o c k _ s i z e ( ) { return block_size_ ; } string NDedup : : b l o c k ( ) { return block_ ; } void NDedup : : s h o w B l o c k I n f o ( ) { / / show B l o c k i n f o r m a t i o n printf (" Block \ n" ) ; p r i n t f ( " begin o f f s e t : %u p r i n t f ( " end offset : %u printf ( " size : %u / / p r i n t f ("% s \ n" , block_begin_offset ( ) ) ; \ n" , block_end_offset ( ) ) ; \ n" , block_size ( ) ) ; \ n " , block ( ) ) ; } void NDedup : : s a v e B l o c k T o F i l e ( ) { ofstream outF ; / / set filename stringstream s ; s t r i n g filename ; s > ] ; output [1] = lut [ c & 15]; output [2] = ’ \0 ’ ; c o u t = ; i p r i n t f ( "%d " , g e t B i t ( num , i ) ) ; } p r i n t f ( " \ n" ) ; ) { } / / get value of a b i t for a position // 3210 ( b i t p o s i t i o n ) / / e g 0100 > bit (2) returns int NDedup : : g e t B i t ( i n t num , i n t p o s ) { i n t mask = ; int i ; f o r ( i = ; i < p o s ; i ++) { mask = ; } / / p r i n t f ( " num = %d , p o s = %d , mask = %d \ n " , num , pos , mask ) ; num &= mask ; r e t u r n num >> p o s ; } / / set value of a b i t for a position // 3210 / / e g num ( ) = 1001 > s e t b i t ( num , , ) > // / / for error , return / / O t h e r w i s e , r e t u r n c h a n g e d number int NDedup : : s e t B i t ( i n t num , i n t pos , i n t b i t V a l u e ) { 0001 254 G Sample Programs Using libnetfilter_queue Library int i ; i f ( b i t V a l u e == ) { i n t mask = b i t V a l u e ; mask = mask =0; i 0) , 1); ) { , 0); r e t u r n num ; } / / add numbers w i t h w r a p p i n g a r o u n d c a r r y / / r e t u r n an added number int NDedup : : addNumbers ( i n t num1 , i n t num2 ) { i n t sum ; int carry ; / / add numbers sum = num1 + num2 ; / / check carry c a r r y = sum >> NUM_BITS ; / / p r i n t f ( " c a r r y = %d \ n " , c a r r y ) ; / / wrap a r o u n d c a r r y G.3 ndedup_main.cc i n t mask = ( i n t ) ( pow ( , NUM_BITS ) i f ( carry > 0) { sum &= mask ; sum += c a r r y ; } r e t u r n sum ; 255 1); } G.3 ndedup_main.cc # i n c l u d e " ndedup h " # i f d e f NDEDUP_TEST v o i d p r i n t B y t e s ( u n s i g n e d char i n p u t , i n t l e n ) { s t a t i c c o n s t char c o n s t l u t = " 0123456789ABCDEF" ; char o u t p u t [ l e n + 1]; f o r ( i n t i = ; i < l e n ; ++ i ) { c o n s t u n s i g n e d char c = i n p u t [ i ] ; output [ i ] = l u t [ c >> ] ; output [ i + 1] = l u t [ c & 15]; } output [ len 2] = ’ \0 ’ ; c o u t ] ; output [1] = lut [ c & 15]; output [2] = ’ \0 ’ ; c o u t h w _ a d d r l e n ) ; p r i n t f ( " hw_src_addr ="); f o r ( i = ; i < h l e n 1; i ++) p r i n t f ("%02 x : " , hwph >hw_addr [ i ] ) ; p r i n t f ("%02 x " , hwph >hw_addr [ h l e n ] ) ; } mark = n f q _ g e t _ n f m a r k ( t b ) ; i f ( mark ) p r i n t f ( " mark=%u " , mark ) ; i f i = nfq_get_indev ( tb ); if ( ifi ) p r i n t f ( " i n d e v=%u " , i f i ) ; i f i = nfq_get_outdev ( tb ); if ( ifi ) p r i n t f ( " o u t d e v=%u " , i f i ) ; i f i = nfq_get_physindev ( tb ); if ( ifi ) p r i n t f ( " p h y s i n d e v=%u " , i f i ) ; i f i = nfq_get_physoutdev ( tb ); if ( ifi ) p r i n t f ( " p h y s o u t d e v=%u " , i f i ) ; r e t = n f q _ g e t _ p a y l o a d ( t b , &n f _ p a c k e t ) ; i f ( r e t >= ) { p r i n t f ( " p a y l o a d _ l e n=%d \ n " , r e t ) ; } fputc ( ’\ n ’ , stdout ); / // / / I n i t i a l i z e NDedup o b j e c t // n f q _ g e t _ p a y l o a d ( t b , &n f _ p a c k e t ) ; nd s e t ( n f _ p a c k e t ) ; // G.3 ndedup_main.cc 257 / / IP I n f o r m a t i o n // nd s h o w I P I n f o ( ) ; // / / TCP I n f o r m a t i o n // i f ( nd i p _ p r o t o c o l ( ) == ) { nd showTCPInfo ( ) ; } // / / UDP I n f o r m a t i o n // / / l a t e r implemented // / / Block Information // i f ( nd b l o c k _ s i z e ( ) > ) { nd s h o w B l o c k I n f o ( ) ; nd s a v e B l o c k T o F i l e ( ) ; n f _ p a y l o a d _ s i z e = nd i p _ t o t _ l e n ( ) ; b l o c k _ b e g i n _ o f f s e t = nd b l o c k _ b e g i n _ o f f s e t ( ) ; block_end_offset = nd b l o c k _ e n d _ o f f s e t ( ) ; } return id ; } s t a t i c i n t cb ( s t r u c t n f q _ q _ h a n d l e qh , s t r u c t nfgenmsg s t r u c t n f q _ d a t a nfa , void d a t a ) { int nf_payload_size = 0; u n s i g n e d char n f _ p a y l o a d ; u_int32_t id ; int i ; int len ; NDedup nd ; unsigned i n t b l o c k _ b e g i n _ o f f s e t , b l o c k _ e n d _ o f f s e t ; / / get nf_payload l e n = n f q _ g e t _ p a y l o a d ( n f a , &n f _ p a y l o a d ) ; nd s e t ( n f _ p a y l o a d ) ; / / g e t i d and n f _ p a y l o a d _ s i z e i d = p r i n t _ p k t ( n f a , &n f _ p a y l o a d _ s i z e , &b l o c k _ b e g i n _ o f f s e t , &b l o c k _ e n d _ o f f s e t ) ; nfmsg , 258 G Sample Programs Using libnetfilter_queue Library if ( nf_payload_size / / p r i n t f (" payload / / p r i n t f (" block / / p r i n t f (" block > 0) { size = %d \ n " , n f _ p a y l o a d _ s i z e ) ; b _ o f f s e t = %u \ n " , b l o c k _ b e g i n _ o f f s e t ) ; e _ o f f s e t = %u \ n " , b l o c k _ e n d _ o f f s e t ) ; // / / payload before m o d i f i c a t i o n // printf (" n f _ p a y l o a d ( I P h e a d e r + TCP h e a d e r + " "TCP d a t a ) \ n" ) ; p r i n t f ( " >>> b e f o r e m o d i f i c a t i o n \ n " ) ; p r i n t f ( " i p checksum = x%0x \ n " , nd ip_computeChecksum ( n f _ p a y l o a d ) ) ; p r i n t f ( " t c p checksum = x%0x \ n " , nd t c p _ c o m p u t e C h e c k s u m ( n f _ p a y l o a d ) ) ; i f ( len ) { p r i n t B y t e s ( nf_payload , len ) ; } / / ########################################################## / / modify nf_payload / / ########################################################## // / / modify data // f o r ( i = b l o c k _ b e g i n _ o f f s e t ; i = ) { p r i n t f ( " pkt received \ n" ) ; / / p r i n t f ("% s " , b u f ) ; n f q _ h a n d l e _ p a c k e t ( h , buf , r v ) ; } p r i n t f ( " u n b i n d i n g from q u e u e \ n " ) ; n f q _ d e s t r o y _ q u e u e ( qh ) ; # i f d e f INSANE / n o r m a l l y , a p p l i c a t i o n s SHOULD NOT i s s u e t h i s command , since i t d e t a c h e s o t h e r p r o g r a m s / s o c k e t s f r o m AF_INET , too ! / p r i n t f ( " u n b i n d i n g from AF_INET \ n " ) ; n f q _ u n b i n d _ p f ( h , AF_INET ) ; # endif p r i n t f ( " closing l i b r a r y handle \ n" ) ; nfq_close (h ) ; exit (0); } # endif References Rabin, M.O.: Fingerprinting by random polynomials Tech Rep Report TR-15-81, Harvard University (1981) Bakker, P.: SHA1 codes http://asf.atmel.com/docs/latest/uc3l/html/sha1_8h.html Mazieres, D.: Rabin Finterprinting Codes https://github.com/okws/sfslite/blob/master/crypt/ rabinpoly.h Glossary ARM-CPU Microprocessor technology that is a family of reduced instruction set computing (RISC) architectures and used mainly for low-power devices Block Fixed-size data accessed with offsets Bloom Filter Small bit array used to check whether duplicate data exist in storage C++ Boost Library Peer-reviewed portable C++ source libraries Chunk Variable-sized data CPLEX optimizer Optimization software to compute a linear program Eclipse Integrated development environment EDMilter Email deduplication mail filter Floodlight Java-based controller system in a software-defined network Granularity The unit of data to be compared to check duplicates Index cache Memory that contains data hash keys Iptables Userspace program that configures firewall in Linux kernel JPEG Popular compression technique for digital photography a lossy compression technique for color images JSON JavaScript Object Notation, a lightweight data-interchange format Libnetfilter_queue Userspace library providing an API to packets that have been queued by the kernel packet filter MPEG Standards for audio and video compression and transmission Netfilter Series of hooks to capture packets at various points in protocol stack NFQUEUE Iptable target © Springer International Publishing Switzerland 2017 D Kim et al., Data Deduplication for Data Optimization for Storage and Network Systems, DOI 10.1007/978-3-319-42280-0 261 262 Glossary Offset-shifting problem Fixed offset is shifted when data before the offset are inserted in a file OpenVSwitch Virtual switch for software-defined network Packet Decoding Reconstructing the encoded packet Packet Encoding Replacing the redundant payload within a packet with an index Rabin Fingerprint Technique to find chunk boundary Redundancy Elimination Solution to remove duplicate data transferred through network REST Software architecture where data are accessed through HTTP API Storage Data Deduplication Solution to remove duplicate data when data are stored in storage Time Complexity Amount of time taken by an algorithm to run Unordered_map Data structure with a pair consisting of a key and value; data are saved in no particular order .. .Data Deduplication for Data Optimization for Storage and Network Systems Daehee Kim • Sejun Song • Baek-Young Choi Data Deduplication for Data Optimization for Storage and Network Systems. .. Deduplication for Data Optimization for Storage and Network Systems, DOI 10.1007/978-3-319-42280-0_1 Introduction Fig 1.1 Data explosion: IDC’s Digital Universe Study [6] computation, storage and networks... of data storage Data storage has evolved from storage devices attached to a single computer (direct-attached storage) into storage devices attached to computer networks (storage area network and