!"I H#C QU$C GIA THÀNH PH$ H% CHÍ MINH TR&'NG !"I H#C BÁCH KHOA PHAN XN THI(N MƠ HÌNH C)NG TÁC H* TR+ !,NH TUY-N TRÊN M"NG NHI.U VÙNG D/A TRÊN OPENFLOW Chuyên nghành: Khoa H!c Máy Tính Mã s": 60.48.01 LU0N V1N TH"C S2 TP H# Chí Minh, Tháng 12 n$m 2012 CƠNG TRÌNH !"#C HỒN THÀNH T$I TR"%NG !$I H&C BÁCH KHOA – !HQG - TPHCM Cán b' h()ng d*n khoa h+c: … PGS TS Tho,i Nam … (Ghi rõ h+, tên, h+c hàm, h+c v- ch k/) Cán b' ch0m nh1n xét 1: … TS Ph,m Tr2n V3 ………………… (Ghi rõ h+, tên, h+c hàm, h+c v- ch k/) Cán b' ch0m nh1n xét 2: … TS Ph,m V4n H1u ………………… (Ghi rõ h+, tên, h+c hàm, h+c v- ch k/) Lu1n v4n th,c s5 6(7c b8o v9 t,i Tr(:ng !,i H+c Bách Khoa, !HQG Tp HCM ngày 24 tháng 12 n4m 2012 Thành ph2n H'i 6;ng 6ánh giá lu1n v4n th,c s5 g;m: (Ghi rõ h+, tên, h+c hàm, h+c v- c k2) TR!:NG KHOA (H% tên ch> k2) "! ! L!I C"M #N Tôi xin g!i l"i c#m $n sâu s%c &'n PGS TS Tho(i Nam &ã t)n tình h*+ng d,n, giúp &- tơi th.c hi/n &0 tài nghiên c1u Tôi c2ng xin c#m $n GS Pierre Kuonen &ã t)n tình h*+ng d,n tơi trình th.c t)p Fribourg, Th4y S5 Sau cùng, xin g!i l"i c#m $n chân thành &'n gia &ình b(n bè &ã ln bên c(nh &6ng viên giúp &- tơi ! ""! TĨM T!T LU"N V#N OpenFlow m!t ki"n trúc m#ng c$i ti"n %ó tách bi&t h& %i'u khi(n m#ng h& truy'n d) li&u, cho phép nhà nghiên kh$ n*ng l+p trình c, s- h# t.ng m#ng %( %i'u khi(n thi"t b/ m#ng ph0c v0 cho trình th1 nghi&m c2a h3 Tuy nhiên, ki"n trúc c2a OpenFlow l& thu!c hoàn toàn vào vi&c s1 d0ng m!t h& %i'u khi(n t+p trung g3i OpenFlow controller ch#y m!t máy tính %,n %( %i'u hành ho#t %!ng c2a t4t c$ thi"t b/ m#ng V5i nh)ng m#ng có s6 l78ng thi"t b/ l5n, t.n su4t g1i yêu c.u t9 thi"t b/ nhi'u vi&c s1 d0ng ch: m!t controller %( x1 l; có th( s< khơng %áp =ng n>i ho?c %áp =ng không hi&u qu$ s6 l78ng yêu c.u l5n g1i t9 thi"t b/ b/ gi5i h#n b-i kh$ n*ng x1 l; c2a m!t máy tính H,n n)a n"u mơi tr7@ng m#ng có ph#m vi phân b6 r!ng th@i gian %áp =ng cho yêu c.u t9 thi"t b/ - xa b/ $nh h7-ng %! trA b*ng thông c2a k"t n6i gi)a controller thi"t b/ m#ng Vì v+y vi&c cho phép nhi'u controller ho#t %!ng %Bng th@i m#ng m!t gi$i pháp h)u hi&u cho v4n %' HC tr8 hi&n t#i c2a OpenFlow cho phép tri(n khai m#ng OpenFlow nhi'u controller %ó mCi controller qu$n l; m!t vùng m#ng m#ng Tuy nhiên hi&n ch7a có c, ch" %( controller c!ng tác v5i vi&c %i'u khi(n thi"t b/ m#ng H& qu$ controller không th( %/nh tuy"n gói tin liên vùng m#ng m!t cách hi&u qu$ Tr75c yêu c.u nh7 v+y, lu+n v*n nghiên c=u %' xu4t m!t mơ hình cho phép controller - vùng m#ng m#ng c!ng tác v5i vi&c %i'u hành m#ng, c, s- %ó xây dDng gi$i pháp nâng cao hi&u qu$ %/nh tuy"n vùng m#ng OpenFlow ! """! ABSTRACT OpenFlow is an innovative network architecture which decouples control plane and data plane, allowing researchers the ability to program their networks, control the behaviour of network switches for their experiments However, current OpenFlow architecture relies completely on the use of a centralized controller to manage all switches connecting to it in the network For networks with large number of switches, depending on a controller to manage all switches in the network becomes unfeasible Thus allowing distributed multiple controllers in managing OpenFlow network is an appropriate solution for OpenFlow scalability Current OpenFlow support allows deploying multi-controllers networks in which each controller is responsible for operating each smaller domain in the network, however there is currently no mechanism for these controllers to cooperate with the each other As a result, the controllers cannot effectively process cross-domain packets This paper proposes building a collaborative model for multi-domains OpenFlow networks that allows different network domains to collaborate with each other in an efficient way In addition, a routing solution is proposed based on the model to achieve higher network performance "#! ! L!I CAM "OAN Tôi xin cam !oan r"ng b#n lu$n v%n t&t nghi'p k(t qu# nghiên c)u th*c s* tơi th*c hi'n Ngo+i tr, k(t qu# tham kh#o t, cơng trình khác nh- !ã ghi rõ lu$n v%n, công vi'c nêu lu$n v%n trung th*c ch-a t,ng !-.c công b& b/t k0 cơng trình khác ho1c !-.c n2p !3 l/y m2t b"ng c/p m2t tr-5ng khác ! "! M!C L!C L!I C"M #N …………………………………………………………………… i TÓM T$T LU%N V&N ………………………………………………………… ii ABSTRACT ………………………………………………………………………iii L!I CAM 'OAN ………………………………………………………………… iv M(C L(C ……………………………………………………………………… v DANH M(C HÌNH ………………………………………………………………ix CH)#NG GI*I THI+U ', TÀI …………………………………………… 1.1 '-t v.n /0 ………………………………………………………………… 1.2 Gi1i thi2u /0 tài …………………………………………………………… 1.2.1 Tên /0 tài ………………………………………………………… 1.2.2 Gi1i h3n c4a /0 tài ………………………………………………… 1.2.3 M5c tiêu /0 tài ……………………………………………………… 1.2.4 ngh7a khoa h8c th9c ti:n ……………………………………… 1.3 Quy trình th9c hi2n …………………………………………………… CH)#NG C# S; L6 THUYp b?o m@t OpenFlow ……………………………… 16 2.2.4 Giao thAc OpenFlow …………………………………………… 16 2.3 NhBng lCi ích c4a m3ng d9a OpenFlow …………………………… 17 ! "#! CH!"NG NH#NG CƠNG TRÌNH NGHIÊN C$U LIÊN QUAN ………… 20 3.1 H% &i'u khi(n phân b) cho m*ng OpenFlow (HyperFlow) …………… 21 3.2 Ki+n trúc m, r-ng cho h% &i'u khi(n m*ng OpenFlow n-i b- (ASIC) … 23 3.3 ' xu/t s0 d1ng nhi'u controller cho m*ng OpenFlow ………………… 25 CH!"NG MƠ HÌNH VÀ GI2I PHÁP 3NH TUY4N XU6T ………… 29 4.1 Mơ hình chia s7 thông tin gi8a vùng m*ng OpenFlow …………… 30 4.2 Gi9i pháp &:nh tuy+n &' xu/t …………………………………………… 32 4.2.1 Các thành ph;n mơ hình RMOF …………………… 33 4.2.2 Quy trình x0 l< mơ hình RMOF …………………………… 39 4.2.3 Ph=>ng pháp dị tìm liên k+t liên vùng ……………………… 42 4.2.4 Ph=>ng pháp tính chi phí &:nh tuy+n gi8a switch liên vùng m-t vùng m*ng ……………………………………… 44 4.2.5 Ph=>ng pháp tính &=?ng &:nh tuy+n toàn c1c …………………… 44 CH!"NG GIAO TH$C TRAO @I THƠNG TIN TRONG MƠ HÌNH XU6T ………………………………………………………… 48 5.1 Các nhóm thơng &i%p giao thAc RMOF …………………… 49 5.2 Header cBa thông &i%p giao thAc RMOF ………………………… 50 5.3 Thông &i%p &)i xAng …………………………………………………… 51 5.3.1 Thông &i%p RMOF_HANDSHAKE ……………………………… 51 5.3.2 Thông &i%p RMOF_ECHO_REQUEST ………………………… 52 5.3.3 Thông &i%p RMOF_ECHO_REPLY …………………………… 52 5.4 Thông &i%p trao &Ci thông tin &Dc tính cBa controller ………………… 52 5.4.1 Thơng &i%p RMOF_CONTROLLER_FEATURES_REQUEST 52 5.4.2 Thông &i%p RMOF_CONTROLLER_FEATURES_REPLY …… 53 ! "##! 5.5 Thơng !i"p c#p nh#t thơng tin topology tồn c$c …………………….… 53 5.5.1 Thông !i"p RMOF_CDLINK_ADD ……………………………… 54 5.5.2 Thông !i"p RMOF_CDLINK_REMOVE ………………………… 55 5.6 Thông !i"p !%nh tuy&n …………………………………………………… 55 5.6.1 Thông !i"p RMOF_ROUTE_INS_REQUEST …………………… 55 5.6.2 Thông !i"p RMOF_ROUTE_INS ………………………………… 56 5.7 Thơng !i"p !%nh v% host !ích ……………………………………… 58 5.7.1 Thông !i"p RMOF_ADDR_AUTHENTICATION_REQUEST … 58 5.7.2 Thơng !i"p RMOF_ADDR_AUTHENTICATION_REPLY ….… 58 CH'(NG HI)N TH*C HĨA MƠ HÌNH VÀ GI+I PHÁP ,-NH TUY.N ,/ XU0T ………………………………………………………….61 6.1 L1a ch2n n3n t4ng controller: Beacon …………………………………… 61 6.2 C4i ti&n n3n t4ng Beacon controller …………………………………… 62 6.3 Hi"n th1c hóa thành ph5n RMOF collaborator ………………………… 66 6.4 Các quy trình x6 l7 h" th8ng ……………………………………… 68 CH'(NG ,ÁNH GIÁ ……………………………………………………… 71 7.1 Tri9n khai m:ng OpenFlow h" !i3u khi9n !;n %ó v)n %( %7nh tuy"n gi2a vùng m#ng m#ng n(n t5ng OpenFlow controller hi1n t#i %(u ch&a có c= ch" hJ tr' vi1c %7nh tuy"n K( tài %ã %( mơ hình cho phép controller m#ng c!ng tác v$i c= s+ %ó xây d9ng gi5i pháp %7nh tuy"n gi2a vùng m#ng OpenFlow Mơ hình cho phép controller chia sM thơng tin m!t c= ch" hi1u qu5 có tính tin c>y gi5i pháp %7nh tuy"n %&'c %( xu)t th9c s9 nâng cao kh5 n*ng %7nh tuy"n gi2a vùng m#ng qua %ó nâng cao hi1u qu5 %7nh tuy"n tồn th4 m#ng Mơ hình %&'c %( xu)t giúp nâng cao kh5 n*ng m+ r!ng c/a m#ng OpenFlow, có th4 %áp ,ng %&'c m#ng có s; l&'ng switch l$n h=n s; request tI thi"t b7 nhi(u h=n K3c bi1t, nh- phân b; nhi(u controller m#ng cho phép chúng kh5 n*ng c!ng tác v$i vi1c qu5n l6 m#ng, %ây m!t gi5i pháp ti(m n*ng có th4 %&'c ,ng dDng tri4n khai cho m#ng OpenFlow có ph#m vi phân b; r!ng Sau %ây k"t qu5 %#t %&'c c/a lu>n v*n nh2ng %i4m cịn thi"u sót (*! ! Nh=ng vi,c )ã ):t );>c: ! Nghiên c,u v( OpenFlow bao gHm c5 ki"n trúc, giao th,c, n(n t5ng controller nghiên c,u d9a OpenFlow ! Nghiên c,u gi5i pháp nâng cao kh5 n*ng m+ r!ng c/a OpenFlow hi1n ! Xác %7nh h&$ng ti"p c>n cho phép nhi(u controller phân b; vùng m#ng qu5n l6 thi"t b7 m#ng nghiên c,u phát tri4n cho h&$ng ti"p c>n ! K&a mơ hình cho phép controller m#ng kh5 n*ng c!ng tác v$i vi1c %i(u hành m#ng c/a chúng ! D9a mơ hình %&a xây d9ng gi5i pháp %7nh tuy"n %4 gi5i quy"t v)n %( %7nh tuy"n gi2a vùng m#ng m#ng ! L9a ch.n nghiên c,u sâu hi1n th9c c/a m!t n(n t5ng OpenFlow controller tiêu bi4u Beacon ! Ti"n hành c5i ti"n m!t s; mô-%un hi1n t#i c/a Beacon %Hng th-i xây d9ng mô-%un ph8n m(m bC sung cho Beacon, ti"n hành xây d9ng mô-%un ph8n m(m cho thành ph8n RMOF collaborator %4 hi1n th9c hóa mơ hình gi5i pháp %7nh tuy"n ! ThE nghi1m h1 th;ng %&'c hi1n th9c %ánh giá gi5i pháp %7nh tuy"n ! Vi"t %&'c m!t báo khoa h.c %&'c ch)p nh>n + m!t h!i ngh7 qu;c t" chuyên ngành d&$i s9 b5o tr' c/a IEEE h!i ngh7 ComManTel2013 (The IEEE International Conference on Computing, Management and Telecommunication) sN diPn vào tháng 1/2013 Nh=ng vi,c cịn thi7u sót: ! Vi1c thE nghi1m h1 th;ng vAn %=n gi5n, m$i ch< thE nghi1m %&'c hi1u su)t c/a vi1c %7nh tuy"n + thông s; v( thông l&'ng %! trP c/a m#ng d&$i s9 %i(u khi4n c/a h1 %i(u khi4n %&'c hi1n th9c )+! ! ! Ch&a thE nghi1m gi5i pháp m!t môi tr&-ng m#ng l$n %4 %ánh giá kh5 n*ng m+ r!ng c/a gi5i pháp hi1u su)t c/a h1 %i(u khi4n %&'c hi1n th9c ! Giao th,c %( xu)t cho vi1c trao %Ci thông tin mơ hình ch&a xây d9ng thơng %i1p báo lJi 8.2 H;+ng phát triAn K4 %ánh giá m!t cách hoàn tồn h=n v( hi1u qu5 c/a mơ hình gi5i pháp %7nh tuy"n %&'c %( xu)t %4 chúng có th4 %&'c l9a ch.n tri4n khai th9c t" h1 th;ng %&'c hi1n th9c c8n %&'c ti"n hành t;i &u l#i thE nghi1m m!t cách toàn di1n h=n %4 %ánh giá %! Cn %7nh, hi1u su)t kh5 n*ng %áp ,ng c/a Do v>y h&$ng phát tri4n ti"p theo c/a %( tài là: ! Ti"n hành làm gi5m %! ph,c t#p gi5i thu>t c/a mã ch&=ng trình c/a mô-%un ph8n m(m %&'c hi1n th9c, %Hng th-i thE nghi1m l9a ch.n gi5i thu>t có th-i gian xE l6 th)p nh)t %4 %&a vào h1 th;ng, %4 tI %ó gi5m t;i %a th-i gian xE l6 cho vi1c tìm %&-ng %i gi2a switch m#ng nâng cao hi1u su)t c/a h1 th;ng ! Xây d9ng thông %i1p báo lJi bC sung cho giao th,c RMOF ! Tìm hi4u xây d9ng b! cơng cD thE nghi1m riêng %4 %ánh giá hi1u su)t c/a h1 %i(u khi4n %&'c hi1n th9c m!t cách toàn di1n h=n ! ThE nghi1m h1 th;ng m!t môi tr&-ng m#ng l$n h=n %4 %ánh giá kh5 n*ng %áp ,ng hi1u su)t c/a nó, %Hng th-i th-i gian thE nghi1m ph5i %/ dài %4 có th4 %ánh giá tính Cn %7nh c/a )"! ! TÀI LI$U THAM KHGO [1] N McKeown, T Anderson, H Balakrishnan, G Parulkar, L Peterson, J Rexford, S Shenker, and J Turner, “Openflow: enabling innovation in campus networks,” SIGCOMM Comput Commun Rev., vol 38, no 2, pp 69–74, 2008 [2] “Intro To OpenFlow,” 2011 [Online] Available: https://www.opennetworking.org/ standards/intro-to-openflow [3] Rob Sherwood, Glen Gibb, Kok-Kiong Yap, Guido Appenzeller, Martin Casado, Nick McKeown, Guru Parulkar, “Flow Visor: A Network Virtualization Layer,” Deutsche Telekom Inc R&D Lab, Stanford University, Nicira Networks, 2009 [4] “Software-Defined Networking: The new norm for networks,” Open Networking Foundation, April 2012 [5] Mark Reitblatt, Nate Foster, Jennifer Rexford, David Walker, “SoftwareDefined Networks: Change you can believe in,” In Hotnets, 2011 [6] “OpenFlow Switch Specification v1.1.0,” 2008 Available: http://www.openflow.org/wp/documents/ [7] Open Networking Foundation https://www.opennetworking.org/ [8] “OpenFlow Switch Specification v1.2,” Open Networking Foundation, December 2011 Available: https://www.opennetworking.org/images/stories/downloads/specification/open flowspecv1.2pdf [9] “OpenFlow Switch Specification v1.3.0,” Open Networking Foundation, 2012 Available: https://www.opennetworking.org/images/stories/downloads/specification/open flow-spec-v1.3.0.pdf )#! ! [10] Amin Tootoonchian, Yashar Ganjali, “HyperFlow: A Distributed Control Plane for OpenFlow,” In Proceedings of NSDI Internet Network Management Workshop/Workshop on Research on Enterprise Networking (INM/WREN), San Jose, CA, April 2010 [11] T Koponen, M Casado, N Gude, J Stribling, P L., M Zhu, R Ramanathan, Y Iwata, H Inouye, T Hama, and S Shenker, “Onix: A distributed control platform for large-scale production networks,” In Operating Systems Design and Implementation USENIX, 2010 [12] N.Gude, T Koponen, J Pettit, B Plaff, M Casado, and N McKeown, “Nox: Towards an operating system for networks,” In ACM SIGCOMM CCR: Editorial note, July 2008 [13] “Beacon: Java-based OpenFlow Control Platform,” 2012 [Online] Available: http://www.beaconcontroller.net/ [14] “Floodlight OpenFlow Control Platform,” 2012 [Online] Available: http://floodlight.openflowhub.org/ [15] “Trema OpenFlow Control Platform,” 2012 [Online] Available: http://trema.github.com/trema/ [16] Bob Lantz, Brandon Heller, Nick McKeown A Network in a Laptop: Rapid Prototyping for Software-Defined Networks In Hotnets, pages 1-6, 2010 [17] Richard Wang, Dana Butnariu, Jennifer Rexford OpenFlow-based Server load-balancing gone wild In Hot-ICE, Mar 2011 [18] “Dijkstra’s shortest path algorithm,” 2012 [Online] Available: http://en.wikipedia.org/wiki/Dijkstra's_algorithm [19] Rodrigo Braga, Edjard Mota, Alexandre Passito, “Lightweight DDoS Flooding Attack Detection Using NOX/OpenFlow,” In 35th Annual IEEE Conference on Local Computer Networks, Colorado, USA, 2010 )$! ! [20] N Foster, R Harrision, M J Freedman, C Monsanto, J Rexford, A Story, and D Walker, “Frenetic: a network programming language,” In ICFP, Japan, September 2011 [21] Hideyuki Shimonishi, Shuji Ishii, Lei Sun, Yoshihiko Kanaumi, “Architecture, Implementation, and Experiments of Programmable Network Using OpenFlow”, IEICE Trans Commun., vol E94-B, no 10, pp 27152722, October 2010 [22] Teemu Koponen, Martin Casado, Natasha Gude, Jeremy Stribling, Leon Poutievski, Min Zhu, “Onix: A Distributed Control Platform for Large-scale Production Networks”, Nicira Network-Google-NEC-International Computer Science Institute (ICSI), 2011 [23] Zheng Cai Alan L Cox T S Eugene Ng, “Maestro: A System for Scalable OpenFlow Control”, Department of Computer Science, Rice University, USA, 2010 [24] M Yu, J Rexford, M J Freedman, and J Wang, “Scalable Flow-based networking with DIFANE,” In Proc SIGCOMM, Aug 2010 [25] “Controller performance comparison,” 2012 [Online] Available: http://www openflow.org/wk/index.php/Controller_Performance_Comparisons [26] “Link Layer Discovery Protocol,” 2012 [Online] Available: http://en.wikipedia.org/wiki/Link_Layer_Discovery_Protocol [27] “Cbench: controller benchmarker,” 2012 [Online] Available: http://www.openflow.org/wk/index.php/Oflops#Current_Benchmarks [28] “Ping (networking utility),” 2012 [Online] http://en.wikipedia.org/wiki/Ping_(networking_utility) Available: [29] “Iperf – measuring network performance in an enterprise environment,” 2012 [Online] Available: http://jaysonbroughton.com/2011/04/iperf-measuring- network-performance-in-an-enterprise-environment/ )%! ! PHU LUC: CH!"NG TRÌNH TRIVN KHAI MWNG OPENFLOW GIG LTP TRONG THX NGHI$M #!/usr/bin/python from mininet.net import Mininet from mininet.node import Controller, OVSKernelSwitch, RemoteController from mininet.cli import CLI from mininet.log import setLogLevel Switch = OVSKernelSwitch def addHost( net, N): “Create host hN and add to net.” name = ‘h%d’ % N ip = ’10.0.0.%d’ % N return net.addHost( name, ip = ip) def multiControllerNet(): “Create a network with multiple controllers.” net = Mininet(controller = Controller, switch = Switch) print “***Creating controllers” c1 = net.addController(‘c1’, defaultIP = ‘192.168.1.11’, port = 6633) c2 = net.addController(‘c2’, defaultIP = ‘192.168.1.14’, port = 6634) c3 = net.addController(‘c3’, defaultIP = ‘192.168.1.13’, port = 6635) print “***Creating switches” s1 = net.addSwitch(‘s1’) s2 = net.addSwitch(‘s2’) s3 = net.addSwitch(‘s3’) s4 = net.addSwitch(‘s4’) s5 = net.addSwitch(‘s5’) s6 = net.addSwitch(‘s6’) s7 = net.addSwitch(‘s7’) s8 = net.addSwitch(‘s8’) s9 = net.addSwitch(‘s9’) s10 = net.addSwitch(‘s10’) s11 = net.addSwitch(‘s11’) s12 = net.addSwitch(‘s12’) s13 = net.addSwitch(‘s13’) s14 = net.addSwitch(‘s14’) print “*** Creating hosts” hosts1 = [addHost(net,n) for n in 1,2] host2 = [addHost(net,n) for n in 3,4] hosts3 = [addHost(net,n) for n in 5,6] host4 = [addHost(net,n) for n in 7,8] hosts5 = [addHost(net,n) for n in 9,10] host6 = [addHost(net,n) for n in 11,12] )&! ! hosts7 = [addHost(net,n) for n in 13,14] host8 = [addHost(net,n) for n in 15,16] hosts9 = [addHost(net,n) for n in 17,18] host10 = [addHost(net,n) for n in 19,20] hosts11 = [addHost(net,n) for n in 21,22] host12 = [addHost(net,n) for n in 23,24] hosts12 = [addHost(net,n) for n in 25,26] host14 = [addHost(net,n) for n in 27,28] print “***Creating links” for h in hosts1: s1.linkTo(h) for h in hosts2: s2.linkTo(h) for h in hosts3: s3.linkTo(h) for h in hosts4: s4.linkTo(h) for h in hosts5: s5.linkTo(h) for h in hosts6: s6.linkTo(h) for h in hosts7: s7.linkTo(h) for h in hosts8: s8.linkTo(h) for h in hosts9: s9.linkTo(h) for h in hosts10: s10.linkTo(h) for h in hosts11: s11.linkTo(h) for h in hosts12: s12.linkTo(h) for h in hosts13: s13.linkTo(h) for h in hosts14: s14.linkTo(h) s1.linkTo(s2) s1.linkTo(s4) s4.linkTo(s3) s1.linkTo(s5) s5.linkTo(s9) s9.linkTo(s7) s9.linkTo(s6) s5.linkTo(s8) s10.linkTo(s1) s10.linkTo(s11) s10.linkTo(s13) s13.linkTo(s14) s13.linkTo(s12) print “***Starting network” net.build() c1.start() c2.start() c3.start() s1.start([c1]) s2.start([c1] s3.start([c1]) s4.start([c1]) s5.start([c2]) s6.start([c2] s7.start([c2]) s8.start([c2]) s9.start([c2]) s10.start([c3]) s11.start([c3]) s12.start([c3] s13.start([c3]) s14.start([c3]) print “***Running CLI” CLI(net) print “Stopping network” net.stop() if name == ‘ main ’: setLogLevel(‘info’) # for CLI output multiControllerNet() A collaborative model for routing in multi-domains OpenFlow networks Xuan Thien Phan, Nam Thoai Pierre Kuonen Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology Ho Chi Minh city, Vietnam XuanThien.Phan@edu.hefr.ch nam@cse.hcmut.edu.vn Institute of Information and Communication Technologies University of Applied Sciences of Western Switzerland of Fribourg Fribourg, Switzerland Pierre.Kuonen@hefr.ch Abstract—OpenFlow is an innovative network architecture which decouples control plane and data plane, allowing researchers the ability to program their networks, control the behaviour of network switches for their experiments However, current OpenFlow architecture relies completely on the use of a centralized controller to manage all switches connecting to it in the network For networks with large number of switches, depending on a controller to manage all switches in the network becomes unfeasible Thus allowing distributed multiple controllers in managing OpenFlow network is an appropriate solution for OpenFlow scalability Current OpenFlow support allows deploying multi-controllers networks in which each controller is responsible for operating each smaller domain in the network, however there is currently no mechanism for these controllers to cooperate with the each other As a result, the controllers cannot effectively process cross-domain packets This paper proposes building a collaborative model for multi-domains OpenFlow networks that allows different network domains to collaborate with each other in an efficient way In addition, a routing solution is proposed based on the model to achieve higher network performance Keywords—OpenFlow; Routing; Scalability Software-defined Networking; I INTRODUCTION Network infrastructure today mostly relies on devices like switches, routers, etc and these platforms are closed by equipment vendors The difficulty in making changes in these closed platforms is the huge obstacle for researchers’ new ideas and network innovations to be tested with sufficiently realistic settings to gain the confidence needed for their widespread deployment Thus this inhibits network development Toward the goal to achieve an open and standard platform, Software-Defined Network (SDN) [4,5] has been proposed enabling researchers the way to run their experiments in production networks In SDN, the data plane and the control plane of the networks are decoupled allowing network applications and services to be directly programmed in a remote controller to control the behaviour of network devices through a well-defined communication interface called OpenFlow [1,2,6,7,8] Some controller platforms have been developed for writing controller applications in different programming 978-1-4673-2088-7/13/$31.00 ©2013 IEEE languages, such as NOX [12], Beacon [13], Onix [11], Floodlight [14], and Trema [15] Further to this, OpenFlow platform is also used to solve a variety of networking problems such as web server load-balancing [17], DDoS flooding attack detection [19], and a network programming language [20] OpenFlow is currently supported by a number of network equipment vendors including hardware (switches, routers, access points) and software (virtual switches) as well as deployment tools like Mininet [16] However, current OpenFlow solutions not support multiple controllers in an OpenFlow network and this can be the lack of scalability In addition, the request processing capability of a single controller is limited [21], thus the response time for requests of large network environments becomes slow due to the latency of control bandwidth The RMOF is proposed for solving the problem This model supports multiple controllers for managing large-scale OpenFlow network concurrently In RMOF, controllers are placed among subdomains of the network and each of them is responsible for operating the subdomain and intra-domain routing effectively The contribution of RMOF is a mechanism that allows controllers associate together and share their management information The current implementation of OpenFlow only supports the assignment switches on local networks In other words, each controller has no information about the other switches existing in the other networks For example, when a controller sends packets to a switch which belongs to another network, it does not know necessary information for providing an optimal routing path In this case, controller mostly applies flooding mechanism to route those packets, and this is certainly not a good routing solution In RMOF model, together with the information of communication channels for controllers, data and applications are also used to calculate routing paths over multiple network domains The routing method can result in a highly efficient network since it can compute most reasonable routes for packets travelling over multiple domains The remainder of the paper is organized as follows Section II presents basis concepts of OpenFlow and provides an overview of related works Section III explains the proposed RMOF model, its elements and applications as well as how they work to solve the problem of routing across multiple network domains In section IV, we give a conclusion of the paper and our plan for future work II BACKGROUND AND RELATED WORKS This section briefly presents the main features of OpenFlow and the related works concerning our research A more detailed description of OpenFlow and OpenFlow Switch specification can be found at [1,7,9] An OpenFlow network includes forwarding plane and control plane The forwarding plane consists of OpenFlow switches (OF switches) that are responsible for directly forward packets in the network in per-flow basis, and the control plane is a remote controller which is responsible for managing OpenFlow switches in the network Figure Architecture of an OpenFlow network Each OpenFlow switch contains one or more Flow Tables, which perform packet look-ups and forwarding, and a Secure Channel that connects the switch to the controller (fig 1) This secure channel is the communication channel which the controller uses to operate all switches in the network via a standard OpenFlow protocol Each table contains Flow Entries which are used as rules to process matching packets The controller can add, update or delete flow entries in the flow table and this is the way it controls flow traffic in the network Each flow entry mainly consists of match fields which define the flow, instructions showing what to with matching packets, and counters for statistical purpose When the OpenFlow switch receives a packet, it will look-up its flow tables to find matching flow entry If matching entry is found, the switch applies instructions defined in the entry to process the packet and update counters If there is no matching entry, it sends the packet to the controller The controller may add a new flow entry into the switch’s flow table with instructions instructing it how to process the packet, or simply drop the packet This mechanism allows the controller to manage OpenFlow switches and control traffic in the OpenFlow network There are some approaches concerning the concept of multiple controllers in OpenFlow network have been proposed recently FlowVisor [3] allows multiple controllers in an OpenFlow network but its mechanism is to slice the network into separate slices and let each slice managed by a controller These controllers are completely independent, each one does not affect the others and the goal of FlowVisor is to allow multiple researchers use the same network resources HyperFlow [10] has another approach It raises the idea that enables interconnecting independently managed OpenFlow networks HyperFlow uses multiple controllers to manage the networks and provides a publish/subscribe system which the controllers use to update the network state and send commands to the other controllers However, this publish/subscribe system relies on the use of a distributed file system, and the events or commands are exchanged between the controllers and this system by a mechanism of reading or writing files in the file system This seems to be an unreliable mechanism for exchanging networking information and the proposed design is just a general idea and does not solve any specific networking problem Moreover, HyperFlow does not support routing over its multiple subnetworks and this can appreciably reduce to the performance of the whole network In the newest version 1.3 of OpenFlow switch specification [9], the concept of multiple controllers is presented According to its idea, each OpenFlow switch can establish the connection with a single controller or with multiple controllers With multiple controllers, the reliability is improved because the switches can continue to process packets in OpenFlow mode if they lose the connection with one controller This idea allows multiple controllers work concurrently in an OpenFlow network but just the primary one is responsible for directly managing the all switches of the network, the others are almost used for recovery purpose to improve the reliability of the network All of the controllers are aimed to take care of the same network and there is still no mechanism that helps the controllers share their managing information, or collaborate all together to operate the network III RMOF MODEL In our model, OpenFlow switches are used as forwarding plane of the network These switches are managed by controllers which are distributed in the network Each switch connects to a nearest controller and is directly operated by that controller Each controller together with switches connecting to it forms a smaller OpenFlow network We note that each of these smaller networks is considered as a network domain in the whole network including all controllers and switches in RMOF network (fig 2) The controllers in RMOF are deployed using similar control platforms and networking applications to make sure that all switches will be operated by similar networking functionalities and services even when they are in different domains In RMOF model, a RMOF collaborator running on an independent computer is used as a communication channel for controllers from different network domains to cooperate with each other The RMOF collaborator maintains a global view of the whole network This global view is the information contributed by controllers of all domains in the network The RMOF collaborator processes the collected information and provides helpful instructions or networking services for controllers By letting controllers independent in operating their local networks, our model prevents the dependence of controllers on the RMOF collaborator Hence it nearly minimizes the frequency of the requests between controllers and the RMOF collaborator and just requires lightweight processing in the RMOF collaborator at a global level Our design allows new network functionalities and services to be easily implemented and deployed to deal with a variety of networking problems which require the collaboration of multiple networks Hence it is a promised solution for OpenFlow scalability information contributed by controllers Furthermore, there is a RMOF global routing application (RMOF Global Routing App) which computes routing paths for packets travelling across multiple domains The routing application (RMOF Local Routing App) is added in each controller as a supplement for its existing routing application to process cross-domain packets, and a Route-Ins Table is installed This table is the place where the controller can look-up to find correspondent routing instructions to route cross-domain packets Figure General design of RMOF model The greatest obstacle in our approach is the problem of routing over multiple network domains The routing mechanism of current OpenFlow implementations does not consider solving cross-domain packet problem in its development goal Thus, the mechanism becomes inefficient when routing packets to or processing received packets from other network domains In case a packet arriving an OpenFlow switch, the switch look-ups its flow tables to find a matching entry If a matching flow entry is found, the switch processes the packet based on the entry If there is no matching entry, it sends a Packet-In message to the controller to ask for route When receiving a Packet-In message from an OpenFlow switch, the controller routes the waiting packet in the switch by computing a routing path and adding flow entries to switches based on this path to form a route for the packet and subsequent ones [9] In a large-scale network environment, each controller of a domain can only control routing within its network range and simply process crossdomain packets by flooding mechanism In this case, the controller simply instructs the switch to flood the crossdomain packets to all of its ports except the incoming one This process will be repeated for the neighbors of that switch and so on until the packets reach their destinations or dropped Obviously the flooding mechanism is not a good way to process packets travelling across multiple domains, thus we build a routing solution for the RMOF model based on the above general design to ensue all domains in the whole network are operated in a reasonable way with effective performance The solution provides routing instructions for the controllers to help them make better routing decisions Additional data and elements that are added to the RMOF as well as the method to process crossdomain packets will be presented in the following subsections A Main components of RMOF model In RMOF model, the communication channels (called RMOF channels) among controllers and the RMOF collaborator are established using a simple protocol designing for specific information exchange between them (fig 3) The RMOF collaborator maintains a global topology view of the whole network that is built on collected topology Figure Main components of RMOF model 1) The Global Topology View The Global Topology View holds the information of all cross-domain links in the network and the estimated routingcosts between them contained in the two following tables: a) Cross-domain Link Table Figure Fields of a Cross-domain Link Entry A Cross-Domain Link is a link between two OpenFlow switches which belongs to two different network domains The Cross-domain Link Table contains the information of all cross-domain links of the network Each entry of this table contains the information representing for a cross-domain link (fig 4), including these fields: • Src Controller ID: The ID of the controller which manages the source switch of the cross-domain link • Src Switch ID, Src Port: The ID and port of the source switch of the cross-domain link • Dst Switch ID, Dst Port, Dst Controller ID: The switch ID, port, and controller ID of the destination switch of the cross-domain link Notice that the role of the source switch and the destination switch of a cross-domain link are equal, so each cross-domain link will be presented by only one entry in the table b) Estimated Routing Cost Table Figure Fields of an Estimated Routing Cost Entry An estimated routing cost is the cost of the shortest routing path between a pair of cross-domain OpenFlow switches in a network domain This routing cost is the number of direct network links (or by the number of switches – 1) of the shortest path between those cross-domain switches which is calculated by the controller of that domain The Estimated Routing Cost Table stores the routing costs of all pairs of cross-domain switches of all domains in the network Each entry in this table presents for an estimated routing cost between pairs of cross-domain switches Figure is an illustration of an entry in the estimated-routing-cost table, including following fields: • Controller ID: The ID of the controller which manages the network domain • Src Switch ID, Dst Switch ID: The IDs of a pair of cross-domain switches in the domain managed by the above controller • Estimated routing cost: The routing cost between the above pair of cross-domain switches 2) Route-Ins Table In RMOF, each controller holds a Route-Ins Table It is a learning table which contains routing instructions for the controller to route cross-domain packets in an efficient way Each entry in this table supplies information of a local part of the routing path computed in a global level by the RMOF collaborator (fig 7), including following fields: • • • • • • • Local Src Switch ID: The ID of the source switch in the local domain Terminal Dst Address: The packet’s destination address Terminal Dst Switch ID: The ID of the destination switch Terminal Dst Controller ID: The ID of the destination domain’s controller Local Dst Switch ID: The ID of the cross-domain switch in the current domain that the controller needs to route the packet to Local Dst Out-Port: The outgoing port number of the above Local Dst Switch, it belongs to a cross-domain link where packets will be forwarded through to the next domain Estimated Routing-cost: The total estimated routingcost of the route from the local source switch of current domain to packet’s destination address Figure Fields of a Route-Ins Entry Each entry in this table instructs the controller to route destination-matching packets to the local switch whose ID is Local Dst Switch ID, and then forward packets out of the port Local Dst Outgoing Port of this switch to the next network domain Similar processes happen in next network domains, and by this mechanism packets are forwarded through domains to their desired destination k Cross-domain OF switch Cross-domain network link Computed path between each pair of cross-domain OF switches and its estimated routing-cost (k) Figure An image of the Global Topology View seen from RMOF collaborator Each controller in the network is responsible for sending switch IDs of all cross-domain switches in its domain and computed routing-costs between these switches to the RMOF collaborator, where this information will be processed together with the information from the other network domains to form the Global Topology View This view is like a ‘weighted graph’ in which nodes are cross-domain switches, edges are computed paths between those switches and weights are estimated routing-costs between them (fig 6) The Route-Ins Table is the key factor to improve routing over multiple network domains since it helps controllers routing cross-domain packets via efficient routes that are computed by the RMOF collaborator instead of flooding them In case the controller does not find any entry in the table that matches the packet’s destination, it can request the RMOF collaborator for routing instruction The routing instruction it receives from the RMOF collaborator then will be added to the Route-Ins Table for subsequent use 3) RMOF Local Routing and RMOF Global Routing applications The RMOF Local Routing application placing at the controller side is responsible for discovering parts of crossdomain links in its domain and computing routing-costs between them in order to report to the RMOF collaborator It also takes on adding or deleting entries in Route-Ins Table, looking up in Route-Ins Table for matching entries, and computing local routes for cross-domain packets which is based on matching routing instructions, as well as contacting the RMOF collaborator for routing instructions when it does not find any matching Route-Ins entry The RMOF Global Routing application placing at the RMOF collaborator is responsible for building up the global topology view based on discrete topology information which is collected from controllers, and computing global routes for cross-domain packets These routing applications collaborate with each other to compute routes for cross-domain packets The algorithm used to compute routes can be a shortest path algorithm such as the Dijkstra’s shortest path algorithm [18] or any other depending on the network features B Discorvering cross-domain network links In this section, we explain how cross-domain network links are discovered based on the topology information shared by controllers In a network domain managed by a controller, after the connections between the controller and the OpenFlow switches were established successfully, each of these switches sends an LLDP packet to all of its neighbor switches This message contains the sending switch’s ID together with its outgoing port number When receiving a LLDP packet from a neighbor switch, the recipient encapsulates and sends it to the controller In the controller, the sending switch ID field in this LLDP message will be extracted and checked if it matches any known switch • • information to so because it knows all of switches in its domain as well as all the direct network links between them The ‘local view’ that the controller observes in its domain is like a weighted graph as showed in fig With this information, the controller can apply an algorithm to compute the path between switches in its domain In routing applications of the model, the Dijkstra’s Shortest Path Algorithm [18] was used to compute the shortest path between those switches D Computing shortest path routes and routing crossdomain packets In this section, we explain how the routing applications compute routes for cross-domain packets and route them When the controller in a network domain receives a PacketIn from one of OpenFlow switches it manages, first it extracts the packet’s header If the packet’s destination is inside the controller’s network domain, the controller processes it normally as specified in OpenFlow Switch specification v1.3 [9] (replies by Packet-Out, drops the packet, ) If it matches a known switch in the controller’s domain, this means there is a network link between known switches (with source switch ID and destination switch ID specified in the LLDP message) in its network domain, the controller will learn that link for future processing This is the current processing of OpenFlow controller to discover all network links between all pairs of OpenFlow switches in its domain [13] If the source Switch ID of a LLDP message does not match any known switch in that domain, the current OpenFlow controller does not have any further processing in this situation In RMOF model, for the above second case, the controller sends this information to the RMOF collaborator The information contains fields of the LLDP message it received from the local switch: a destination switch ID, destination incoming port number (of the known switch in its domain) and the source switch ID, source outgoing port number (of a switch somewhere in the world outside the controller’s domain) When receiving that information from the controller, together with the other information receiving from the other controllers, the RMOF collaborator will combine them to build up the Cross-Domain Link Table C Computing routing-costs between cross-domain switches in a network domain This section explains how controllers compute routingcosts between cross-domain switches in their local network domains When an OpenFlow switch in a network domain receives a LLDP message from an unknown OF Switch outside its domain which the local controller has no idea about (this internal OpenFlow switch in this current domain is called ‘cross-domain switch’), before sending this network link information to the RMOF collaborator, the controller computes the routing-costs from this cross-domain switch to the other known cross-domain switches in its domain This information will be encapsulated and sent to the RMOF collaborator to help it build the estimated-routing-cost table A controller which manages a network domain has enough Normal OF switch Computed path between a normal OF switch and a cross-domain switch in a local network domain and its estimated routing-cost (k) Routing direction Figure An example of global route computing in RMOF collaborator If the controller has no information about the packet’s destination (in case of cross-domain packets where their destinations belong to other network domains outside the current controller’s domain), it passes the packet to the RMOF Local Routing application This application will lookup in the Route-Ins Table for destination-matching entry If an entry found, the application will route the packet based on the information in this entry In case no match is found in the Route-Ins Table, the application contacts the RMOF collaborator to ask for routing-instruction First, the controller computes routing-costs between the OF source switch (switch S x in fig 8) to all of the crossdomain switches in its domain, and then send them together with the information of packet’s destination to the RMOF collaborator In RMOF collaborator, the RMOF Global ! locates the packet’s destination by Routing application asking the other controllers except the one that it received routing-instruction request from Replies from controllers help it know where packet’s destination is (switch S y in fig ! 8) and also the estimated routing-costs from that destination to cross-domain switches in destination domain (computed by the controller who manages it) Now the RMOF Global Routing application has enough information to compute a global route for the packet What it observes is like a weighted graph in which nodes are switches (including all cross-domain switches, the normal source switch and destination switch), edges are computed paths between them, and weights are estimated routing-costs of these paths (fig 8) Based on that information the application computes an efficient global route for the packet, for example the route S x – S1 – S – S – S – S – S y in fig This global route can be the shortest one if a shortest path algorithm is applied for route computing ! Routing !instructions ! ! !will ! be !sent! to correspondent controllers along the computed route, and those instructions will help them route the packet and subsequent destinationmatching packets in an efficient way For example in fig 8, packets will be routed from switch S x to switch S1 in the source network domain D1, then forwarded through the crossdomain link S1 S to switch S of network domain D Similar routing processes happen in the network domain D , ! ! D until packets reach switch network S of destination ! domain D Here the controller in D knows how to route ! ! ! ! S packets from to S y and forward out the destination host ! We note that controllers route packets by directly adding ! flow entries to flow tables of OpenFlow switches along the !computed global routes Moreover, ! these flow entries are ! ! added concurrently in all correspondent network domains, thus packets will be forwarded directly from the source host to the destination host without waiting in the intermediate network domains IV CONCLUSION AND FUTURE WORK In this paper, we proposed RMOF model that allows multiple controllers in different OpenFlow subdomains of a network associate with each other, and a solution for routing cross-domain packets based on the model By letting controllers independently manage their local networks, RMOF keeps all advantages of OpenFlow architecture while providing a mechanism allowing them to exchange necessary information to make the networks operate more efficiently Our design is a promising solution for OpenFlow scalability since it enables control planes of multiple OpenFlow networks the ability to collaborate together to operate networks as a whole one It’s also reliable because the communication between controllers and the RMOF collaborator bases on the use of a secure channel Our model can be used to improve networking functionalities due to the global view maintained as an observation over all networks The routing solution we presented is a case study for the applicability of our model The global view holds topology information about cross-network links together with estimated routing-costs between cross-network switches therefore routing applications have enough information to calculate routes for cross-domain packets Our solution appreciably improves the functionality of routing over multiple networks because routes are efficient ones due to the global topology information obtained in the global topology view and the use of an effective path-computing algorithm We are in working progress to provide a complete implementation for our design as well as run experiments in a large testbed to evaluate its efficiency, performance and scalability In the future we also plan to extend the model to deal with the other networking problems that need the collaboration of multiple controllers to solve such as loadbalance over multiple networks The direction is to extend the global topology view, allowing controllers to share information concerning their networks more completely, including other information such as the traffic load of partner networks, instead of just topology information as currently REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] N McKeown, T Anderson, H Balakrishnan, G Parulkar, L Peterson, J Rexford, S Shenker, and J Turner, “Openflow: enabling innovation in campus networks,” In SIGCOMM Comput Commun Rev., vol 38, no 2, pp 69–74, 2008 “Intro to OpenFlow,” 2011 [Online] Available: https://www.opennetworking.org/standards/intro-to-openflow Rob Sherwood, Glen Gibb, Kok-Kiong Yap, Guido Appenzeller, Martin Casado, Nick McKeown, Guru Parulkar, “Flow Visor: a network virtualization layer,” Deutsche Telekom Inc R&D Lab, Stanford University, Nicira Networks, 2009 “Software-Defined Networking: the new norm for networks,” Open Networking Foundation, April 2012 Mark Reitblatt, Nate Foster, Jennifer Rexford, David Walker, “Software-Defined Networks: change you can believe in,” In Hotnets, 2011 “OpenFlow switch specification v1.1.0,” 2008 Available: http://www.openflow.org/wp/documents/ Open Networking Foundation https://www.opennetworking.org/ “OpenFlow switch specification v1.2,” Open Networking Foundation, December 2011 Available: https://www.opennetworking.org/images/stories/downloads/specificat ion/openflowspecv1.2pdf “OpenFlow switch specification v1.3.0,” Open Networking Foundation, 2012 Available: https://www.opennetworking.org/images/stories/downloads/specificat ion/openflow-spec-v1.3.0.pdf Amin Tootoonchian, Yashar Ganjali, “HyperFlow: a distributed control plane for OpenFlow,” In Proceedings of NSDI Internet Network Management Workshop/Workshop on Research on Enterprise Networking (INM/WREN), San Jose, CA, April 2010 T Koponen, M Casado, N Gude, J Stribling, P L., M Zhu, R Ramanathan, Y Iwata, H Inouye, T Hama, and S Shenker, “Onix: a distributed control platform for large-scale production networks,” In Operating Systems Design and Implementation USENIX, 2010 N.Gude, T Koponen, J Pettit, B Plaff, M Casado, and N McKeown, “Nox: towards an operating system for networks,” In ACM SIGCOMM CCR: Editorial note, July 2008 “Beacon: java-based OpenFlow control platform,” 2012 [Online] Available: http://www.beaconcontroller.net/ “Floodlight OpenFlow control platform,” 2012 [Online] Available: http://floodlight.openflowhub.org/ “Trema OpenFlow control platform,” 2012 [Online] Available: http://trema.github.com/trema/ Bob Lantz, Brandon Heller, Nick McKeown, “A network in a laptop: rapid prototyping for Software-Defined Networks,” In Hotnets, pages 1-6, 2010 Richard Wang, Dana Butnariu, Jennifer Rexford, “OpenFlow-based server load-balancing gone wild,” In Hot-ICE, Mar 2011 “Dijkstra’s shortest path algorithm,” 2012 [Online] Available: http://en.wikipedia.org/wiki/Dijkstra's_algorithm Rodrigo Braga, Edjard Mota, Alexandre Passito, “Lightweight DDoS flooding attack detection using NOX/OpenFlow,” In 35th Annual IEEE Conference on Local Computer Networks, Colorado, USA, 2010 N Foster, R Harrision, M J Freedman, C Monsanto, J Rexford, A Story, and D Walker, “Frenetic: a network programming language,” In ICFP, Japan, September 2011 Pingping Lin, Jun Bi, Hongyu Hu, “ASIC: an architectur for scalable intra-domain control in OpenFlow,” In CFI, Korea, September 2012 PH!N L" L#CH TRÍCH NGANG H! tên: Phan Xuân Thi"n Ngày, tháng, n#m sinh: 03/3/1987 N$i sinh: Th%a Thiên Hu& '(a ch) liên l*c: 6/13 Hàn M*c T+, P V, D*, TP Hu&, T)nh T-T-Hu& QUÁ TRÌNH $ÀO T%O H!c '*i h!c t*i Tr-.ng '*i H!c Bách Khoa Tp HCM, Khoa Khoa H!c K/ Thu0t Máy Tính (t% n#m 2005 1&n n#m 2010) H!c ch-$ng trình Th*c S, t*i Tr-.ng '*i H!c Bách Khoa Tp HCM, Khoa Khoa H!c K/ Thu0t Máy Tính (t% n#m 2010 1&n n#m 2012) ... quy"t v)n %( toàn cDc Trên c= s+ c/a mơ hình này, tơi %( xu)t gi5i pháp hoàn ch