Python Network Programming Cookbook Second Edition Overcome real-world networking challenges Pradeeban Kathiravelu Dr M O Faruque Sarker BIRMINGHAM - MUMBAI Python Network Programming Cookbook Second Edition Copyright © 2017 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews Every effort has been made in the preparation of this book to ensure the accuracy of the information presented However, the information contained in this book is sold without warranty, either express or implied Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals However, Packt Publishing cannot guarantee the accuracy of this information First published: March 2014 Second edition: August 2017 Production reference: 1080817 Published by Packt Publishing Ltd Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78646-399-9 www.packtpub.com How to it As you have installed ldns, now you may check out the Dyn's dns_lg source code from its source code repository: git clone git@github.com:dyninc/dns_lg.git cd dns_lg/ Now, you may run the application simply by executing api.py Running it produces the following output: $ python api.py * Running on http://0.0.0.0:8185/ (Press CTRL+C to quit) Now open another console window to run a curl: curl http://0.0.0.0:8185/cnn.com/ This will output a line to the preceding api.py console: 127.0.0.1 - - [15/Jul/2017 23:33:40] "GET /cnn.com/ HTTP/1.1" 200 - The curl command produces the following output with the detailed DNS information from the looking glass implementation: $ curl http://0.0.0.0:8185/cnn.com/ { "AdditionalSection": [ { "Address": "205.251.192.47", "Class": "IN", "Name": "ns-47.awsdns-05.com.", "TTL": "20545", "Type": "A" }, { "Address": "205.251.194.64", "Class": "IN", "Name": "ns-576.awsdns-08.net.", "TTL": "20545", "Type": "A" } ], "AnswerSection": [], "AuthoritySection": [ { "Class": "IN", "Name": "cnn.com.", "TTL": "20545", "Target": "ns-47.awsdns-05.com.", "Type": "NS" }, { "Class": "IN", "Name": "cnn.com.", "TTL": "20545", "Target": "ns-1086.awsdns-07.org.", "Type": "NS" }, { "Class": "IN", "Name": "cnn.com.", "TTL": "20545", "Target": "ns-576.awsdns-08.net.", "Type": "NS" }, { "Class": "IN", "Name": "cnn.com.", "TTL": "20545", "Target": "ns-1630.awsdns-11.co.uk.", "Type": "NS" } ], "None": "true", "Query": { "Duration": 118, "Server": "", "ServerIP": "127.0.1.1", "Versions": "Dyn DNS Looking Glass 1.0.0" }, "QuestionSection": { "Qclass": "IN", "Qname": "cnn.com.", "Qtype": "A" }, "ReturnCode": "NOERROR" } The preceding output shows Dyn's dns_lg and curl in action Please note that currently Dyn's dns_lg works only in Python 2.x However, with some minor fixes, this can easily be ported to Python 3.x As we ported bgperf to Python 3.x in a previous recipe, this is left as an exercise for those who like to port this to Python 3.x How it works Looking glass offers you an opportunity to see how you can connect to another part of the internet through the routers of any given IXP Similar to the functionality of traceroute, lg implementations show you the connectivity in the internet scale They are deployed by the IXPs to demonstrate the IXP performance to the potential customers Understanding the internet ecosystem with Python When network traffic is sent to the internet, it passes through various ASes and IXPs Tools such as traceroute and tcptraceroute can be used to trace how a particular network node in the internet can be accessed from your computer through your internet provider Various tools developed in Python can be used to understand the nature of the internet traIXroute (https://pypi.python.o rg/pypi/traixroute) is a tool developed on Python 3, which identifies the IXPs on the traceroute path Getting ready You may install traIXroute through pip: $ sudo pip install traixroute To measure the performance and topologies of the internet, you also need to install scamper (https://www.caida.org/tools/measurement/sc amper/), a parallel measurement utility for the internet: $ sudo scamper-install Your traIXroute is now ready to analyze the internet connectivity through the IXPs You may confirm your successful install by running the help command, which will produce the output as follows: $ traixroute help usage: traixroute [-h] [-dns] [-asn] [-db] [-rule] [-u] [-m] [-o OUTPUT] {probe,ripe,import} positional arguments: {probe,ripe,import} probe ripe import probe help ripe help import help optional arguments: -h, help -dns, enable-dns-print show this help message and exit Enables printing the domain name of each IP hop in the traceroute path -asn, enable-asn-print Enables printing the ASN of each IP hop in the traceroute path -db, store-database Enables printing the database information -rule, enable-rule-print Enables printing the hit IXP detection rule(s -u, update Updates the database with up-to-date datasets -m, merge Exports the database to distinct files, the -o OUTPUT, output OUTPUT Specifies the output file name to redirect th -v, version show program's version number and exit How to it Now you may run traixroute to see the IXPs in your path Running traixroute for the first time takes a few minutes, as it has to perform a few initialization actions, downloading the datasets: $ traixroute probe -dest cnn.com -s="-m 12" Dataset files are missing Updating the database Started downloading PDB dataset Started downloading PCH dataset Started downloading RouteViews dataset Routeviews has been updated successfully PDB dataset has been updated successfully PCH dataset has been updated successfully Database has been updated successfully Imported 13 IXP Detection Rules from /configuration/rules.txt Loading from PCH, PDB, Routeviews and additional_info.txt traIXroute using scamper with "-m 12" options [15:08:06:001] scamper_privsep_init: could not mkdir /var/empty: Permissi Scamper failed Trying to run with sudo [sudo] password for pradeeban: traIXroute to cnn.com (151.101.1.67) 1) (62.4.224.1) 15.465 ms 2) (91.183.241.176) 18.642 ms 3) (91.183.246.112) 12.178 ms 4) (62.115.40.97) 20.216 ms 5) (213.155.136.216) 20.027 ms 6) (80.91.253.163) 12.127 ms 7) (*) 8) (*) 9) (*) 10) (*) 11) (*) - This did not indicate any IXP in the path between my network and http://edition.cnn.com/ Let's try once more towards register.bg: $ sudo traixroute probe -dest register.bg -s="-m 12" Imported 13 IXP Detection Rules from /configuration/rules.txt Loading from Database traIXroute using scamper with "-m 12" options traIXroute to register.bg (192.92.129.35) 1) (62.4.224.1) 21.699 ms 2) (91.183.241.176) 7.769 ms 3) (91.183.246.114) 8.056 ms 4) (BNIX)->AS9002 (194.53.172.71) 7.417 ms 5) (87.245.234.130) 51.538 ms 6) (87.245.240.146) 51.324 ms 7) (178.132.82.126) 44.711 ms 8) (193.68.0.150) 46.406 ms 9) (193.68.0.181) 44.492 ms 10) (192.92.129.35) 44.777 ms IXP hops: 3) 91.183.246.114 < - BNIX (BE,Brussels) -> 4) 194.53.172.71 This shows that my request had an IXP hops (BNIX in Brussels) in between If you repeat the request, you may notice that the IXP hops most certainly remained the same while other hops may have changed You may repeat with other websites to see which IXPs that your network traffic passes through How it works The -m flag indicates the maximum time-to-live (TTL) between the hops The * in the output logs indicates failure to trace a node within the given TTL, as no response was received The -m flag dictates the maximum number of hops to be traced It can be a value between and 255, with producing just hop in between, where 255 produces up to 255 hops towards the end point However, note that it is unlikely to have such a long path in the internet, and if exists, it is even more unlikely to retrieve the exact IP addresses through traceroute or traixroute (you will more likely receive * for the latter hops) Establishing BGP connections with yabgp Yabgp is a Python implementation for BGP protocol that supports establishing BGP connections from various routers It can be used for various advanced use cases such as future analysis In this recipe, we will install yabgp using virtualenv virtual environment for Python programs Getting ready First, get the sources of yabgp: $ git clone https://github.com/smartbgp/yabgp Now to build yabgp: $ cd yabgp Install the requirements following this command, and observe the following logs: $ pip install -r requirements.txt Successfully installed Twisted Flask Flask-HTTPAuth netaddr zope.interfac Cleaning up Now you may confirm the correct installation of yabgpd by using the following command: $ cd bin $ python yabgpd -h This will output detailed help information on yabgpd How to it is a BGP agent that can orchestrate the BGP routers You may start the agent as a Python application Make sure to update the correct values for the BGP local and remote addresses, and the local and remote BGP autonomous system values The program will print a set of log lines as follows: yabgpd $ python yabgpd bgp-local_addr=172.31.0.232 bgp-local_as=23650 \ bgp-remote_addr=52.58.130.47 bgp-remote_as=23650 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] Log (Re)opened 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] Configuration: 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] ****************** 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] Configuration opti 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] command line args: 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] config files: [] 2017-07-16 16:19:05,837.837 78465 INFO yabgp.agent [-] ================== 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -remote_as = 236 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -remote_addr = 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -local_as = 2365 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -local_addr = 17 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -capability = {' 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -afi_safi = ['ip 2017-07-16 16:19:05,840.840 78465 INFO yabgp.agent [-] -md5 = None 2017-07-16 16:19:05,840.840 78465 INFO yabgp.handler.default_handler [-] 2017-07-16 16:19:05,840.840 78465 INFO yabgp.handler.default_handler [-] 2017-07-16 16:19:05,840.840 78465 INFO yabgp.handler.default_handler [-] 2017-07-16 16:19:05,840.840 78465 INFO yabgp.handler.default_handler [-] 2017-07-16 16:19:05,840.840 78465 INFO yabgp.handler.default_handler [-] 2017-07-16 16:19:05,841.841 78465 INFO yabgp.agent [-] Create BGPPeering 2017-07-16 16:19:05,841.841 78465 INFO yabgp.core.factory [-] Init BGPPee 2017-07-16 16:19:05,841.841 78465 INFO yabgp.agent [-] Prepare RESTAPI se 2017-07-16 16:19:05,842.842 78465 INFO yabgp.agent [-] serving RESTAPI on 2017-07-16 16:19:05,842.842 78465 INFO yabgp.agent [-] Starting BGPPeerin 2017-07-16 16:19:05,842.842 78465 INFO yabgp.core.fsm [-] Automatic start 2017-07-16 16:19:05,842.842 78465 INFO yabgp.core.fsm [-] Do not need Idl 2017-07-16 16:19:05,842.842 78465 INFO yabgp.core.fsm [-] Connect retry c 2017-07-16 16:19:05,843.843 78465 INFO yabgp.core.fsm [-] Connect retry t 2017-07-16 16:19:05,843.843 78465 INFO yabgp.core.fsm [-] [10.124.1.245]S 2017-07-16 16:19:05,843.843 78465 INFO yabgp.core.factory [-] (Re)connect As can be seen from the logs, the BGP message file is created in the folder, /home/ubuntu/data/bgp/10.124.1.245 By analyzing the logs, you may notice logs are stored from both the remote and local BGP addresses: $ tree /home/ubuntu/data/bgp /home/ubuntu/data/bgp ├── 10.124.1.245 │ └── msg │ └── 1500221945.84.msg └── 52.58.130.47 └── msg └── 1500221444.73.msg How it works is a BGP agent that can establish BGP connections with various routers The agent receives the BGP messages and is capable of using them for further uses, such as future analysis Running these applications require access to BGP routers to route traffic between the autonomous systems These recipes illustrate the capability of Python to build large-scale complex network applications in the internet scale yabgpd This book was downloaded from AvaxHome! Visit my blog for more new books: www.avxhm.se/blogs/AlenMiler ... Python Network Programming Cookbook Second Edition Overcome real-world networking challenges Pradeeban Kathiravelu Dr M O Faruque Sarker BIRMINGHAM - MUMBAI Python Network Programming Cookbook. .. few book chapters He has also worked on OpenDaylight Cookbook and Learning OpenDaylight as a technical reviewer Python Network Programming Cookbook, Second Edition (2017) is his first book as... and Tahoe-LAFS open source projects He is the author of Python Network Programming Cookbook and Learning Python Network Programming both by Packt Publishing He received his Ph.D in multi-robot