ATTACKING NETWORK PROTOCOLS A Hacker’s Guide to Capture, Analysis, and Exploitation by James Forshaw San Francisco ATTACKING NETWORK PROTOCOLS Copyright © 2018 by James Forshaw All rights reserved No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher ISBN-10: 1-59327-750-4 ISBN-13: 978-1-59327-750-5 Publisher: William Pollock Production Editor: Laurel Chun Cover Illustration: Garry Booth Interior Design: Octopod Studios Developmental Editors: Liz Chadwick and William Pollock Technical Reviewers: Cliff Janzen Additional Technical Reviewers: Arrigo Triulzi and Peter Gutmann Copyeditor: Anne Marie Walker Compositors: Laurel Chun and Meg Sneeringer Proofreader: Paula L Fleming Indexer: BIM Creatives, LLC For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc directly: No Starch Press, Inc 245 8th Street, San Francisco, CA 94103 phone: 1.415.863.9900; info@nostarch.com www.nostarch.com Library of Congress Control Number: 2017954429 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc Other product and company names mentioned herein may be the trademarks of their respective owners Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark The information in this book is distributed on an “As Is” basis, without warranty While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it About the Author James Forshaw is a renowned computer security researcher at Google Project Zero, with more than ten years of experience in analyzing and exploiting application network protocols His skills range from cracking game consoles to exposing complex design issues in operating systems, especially Microsoft Windows, which earned him the top bug bounty of $100,000 and placed him as the #1 researcher on Microsoft Security Response Center’s (MSRC) published list He’s the creator of the network protocol analysis tool, Canape, which was developed from his years of experience He’s been invited to present his novel security research at global security conferences such as BlackHat, CanSecWest and Chaos Computer Congress About the Technical Reviewer Since the early days of Commodore PET and VIC-20, technology has been a constant companion (and sometimes an obsession!) to Cliff Janzen Cliff discovered his career passion when he moved to information security in 2008 after a decade of IT operations Since then, Cliff has had the great fortune to work with and learn from some of the best people in the industry, including Mr Forshaw and the fine people at No Starch during the production of this book He is happily employed as a security consultant, doing everything from policy review to penetration tests He feels lucky to have a career that is also his favorite hobby and a wife who supports him BRIEF CONTENTS Foreword by Katie Moussouris Acknowledgments Introduction Chapter 1: The Basics of Networking Chapter 2: Capturing Application Traffic Chapter 3: Network Protocol Structures Chapter 4: Advanced Application Traffic Capture Chapter 5: Analysis from the Wire Chapter 6: Application Reverse Engineering Chapter 7: Network Protocol Security Chapter 8: Implementing the Network Protocol Chapter 9: The Root Causes of Vulnerabilities Chapter 10: Finding and Exploiting Security Vulnerabilities Appendix: Network Protocol Analysis Toolkit Index CONTENTS IN DETAIL FOREWORD by Katie Moussouris ACKNOWLEDGMENTS INTRODUCTION Why Read This Book? What’s in This Book? How to Use This Book Contact Me THE BASICS OF NETWORKING Network Architecture and Protocols The Internet Protocol Suite Data Encapsulation Headers, Footers, and Addresses Data Transmission Network Routing My Model for Network Protocol Analysis Final Words CAPTURING APPLICATION TRAFFIC Passive Network Traffic Capture Quick Primer for Wireshark Alternative Passive Capture Techniques System Call Tracing The strace Utility on Linux Monitoring Network Connections with DTrace Process Monitor on Windows Advantages and Disadvantages of Passive Capture Active Network Traffic Capture Network Proxies Port-Forwarding Proxy SOCKS Proxy HTTP Proxies Forwarding an HTTP Proxy Reverse HTTP Proxy Final Words NETWORK PROTOCOL STRUCTURES Binary Protocol Structures Numeric Data Booleans Bit Flags Binary Endian Text and Human-Readable Data Variable Binary Length Data Dates and Times POSIX/Unix Time Windows FILETIME Tag, Length, Value Pattern Multiplexing and Fragmentation Network Address Information Structured Binary Formats Text Protocol Structures Numeric Data Text Booleans Dates and Times Variable-Length Data Structured Text Formats Encoding Binary Data Hex Encoding Base64 Final Words ADVANCED APPLICATION TRAFFIC CAPTURE Rerouting Traffic Using Traceroute Routing Tables Configuring a Router Enabling Routing on Windows Enabling Routing on *nix Network Address Translation Enabling SNAT Configuring SNAT on Linux Enabling DNAT Forwarding Traffic to a Gateway DHCP Spoofing ARP Poisoning Final Words ANALYSIS FROM THE WIRE The Traffic-Producing Application: SuperFunkyChat Starting the Server Starting Clients Communicating Between Clients A Crash Course in Analysis with Wireshark Generating Network Traffic and Capturing Packets Basic Analysis Reading the Contents of a TCP Session Identifying Packet Structure with Hex Dump Viewing Individual Packets Determining the Protocol Structure Testing Our Assumptions Dissecting the Protocol with Python Developing Wireshark Dissectors in Lua Creating the Dissector The Lua Dissection Parsing a Message Packet Using a Proxy to Actively Analyze Traffic Setting Up the Proxy Protocol Analysis Using a Proxy Adding Basic Protocol Parsing Changing Protocol Behavior Final Words APPLICATION REVERSE ENGINEERING Compilers, Interpreters, and Assemblers Interpreted Languages Compiled Languages Static vs Dynamic Linking The x86 Architecture The Instruction Set Architecture CPU Registers Program Flow Operating System Basics Executable File Formats Sections Processes and Threads Operating System Networking Interface Application Binary Interface Static Reverse Engineering A Quick Guide to Using IDA Pro Free Edition Analyzing Stack Variables and Arguments Identifying Key Functionality Dynamic Reverse Engineering Setting Breakpoints Debugger Windows Where to Set Breakpoints? Reverse Engineering Managed Languages NET Applications Using ILSpy Java Applications Dealing with Obfuscation Reverse Engineering Resources Final Words NETWORK PROTOCOL SECURITY Encryption Algorithms Substitution Ciphers XOR Encryption Random Number Generators Symmetric Key Cryptography Block Ciphers Block Cipher Modes Block Cipher Padding HTTP, 29–35 man-in-the-middle, 20 passive method, 12–20 port-forwarding, 21–24 proxies, 20–35 SOCKS, 24–29 system call tracing, 14–19 capturing tools Dtrace, 17–18 Netcat, 180–182 Process Monitor tool, 18–19 strace, 16 generating, 83–84 outbound, 89 Transmission Control Protocol (TCP), 2–3, 21 bit flags, 41 client connection to server, 121–123 header, 5, 87 HTTP proxy, 30 packets, 87–88 port numbers, port-forwarding proxy, 21–22, 201 reading contents of sessions, 85–86 reverse shell, 265–266 SOCKS proxy, 24–28 stream, 13–14 transport layer, 3, 6, 8–10 Transport Layer Security (TLS) certificate pinning, 177 client certificate, 175 decryption, 201–202 encryption, 175–176, 200–201 endpoint authentication, 174–175 forcing TLS 1.2, 202 handshake, 172–173 initial negotiation, 173 perfect forward secrecy, 177 replacing certificate in, 202–206 security requirements, 176–177 TLS Record protocol, 172 trapdoor functions, 160 Triple DES, 151 true, 55 trusted root certification authorities, 204 Tshark, 180–182 TVB (testy virtual buffer), 99 Twofish, 152 two’s complement, 39 U UCS (Universal Character Set), 44–45 UDP See User Datagram Protocol (UDP) UI (user interface), uname command, 263–264 Unicode character encoding, 44–45 character mapping, 44–45 UCS-2/UTF-16, 45 UCS-4/UTF-32, 45 Unicode Transformation Format (UTF), 44–45 Unified Sniffing mode (Ettercap), 76 Uniform Request Identifier (URI), 30, 32 uninitialized data, 120 Universal Character Set (UCS), 44–45 Unix-like systems, ASLR implementation flaws in, 272 AT&T syntax, 116 command injection, 228 command line utilities on, 31 configuring DNAT on, 70 Dtrace, 16 enabling routing on, 67 error codes, 262 executable format, 120 hosts file, 23 read and write calls, 122 routing tables on, 65 system calls, 15–16, 122 traceroute, 64 Unk2 value, 93–95 unmanaged executables, 195–199 dynamic libraries, 195–196 unsafe keyword, 210 unsigned integers, 38 UPX, 134 URI (Uniform Request Identifier), 30, 32 User Datagram Protocol (UDP), captured traffic, 182–183 dissectors, 98–99 payload and header, port forwading, 21 socket, 122 user enumeration, 218–219 user interface (UI), user mode, 14 user-after-free vulnerability, 249–250 UTF (Unicode Transformation Format), 44–45 UTF-8, 45–46 V variable binary length data implicit-length data, 48–49 length-prefixed data, 48 padded data, 49 terminated data, 47–48 variable-length buffer overflows, 211, 213–214 variable-length data, 56 variable-length integers, 39–40 verbose errors, 221–222 Verisign, 170 virtual function table, 242, 248–249 virtual hosts, 24 virtual machine, 137 VirtualAlloc, 250 Visual C++, 129 vulnerabilities authentication checking, 226 classes authentication bypass, 209 authorization bypass, 209–210 denial-of-service, 208 information disclosure, 209 remote code execution, 208 command injection, 228 CPU exhaustion attacks algorithmic complexity, 224–225 configurable cryptography, 224–225 default or hardcoded credentials, 218 exploiting arbitrary writing of memory, 253–254 defined memory pool allocations, 252–253 heap layout manipulation, 249–250 heap memory storage, 253 high-privileged file writes, 254–256 low-privileged file writes, 255 memory corruption, 245–253 user-after-free vulnerability, 249–250 format string, 227 fuzz testing, 234–236 incorrect resource access canonicalization, 220–221 verbose errors, 221–222 memory corruption buffer overflows, 210–215 data expansion attack, 217 dynamic memory allocation failures, 217 exploit mitigations, 267–268 memory-safe vs memory-unsafe languages, 210 out-of-bounds buffer indexing, 216–217 memory exhaustion attacks, 222–223 shell code, 255–266 SQL injection, 228–229 storage exhaustion attacks, 223–224 text-encoding character replacement, 229–231 triaging, 236–245 user enumeration, 218–219 W W3C, 58 web application testing tools, 283–285 Burp Suite, 283–284 Mitmproxy, 284–285 Zed Attack Proxy, 284 web of trust (WOT), 169 wget, 31 windll, 199 Windows ASLR implementation flaws in, 272 calling functions with Python on, 199 certificate manager, 203 debug symbols, 129 debugger, 236–241, 244–245 dynamic link libraries, 196 enabling routing on, 67 FILETIME, 50 loading library on, 197 Page Heap, 244–245 registry, 67 Winsock library, 121 XP SP2, 270 WinDump, 278 WinPcap, 278 Winsock, 121 Wireshark, 12–14, 81, 279–280 basic analysis, 84–85 capture interfaces dialog, 82–83 Conversations window, 84–85 dissectors, 95–103 generating network traffic in, 83–84 Hex Dump view, 86–95 main window, 82 reading contents of TCP sessions in, 85–86 Tshark command line version, 180–182 WOT (web of trust), 169 write system call, 15, 18, 122, 261–263 WriteData() function, 108 WritePackets() method, 22 ws2_32.dll Windows network library, 130–131 X X.509 certificates, 53–54, 169–171, 173 X.680 series, 53 x86 architecture, 42, 125 history, 114 instruction mnemonics, 115 instruction set architecture, 114–116 mnemonic forms, 115 program flow, 118–119 registers, 116–118 xcalc, 228 XML Schema, 58 XOR encryption, 108–109, 148–149, 153–154 XOR instruction, 115 XOR parameter, 108–109 xp_cmdshell function, 229 xxd tool, 90, 181 Z Zed Attack Proxy (ZAP), 284 zero flag, 117 ZLib compression library, 132 RESOURCES Visit https://www.nostarch.com/networkprotocols/ for resources, errata, and more information More no-nonsense books from NO STARCH PRESS ROOTKITS AND BOOTKITS Reversing Modern Malware and Next Generation Threats by ALEX MATROSOV, EUGENE RODIONOV, and SERGEY BRATUS 2018, 504 PP., $49.95 ISBN 978-1-59327-716-1 SPRING SERIOUS CRYPTOGRAPHY A Practical Introduction to Modern Encryption by JEAN-PHILIPPE AUMASSON 2017, 312 PP., $49.95 ISBN 978-1-59327-826-7 NOVEMBER GRAY HAT C# A Hacker’s Guide to Creating and Automating Security Tools by BRANDON PERRY JUNE 2017, 304 PP., $39.95 ISBN 978-1-59327-759-8 PRACTICAL PACKET ANALYSIS, 3RD EDITION Using Wireshark to Solve Real-World Network Problems by CHRIS SANDERS 2017, 368 PP., $49.95 ISBN 978-1-59327-802-1 APRIL THE HARDWARE HACKER Adventures in Making and Breaking Hardware by ANDREW “BUNNIE” HUANG 2017, 416 PP., $29.95 ISBN 978-1-59327-758-1 hardcover MARCH BLACK HAT PYTHON Python Programming for Hackers and Pentesters by JUSTIN SEITZ 2014, 192 PP., $34.95 ISBN 978-1-59327-590-7 DECEMBER PHONE: 1.800.420.7240 OR +1.415.863.9900 EMAIL: sales@nostarch.com WEB: www.nostarch.com “James can see the Lady in the Red Dress, as well as the code that rendered her, in the Matrix.” — Katie Moussouris, founder and CEO, Luta Security Attacking Network Protocols is a deep dive into network protocol security from James Forshaw, one of the world’s leading bug hunters This comprehensive guide looks at networking from an attacker’s perspective to help you discover, exploit, and ultimately protect vulnerabilities You’ll start with a rundown of networking basics and protocol traffic capture before moving on to static and dynamic protocol analysis, common protocol structures, cryptography, and protocol security Then you’ll turn your focus to finding and exploiting vulnerabilities, with an overview of common bug classes, fuzzing, debugging, and exhaustion attacks Learn how to: • Capture, manipulate, and replay packets • Develop tools to dissect traffic and reverse engineer code to understand the inner workings of a network protocol • Discover and exploit vulnerabilities such as memory corruptions, authentication bypasses, and denials of service • Use capture and analysis tools like Wireshark and develop your own custom network proxies to manipulate network traffic Attacking Network Protocols is a must-have for any penetration tester, bug hunter, or developer looking to understand and discover network vulnerabilities About the Author James Forshaw is a renowned computer security researcher at Google Project Zero and the creator of the network protocol analysis tool Canape His discovery of complex design issues in Microsoft Windows earned him the top bug bounty of $100,000 and placed him as the #1 researcher on the published list from Microsoft Security Response Center (MSRC) He’s been invited to present his novel security research at global security conferences such as BlackHat, CanSecWest, and Chaos Computer Congress THE FINEST IN GEEK ENTERTAINMENT™ www.nostarch.com Footnotes Chapter 2: Capturing Application Traffic A proxy loop occurs when a proxy repeatedly connects to itself, causing a recursive loop The outcome can only end in disaster, or at least running out of available resources Chapter 3: Network Protocol Structures Just ask those who have tried to parse HTML for errant script code how difficult that task can be without a strict format Chapter 6: Application Reverse Engineering Apple moved to the x86 architecture in 2006 Prior to that, Apple used the PowerPC architecture PCs, on the other hand, have always been based on x86 architecture This isn’t completely accurate: many network cards can perform some processing in hardware ... PROTOCOL ANALYSIS TOOLKIT Passive Network Protocol Capture and Analysis Tools Microsoft Message Analyzer TCPDump and LibPCAP Wireshark Active Network Capture and Analysis Canape Canape Core Mallory Network. . .ATTACKING NETWORK PROTOCOLS A Hacker’s Guide to Capture, Analysis, and Exploitation by James Forshaw San Francisco ATTACKING NETWORK PROTOCOLS Copyright © 2018 by James Forshaw All rights... payload and header are commonly called a segment, whereas a UDP payload and header are commonly called a datagram The IP protocol uses a source and a destination address ➋ The destination address allows