ZeroMQ docx

www.it-ebooks.info www.it-ebooks.info Pieter Hintjens ZeroMQ www.it-ebooks.info ZeroMQ by Pieter Hintjens Copyright © 2013 Pieter Hintjens. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or corporate@oreilly.com. Editors: Andy Oram and Maria Gulick Production Editor: Christopher Hearse Copyeditor: Gillian McGarvey Proofreader: Rachel Head Indexer: Angela Howard Cover Designer: Randy Comer Interior Designer: David Futato Illustrator: Rebecca Demarest and Kara Ebrahim March 2013: First Edition Revision History for the First Edition: 2013-03-11: First release See http://oreilly.com/catalog/errata.csp?isbn=9781449334062 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. ZeroMQ, the image of a fourhorn sculpin, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐ mark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-33406-2 [LSI] www.it-ebooks.info To Noémie, Freeman, and Gregor. www.it-ebooks.info www.it-ebooks.info Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Part I. Learning to Work with ØMQ 1. Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Fixing the World 3 Audience for This Book 5 Getting the Examples 5 Ask and Ye Shall Receive 5 A Minor Note on Strings 10 Version Reporting 11 Getting the Message Out 11 Divide and Conquer 16 Programming with ØMQ 21 Getting the Context Right 21 Making a Clean Exit 22 Why We Needed ØMQ 23 Socket Scalability 27 Upgrading from ØMQ v2.2 to ØMQ v3.2 27 Warning: Unstable Paradigms! 28 2. Sockets and Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 The Socket API 32 Plugging Sockets into the Topology 32 Using Sockets to Carry Data 34 Unicast Transports 35 ØMQ Is Not a Neutral Carrier 35 I/O Threads 36 Messaging Patterns 37 v www.it-ebooks.info High-Level Messaging Patterns 38 Working with Messages 39 Handling Multiple Sockets 41 Multipart Messages 44 Intermediaries and Proxies 45 The Dynamic Discovery Problem 45 Shared Queue (DEALER and ROUTER Sockets) 48 ØMQ’s Built-in Proxy Function 53 Transport Bridging 54 Handling Errors and ETERM 56 Handling Interrupt Signals 61 Detecting Memory Leaks 62 Multithreading with ØMQ 63 Signaling Between Threads (PAIR Sockets) 68 Node Coordination 70 Zero-Copy 74 Pub-Sub Message Envelopes 75 High-Water Marks 77 Missing Message Problem Solver 78 3. Advanced Request-Reply Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 The Request-Reply Mechanisms 81 The Simple Reply Envelope 82 The Extended Reply Envelope 82 What’s This Good For? 85 Recap of Request-Reply Sockets 85 Request-Reply Combinations 86 The REQ to REP Combination 87 The DEALER to REP Combination 87 The REQ to ROUTER Combination 87 The DEALER to ROUTER Combination 88 The DEALER to DEALER Combination 88 The ROUTER to ROUTER Combination 88 Invalid Combinations 88 Exploring ROUTER Sockets 89 Identities and Addresses 89 ROUTER Error Handling 91 The Load-Balancing Pattern 91 ROUTER Broker and REQ Workers 92 ROUTER Broker and DEALER Workers 94 A Load-Balancing Message Broker 96 A High-Level API for ØMQ 102 vi | Table of Contents www.it-ebooks.info Features of a Higher-Level API 104 The CZMQ High-Level API 105 The Asynchronous Client/Server Pattern 111 Worked Example: Inter-Broker Routing 116 Establishing the Details 116 Architecture of a Single Cluster 117 Scaling to Multiple Clusters 118 Federation Versus Peering 121 The Naming Ceremony 122 Prototyping the State Flow 123 Prototyping the Local and Cloud Flows 126 Putting It All Together 133 4. Reliable Request-Reply Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 What Is “Reliability”? 141 Designing Reliability 142 Client-Side Reliability (Lazy Pirate Pattern) 144 Basic Reliable Queuing (Simple Pirate Pattern) 148 Robust Reliable Queuing (Paranoid Pirate Pattern) 151 Heartbeating 159 Shrugging It Off 160 One-Way Heartbeats 160 Ping-Pong Heartbeats 161 Heartbeating for Paranoid Pirate 161 Contracts and Protocols 163 Service-Oriented Reliable Queuing (Majordomo Pattern) 164 Asynchronous Majordomo Pattern 186 Service Discovery 191 Idempotent Services 193 Disconnected Reliability (Titanic Pattern) 194 High-Availability Pair (Binary Star Pattern) 206 Detailed Requirements 208 Preventing Split-Brain Syndrome 211 Binary Star Implementation 211 Binary Star Reactor 218 Brokerless Reliability (Freelance Pattern) 223 Model One: Simple Retry and Failover 225 Model Two: Brutal Shotgun Massacre 228 Model Three: Complex and Nasty 233 Conclusion 244 5. Advanced Publish-Subscribe Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Table of Contents | vii www.it-ebooks.info Pros and Cons of Publish-Subscribe 245 Pub-Sub Tracing (Espresso Pattern) 247 Last Value Caching 250 Slow Subscriber Detection (Suicidal Snail Pattern) 254 High-Speed Subscribers (Black Box Pattern) 258 Reliable Publish-Subscribe (Clone Pattern) 260 Centralized Versus Decentralized 261 Representing State as Key-Value Pairs 261 Getting an Out-of-Band Snapshot 271 Republishing Updates from Clients 276 Working with Subtrees 281 Ephemeral Values 284 Using a Reactor 292 Adding the Binary Star Pattern for Reliability 296 The Clustered Hashmap Protocol 306 Building a Multithreaded Stack and API 310 Part II. Software Engineering Using ØMQ 6. The ØMQ Community. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Architecture of the ØMQ Community 326 How to Make Really Large Architectures 327 Psychology of Software Architecture 328 The Contract 330 The Process 332 Crazy, Beautiful, and Easy 332 Stranger, Meet Stranger 333 Infinite Property 333 Care and Feeding 334 The ØMQ Process: C4 335 Language 335 Goals 336 Preliminaries 338 Licensing and Ownership 339 Patch Requirements 340 Development Process 342 Creating Stable Releases 345 Evolution of Public Contracts 347 A Real-Life Example 349 Git Branches Considered Harmful 352 Simplicity Versus Complexity 353 viii | Table of Contents www.it-ebooks.info [...]... 481 482 484 Index 485 Table of Contents www.it-ebooks.info | xi www.it-ebooks.info Preface ØMQ in a Hundred Words ØMQ (also known as ZeroMQ, 0MQ, or zmq) looks like an embeddable networking library, but acts like a concurrency framework It gives you sockets that carry atomic messages across various transports, like in-process, inter-process,... in each case how to run each example Preface www.it-ebooks.info | xv We appreciate, but do not require, attribution An attribution usually includes the title, author, publisher, and ISBN For example: ZeroMQ by Pieter Hintjens (O’Reilly) Copyright 2013 Pieter Hintjens, 978-1-449-33406-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact... Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information You can access this page at http://bit.ly /ZeroMQ- OReilly To comment or ask technical questions about this book, send email to bookques tions@oreilly.com xvi | Preface www.it-ebooks.info For more information about our books, courses, conferences,... Aarts, Marc Harter, Iskren Ivov Chernev, Jay Han, Sonia Hamilton, Nathan Stocks, Naveen Palli, and Zed Shaw for their contributions to this work Thanks to Martin Sustrik for his years of incredible work on ZeroMQ Thanks to Stathis Sideris for Ditaa Preface www.it-ebooks.info | xvii www.it-ebooks.info PART I Learning to Work with ØMQ In the first part of this book, you’ll learn how to use ØMQ We’ll cover . www.it-ebooks.info www.it-ebooks.info Pieter Hintjens ZeroMQ www.it-ebooks.info ZeroMQ by Pieter Hintjens Copyright © 2013 Pieter Hintjens. All rights reserved. Printed. Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. ZeroMQ, the image of a fourhorn sculpin, and related trade dress are trademarks of O’Reilly Media,. Contents | xi www.it-ebooks.info www.it-ebooks.info Preface ØMQ in a Hundred Words ØMQ (also known as ZeroMQ, 0MQ, or zmq) looks like an embeddable networking library, but acts like a concurrency framework.