Additional Resources Microservices AntiPatterns and Pitfalls Mark Richards Microservices Antipatterns and Pitfalls by Mark Richards Copyright © 2016 O’Reilly Media, Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://safaribooksonline.com) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Brian Foster Production Editor: Melanie Yarbrough Copyeditor: Christina Edwards Proofreader: Amanda Kersey Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest July 2016: First Edition Revision History for the First Edition 2016-07-06: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Microservices AntiPatterns and Pitfalls, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights 978-1-491-96331-9 [LSI] Preface In late 2006 service-oriented architecture (SOA) was all the craze Companies were jumping on the bandwagon and embracing SOA before fully understanding the advantages and disadvantages of this very complex architecture style Those companies that embarked on SOA projects often found constant struggles with service granularity, performance, data migrations, and in particular the organizational change that comes about with SOA As a result, many companies either abandoned their SOA efforts or built hybrid architectures that did not fulfill all of the promises of SOA Today we are poised to repeat this same experience with a relatively new architecture style known as microservices Microservices is a current trend in the industry right now, and like SOA back in the mid 2000s, is all the craze As a result, many companies are looking toward this architecture style to leverage the benefits provided by microservices such as ease of testing, fast and easy deployments, fine-grained scalability, modularity, and overall agility However, like SOA, companies developing microservices are finding themselves struggling with things like service granularity, data migration, organizational change, and distributed processing challenges As with any new technology, architecture style, or practice, antipatterns, and pitfalls usually emerge as you learn more about it and experience the many “lessons learned” during the process While antipatterns and pitfalls may seem like the same thing, there is a subtle difference between them Andrew Koenig defines an antipattern as something that seems like a good idea when you begin, but leads you into trouble, whereas my friend Neal Ford defines a pitfall as something that was never a good idea, even from the start This is an important distinction because you may not experience the negative results from an antipattern until you are well into the development lifecycle or even well into production, whereas with a pitfall you usually find out you are headed down the wrong path relatively quickly This report will introduce several of the more common antipatterns and pitfalls that emerge when using microservices My goal with this report is to help you avoid costly mistakes by not only helping you understand when an antipattern or pitfall is occurring, but more importantly helping you understand the techniques and practices for avoiding these antipatterns and pitfalls While I don’t have time in this report to cover the details of all of the various antipatterns and pitfalls you might encounter with microservices, I cover some of the more common ones These include antipatterns and pitfalls related to service granularity (Chapter 5, Grains of Sand Pitfall), data migration (Chapter 1, Data-Driven Migration AntiPattern), remote access latency (Chapter 9, Are We There Yet Pitfall), reporting (Chapter 4, Reach-in Reporting AntiPattern), contract versioning (Chapter 8, The Static Contract Pitfall), service responsiveness (Chapter 2, The Timeout AntiPattern), and many others I recently recorded a video for O’Reilly called Microservices AntiPatterns and Pitfalls: Learning to Avoid Costly Mistakes that contains the complete set of antipatterns and pitfalls you may encounter when using microservices, as well as a more in-depth look into each one Included in the video is a selfassessment workbook containing analysis tasks and goals oriented around analyzing your current application You can use this assessment workbook to determine whether you are experiencing any of the antipatterns and pitfalls introduced in the video and ways to avoid them Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords Constant width bold Shows commands or other text that should be typed literally by the user Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context Safari® Books Online NOTE Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more For more information about Safari Books Online, please visit us online Notice that the the schema actually contains the version number field (version) as an integer value Because you only have one schema now, you will need to add all of the combinations of possibilities to the schema In the example above both the CUSIP and SEDOL are added to the schema because that is what varies between the versions The big advantage of this technique is that the schema (including the version) is independent of the remote access protocol This means that the same exact schema can be used by multiple protocols For example, the same schema can be used by REST and JMS 2.0 without any modifications to the remote access protocol headers: POST /trade/buy Accept: application/json { "version": "2", "acct": "12345", "sedol": "2046251", "shares": "1000" } String msg = createJSON( "version","2", "acct","12345", "sedol","2046251", "shares","1000")}; jmsContext.createProducer().send(queue, msg); Unfortunately this technique has a lot of disadvantages associated with it First, you must parse the actual payload of the message to extract the version number This precludes using things like XML appliances (e.g., DataPower) to routing, and also might present issues when trying to parse the schema (particularly with XML) Secondly, the schemas can get quite complex, making it difficult to automated conversions of the schema (e.g., JSON to Java object) Finally, custom validations may be required in the service to validate the schema In the example above, the service would have to validate that either the CUSIP or SEDOL is filled in based on the version number Chapter Are We There Yet Pitfall With the microservices architecture every service is deployed as a separate application, meaning all of the communication to a microservice from the client or API layer, as well as communication between services, requires a remote call This pitfall occurs when you don’t know how long the remote access call takes You might assume the latency it around 50 milliseconds, but have you ever measured it? Do you know what the average latency is for your particular environment? Do you know what the “long tail” latency is (e.g., 95, 99, 99.5 percentiles) for your environment? Measuring both of these metrics is important, because even with good average latency, bad long-tail latency can destroy you Measuring Latency Measuring the remote access latency under load in your production environment (or production-like environment) is critical for understanding the performance profile of your application For example, let’s say a particular business request requires the coordination of four microservices Assuming that your remote access latency is 100 milliseconds, that particular business request would consume 500 milliseconds just in remote access latency alone (the initial request plus four remote calls between services) That is a half a second of request time without one single line of source code being executed for the actual business request processing Most applications simply cannot absorb that sort of latency You might think the way to avoid this pitfall is to simply measure and determine your average latency for your chosen remote access protocol (e.g., REST) However, that only provides you with one piece of information — the average latency of the particular remote access protocol you are using The other task is to investigate the comparative latency using other remote access protocols such as Java Message Service (JMS), Advanced Message Queuing Protocol (AMQP), and Microsoft Message Queue (MSMQ) Comparing Protocols The comparative latency will vary greatly based on both your environment and the nature of the business request, so it is important to establish these benchmarks on a variety of business requests with different load profiles Figure 9-1 Comparing remote access latency In looking at the hypothetical example in Figure 9-1 you notice that AMQP is in fact almost twice as fast as REST You can now leverage this information to make intelligent choices as to which requests should use which remote access protocol For example, you may choose to use REST for all communications from client requests to your microservices and AMQP for all interservice communication in order to increase performance within your application Performance is not the only consideration when selecting your remote access protocol As you will see in Chapter 10, you may want to leverage messaging to provide additional capabilities to your application as well Chapter 10 Give It a Rest Pitfall Using REST is by far the most popular choice for accessing microservices and communicating between services It’s so common a choice that most of the popular template frameworks (e.g., DropWizard, Spring Boot, etc.) have REST access already built into the service templates If REST is such a popular choice, then why is it a pitfall? The give it a rest pitfall is about using REST as the only communication protocol and ignoring the power of messaging to enhance your microservices architecture For example, in a RESTful microservices architecture, how would you handle asynchronous communications? What about the need for broadcast capabilities? What you if you need to manage multiple remote RESTful calls within a transactional unit of work? There are two types of messaging standards you should be aware of when considering using messaging for your microservices architecture — platformspecific standards and platform-independent standards Platform-specific standards include the JMS for the Java platform and MSMQ for the NET platform Both describe a standard API used within the platform, independent of the messaging provider (vendor) you are using For example, in the Java platform you can swap out brokers (e.g., ActiveMQ, HornetQ, etc.) with no API changes While the API is standard and remains the same, it’s the underlying proprietary protocol between these brokers that is different (which is why you need to have the same client JAR and server JAR for the same vendor) With platform-standard messaging protocols you are more concerned about portability via a common API rather than the actual vendor product you are using or the wire-level protocols used The current platform-independent standard is AMQP AMQP, which standardizes on the wire-level protocol, not the API This allows heterogeneous platforms to communicate with one another, regardless of the vendor product you are using A client using RabbitMQ, for example, can easily communicate with a StormMQ server (assuming they are using the same protocol version) AMQP using RabbitMQ is currently the most popular choice for messaging within a microservices architecture, mostly because of its platform-independent nature Asynchronous Requests The first consideration for using messaging within your microservices architecture is asynchronous communication With asynchronous requests the service caller does not need to wait for a response from the service when making a request, as illustrated in Figure 10-1 This is sometimes referred to as “fire-and-forget” processing Figure 10-1 Asynchronous communications using messaging Not only does asynchronous processing increase overall performance, but it also adds an element of reliability to your system Performance is increased because callers don’t have to wait for a response if none is needed Through guaranteed delivery, the message broker ensures that the service will eventually receive the message Reliability is increased because the caller doesn’t need to worry about setting timeout values or using the circuit breaker pattern when communicating with a service (see Chapter 2) Broadcast Capabilities Another very powerful feature of messaging that is not available within REST is the capability to broadcast a message to multiple services This is known in messaging as “publish-and-subscribe” messaging, and usually involves topics and subscribers (depending on the messaging standard you are using) Figure 10-2 illustrates the basic behavior of broadcast messaging Figure 10-2 Broadcast capabilities using messaging Broadcast messaging involves a message producer sending out the same message to multiple message receivers (i.e., services) The message producer generally doesn’t know who is accepting the message or what they are going to with it For example, a message producer may broadcast a message informing consumers about a stock split for Apple stock (AAPL) The message producer only has the responsibility of publishing a message to a topic (JMS), a fanout or topic exchange (AMQP), or a multicast queue (MSMQ) The stock split message may be picked up by any number of consumers, or no consumers at all Transacted Requests Messaging systems support the notion of transacted messages, meaning that if messages are sent to multiple queues or topics within the context of a transaction, the messages are not actually received by the services until the sender does a commit on that transaction The service consumer sends a message to the first service and then sends another message to the second service, as illustrated in Figure 10-3 Until the service consumer performs a commit, those messages are held in the queues Once the service consumer performs a commit, both messages are then released Figure 10-3 Transaction capabilities of messaging If the service consumer in Figure 10-3 sends a message to the first queue, but then experiences some sort of error, the service consumer can perform a rollback on the messaging transaction, which would effectively remove the message from the first queue Implementing this sort of transaction capability using REST would be very difficult, essentially requiring the service consumer to issue compensating requests to reverse the updates made by each request Therefore, it is a good idea to consider using transacted messaging any time a service consumer needs to orchestrate multiple remote requests About the Author Mark Richards is an experienced, hands-on software architect involved in the architecture, design, and implementation of microservices architectures, service-oriented architectures, and distributed systems in J2EE and other technologies He has been in the software industry since 1983 and has significant experience and expertise in application, integration, and enterprise architecture Mark served as the president of the New England Java Users Group from 1999 through 2003 He is the author of numerous technical books and videos, including the Software Architecture Fundamentals Video Series (O’Reilly video), Enterprise Messaging (O’Reilly video), Java Message Service, 2nd Edition (O’Reilly), and a contributing author to 97 Things Every Software Architect Should Know (O’Reilly) Mark has a master’s degree in computer science and numerous architect and developer certifications from IBM, Sun, The Open Group, and BEA He is a regular conference speaker at the No Fluff Just Stuff (NFJS) Symposium Series and has spoken at more than 100 conferences and user groups around the world on a variety of enterprise-related technical topics When he is not working, Mark can usually be found teaching architecture fundamentals classes and hiking in the White Mountains of New Hampshire and along the Appalachian Trail Preface Conventions Used in This Book Safari® Books Online How to Contact Us Acknowledgments Data-Driven Migration AntiPattern Too Many Data Migrations Functionality First, Data Last The Timeout AntiPattern Using Timeout Values Using the Circuit Breaker Pattern The “I Was Taught to Share” AntiPattern Too Many Dependencies Techniques for Sharing Code Reach-in Reporting AntiPattern Issues with Microservices Reporting Asynchronous Event Pushing Grains of Sand Pitfall Analyzing Service Scope and Function Analyzing Database Transactions Analyzing Service Choreography Developer Without a Cause Pitfall Making the Wrong Decisions Understanding Business Drivers Jump on the Bandwagon Pitfall Advantages and Disadvantages Matching Business Needs Other Architecture Patterns The Static Contract Pitfall Changing a Contract Header Versioning Schema Versioning Are We There Yet Pitfall Measuring Latency Comparing Protocols 10 Give It a Rest Pitfall Asynchronous Requests Broadcast Capabilities Transacted Requests ... friend and partner-in-crime Neal Ford for helping me understand the differences between pitfalls and antipatterns and for also helping me properly classify each of the antipatterns and pitfalls. .. understand the techniques and practices for avoiding these antipatterns and pitfalls While I don’t have time in this report to cover the details of all of the various antipatterns and pitfalls. .. and many others I recently recorded a video for O’Reilly called Microservices AntiPatterns and Pitfalls: Learning to Avoid Costly Mistakes that contains the complete set of antipatterns and pitfalls