www.it-ebooks.info www.it-ebooks.info Building the Realtime User Experience www.it-ebooks.info www.it-ebooks.info Building the Realtime User Experience Ted Roden Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo www.it-ebooks.info Building the Realtime User Experience by Ted Roden Copyright © 2010 Ted Roden. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com. Editor: Simon St.Laurent Production Editor: Kristen Borg Copyeditor: Genevieve d’Entremont Proofreader: Teresa Barensfeld Production Services: Molly Sharp Indexer: Ellen Troutman Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Robert Romano Printing History: July 2010: First Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Building the Realtime User Experience, the image of a myna bird, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information con- tained herein. TM This book uses RepKover™, a durable and flexible lay-flat binding. ISBN: 978-0-596-80615-6 [M] 1277230732 www.it-ebooks.info Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is Realtime? 2 Changing Interactions 2 Push Versus Pull 4 Prerequisites 4 Python 6 JavaScript 6 JavaScript Object Notation 6 Google’s App Engine 6 The Rest 7 2. Realtime Syndication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Simple Update Protocol (SUP) 11 The SUP file 12 Subscribing with SUP 13 Publishing with SUP 21 PubSubHubbub 26 The Protocol 27 Subscribing with PubSubHubbub 30 Publishing with PubSubHubbub 36 3. The Dynamic Homepage (Widgets in Pseudorealtime) . . . . . . . . . . . . . . . . . . . . . . . 39 The Basic Widgets 39 HTML 40 Setting Up the JavaScript 41 FriendFeed in Realtime 48 Live Images 51 It Was All a Setup! 54 The Old Versus the New 55 v www.it-ebooks.info 4. River of Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 A Crash Course in Server Push 57 Long Polling 58 The Bayeux Protocol 58 Cometd 60 Setting Up Your Cometd Environment 60 Putting Everything in Its Place 62 A Realtime Live Blog 66 The Two-Connection Limit 72 Server-Side Filters (with Java) 73 Integrating Cometd into Your Infrastructure 77 5. Taming the Firehose with Tornado . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Tornado 79 Installing Tornado 80 The Basic Framework 80 Building an Application 82 Asynchronous Tornado in Realtime 84 Twitter’s Realtime Streaming APIs 87 From the Firehose to the Web Browser 91 Templates in Tornado 91 Creating a Template for This Project 93 The JavaScript Parts 96 6. Chat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Setting Up the Basic Code 101 The Basic HTML Frame 101 The Python Shell 103 The JavaScript Base 105 Checking the Progress 106 Logging In 107 On the Server Side 107 JavaScript 112 Basic Chatting 117 Chatting on the Server Side 117 Sending Messages 118 Receiving Messages 121 Acting Natural 121 7. Instant Messaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Getting Started with Google App Engine 130 Setting Up an Account 130 Creating an Application with the SDK 131 vi | Table of Contents www.it-ebooks.info Taking Advantage of Google 133 Keeping Track of the User 135 The Deploy Button 135 The Dashboard 137 Receiving Instant Messages 138 Sending Instant Messages 141 Responding Intelligently 143 The Basic Commands 143 Checking Authentication via Instant Messenger 145 Introducing a Third Party 146 Setting Up an API 148 8. SMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 The SMS Landscape 153 Email to SMS 153 SMS APIs 156 Building the Basic Application 158 Extending the Instant Messaging Application 158 An SMS Service Class 160 The Base Handler 162 Preparing to Accept Messages 163 Setting Up the Server 164 Sending and Receiving the Messages 165 TextMarks 165 Zeep Mobile 172 Authenticating Users 176 Building an SMS API 180 9. Measuring User Engagement: Analytics on the Realtime Web . . . . . . . . . . . . . . . . 185 Realtime Analytics Services 185 Chartbeat 186 Woopra 187 Customized Analytics 189 Sending Tracking Pings with JavaScript 189 Catching the Statistics on the Server 193 Making Sense of the Traffic 198 Viewing the Traffic 201 Tracking Backend Traffic and Custom Data 208 Sending Out Alerts 214 10. Putting It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 The Game 217 Caveats 217 Table of Contents | vii www.it-ebooks.info Getting Set Up 218 Google App Engine 218 Google Maps API Key 219 EC2 or Other Hosted Server 220 GeoModel 220 The Basic Models 221 UserInfo 222 Disease 224 Germ 225 CommandCenter 227 Textmark 229 Messenger 230 UserThreats 232 GameUtils 234 Building the Game Itself 235 User Authentication 235 Geolocation 241 Spreading Germs 244 Loading Germs 251 Threats 253 Spreading Germs 267 Fighting Germs 269 Realtime Syndication 271 The Command Center 275 In Review 295 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 viii | Table of Contents www.it-ebooks.info [...]... it was much the same This has changed No longer can a website wait for users to navigate to the right URL; the website must contact the user wherever that user may be The paradigm has shifted from a website-centric model, where the website was at the center of the interaction, to a user- centric model Now all interactions start and end at the user (see Figure 1-2), whether she is visiting the website... long as the Web has been around, there have been two main ways of getting content to a user: push and pull Pull is the method in which most interactions have worked the user clicks a link and the browser pulls the content down from the server If the server wants to send additional messages to the user after the data has been pulled down, it just waits and queues them up until the client makes another... connecting to the database, this file starts by making the active.users API call That function returns a serialized JSON object with the list of active users We loop through those users to create an array called $feeds containing only the location of the Atom feeds The result of the active.users API call will give us various pieces of information about recently active users One of these fields will be the URL... the base and the SUP-ID Then, we build a simple PHP object to map the fields from the feeds table to the Atom feed URL, the SUP URL, and the SUP-ID itself The insert_array function takes this data, turns it into SQL, and inserts it into the database The sup_discover function is the function that does most of the work in this file This function uses PHP’s very handy SimpleXML extension to download the. .. permission before you do some of these things, but your application needs to offer them 2 | Chapter 1: Introduction www.it-ebooks.info Figure 1-1 In the past, users visited websites Figure 1-2 Websites must reach out to users wherever they are What Is Realtime? | 3 www.it-ebooks.info I mentioned SMS, but the mobile experience does not end there These days, users have phones in their pockets with full-fledged... seconds onto the age variable The extra time gives subscribers some leeway when checking the feeds The first function in the file, generate_sup_id, is used to create a unique SUP-ID for each user This function combines a “secret” string along with the user s unique ID and generates the MD5 hash of it Then it returns the first 10 characters of the hash by running it through PHP’s substr function There is... Atom feeds and their corresponding SUP IDs Rather than scour the Web for sites that support SUP and then manually finding the users on that site that are still active, we’re going to ask enjoysthin.gs for that list Enjoysthin.gs provides a public API function to get a list of recently active users That API function returns, amongst other things, the URL of the Atom feed for that user The following PHP... run, you can follow along with the text and test the application online I view this book as realtime experience in its own way Not only do I plan to keep the code updated, but I also plan to continue the conversation about these topics online through the website, Twitter, and any new service as it pops up The official Twitter account for this book is @therealtimebook There is a lot of content being... check each of them to see whether they have a SUP feed associated with them They all do have a SUP feed defined, but if you run this code against other URLs, this is a good test Once we determine that the feed does have a SUP-ID associated with it, we save it to the database The SUP specification says that the SUP-ID can be specified in one of two ways Either it can be a tag in the feed itself,... “pinging the blogosphere.” These work by pinging known URLs as things are published After being pinged, these services can then download the new feed However, the basic method of pinging doesn’t supply link to the actual feed URL, so the server must parse the site to find the usable RSS/ Atom feed This protocol also doesn’t allow for arbitrary subscribers to receive pings or get the data any faster than they . www.it-ebooks.info www.it-ebooks.info Building the Realtime User Experience www.it-ebooks.info www.it-ebooks.info Building the Realtime User Experience Ted Roden Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo www.it-ebooks.info Building. wait for users to navigate to the right URL; the website must contact the user wherever that user may be. The paradigm has shifted from a website-centric model, where the website was at the center. Service (SMS) updates. A truly realtime experience exists anywhere the user is at a given moment. If the user is interacting with the web browser, then that’s the place to contact her. If she’s