CuuDuongThanCong.com https://fb.com/tailieudientucntt Node.js in Practice Alex Young and Marc Harter CuuDuongThanCong.com https://fb.com/tailieudientucntt Copyright For online information and ordering of this and other Manning books, please visit www.manning.com The publisher offers discounts on this book when ordered in quantity For more information, please contact Special Sales Department Manning Publications Co 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2015 by Manning Publications Co All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine Manning Publications Co 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Development editor: Cynthia Kane Technical development editor: Jose Maria Alvarez Rodriguez Copyeditor: Benjamin Berg Proofreader: Katie Tennant Typesetter: Gordan Salinovic Cover designer: Marija Tudor ISBN 9781617290930 Printed in the United States of America 10 – EBM – 19 18 17 16 15 14 CuuDuongThanCong.com https://fb.com/tailieudientucntt Brief Table of Contents Copyright Brief Table of Contents Table of Contents Foreword Preface Acknowledgments About this Book About the Cover Illustration Node fundamentals Chapter Getting started Chapter Globals: Node’s environment Chapter Buffers: Working with bits, bytes, and encodings Chapter Events: Mastering EventEmitter and beyond Chapter Streams: Node’s most powerful and misunderstood feature Chapter File system: Synchronous and asynchronous approaches to files Chapter Networking: Node’s true “Hello, World” Chapter Child processes: Integrating external applications with Node Real-world recipes Chapter The Web: Build leaner and meaner web applications Chapter 10 Tests: The key to confident code Chapter 11 Debugging: Designing for introspection and resolving issues Chapter 12 Node in production: Deploying applications safely Writing modules Chapter 13 Writing modules: Mastering what Node is all about Community Index List of Figures List of Tables CuuDuongThanCong.com https://fb.com/tailieudientucntt List of Listings CuuDuongThanCong.com https://fb.com/tailieudientucntt Table of Contents Copyright Brief Table of Contents Table of Contents Foreword Preface Acknowledgments About this Book About the Cover Illustration Node fundamentals Chapter Getting started 1.1 Getting to know Node 1.1.1 Why Node? 1.1.2 Node’s main features 1.2 Building a Node application 1.2.1 Creating a new Node project 1.2.2 Making a stream class 1.2.3 Using a stream 1.2.4 Writing a test 1.3 Summary Chapter Globals: Node’s environment 2.1 Modules Technique Installing and loading modules Technique Creating and managing modules Technique Loading a group of related modules Technique Working with paths 2.2 Standard I/O and the console object Technique Reading and writing to standard I/O Technique Logging messages CuuDuongThanCong.com https://fb.com/tailieudientucntt Technique Benchmarking a program 2.3 Operating system and command-line integration Technique Getting platform information Technique Passing command-line arguments Technique 10 Exiting a program Technique 11 Responding to signals 2.4 Delaying execution with timers Technique 12 Executing functions after a delay with setTimeout Technique 13 Running callbacks periodically with timers Technique 14 Safely managing asynchronous APIs 2.5 Summary Chapter Buffers: Working with bits, bytes, and encodings 3.1 Changing data encodings Technique 15 Converting buffers into other formats Technique 16 Changing string encodings using buffers 3.2 Converting binary files to JSON Technique 17 Using buffers to convert raw data 3.3 Creating your own binary protocol Technique 18 Creating your own network protocol 3.4 Summary Chapter Events: Mastering EventEmitter and beyond 4.1 Basic usage Technique 19 Inheriting from EventEmitter Technique 20 Mixing in EventEmitter 4.2 Error handling Technique 21 Managing errors Technique 22 Managing errors with domains 4.3 Advanced patterns CuuDuongThanCong.com https://fb.com/tailieudientucntt Technique 23 Reflection Technique 24 Detecting and exploiting EventEmitter Technique 25 Categorizing event names 4.4 Third-party modules and extensions Technique 26 Alternatives to EventEmitter 4.5 Summary Chapter Streams: Node’s most powerful and misunderstood feature 5.1 Introduction to streams 5.1.1 Types of streams 5.1.2 When to use streams 5.1.3 History 5.1.4 Streams in third-party modules 5.1.5 Streams inherit from EventEmitter 5.2 Built-in streams Technique 27 Using built-in streams to make a static web server Technique 28 Stream error handling 5.3 Third-party modules and streams Technique 29 Using streams from third-party modules 5.4 Using the stream base classes Technique 30 Correctly inheriting from the stream base classes Technique 31 Implementing a readable stream Technique 32 Implementing a writable stream Technique 33 Transmitting and receiving data with duplex streams Technique 34 Parsing data with transform streams 5.5 Advanced patterns and optimization Technique 35 Optimizing streams Technique 36 Using the old streams API Technique 37 Adapting streams based on their destination Technique 38 Testing streams CuuDuongThanCong.com https://fb.com/tailieudientucntt 5.6 Summary Chapter File system: Synchronous and asynchronous approaches to files 6.1 An overview of the fs module 6.1.1 POSIX file I/O wrappers 6.1.2 Streaming 6.1.3 Bulk file I/O 6.1.4 File watching 6.1.5 Synchronous alternatives Technique 39 Loading configuration files Technique 40 Using file descriptors Technique 41 Working with file locking Technique 42 Recursive file operations Technique 43 Writing a file database Technique 44 Watching files and directories 6.2 Summary Chapter Networking: Node’s true “Hello, World” 7.1 Networking in Node 7.1.1 Networking terminology 7.1.2 Node’s networking modules 7.1.3 Non-blocking networking and thread pools 7.2 TCP clients and servers Technique 45 Creating a TCP server and tracking clients Technique 46 Testing TCP servers with clients Technique 47 Improve low-latency applications 7.3 UDP clients and servers Technique 48 Transferring a file with UDP Technique 49 UDP client server applications 7.4 HTTP clients and servers Technique 50 HTTP servers Technique 51 Following redirects CuuDuongThanCong.com https://fb.com/tailieudientucntt Technique 52 HTTP proxies 7.5 Making DNS requests Technique 53 Making a DNS request 7.6 Encryption Technique 54 A TCP server that uses encryption Technique 55 Encrypted web servers and clients 7.7 Summary Chapter Child processes: Integrating external applications with Node 8.1 Executing external applications Technique 56 Executing external applications 8.1.1 Paths and the PATH environment variable 8.1.2 Errors when executing external applications Technique 57 Streaming and external applications 8.1.3 Stringing external applications together Technique 58 Executing commands in a shell 8.1.4 Security and shell command execution Technique 59 Detaching a child process 8.1.5 Handing I/O between the child and parent processes 8.1.6 Reference counting and child processes 8.2 Executing Node programs Technique 60 Executing Node programs Technique 61 Forking Node modules Technique 62 Running jobs 8.2.1 Job pooling 8.2.2 Using the pooler module 8.3 Working synchronously Technique 63 Synchronous child processes 8.4 Summary Real-world recipes CuuDuongThanCong.com https://fb.com/tailieudientucntt scripts, npm seams secure event Secure Sockets Layer See SSL segments send method, 2nd send module Sequelize library serve-favicon module serve-index module SERVER_CONFIG_FILE value serverless apps setBreakpoint function setInterval method setTimeout method shell commands overview security and should.js shrinkwrap command SIGHUP signal silent option Simple Object Access Protocol See SOAP single page applications single responsibility principle single-threaded programs Sinon.JS slice method slow flag slug size SOAP (Simple Object Access Protocol) Socket.IO sockets, 2nd SocketStream SOLID principles source maps spawn method, 2nd, 3rd spawnSync method Splunk Square SRV records SSL (Secure Sockets Layer) Stack Exchange stack traces stat method, 2nd stateless, defined static middleware statSync method, 2nd STATUS_CODES object statusCode property stderr stream stdin stream stdout stream strace command, 2nd strictEqual method CuuDuongThanCong.com https://fb.com/tailieudientucntt stringify method strings Stripe StrongLoop stubs, 2nd SuperTest module, 2nd symlink method synchronous functions, speed of synchronous logging SyntaxError Syslog Protocol T tap module tap-results module TCP/IP suite TDD (test-driven development) terminology layers overview sockets TCP/IP UDP Test Anything Protocol, 2nd test method test-driven development See TDD text method thread pools throw statement throws method time method time-based conditional request timeEnd method TLS (Transport Layer Security) tls module toString method trace method training Transform class transform streams, 2nd Transmission Control Protocol See TCP transport layer Transport Layer Security See TLS Travis CI truncate method try/catch blocks TTY (user shell) Twitter, 2nd TXT records U CuuDuongThanCong.com https://fb.com/tailieudientucntt UDP (User Datagram Protocol) defined, 2nd dgram module and overview sending messages to client transferring files uncaught exceptions overview using domains uniform resource locators See URLs unlink method unloading modules unref method, 2nd unstable versions unwatch command Upstart url module URLs (uniform resource locators) useGlobal property User Datagram Protocol See UDP UTF-8 encoding util.format method util.inherits method, 2nd util.pump method utimes method V V8 version command vhost module W warn method watch method, 2nd watchers command watchFile method web scraping Web Workers WebMatrix WebOps websocket-server module WebSockets, 2nd Winston module winston-loggly transport wkhtmltopdf WorldPay Writable class writable streams, 2nd write method Writeable stream writeFile method CuuDuongThanCong.com https://fb.com/tailieudientucntt ws module, 2nd X x509 command XML (Extensible Markup Language) xmllint CuuDuongThanCong.com https://fb.com/tailieudientucntt List of Figures Chapter Getting started Figure 1.1 An advertising server built with Node Figure 1.2 Node’s key parts in context Figure 1.3 The three steps to creating a new Node project Chapter Globals: Node’s environment Figure 2.1 Folders as modules Figure 2.2 Data flows in a simple program that uses stdio Figure 2.3 Signals originate from a process, and are handled with an event listener Figure 2.4 Scheduling nextTick on the event loop Chapter Buffers: Working with bits, bytes, and encodings Figure 3.1 Data URI read in a browser displays the monkey as an image Figure 3.2 Generated secondmonkey.png file from a data URI Figure 3.3 The transformation of binary data into a more usable/programmable format Figure 3.4 Binary data is read using FileSystem API into Node.js, transformed using the Buffer API into an easier-to-use JSON format Figure 3.5 Final result of our transformation Figure 3.6 The header: transformation from the specification to code using the Node Buffer API Figure 3.7 The field descriptor array: transformation from the specification to code using the Node Buffer API Figure 3.8 The records: transformation from the specification to code using the Node Buffer API Figure 3.9 The full set of code for parsing a DBF file into JSON Chapter Events: Mastering EventEmitter and beyond Figure 4.1 Domains help catch errors and handle them with an EventEmitter-style API Chapter Streams: Node’s most powerful and misunderstood feature Figure 5.1 Using streamable APIs means I/O operations potentially use less memory Figure 5.2 The baudio module by James Halliday (substack) supports the generation of audio streams (from https://github.com/substack/baudio) Figure 5.3 The network inspector confirms the content was compressed CuuDuongThanCong.com https://fb.com/tailieudientucntt Figure 5.4 A duplex stream Figure 5.5 A graphical representation of the memory usage of streams Chapter File system: Synchronous and asynchronous approaches to files Figure 6.1 Advisory locking using a lockfile between cooperating processes Chapter Networking: Node’s true “Hello, World” Figure 7.1 Protocols are grouped into seven logical layers Packets are wrapped by protocols at consecutive layers Figure 7.2 Network layer wrapping Figure 7.3 Node’s threads when making HTTP requests Figure 7.4 When Nagle’s algorithm is used, smaller packets are collected into a larger payload Figure 7.5 Even though UDP isn’t full-duplex, it’s possible to create connections in two directions given a port number at both sides Figure 7.6 Redirection is cyclical, and requests will be made until a 200 status is encountered Figure 7.7 To use the Node proxy we’ve created, set localhost:8080 as the Web Proxy Server Chapter Child processes: Integrating external applications with Node Figure 8.1 Choosing the right method Figure 8.2 The execFile method buffers the result and provides a callback interface Figure 8.3 Common child process errors Figure 8.4 The spawn method returns a streaming interface for I/O Figure 8.5 Stringing external applications together with spawn Figure 8.6 The exec method runs our commands in a subshell Figure 8.7 Detached child process exists independent of the Node process Figure 8.8 The fork command runs a Node module in a separate process and sets up a communications channel Chapter The Web: Build leaner and meaner web applications Figure 9.1 Glance has built-in pages for errors Figure 9.2 Projects that use Grunt typically have a package.json and a Gruntfile.js Figure 9.3 Making requests to a REST API Figure 9.4 Requests can pass through several callbacks until the final response is sent Figure 9.5 Applications can be easier to understand if organized according to the SOLID principles CuuDuongThanCong.com https://fb.com/tailieudientucntt Figure 9.6 A Node web application should support both standard HTTP requests and WebSockets Figure 9.7 Accessing sessions from WebSockets Figure 9.8 You can test authenticated routes by catching cookies Figure 9.9 Cubism.js shows time series values in real time Chapter 10 Tests: The key to confident code Figure 10.1 Node can run a web server and requests against it to support web application testing Figure 10.2 node-tap uses several reusable submodules to orchestrate tests Figure 10.3 Travis CI running tests Chapter 11 Debugging: Designing for introspection and resolving issues Figure 11.1 Leaking resources when using uncaughtException Figure 11.2 Error screen when no debugging agent is found Figure 11.3 Node inspector connected to the debugger Figure 11.4 Using the debug-brk flag Figure 11.5 Loading a heap snapshot into the Chrome DevTools Figure 11.6 Loading a second snapshot for comparison Figure 11.7 Using the comparison view Figure 11.8 Examining memory allocations between the snapshots Figure 11.9 Drilling down to the types of data being created in memory Figure 11.10 Sample Node REPL session Figure 11.11 Using Netcat against a REPL server Chapter 12 Node in production: Deploying applications safely Figure 12.1 The jitsu command-line client allows you to sign in Figure 12.2 The WebOps management interface Figure 12.3 Signing in with Heroku Figure 12.4 The Azure CLI tool Figure 12.5 Azure’s web interface after creating a website Figure 12.6 A Node program running alongside Apache or nginx Figure 12.7 Browsers either use the local cache or make a conditional request, based on the previous request’s headers CuuDuongThanCong.com https://fb.com/tailieudientucntt Figure 12.8 A Node process running on a single core Figure 12.9 Take advantage of more cores by running multiple processes Figure 12.10 Loggly’s dashboard Chapter 13 Writing modules: Mastering what Node is all about Figure 13.1 Node avoids dependency hell Figure 13.2 npmjs.com package details page Figure 13.3 The different types of dependencies Figure 13.4 Semantic versioning Figure 13.5 Sample output from fastfibserver Appendix Community Figure A.1 James Halliday’s blog about Node and testing Figure A.2 Learn Node with NodeSchool CuuDuongThanCong.com https://fb.com/tailieudientucntt List of Tables Chapter Globals: Node’s environment Table 2.1 Formatting placeholders Chapter Buffers: Working with bits, bytes, and encodings Table 3.1 DBase 5.0 header specification Table 3.2 DBase 5.0 field descriptor array specification Table 3.3 Field types specification Table 3.4 Simple key-value database protocol Chapter Streams: Node’s most powerful and misunderstood feature Table 5.1 A summary of the classes available in streams2 Table 5.2 Selecting a streams base class Chapter File system: Synchronous and asynchronous approaches to files Table 6.1 Supported POSIX file methods in Node Table 6.2 Common file descriptors Chapter Networking: Node’s true “Hello, World” Table 7.1 Networking concepts Table 7.2 DNS record types Chapter The Web: Build leaner and meaner web applications Table 9.1 Choosing the correct HTTP verbs Table 9.2 Mapping routes to responses Table 9.3 Migrating Express middleware components Chapter 10 Tests: The key to confident code Table 10.1 Node testing concepts Table 10.2 When to use integration tests, mocks, and stubs Chapter 12 Node in production: Deploying applications safely Table 12.1 Setting environmental variables CuuDuongThanCong.com https://fb.com/tailieudientucntt Table 12.2 Comparing server options CuuDuongThanCong.com https://fb.com/tailieudientucntt List of Listings Chapter Getting started Listing 1.1 A writable stream that counts Listing 1.2 Using the CountStream class Listing 1.3 Using the CountStream class Chapter Globals: Node’s environment Listing 2.1 Using npm Listing 2.2 Exporting modules Listing 2.3 Exporting multiple objects, methods, and values Listing 2.4 Loading modules with require Listing 2.5 The group/index.js file Listing 2.6 The group/one.js file Listing 2.7 A file loading the group of modules Listing 2.8 A package.json file for a directory containing a module Listing 2.9 Path variables Listing 2.10 Path variables Listing 2.11 Path variables Listing 2.12 Benchmarking a function Listing 2.13 Branching based on architecture Listing 2.14 Manipulating command-line arguments Listing 2.15 Returning meaningful exit status codes Listing 2.16 Adding a listener for a POSIX signal Listing 2.17 Combining setTimeout with Function.prototype.bind Listing 2.18 Using clearTimeout to prevent scheduled functions Listing 2.19 Using setInterval and setTimeout together Listing 2.20 Keeping a timer alive until the program cleanly exits Listing 2.21 Incorrectly triggering asynchronous methods with events Listing 2.22 Triggering events inside process.nextTick Listing 2.23 Creating the illusion of an always asynchronous API Chapter Events: Mastering EventEmitter and beyond CuuDuongThanCong.com https://fb.com/tailieudientucntt Listing 4.1 Inheriting from EventEmitter Listing 4.2 Inheriting from EventEmitter Listing 4.3 Adding multiple listeners Listing 4.4 Removing listeners Listing 4.5 Mixing in EventEmitter Listing 4.6 utils.merge from Connect Listing 4.7 Event-based errors Listing 4.8 Managing errors with domain Listing 4.9 Keeping tabs on new listeners Listing 4.10 Automatically triggering events based on new listeners Listing 4.11 Querying listeners Listing 4.12 Reusing EventEmitter in Express Listing 4.13 Reusing EventEmitter in the redis module Listing 4.14 Categorizing event names using an object Listing 4.15 Using RabbitMQ with Node Listing 4.16 Using ØMQ with Node Listing 4.17 Using Redis Pub/Sub with Node Listing 4.18 Using Redis Pub/Sub with Node Chapter Streams: Node’s most powerful and misunderstood feature Listing 5.1 A simple static web server that uses streams Listing 5.2 A static web server with gzip Listing 5.3 Catching errors during streaming Listing 5.4 An Express application that uses streams Listing 5.5 Inheriting from the stream.Readable base class Listing 5.6 A JSON line parser Listing 5.7 A stream configured to use objectMode Listing 5.8 An example implementation of a writable stream Listing 5.9 A duplex stream Listing 5.10 A CSV parser implemented using a transform stream Listing 5.11 Benchmarking streams Listing 5.12 An old-style stream that has been wrapped Listing 5.13 Using isTTY to adapt stream behavior Listing 5.14 The CSVParser stream CuuDuongThanCong.com https://fb.com/tailieudientucntt Listing 5.15 Testing the CSVParser stream Chapter Networking: Node’s true “Hello, World” Listing 7.1 A simple TCP server Listing 7.2 Creating TCP clients to test servers Listing 7.3 Turning off Nagle’s algorithm Listing 7.4 A UDP client and server Listing 7.5 Sending messages back to clients Listing 7.6 A simple HTTP server Listing 7.7 Making an HTTP GET request that follows redirects Listing 7.8 Using the http module to create a proxy Listing 7.9 A TCP server that uses TLS for encryption Listing 7.10 A TCP client that uses TLS Listing 7.11 A basic HTTP server that uses TLS for encryption Listing 7.12 An example HTTPS client Chapter The Web: Build leaner and meaner web applications Listing 9.1 A quick static web server Listing 9.2 A Gruntfile for serving static files Listing 9.3 Scraping a web page with cheerio Listing 9.4 Node modules in the browser Listing 9.5 Node modules in the browser Listing 9.6 A routing module without the rest of the application Listing 9.7 A refactored app.js file Listing 9.8 Reloading a Node process Listing 9.9 Nodemon’s configuration file Listing 9.10 Configuring an Express application Listing 9.11 A JSON configuration file loader Listing 9.12 Loading the configuration directory Listing 9.13 Using nconf to configure an Express application Listing 9.14 Loading nconf elsewhere in the application Listing 9.15 Passing errors to middleware Listing 9.16 Inheriting errors and including status codes CuuDuongThanCong.com https://fb.com/tailieudientucntt Listing 9.17 Using an error-handling middleware component Listing 9.18 A RESTful resource in Express Listing 9.19 RESTful route handlers Listing 9.20 A restify application Listing 9.21 Restify routes Listing 9.22 Three types of middleware Listing 9.23 Using events to structure an application Listing 9.24 Emitting events Listing 9.25 An Express application that uses WebSockets Listing 9.26 The client-side WebSocket implementation Listing 9.27 Express middleware Listing 9.28 Express middleware Listing 9.29 Testing authenticated requests Listing 9.30 Taking control of middleware Listing 9.31 Injecting new behavior during tests Listing 9.32 A small web store that uses PayPal Listing 9.33 Mocking PayPal’s IPN requests Listing 9.34 Testing PayPal Chapter 10 Tests: The key to confident code Listing 10.1 The assert module Listing 10.2 Testing object equality Listing 10.3 Handling errors from asynchronous APIs Listing 10.4 Ensuring that exceptions are raised Listing 10.5 A custom assertion Listing 10.6 A package.json with a test script Listing 10.7 An example test file Listing 10.8 Running tests in a prescribed manner Listing 10.9 A simple Mocha test Listing 10.10 A sample module to test Listing 10.11 The Mocha sample project’s JSON file Listing 10.12 A Mocha test for a web application Listing 10.13 A web application that can square numbers Listing 10.14 The refactored Mocha test that uses SuperTest CuuDuongThanCong.com https://fb.com/tailieudientucntt Listing 10.15 Testing with TAP Listing 10.16 A simple test to try with Travis CI Listing 10.17 A basic package.json file Listing 10.18 Travis CI configuration Listing 10.19 The assert module Listing 10.20 Preparing test data with an ORM Listing 10.21 Stubbing a database Chapter 12 Node in production: Deploying applications safely Listing 12.1 Proxying requests to a Node application with Apache Listing 12.2 Running a program with runit Listing 12.3 Proxying requests to a Node application with nginx Listing 12.4 Managing a Node program with Upstart Listing 12.5 Adding WebSocket support to nginx Listing 12.6 Using HAProxy with a Node application Listing 12.7 Using HAProxy with WebSockets Listing 12.8 Redirecting traffic to another port with http-proxy Listing 12.9 Routing WebSocket connections separately Listing 12.10 Scaling using multiple instances of a server Listing 12.11 Clustering a Node web application Listing 12.12 Recovering from untimely worker death Listing 12.13 logrotate configuration CuuDuongThanCong.com https://fb.com/tailieudientucntt ... third-party modules 5.1.5 Streams inherit from EventEmitter 5.2 Built-in streams Technique 27 Using built-in streams to make a static web server Technique 28 Stream error handling 5.3 Third-party... with 2> error-file.log, the error messages would be redirected to error-file.log The other messages would be printed to the console as usual: node listings/globals/console-1.js 2> errors-file.log... Node’s non-blocking I/O would be an extremely cost-effective solution for this, because the server could make the best use of available I/O without you needing to write special low-level code