www.it-ebooks.info The world is parallel. If we want to write programs that behave as other objects behave in the real world, then these programs will have a concurrent st ructure. Use a language that was designed for writing concurrent applications, and development becomes a lot easier. Erlang programs model how we think and interact. Joe Armstrong www.it-ebooks.info Programmin g Erlang Software for a Concurrent World Joe Armstrong The Pragmatic Bookshelf Raleigh, North Carolina Dallas, Texas www.it-ebooks.info Many of the designations used by manufacturers and sellers to distinguish their prod- ucts are claimed as trademarks. Where those designations appear in this book, and The Pragmatic Progra mmers, LLC was aware of a trademark claim, the designations have been printed in initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer, Pragmatic Programming, Pragmatic Bookshelf and the linking g device are trademarks of The Pragmatic Programmers, LLC. Every precaution was taken in the preparation of this book. However, the publisher assumes no re sponsibility for errors or omissions, or for damages that may result from the use of information (including program listings) contained herein. Our Pragmatic courses, workshops, and other products can help you and your team create better software and ha ve more fun. For more information, as well as the latest Pragmatic titles, please visit us at http://www.pragmaticprogrammer.com Copyright © 2 007 armstrongonsoftware. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmit- ted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. Printed in the United States of America. ISBN-10: 1-9343560-0-X ISBN-13: 978-1-934356-00-5 Printed on acid-free paper with 50% recycled, 15% post-consumer content. P1.1 printing, July, 2007 Version: 2007-7-17 www.it-ebooks.info Contents 1 Begin 12 1.1 Road Map . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.2 Begin Again . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 17 2 Getting Started 18 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2 Installing Erlang . . . . . . . . . . . . . . . . . . . . . . 21 2.3 The Code in This Book . . . . . . . . . . . . . . . . . . . 23 2.4 Starting the Shell . . . . . . . . . . . . . . . . . . . . . . 24 2.5 Simple Integer Arithmetic . . . . . . . . . . . . . . . . . 25 2.6 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.7 Floating-Point Numbers . . . . . . . . . . . . . . . . . . 32 2.8 Atoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.9 Tuples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.10 Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.11 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.12 Patter n Matching Again . . . . . . . . . . . . . . . . . . 41 3 Sequential Programming 43 3.1 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.2 Back to Shopping . . . . . . . . . . . . . . . . . . . . . . 49 3.3 Functions with the Same Name and Different Arity . . 52 3.4 Funs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.5 Simple List Processing . . . . . . . . . . . . . . . . . . . 58 3.6 List Comprehensions . . . . . . . . . . . . . . . . . . . . 61 3.7 Arithmetic Expressions . . . . . . . . . . . . . . . . . . 64 3.8 Guards . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.9 Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.10 case and if Expressions . . . . . . . . . . . . . . . . . . 72 3.11 Building Lists in Natural Order . . . . . . . . . . . . . . 73 3.12 Accumulators . . . . . . . . . . . . . . . . . . . . . . . . 74 www.it-ebooks.info CONTENTS 6 4 Exceptions 76 4.1 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . 76 4.2 Raising an Exception . . . . . . . . . . . . . . . . . . . . 77 4.3 try catch . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.4 catch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.5 Improving Error Messages . . . . . . . . . . . . . . . . . 82 4.6 Programming Style with try catch . . . . . . . . . . . . 82 4.7 Catching Every Possible Exception . . . . . . . . . . . . 83 4.8 Old- and New-Style Exception Handling . . . . . . . . . 84 4.9 Stack Traces . . . . . . . . . . . . . . . . . . . . . . . . . 84 5 Advanced Sequential Programming 86 5.1 BIFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.2 Binaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.3 The Bit Syntax . . . . . . . . . . . . . . . . . . . . . . . 89 5.4 Miscellaneous Short Topics . . . . . . . . . . . . . . . . 98 6 Compiling and Running Your Program 118 6.1 Starting and Stopping the Erlang Shell . . . . . . . . . 118 6.2 Modifying the Development Environment . . . . . . . . 119 6.3 Different Ways to Run Your Program . . . . . . . . . . . 122 6.4 Automating Compilation with Makefiles . . . . . . . . . 127 6.5 Command Editing in the Erlang Shell . . . . . . . . . . 130 6.6 Getting Out of Trouble . . . . . . . . . . . . . . . . . . . 131 6.7 When Things Go Wrong . . . . . . . . . . . . . . . . . . 131 6.8 Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.9 Tweaking the Environment . . . . . . . . . . . . . . . . 135 6.10 The Crash Dump . . . . . . . . . . . . . . . . . . . . . . 136 7 Concurrency 137 8 Concurrent Programming 141 8.1 The Concurrency Primitives . . . . . . . . . . . . . . . . 142 8.2 A Simple Example . . . . . . . . . . . . . . . . . . . . . 143 8.3 Client-Server—An Introduction . . . . . . . . . . . . . . 144 8.4 How Long Does It Take to Create a Process? . . . . . . 148 8.5 Receive with a Timeout . . . . . . . . . . . . . . . . . . 150 8.6 Selective Receive . . . . . . . . . . . . . . . . . . . . . . 153 8.7 Registered Processes . . . . . . . . . . . . . . . . . . . . 154 8.8 How Do We Write a Concurrent Program? . . . . . . . . 156 8.9 A Word About Tail R ecursion . . . . . . . . . . . . . . . 156 8.10 Spawning with MFAs . . . . . . . . . . . . . . . . . . . . 157 8.11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Report erratum this copy is (P1.1 printing, July, 2007) www.it-ebooks.info CONTENTS 7 9 Errors in Concurrent Programs 159 9.1 Linking Processes . . . . . . . . . . . . . . . . . . . . . . 159 9.2 An on_exit Handler . . . . . . . . . . . . . . . . . . . . . 160 9.3 Remote Handling of Errors . . . . . . . . . . . . . . . . 162 9.4 The Details of Error Handling . . . . . . . . . . . . . . . 162 9.5 Error Handling Primitives . . . . . . . . . . . . . . . . . 170 9.6 Sets of Linked Processes . . . . . . . . . . . . . . . . . . 172 9.7 Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 9.8 A Keep-Alive Process . . . . . . . . . . . . . . . . . . . . 173 10 Distributed Programming 175 10.1 The Name Server . . . . . . . . . . . . . . . . . . . . . . 177 10.2 The Distribution Primitives . . . . . . . . . . . . . . . . 182 10.3 Libraries for Distributed Pr ogramming . . . . . . . . . 185 10.4 The Cookie Protection System . . . . . . . . . . . . . . . 186 10.5 Socket-Based Distri bution . . . . . . . . . . . . . . . . . 187 11 IRC Lite 191 11.1 Message Sequence Diagrams . . . . . . . . . . . . . . . 193 11.2 The User Interface . . . . . . . . . . . . . . . . . . . . . 194 11.3 Client-Side Software . . . . . . . . . . . . . . . . . . . . 195 11.4 Server-Side Software . . . . . . . . . . . . . . . . . . . . 199 11.5 Running the Application . . . . . . . . . . . . . . . . . . 203 11.6 The Chat Program Source Code . . . . . . . . . . . . . . 204 11.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 12 Interfacing Techniques 212 12.1 Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 12.2 Interfacing an External C Program . . . . . . . . . . . . 214 12.3 open_port . . . . . . . . . . . . . . . . . . . . . . . . . . 220 12.4 Linked-in Drivers . . . . . . . . . . . . . . . . . . . . . . 221 12.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 13 Programming with Files 226 13.1 Organization of the Libraries . . . . . . . . . . . . . . . 226 13.2 The Different Ways of Reading a File . . . . . . . . . . . 227 13.3 The Different Ways of Writing to a File . . . . . . . . . . 235 13.4 Directory Operations . . . . . . . . . . . . . . . . . . . . 239 13.5 Finding Information About a File . . . . . . . . . . . . . 240 13.6 Copying and Deleting Files . . . . . . . . . . . . . . . . 241 13.7 Bits and Pieces . . . . . . . . . . . . . . . . . . . . . . . 241 13.8 A Find Utility . . . . . . . . . . . . . . . . . . . . . . . . 242 Report erratum this copy is (P1.1 printing, July, 2007) www.it-ebooks.info CONTENTS 8 14 Programming with Sockets 245 14.1 Using TCP . . . . . . . . . . . . . . . . . . . . . . . . . . 246 14.2 Control Issues . . . . . . . . . . . . . . . . . . . . . . . . 255 14.3 Where Did That Connection Come From? . . . . . . . . 258 14.4 Error Handling with Sockets . . . . . . . . . . . . . . . 259 14.5 UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 14.6 Broadcasting to Multiple Machines . . . . . . . . . . . 263 14.7 A SHOUTcast Server . . . . . . . . . . . . . . . . . . . . 265 14.8 Digging Deeper . . . . . . . . . . . . . . . . . . . . . . . 272 15 ETS and DETS: Large Data Storage Mechanisms 273 15.1 Basic Operations on Tables . . . . . . . . . . . . . . . . 274 15.2 Types of Table . . . . . . . . . . . . . . . . . . . . . . . . 275 15.3 ETS Table Efficiency Considerations . . . . . . . . . . . 276 15.4 Creating an ETS Table . . . . . . . . . . . . . . . . . . . 277 15.5 Example Progr ams with ETS . . . . . . . . . . . . . . . 279 15.6 DETS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 15.7 What Haven’t We Talked About? . . . . . . . . . . . . . 287 15.8 Code Listin gs . . . . . . . . . . . . . . . . . . . . . . . . 288 16 OTP Introduction 291 16.1 The Road to the Generic Server . . . . . . . . . . . . . . 292 16.2 Getting Started with gen_server . . . . . . . . . . . . . 301 16.3 The gen_server Callback Structure . . . . . . . . . . . . 305 16.4 Code and Templates . . . . . . . . . . . . . . . . . . . . 309 16.5 Digging Deeper . . . . . . . . . . . . . . . . . . . . . . . 312 17 Mnesia: The Erlang Database 313 17.1 Database Queries . . . . . . . . . . . . . . . . . . . . . . 313 17.2 Adding and Removing Data in the Database . . . . . . 317 17.3 Mnesia Transactions . . . . . . . . . . . . . . . . . . . . 319 17.4 Storing Complex Data in Tables . . . . . . . . . . . . . 323 17.5 Table Types and Location . . . . . . . . . . . . . . . . . 325 17.6 Creating the Initial Database . . . . . . . . . . . . . . . 328 17.7 The Table Viewer . . . . . . . . . . . . . . . . . . . . . . 329 17.8 Digging Deeper . . . . . . . . . . . . . . . . . . . . . . . 329 17.9 Listings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 Report erratum this copy is (P1.1 printing, July, 2007) www.it-ebooks.info CONTENTS 9 18 Making a System with OTP 335 18.1 Generic Event Handling . . . . . . . . . . . . . . . . . . 336 18.2 The Error Logger . . . . . . . . . . . . . . . . . . . . . . 339 18.3 Alarm Management . . . . . . . . . . . . . . . . . . . . . 346 18.4 The Application Ser vers . . . . . . . . . . . . . . . . . . 348 18.5 The Supervision Tree . . . . . . . . . . . . . . . . . . . . 351 18.6 Starting the System . . . . . . . . . . . . . . . . . . . . 354 18.7 The Application . . . . . . . . . . . . . . . . . . . . . . . 358 18.8 File System Organization . . . . . . . . . . . . . . . . . 360 18.9 The Application Monitor . . . . . . . . . . . . . . . . . . 361 18.10 Digging Deeper . . . . . . . . . . . . . . . . . . . . . . . 361 18.11 How Did We Make That Prime? . . . . . . . . . . . . . . 363 19 Multicore Prelude 365 20 Programming Multicore CPUs 367 20.1 How to Make Progr ams Run Efficiently on a Multicore CPU 368 20.2 Parallelizing Sequential Code . . . . . . . . . . . . . . . 372 20.3 Small Messages, Big Computations . . . . . . . . . . . 375 20.4 mapreduce and Indexing Our Disk . . . . . . . . . . . . 379 20.5 Growing Into the Future . . . . . . . . . . . . . . . . . . 389 A Documenting Our Program 390 A.1 Erlang Type Notation . . . . . . . . . . . . . . . . . . . . 391 A.2 Tools That Use Types . . . . . . . . . . . . . . . . . . . . 394 B Erlang on Microsoft Windows 396 B.1 E rlang . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 B.2 Fetch and Install MinGW . . . . . . . . . . . . . . . . . 396 B.3 Fetch and Install MSYS . . . . . . . . . . . . . . . . . . 397 B.4 Install th e MSYS Developer Toolkit (Optional) . . . . . 397 B.5 E macs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 C Resources 399 C.1 Online Documentation . . . . . . . . . . . . . . . . . . . 399 C.2 Books and Theses . . . . . . . . . . . . . . . . . . . . . 400 C.3 Li nk Collections . . . . . . . . . . . . . . . . . . . . . . . 400 C.4 Blogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 C.5 Forums, Onlin e Communities, and Social Sites . . . . 401 C.6 Conferences . . . . . . . . . . . . . . . . . . . . . . . . . 401 C.7 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 C.8 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . 402 Report erratum this copy is (P1.1 printing, July, 2007) www.it-ebooks.info CONTENTS 10 D A Socket Application 403 D.1 An Example . . . . . . . . . . . . . . . . . . . . . . . . . 403 D.2 How l i b_chan Works . . . . . . . . . . . . . . . . . . . . 406 D.3 The lib_chan Code . . . . . . . . . . . . . . . . . . . . . 409 E Miscellaneous 419 E.1 Analysis and Profiling Tools . . . . . . . . . . . . . . . . 419 E.2 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . 422 E.3 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 E.4 Dynamic Code Loading . . . . . . . . . . . . . . . . . . . 435 F Module and Function Reference 439 F.1 Module: application . . . . . . . . . . . . . . . . . . . . . 439 F.2 Module: base64 . . . . . . . . . . . . . . . . . . . . . . . 440 F.3 Module: beam_lib . . . . . . . . . . . . . . . . . . . . . . 441 F.4 Module: c . . . . . . . . . . . . . . . . . . . . . . . . . . 441 F.5 Module: calendar . . . . . . . . . . . . . . . . . . . . . . 443 F.6 Module: code . . . . . . . . . . . . . . . . . . . . . . . . 444 F.7 Module: dets . . . . . . . . . . . . . . . . . . . . . . . . . 445 F.8 Module: dict . . . . . . . . . . . . . . . . . . . . . . . . . 448 F.9 Module: digraph . . . . . . . . . . . . . . . . . . . . . . . 449 F.10 Module: digraph_utils . . . . . . . . . . . . . . . . . . . 450 F.11 Module: disk_log . . . . . . . . . . . . . . . . . . . . . . 451 F.12 Module: epp . . . . . . . . . . . . . . . . . . . . . . . . . 452 F.13 Module: erl_eval . . . . . . . . . . . . . . . . . . . . . . . 453 F.14 Module: erl_parse . . . . . . . . . . . . . . . . . . . . . . 453 F.15 Module: erl_pp . . . . . . . . . . . . . . . . . . . . . . . 454 F.16 Module: erl_scan . . . . . . . . . . . . . . . . . . . . . . 454 F.17 Module: erl_tar . . . . . . . . . . . . . . . . . . . . . . . 454 F.18 Module: erlang . . . . . . . . . . . . . . . . . . . . . . . 455 F.19 Module: error_handler . . . . . . . . . . . . . . . . . . . 464 F.20 Module: error_logger . . . . . . . . . . . . . . . . . . . . 464 F.21 Module: ets . . . . . . . . . . . . . . . . . . . . . . . . . 465 F.22 Module: file . . . . . . . . . . . . . . . . . . . . . . . . . 468 F.23 Module: file_sorter . . . . . . . . . . . . . . . . . . . . . 470 F.24 Module: filelib . . . . . . . . . . . . . . . . . . . . . . . . 471 F.25 Module: filename . . . . . . . . . . . . . . . . . . . . . . 471 F.26 Module: gb_sets . . . . . . . . . . . . . . . . . . . . . . . 472 F.27 Module: gb_trees . . . . . . . . . . . . . . . . . . . . . . 474 F.28 Module: gen_event . . . . . . . . . . . . . . . . . . . . . 475 F.29 Module: gen_fsm . . . . . . . . . . . . . . . . . . . . . . 476 Report erratum this copy is (P1.1 printing, July, 2007) www.it-ebooks.info [...]... probably met before, Erlang is a concurrent programming language—this makes it particularly suited for writing distributed programs and for programming modern multicore and SMP2 computers Most Erlang programs will just run faster when run on a multicore or SMP machine Erlang programming involves using a programming paradigm that I call concurrency-oriented programming (COP) 1 2 Open Telecom Platform... functional programming Functional programming forbids code with side effects Side effects and concurrency don’t mix You can have sequential code with side effects, or you can have code and concurrency that is free from side effects You have to choose There is no middle way Erlang is a language where concurrency belongs to the programming language and not the operating system Erlang makes parallel programming. .. page 365 is a short introduction to why Erlang is suited for programming multicore computers We talk in general terms about shared memory and message passing concurrency and why we strongly believe that languages with no mutable state and concurrency are ideally suited to programming multicore computers • Chapter 20, Programming Multicore CPUs, on page 367 is about programming multicore computers We talk... Interfacing Techniques, on page 212 is all about interfacing Erlang to foreign-language code • Chapter 13, Programming with Files, on page 226 has numerous examples of programming with files • Chapter 14, Programming with Sockets, on page 245 shows you how to program with sockets We’ll look at how to build sequential and parallel servers in Erlang We finish this chapter with the second sizable application:... building the system Use CEAN The Comprehensive Erlang Archive Network (CEAN) is an attempt to gather all the major Erlang applications in one place with a common installer The advantage of using CEAN is that it manages not only the basic Erlang system but a large number of packages written in Erlang This means that as well as being able to keep your basic Erlang installation up-to-date, you’ll be able... Command Editing in the Erlang Shell, on page 130 Best of all, when you start writing distributed programs, you will find that you can attach a shell to a running Erlang system on a different Erlang node in a cluster or even make an secure shell (ssh) connection directly to an Erlang system running on a remote computer Using this, you can interact with any program on any node in a system of Erlang nodes Warning:... Chapter 3, Sequential Programming, on page 43 is the first of two chapters on sequential programming It introduces the ideas of pattern matching and of nondestructive assignments • Chapter 4, Exceptions, on page 76 is about exception handling No program is error free This chapter is about detecting and handling errors in sequential Erlang programs • Chapter 5, Advanced Sequential Programming, on page... you need to remember throughout this book: programming is fun And I personally think programming distributed applications such as chat programs or instant messaging applications is a lot more fun than programming conventional sequential applications What you can do on one computer is limited, but what you can do with networks of computers becomes unlimited Erlang provides an ideal environment for experimenting... www.it-ebooks.info I NSTALLING E RLANG 2.2 Installing Erlang Before you can do anything, you have to make sure you have a functioning version of Erlang on your system Go to a command prompt, and type erl: $ erl Erlang (BEAM) emulator version 5.5.2 [source] [kernel-poll:false] Eshell V5.5.2 1> (abort with ^G) On a Windows system, the command erl works only if you have installed Erlang and changed the PATH environment... variable to refer to the program Assuming you’ve installed the program in the standard way, you’ll invoke Erlang through the Start > All Programs > Erlang OTP menu In Appendix B, on page 396, I’ll describe how I’ve rigged Erlang to run with MinGW and MSYS Note: I’ll show the banner (the bit that says Erlang (BEAM) (abort with ∧G)”) only occasionally This information is useful only if you want to report . handling errors in sequential Erlang programs. • Chapter 5, A dvanced Sequential Programming, on page 86 is the second chapter on sequential Erlang programming . It. inter- facing Erlang to foreign-language code. • Chapter 13, Programming with Files, on page 226 has numerous examples of programming with files. • Chapter 14, Programming