www.it-ebooks.info www.it-ebooks.info Simon Marlow Parallel and Concurrent Programming in Haskell www.it-ebooks.info Parallel and Concurrent Programming in Haskell by Simon Marlow Copyright © 2013 Simon Marlow. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or corporate@oreilly.com. Editors: Andy Oram and Maria Gulick Production Editor: Melanie Yarbrough Copyeditor: Gillian McGarvey Proofreader: Julie Van Keuren Indexer: WordCo Indexing Services Cover Designer: Randy Comer Interior Designer: David Futato Illustrator: Rebecca Demarest July 2013: First Edition Revision History for the First Edition: 2013-07-10: First release See http://oreilly.com/catalog/errata.csp?isbn=9781449335946 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Parallel and Concurrent Programming in Haskell, the image of a scrawled butterflyfish, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐ mark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-33594-6 [LSI] www.it-ebooks.info Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Terminology: Parallelism and Concurrency 2 Tools and Resources 3 Sample Code 4 Part I. Parallel Haskell 2. Basic Parallelism: The Eval Monad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Lazy Evaluation and Weak Head Normal Form 9 The Eval Monad, rpar, and rseq 15 Example: Parallelizing a Sudoku Solver 19 Deepseq 29 3. Evaluation Strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Parameterized Strategies 32 A Strategy for Evaluating a List in Parallel 34 Example: The K-Means Problem 35 Parallelizing K-Means 40 Performance and Analysis 42 Visualizing Spark Activity 46 Granularity 47 GC’d Sparks and Speculative Parallelism 48 Parallelizing Lazy Streams with parBuffer 51 Chunking Strategies 54 The Identity Property 55 4. Dataflow Parallelism: The Par Monad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 iii www.it-ebooks.info Example: Shortest Paths in a Graph 61 Pipeline Parallelism 65 Rate-Limiting the Producer 68 Limitations of Pipeline Parallelism 69 Example: A Conference Timetable 70 Adding Parallelism 74 Example: A Parallel Type Inferencer 77 Using Different Schedulers 82 The Par Monad Compared to Strategies 82 5. Data Parallel Programming with Repa. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Arrays, Shapes, and Indices 86 Operations on Arrays 88 Example: Computing Shortest Paths 90 Parallelizing the Program 93 Folding and Shape-Polymorphism 95 Example: Image Rotation 97 Summary 101 6. GPU Programming with Accelerate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Overview 104 Arrays and Indices 105 Running a Simple Accelerate Computation 106 Scalar Arrays 108 Indexing Arrays 108 Creating Arrays Inside Acc 109 Zipping Two Arrays 111 Constants 111 Example: Shortest Paths 112 Running on the GPU 115 Debugging the CUDA Backend 116 Example: A Mandelbrot Set Generator 116 Part II. Concurrent Haskell 7. Basic Concurrency: Threads and MVars. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 A Simple Example: Reminders 126 Communication: MVars 128 MVar as a Simple Channel: A Logging Service 130 MVar as a Container for Shared State 133 MVar as a Building Block: Unbounded Channels 135 iv | Table of Contents www.it-ebooks.info Fairness 140 8. Overlapping Input/Output. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Exceptions in Haskell 146 Error Handling with Async 151 Merging 152 9. Cancellation and Timeouts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Asynchronous Exceptions 156 Masking Asynchronous Exceptions 158 The bracket Operation 162 Asynchronous Exception Safety for Channels 162 Timeouts 164 Catching Asynchronous Exceptions 166 mask and forkIO 168 Asynchronous Exceptions: Discussion 170 10. Software Transactional Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Running Example: Managing Windows 173 Blocking 177 Blocking Until Something Changes 179 Merging with STM 181 Async Revisited 182 Implementing Channels with STM 184 More Operations Are Possible 185 Composition of Blocking Operations 185 Asynchronous Exception Safety 186 An Alternative Channel Implementation 187 Bounded Channels 189 What Can We Not Do with STM? 191 Performance 193 Summary 195 11. Higher-Level Concurrency Abstractions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Avoiding Thread Leakage 197 Symmetric Concurrency Combinators 199 Timeouts Using race 201 Adding a Functor Instance 202 Summary: The Async API 203 12. Concurrent Network Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 A Trivial Server 205 Table of Contents | v www.it-ebooks.info Extending the Simple Server with State 209 Design One: One Giant Lock 209 Design Two: One Chan Per Server Thread 210 Design Three: Use a Broadcast Chan 211 Design Four: Use STM 212 The Implementation 213 A Chat Server 216 Architecture 217 Client Data 217 Server Data 218 The Server 219 Setting Up a New Client 219 Running the Client 222 Recap 223 13. Parallel Programming Using Threads. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 How to Achieve Parallelism with Concurrency 225 Example: Searching for Files 226 Sequential Version 226 Parallel Version 228 Performance and Scaling 230 Limiting the Number of Threads with a Semaphore 231 The ParIO monad 237 14. Distributed Programming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 The Distributed-Process Family of Packages 242 Distributed Concurrency or Parallelism? 244 A First Example: Pings 244 Processes and the Process Monad 245 Defining a Message Type 245 The Ping Server Process 246 The Master Process 248 The main Function 249 Summing Up the Ping Example 250 Multi-Node Ping 251 Running with Multiple Nodes on One Machine 252 Running on Multiple Machines 253 Typed Channels 254 Merging Channels 257 Handling Failure 258 The Philosophy of Distributed Failure 261 A Distributed Chat Server 262 vi | Table of Contents www.it-ebooks.info Data Types 263 Sending Messages 265 Broadcasting 265 Distribution 266 Testing the Server 269 Failure and Adding/Removing Nodes 269 Exercise: A Distributed Key-Value Store 271 15. Debugging, Tuning, and Interfacing with Foreign Code. . . . . . . . . . . . . . . . . . . . . . . . . 275 Debugging Concurrent Programs 275 Inspecting the Status of a Thread 275 Event Logging and ThreadScope 276 Detecting Deadlock 278 Tuning Concurrent (and Parallel) Programs 280 Thread Creation and MVar Operations 281 Shared Concurrent Data Structures 283 RTS Options to Tweak 284 Concurrency and the Foreign Function Interface 286 Threads and Foreign Out-Calls 286 Asynchronous Exceptions and Foreign Calls 288 Threads and Foreign In-Calls 289 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Table of Contents | vii www.it-ebooks.info www.it-ebooks.info [...]... chapters require the installation of additional packages Instructions for installing the extra dependencies can be found in “Sample Code” on page 4 Additionally, I recommend installing ThreadScope ThreadScope is a tool for visualizing the execution of Haskell programs and is particularly useful for gaining insight into the behavior of Parallel and Concurrent Haskell code On a Linux system, ThreadScope... Even this, however, has not been terribly successful in practice Fully automatic parallelization is still a pipe dream However, the parallel programming models provided by Haskell do succeed in eliminating some mundane or error-prone aspects traditionally associated with parallel programming: • Parallel programming in Haskell is deterministic: The parallel program always pro‐ duces the same answer,... expert content in both book and video form from the world’s lead‐ ing authors in technology and business Technology professionals, software developers, web designers, and business and crea‐ tive professionals use Safari Books Online as their primary resource for research, prob‐ lem solving, learning, and certification training Safari Books Online offers a range of product mixes and pricing programs for... do have side effects, and then there is no alternative but to use nondeterministic parallel or concurrent programming Finally, it is entirely reasonable to want to mix parallelism and concurrency in the same program Most interactive programs need to use concurrency to maintain a responsive user interface while compute-intensive tasks are being performed in the background Tools and Resources To try out... to use parallelism and concurrency in Haskell were con‐ spicuously absent Over the last couple of years, I was fortunate to have had the oppor‐ tunity to teach two summer school courses on parallel and concurrent programming in Haskell: one at the Central European Functional Programming (CEFP) 2011 summer school in Budapest, and the other the CEA/EDF/INRIA 2012 Summer School at Ca‐ darache in the south... implementation, and I’m sure that this decision was instrumental in bringing about GHC’s solid and lightning-fast concurrency support Haskell has a long tradition of being associated with parallelism To name just a few of the projects, there was the pH variant of Haskell derived from the Id language, which was designed for parallelism, the GUM system for running parallel Haskell programs on multiple machines in. .. distinction is between deterministic and nondeterministic programming mod‐ els A deterministic programming model is one in which each program can give only one result, whereas a nondeterministic programming model admits programs that may have different results, depending on some aspect of the execution Concurrent pro‐ gramming models are necessarily nondeterministic because they must interact with external... indicates where the parallelism is, and the details of actually running the program in parallel are left to the runtime system This is both a blessing and a curse: — By embodying fewer operational details, parallel Haskell programs are abstract and are therefore likely to work on a wide range of parallel hardware — Parallel Haskell programs can take advantage of existing highly tuned technol‐ ogy in. .. describe the various parallel programming models that Haskell provides: • Chapters 2 and 3 introduce the Eval monad and Evaluation Strategies, which are suitable for expressing parallelism in Haskell programs that are not heavily nu‐ merical or array-based These programming models are well established, and there are many good examples of using them to achieve parallelism • Chapter 4 introduces the Par... parallel programming in any language Hopefully by the end you’ll have built up enough of an intuition for parallel programming that you’ll be able to achieve decent parallel speedups in your own code using the techniques covered Keep in mind while reading this part of the book that obtaining reliable results with parallelism is inherently difficult because in today’s complex computing devices, per‐ formance . www.it-ebooks.info www.it-ebooks.info Simon Marlow Parallel and Concurrent Programming in Haskell www.it-ebooks.info Parallel and Concurrent Programming in Haskell by Simon Marlow Copyright. school courses on parallel and concurrent programming in Haskell: one at the Central European Functional Programming (CEFP) 2011 summer school in Budapest, and the other the CEA/EDF/INRIA 2012 Summer. to install the Haskell Platform. The Haskell Platform includes the GHC compiler and all the important libraries, including the parallel and concurrent libraries we shall be using. The code in