• • • • • • Table of Contents Index Reviews Reader Reviews Errata Academic High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI By Joseph D Sloan Publisher : O'Reilly Pub Date : November 2004 ISBN : 0-596-00570-9 Pages : 360 This new guide covers everything you need to plan, build, and deploy a high-performance Linux cluster You'll learn about planning, hardware choices, bulk installation of Linux on multiple systems, and other basic considerations Learn about the major free software projects and how to choose those that are most helpful to new cluster administrators and programmers Guidelines for debugging, profiling, performance tuning, and managing jobs from multiple users round out this immensely useful book • • • • • • Table of Contents Index Reviews Reader Reviews Errata Academic High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI By Joseph D Sloan Publisher : O'Reilly Pub Date : November 2004 ISBN : 0-596-00570-9 Pages : 360 Copyright Preface Audience Organization Conventions Using Code Examples How to Contact Us Acknowledgments Part I: An Introduction to Clusters Chapter 1 Cluster Architecture Section 1.1 Modern Computing and the Role of Clusters Section 1.2 Types of Clusters Section 1.4 Limitations Section 1.3 Distributed Computing and Clusters Section 1.5 My Biases Chapter 2 Cluster Planning Section 2.1 Design Steps Section 2.3 Architecture and Cluster Software Section 2.5 CD-ROM-Based Clusters Section 2.2 Determining Your Cluster's Mission Section 2.4 Cluster Kits Section 2.6 Benchmarks Chapter 3 Cluster Hardware Section 3.2 Environment Section 3.1 Design Decisions Chapter 4 Linux for Clusters Section 4.1 Installing Linux Section 4.2 Configuring Services Section 4.3 Cluster Security Part II: Getting Started Quickly Chapter 5 openMosix Section 5.1 What Is openMosix? Section 5.3 Selecting an Installation Approach Section 5.5 Using openMosix Section 5.7 Is openMosix Right for You? Section 5.2 How openMosix Works Section 5.4 Installing a Precompiled Kernel Section 5.6 Recompiling the Kernel Chapter 6 OSCAR Section 6.1 Why OSCAR? Section 6.3 Installing OSCAR Section 6.5 Using switcher Section 6.2 What's in OSCAR Section 6.4 Security and OSCAR Section 6.6 Using LAM/MPI with OSCAR Chapter 7 Rocks Section 7.1 Installing Rocks Section 7.2 Managing Rocks Section 7.3 Using MPICH with Rocks Part III: Building Custom Clusters Chapter 8 Cloning Systems Section 8.1 Configuring Systems Section 8.2 Automating Installations Section 8.3 Notes for OSCAR and Rocks Users Chapter 9 Programming Software Section 9.2 Selecting a Library Section 9.4 MPICH Section 9.6 Notes for OSCAR Users Section 9.1 Programming Languages Section 9.3 LAM/MPI Section 9.5 Other Programming Software Section 9.7 Notes for Rocks Users Chapter 10 Management Software Section 10.1 C3 Section 10.2 Ganglia Section 10.3 Notes for OSCAR and Rocks Users Chapter 11 Scheduling Software Section 11.1 OpenPBS Section 11.2 Notes for OSCAR and Rocks Users Chapter 12 Parallel Filesystems Section 12.1 PVFS Section 12.2 Using PVFS Section 12.3 Notes for OSCAR and Rocks Users Part IV: Cluster Programming Chapter 13 Getting Started with MPI Section 13.1 MPI Section 13.3 An MPI Solution Section 13.5 Broadcast Communications Section 13.2 A Simple Problem Section 13.4 I/O with MPI Chapter 14 Additional MPI Features Section 14.1 More on Point-to-Point Communication Section 14.3 Managing Communicators Section 14.2 More on Collective Communication Section 14.4 Packaging Data Chapter 15 Designing Parallel Programs Section 15.2 Problem Decomposition Section 15.4 Other Considerations Section 15.1 Overview Section 15.3 Mapping Tasks to Processors Chapter 16 Debugging Parallel Programs Section 16.1 Debugging and Parallel Programs Section 16.2 Avoiding Problems Section 16.3 Programming Tools Section 16.5 Tracing with printf Section 16.7 Using gdb and ddd with MPI Section 16.4 Rereading Code Section 16.6 Symbolic Debuggers Section 16.8 Notes for OSCAR and Rocks Users Chapter 17 Profiling Parallel Programs Section 17.1 Why Profile? Section 17.2 Writing and Optimizing Code Section 17.4 Timing C Code Segments Section 17.6 MPE Section 17.3 Timing Complete Programs Section 17.5 Profilers Section 17.7 Customized MPE Logging Section 17.8 Notes for OSCAR and Rocks Users Part V: Appendix Appendix A References Section A.1 Books Section A.2 URLs Colophon Index Copyright © 2005 O'Reilly Media, Inc All rights reserved Printed in the United States of America Published by O'Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O'Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://safari.oreilly.com) For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly Media, Inc The Linux series designations, High Performance Linux Clusters with OSCAR, Rocks, openMosix, and MPI, images of the American West, and related trade dress are trademarks of O'Reilly Media, Inc Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in this book, and O'Reilly Media, Inc was aware of a trademark claim, the designations have been printed in caps or initial caps While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein Preface Clusters built from open source software, particularly based on the GNU/Linux operating system, are increasingly popular Their success is not hard to explain because they can cheaply solve an ever-widening range of number-crunching applications A wealth of open source or free software has emerged to make it easy to set up, administer, and program these clusters Each individual package is accompanied by documentation, sometimes very rich and thorough But knowing where to start and how to get the different pieces working proves daunting for many programmers and administrators This book is an overview of the issues that new cluster administrators have to deal with in making clusters meet their needs, ranging from the initial hardware and software choices through long-term considerations such as performance This book is not a substitute for the documentation that accompanies the software that it describes You should download and read the documentation for the software Most of the documentation available online is quite good; some is truly excellent In writing this book, I have evaluated a large number of programs and selected for inclusion the software I believe is the most useful for someone new to clustering While writing descriptions of that software, I culled through thousands of pages of documentation to fashion a manageable introduction This book brings together the information you'll need to get started After reading it, you should have a clear idea of what is possible, what is available, and where to go to get it While this book doesn't stand alone, it should reduce the amount of work you'll need to do I have tried to write the sort of book I would have wanted when I got started with clusters The software described in this book is freely available, open source software All of the software is available for use with Linux; however, much of it should work nicely on other platforms as well All of the software has been installed and tested as described in this book However, the behavior or suitability of the software described in this book cannot be guaranteed While the material in this book is presented in good faith, neither the author nor O'Reilly Media, Inc makes any explicit or implied warranty as to the behavior or suitability of this software We strongly urge you to evaluate the software and information provided in this book as appropriate for your own circumstances One of the more important developments in the short life of high performance clusters has been the creation of cluster installation kits such as OSCAR and Rocks With software packages like these, it is possible to install everything you need and very quickly have a fully functional cluster For this reason, OSCAR and Rocks play a central role in this book OSCAR and Rocks are composed of a number of different independent packages, as well as customizations available only with each kit A fully functional cluster will have a number of software packages each addressing a different need, such as programming, management, and scheduling OSCAR and Rocks use a best-in-category approach, selecting the best available software for each type of cluster-related task In addition to the core software, other compatible packages are available as well Consequently, you will often have several products to choose from for any given need Most of the software included in OSCAR or Rocks is significant in its own right Such software is often nontrivial to install and takes time to learn to use to its full potential While both OSCAR and Rocks automate the installation process, there is still a lot to learn to effectively use either kit Installing OSCAR or Rocks is only the beginning After some basic background information, this book describes the installation of OSCAR and then Rocks The remainder of the book describes in greater detail much of the software found in these packages In each case, I describe the installation, configuration, and use of the software apart from OSCAR or Rocks This should provide the reader with the information he will need to customize the software or even build a custom cluster bypassing OSCAR or Rocks completely, if desired I have also included a chapter on openMosix in this book, which may seem an odd choice to some But there are several compelling reasons for including this information First, not everyone needs a world-class high-performance cluster If you have several machines and would like to use them together, but don't want the headaches that can come with a full cluster, openMosix is worth investigating Second, openMosix is a nice addition to some more traditional clusters Including openMosix also provides an opportunity to review recompiling the Linux kernel and an alternative kernel that can be used to demonstrate OSCAR's kernel_picker Finally, I think openMosix is a really nice piece of software In a sense, it represents the future, or at least one possible future, for clusters I have described in detail (too much, some might say) exactly how I have installed the software Unquestionably, by the time you read, this some of the information will be dated I have decided not to follow the practice of many authors in such situations, and offer just vague generalities I feel that readers benefit from seeing the specific sorts of problems that appear in specific installations and how to think about their solutions asymptotic analysis code optimization and gcc and -O compiler flag justification for MPE MPI, gprof, and gcov OSCAR and Rocks and timing C-language code segments gettimeofday function MPI, manual timing [See MPI, code timing with] time system call timing programs profilers [See profilers] writing code versus optimization programming software 2nd choosing debuggers HDF5 LAM/MPI installing testing user configuration using library selection MPI PVM MPICH installing MPE (Multi-Processing Environment) on Windows systems testing user configuration using OSCAR and programming languages Rocks and SPRNG proprietary clusters ps command pseudorandom number generators public interface purify pushimage command PVFS (Parallel Virtual File System) 2nd 3rd advantages and disadvantages architecture client setup mknod command pvfstab file useful utilities cluster partitioning downloading head node configuration head node, installing on I/O server setup metadata server, configuring needed patches OSCAR and Rocks ownerships and permissions, setting running daemons, starting up insmod command iod-ping and mgr-ping utilities mount command ps command troubleshooting striping scheme and parameters using pvstat utility ROMIO interface u2p utility ỊUsing the Parallel Virtual File SystemĨ documentation web site pvfstab file (PVFS) PVM (Parallel Virtual Machine) library pvstat utility (PVFS) [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] qdel command (PBS) qmgr command (PBS) QsNet qstat command (PBS) qsub command (PBS) [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] racks versus shelving random numbers in parallel program design libraries for random number generation SPRNG (Scalable Parallel Random Number Generators) recon tool Red Hat Linux Rocks and replace-auto-partition.xml replicant client images RISC (reduced instruction set computer) architecture Rocks 2nd 3rd 4th cluster-fork command 2nd ddd and diskless clusters and downloading available rolls frontend or head node Ganglia and gdb and Grids and insert-ethers program installing 2nd compute nodes 2nd compute nodes, customizing default installations disk partitioning, compute nodes frontend frontend, customizing required software downloads Kickstart and managing web-based management tools MPICH programming network support OSCAR, compared to profiling notes programming software and public and private interfaces PVFS and Red Hat Linux and scheduling software and supported processors web-based management tools links X Window System ROMIO ROMIO interface round-robin databases Round-Robin DNS RRDtool rsync 2nd anonymous rsync [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] SAN (storage area network) Scalable Cluster Environment (SCE) roll (Rocks) Scalable Parallel Random Number Generators (SPRNG) 2nd scheduling software 2nd available systems Maui OpenPBS architecture configuring graphical user interface installing managing using OSCAR and Rocks and schemas scientific data, HDF5 storage format scram switch Scyld Beowulf security checking services OSCAR SSH physical environment servers service systemimager start command SETI@Home setpe (openMosix tool) shared clusters shelving versus racks shoot-node command 2nd showmap command SIMPLE_SPRNG macro single system image (SSI) single system image clustering SIS (Systems Installation Suite) mkbootserver command OSCAR and supported Linux distributions SystemConfigurator SystemImager [See SystemImager] SystemInstaller skeleton.xml slog SLOG log file format slog_print and clog_print utility slurpdisk command SMP (symmetric multiprocessors) 2nd snooping software compatibility control and management software image copying software [See image copying software] management software [See management software] operating system compatiblity programming software 2nd [See programming software] scheduling software 2nd [See scheduling software] system software splint SPRNG (Scalable Parallel Random Number Generators) 2nd SSH (Secure Shell) OSCAR and SSI (single system image) SSI (System Service Interface) ssize striping parameter (PVFS) static client images Sterling, Thomas storage area network (SAN) stream parallelism Sun Grid Engine (SGE) roll (Rocks) supercomputers superscalar architectures switcher symbolic debuggers ddd gdb [See gdb] symmetric clusters symmetric multiprocessors [See SMP] sync_users script (OPIUM) synchronization problems, debugging system context system integrators System Service Interface (SSI) SystemImager 2nd client maintenance using cloning the system DHCP address assignment golden client 2nd image retrieval image server local.cfg file SystemImager, mkdhcpserver script Systems Installation Suite [See SIS] [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] task partitioning tasks and task granularity Tcl/Tk 8.0 and PBS temperature recommendations, operating environment time command time system call TotalView tping command tstmachines script [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] u2p utility (PVFS) UHN (unique home node) UMA (uniform memory access) uniform memory access (UMA) uniprocessor computers unique home node (UHN) updateclient script uploaddisk command UPS (uninterruptible power supply) upshot 2nd USE_MPI macro user context [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] viewers (MPE) VNC (Virtual Network Computing) von Neumann bottleneck von Neumann computer [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] wall-clock tim watchpoints wattage, conversion to BTUs work pools [SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] X Window System Rocks and xpbs command (PBS) xpbsmon command (PBS) ... Reader Reviews Errata Academic High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI By Joseph D Sloan Publisher : O''Reilly Pub Date : November 2004 ISBN : 0-596-00570-9 Pages : 360... An attribution usually includes the title, author, publisher, and ISBN For example: "High Performance Linux Clusers with OSCAR, Rocks, openMosix, and MPI, by Joseph Sloan Copyright 2005 O''Reilly, 0-596-00570-9."... 998-9938 or corporate @oreilly. com Nutshell Handbook, the Nutshell Handbook logo, and the O''Reilly logo are registered trademarks of O''Reilly Media, Inc The Linux series designations, High Performance Linux Clusters