Self-Service Linux® B RUCE P ERENS ’ O PEN S OURCE S ERIES www.phptr.com/perens ◆ Java™ Application Development on Linux® Carl Albing and Michael Schwarz ◆ C++ GUI Programming with Qt 3 Jasmin Blanchette and Mark Summerfield ◆ Managing Linux Systems with Webmin: System Administration and Module Development Jamie Cameron ◆ Understanding the Linux Virtual Memory Manager Mel Gorman ◆ PHP 5 Power Programming Andi Gutmans, Stig Bakken, and Derick Rethans ◆ Linux® Quick Fix Notebook Peter Harrison ◆ Implementing CIFS: The Common Internet File System Christopher Hertel ◆ Open Source Security Tools: A Practical Guide to Security Applications Tony Howlett ◆ Apache Jakarta Commons: Reusable Java™ Components Will Iverson ◆ Embedded Software Development with eCos Anthony Massa ◆ Rapid Application Development with Mozilla Nigel McFarlane ◆ Subversion Version Control: Using the Subversion Version Control System in Development Projects William Nagel ◆ Intrusion Detection with SNORT: Advanced IDS Techniques Using SNORT, Apache, MySQL, PHP, and ACID Rafeeq Ur Rehman ◆ Cross-Platform GUI Programming with wxWidgets Julian Smart and Kevin Hock with Stefan Csomor ◆ Samba-3 by Example, Second Edition: Practical Exercises to Successful Deployment John H. Terpstra ◆ The Official Samba-3 HOWTO and Reference Guide, Second Edition John H. Terpstra and Jelmer R. Vernooij, Editors ◆ Self-Service Linux®: Mastering the Art of Problem Determination Mark Wilding and Dan Behman perens_series_7x9.25.fm Page 1 Tuesday, August 16, 2005 2:17 PM Mark Wilding and Dan Behman PRENTICE HALL Professional Technical Reference Upper Saddle River, NJ ● Boston ● Indianapolis ● San Francisco ● New York ● Toronto ● Montreal ● London ● Munich ● Paris ● Madrid ● Capetown ● Sydney ● Tokyo ● Singapore ● Mexico City Self-Service Linux® Mastering the Art of Problem Determination Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact: U. S. Corporate and Government Sales (800) 382-3419 corpsales@pearsontechgroup.com For sales outside the U. S., please contact: International Sales international@pearsoned.com Visit us on the Web: www.phptr.com Library of Congress Number: 2005927150 Copyright © 2006 Pearson Education, Inc. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is presently available at http:// www.opencontent.org/openpub/). ISBN 0-13-147751-X Text printed in the United States on recycled paper at R.R. Donnelley in Crawfordsville, Indiana. First printing, September, 2005 I would like to dedicate this book to my wife, Caryna, whose relentless nagging and badgering forced me to continue working on this book when nothing else could. Just kidding Without Caryna’s support and understanding, I could never have written this book. Not only did she help me find time to write, she also spent countless hours formatting the entire book for production. I would also like to dedicate this book to my two sons, Rhys and Dylan, whose boundless energy acted as inspiration throughout the writing of this book. Mark Wilding Without the enduring love and patience of my wife Kim, this laborous project would have halted long ago. I dedicate this book to her, as well as to my beautiful son Nicholas, my family, and all of the Botzangs and Mayos. Dan Behman Gutmans_Frontmatter Page vi Thursday, September 23, 2004 9:05 AM Contents Preface Chapter 1: Best Practices and Initial Investigation Chapter 2: strace and System Call Tracing Explained Chapter 3: The /proc Filesystem Chapter 4: Compiling Chapter 5: The Stack Chapter 6: The GNU Debugger (GDB) Chapter 7: Linux System Crashes and Hangs Chapter 8: Kernel Debugging with KDB Chapter 9: ELF: Executable and Linking Format A: The Toolbox B: Data Collection Script Index Contents Preface xvii 1 Best Practices and Initial Investigation 1 1.1 Introduction 1 1.2 Getting Your System(s) Ready for Effective Problem Determination 2 1.3 The Four Phases of Investigation 3 1.3.1 Phase #1: Initial Investigation Using Your Own Skills 5 1.3.2 Phase #2: Searching the Internet Effectively 9 1.3.3 Phase #3: Begin Deeper Investigation (Good Problem Investigation Practices) 12 1.3.4 Phase #4: Getting Help or New Ideas 21 1.4 Technical Investigation 28 1.4.1 Symptom Versus Cause 28 1.5 Troubleshooting Commercial Products 38 1.6 Conclusion 39 2 strace and System Call Tracing Explained 41 2.1 Introduction 41 2.2 What Is strace? 41 2.2.1 More Information from the Kernel Side 45 2.2.2 When to Use It 48 2.2.3 Simple Example 49 2.2.4 Same Program Built Statically 53 2.3 Important strace Options 54 2.3.1 Following Child Processes 54 2.3.2 Timing System Call Activity 55 2.3.3 Verbose Mode 57 2.3.4 Tracing a Running Process 59 2.4 Effects and Issues of Using strace 60 2.4.1 strace and EINTR 61 2.5 Real Debugging Examples 62 2.5.1 Reducing Start Up Time by Fixing LD_LIBRARY_PATH 62 2.5.2 The PATH Environment Variable 65 2.5.3 stracing inetd or xinetd (the Super Server) 66 2.5.4 Communication Errors 68 2.5.5 Investigating a Hang Using strace 69 2.5.6 Reverse Engineering (How the strace Tool Itself Works) 71 2.6 System Call Tracing Examples 74 2.6.1 Sample Code 75 2.6.2 The System Call Tracing Code Explained 87 2.7 Conclusion 88 3 The /proc Filesystem 89 3.1 Introduction 89 3.2 Process Information 90 3.2.1 /proc/self 90 3.2.2 /proc/<pid> in More Detail 91 3.2.3 /proc/<pid>/cmdline 107 3.2.4 /proc/<pid>/environ 107 3.2.5 /proc/<pid>/mem 107 3.2.6 /proc/<pid>/fd 108 3.2.7 /proc/<pid>/mapped base 108 3.3 Kernel Information and Manipulation 109 3.3.1 /proc/cmdline 109 3.3.2 /proc/config.gz or /proc/sys/config.gz 109 3.3.3 /proc/cpufreq 109 3.3.4 /proc/cpuinfo 110 3.3.5 /proc/devices 110 3.3.6 /proc/kcore 111 3.3.7 /proc/locks 111 3.3.8 /proc/meminfo 111 3.3.9 /proc/mm 111 3.3.10 /proc/modules 112 3.3.11 /proc/net 112 3.3.12 /proc/partitions 112 3.3.13 /proc/pci 113 3.3.14 /proc/slabinfo 113 x Contents [...]... causing no sideeffect In the other xterm, the memory corruption landed on a pointer on the stack (the long description of the problem is beyond the scope of this chapter) The pointer was dereferenced by the application, and the trap occurred This is a very rare problem but is a good example of how small and seemingly unrelated changes or differences can affect a problem If the problem is serious and... complex problems: 1 Collect relevant information when the problem occurs 2 Keep a log of what you’ve done and what you think the problem might be 3 Be detailed and avoid qualitative information 4 Challenge assumptions until they are proven 5 Narrow the scope of the problem 6 Work to prove or disprove theories about the problem The best practices listed here are particularly important for complex problems... and back in, the problem could not be reproduced A few days later the problem came back again, only in one xterm After a very complex investigation, it turned out that an environment variable PWD was the difference that caused the problem to occur This isn’t as simple as it sounds The contents of the PWD environment variable was not the cause of the problem, although the difference in size of PWD variables... (for example, a software upgrade) The change could introduce a bug or a change in the (expected) behavior of the operating system Either can affect the software that runs on the system 1.3 The Four Phases of Investigation 9 ☞ Changes to kernel parameters or tunable values can cause changes to behavior of the operating system, which can in turn cause problems for software that runs on the system ☞ Hardware... resolution of a problem The second reason is that your patience has run out, and the problem is going in a direction that will take a long time to investigate The third is that the type of problem is such that investigating it on your own is not going to build useful skills for the future Using what you’ve learned about the problem in the first phase of investigation, you can search online for similar problems,... xxii OTHER The history and evolution of the Linux operating system is fascinating and certainly still being written with new twists popping up all the time Linux itself comprises only the kernel of the whole operating system Granted, this is the single most important part, but everything else surrounding the Linux kernel is made up mostly of GNU free software There are two major things that GNU software... when the problem occurred Were you installing software? Were you trying to start a Web server? ☞ A problem description This should include a description of what happened and a description of what was supposed to happen In other words, how do you know there was a problem? ☞ Anything that may have triggered the problem This will be pretty problem- specific, but it’s worthwhile to think about it when the problem. .. community of experts, it includes industry leading problem determination tools, and of course, the product itself includes the source code These resources are in addition to the professional Linux support services that are available from companies, such as IBM, and the various Linux vendors, such as Redhat and SUSE Making the most of these additional resources is called self-service and is the main... reproduced, you should start writing things down because you might need to refer to the information weeks or months into the future For these types of problems, it may be worthwhile to collect a lot of information about the OS (including the software versions that are installed on it) considering that the problem could be related to something else that may change over weeks or months of time Problems that take... practices for problem determination and points to the more in-depth information found in the chapters throughout this book The first step is to ensure that your Linux system(s) are configured for effective problem determination 1.2 GETTING YOUR SYSTEM(S) READY DETERMINATION FOR EFFECTIVE PROBLEM The Linux problem determination tools and facilities are free, which begs the question: Why not install them? Without . Mexico City Self-Service Linux® Mastering the Art of Problem Determination Many of the designations used by manufacturers and sellers to distinguish their products are. (as is often the case), whether you need to generate an assembly language listing for a tough problem (that is, to find the line of code for a trap), or whether