Tài liệu Memory Dump Analysis Anthology- P2 doc

Hangs Explained 31 HANGS EXPLAINED Another category of problems happens very often where we also need a dump for analysis: hangs. There is some confusion exists in understanding the difference be- tween these two categories: crash and hang. Although sometimes a hang is a direct consequence of a crash most of the time hangs happen independently. They also manif- est themselves differently. Let’s look at application (process) crashes and hangs first. When a crash happens an application (process) often disappears. When a hang hap- pens an application (process) is still in memory: we can see it in Task Manager, for example, but it doesn’t respond to user commands or to any other requests like ping- ing a TCP/IP port. If we have a crash in OS then the most visible manifestation is blue screen and/or reboot. If we have a hang then everything freezes. Application or system hang happens because from the high level of view the interaction between applications and OS components (modules) is done via messages. One component sends a message to another and waits for a response. Some components are critical, for example, a registry. The following hand-made picture depicts very common system hang situations when the register component stops responding. Then every running application (process) stops responding if its execution path depends on registry access. Registry Internet Explorer ServiceDriver Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 32 PART 1: Crash Dumps for Beginners The very common reason for hang is the so called deadlock when two running applications, their execution paths or threads are waiting for each other. Here is an anal- ogy with a blocked road: Car 1 blocked the road, waiting for service Car 2 car car Service Car 2 – waiting for Car 1 to unblock the road In order to see what’s inside the process or OS which caused a hang we need a memory dump. Usually this dump is called a crash dump too because in order to get it the usual method is to make some sort of a trap which causes an application or OS to crash and to save the dump. I personally prefer to call these crash dumps just memory dumps to avoid confusion. Some FAQ: Q. How can we get a memory dump if our application or service hangs? A. It is possible to do using various methods: by using NTSD command line options (remember that NTSD is always present on pre-Vista system) by using userdump.exe by attaching and saving the dump interactively via NTSD (http://support.citrix.com/article/CTX108173) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Hangs Explained 33 by attaching and saving the dump interactively via WinDbg (http://support.citrix.com/article/CTX106566) by using ADPlus in a hang mode (http://support.microsoft.com/default.aspx?scid=kb;en-us;286350) Q. How can we get a memory dump if our system hangs? A. Two common methods are: manually via keyboard (http://support.microsoft.com/kb/244139/EN-US/) by using Citrix SystemDump tool remotely or via GUI if some session is still alive (http://support.citrix.com/article/CTX111072) For most system hangs choosing Kernel memory dump option in Control Panel\System\Advanced\Startup and Recovery applet is sufficient. Kernel memory dumps are smaller and less susceptible to corruption or truncation due to small page file size. If you discover that you need to peer inside running user applications then you can always ask for another complete memory dump when the problem happens again. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 34 PART 1: Crash Dumps for Beginners SYMBOL FILES EXPLAINED Symbol files are usually called PDB files because they have .PDB extension al- though the older ones can have .DBG extension. PDB files are needed to read dump files properly. Without PDB files the dump file data is just a collection of numbers, the con- tents of memory, without any meaning. PDB files help tools like WinDbg to interpret the data and present it in a human-readable format. Roughly speaking, PDB files contain associations between numbers and their meanings expressed in short text strings: Dump data (memory contents) . . . 773f8ea4 0012f9f4 6be82f08 00000000 7e4188da 00000000 0012fa80 7fffffff 000003e8 00406258 00000000 00000001 00da00ab 00aa00f3 00dc0000 . . . comctl32.pdb . 773f8ea4 Button_WndProc . ProductA.pdb . 0012f9f4 ProcessPayment . imgutil.pdb . 6be82f08 DrawImage . Because these associations are changed when we have a fix or a service pack on a computer and we have a crash dump from it we need newer PDB files that correspond to updated components such as DLLs or drivers. Long time ago we had to download symbol files manually from Microsoft or get them from CDs. Now Microsoft has its dedicated internet symbol server and WinDbg can download PDB files automatically. However we need to specify Microsoft symbol Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Symbol Files Explained 35 server location in File\Symbol File Path… dialog and check Reload. The location is usually (check http://windbg.org): SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols If we don’t remember the location when we run WinDbg for the first time or on a new computer we can enter .symfix command to set Microsoft symbol server path automatically and specify the location where to download symbol files. We can check our current symbol search path by using .sympath command and then reload symbols by entering .reload command: 0:000> .symfix No downstream store given, using C:\Program Files\Debugging Tools for Windows\sym 0:000> .sympath Symbol search path is: SRV**http://msdl.microsoft.com/download/symbols 0:000> .symfix c:\websymbols 0:000> .sympath Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols 0:000> .reload Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 36 PART 1: Crash Dumps for Beginners CRASHES AND HANGS DIFFERENTIATED In the articles Crashes Explained (page 28) and Hangs Explained (page 31) I highlighted the difference between crashes and hangs. In this part I will elaborate on this terminology a bit further. First of all, we have to unify them as manifestations of a functional failure. Considering a computer as a system of components having certain functions we shall subdivide failures into system and component failures. Of course, systems themselves may be components in some larger hierarchy, like in the case of virtualization. Application and service process failures fall under component failures category. Blue screen and server freezes fall under system failures category. Now it is obvious why most computer users confuse crashes and hangs. They are just failures and often the distinction between them is blurred from the user perspective. Software developers tend to make sharp distinction between crashes and hangs because they consider a situation when a computer accesses wrong memory or gets and executes an invalid instruction as a crash. However, after such situation a com- puter system may or may not terminate that application or service. Therefore, I propose to consider crashes as situations when a system or a compo- nent is not observed anymore. For example, a running application or service disappears from Task Manager, computer system shows blue screen or reboots. In hang situations we can observe that existence of a failed component in Task Manager or a computer system doesn’t reboot automatically and shows some screen image different from BSOD or panic message. The so called sluggish behavior or long response time can also be considered as hang situations. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Crashes and Hangs Differentiated 37 Here is a simple rough diagram I devised to illuminate the proposed terminologi- cal difference: Functional failure System failure Component failure Visibility Crash Hang T F Based on the clarification above the task of collecting memory or crash dumps is much simpler and clearer. In the case of a system crash or hang we need to setup correct crash dump op- tions in Advanced System Settings in Control Panel and check page file size in case of the complete memory dump option. A system crash will save the dump automatically. For system hangs we need to actively trigger crash dump saving procedure using either standard keyboard method, SystemDump tool or live system debugging. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 38 PART 1: Crash Dumps for Beginners In the case of an application crash we need to set up a postmortem debugger, get WER report or attach a debugger to a component and wait for a failure to happen. In the case of a hang we save a memory dump manually either by using process dumpers like userdump.exe or attaching a debugger. Links to some dump collection techniques can be found in previous Crashes Ex- plained and Hangs Explained articles. Forthcoming Windows® Crash Dump Analysis book (ISBN-13: 978-0-9558328-2-6) will discuss all memory dump collection methods tho- roughly and in detail. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Proactive Crash Dumps 39 PROACTIVE CRASH DUMPS In Crashes and Hangs Differentiated article (page 36) I introduced clear separa- tion between crashes and hangs and outlined memory dump capturing methods for each category. However, looking from user point of view we need to tell them what is the best way to capture a dump based on observations they have and their failure level, system or component. The latter failure type usually happens with user applications and services. For user applications the best way is to get a memory dump proactively or put in another words, manually, and do not rely on a postmortem debugger that may not be set up correctly on a problem server in one hundred server farm. If any error message box appears with a message that an application stopped working or that it has encoun- tered an application error then we can use process dumpers like userdump.exe. Suppose we have the following error message when TestDefaultDebugger application crashes on Vista x64 (the same technique is applicable to earlier OS too): Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 40 PART 1: Crash Dumps for Beginners Then we can dump the process while it displays the problem message if we know its process ID: In Vista this can be done even more easily by dumping the process from Task Manager directly: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... manual process dumpers Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Minidump Analysis 43 PART 2: PROFESSIONAL CRASH DUMP ANALYSIS MINIDUMP ANALYSIS SCRIPTS AND WINDBG COMMANDS Small Memory Dumps, also referred to as minidumps because they are stored in %SystemRoot% \ Minidump folder, contain only bugcheck information, kernel mode stack data and the list of loaded drivers... Split-Merge on www.verypdf.com to remove this watermark 44 PART 2: Professional Crash Dump Analysis Here is listing of our WinDbg script: $$ $$ MiniDmp2Txt: Dump information from minidump into log $$ logopen /d /u echo "command> ||" || echo "command> vertarget" vertarget echo "command> r (before analysis) " r echo "command> kv (before analysis) " kv 100 echo "command> !analyze -v" !analyze -v echo "command> r"... nt!ExFreePoolWithTag=Dynamic memory corruption detected when freeing memory Now the analysis reports our custom follow up message: 3: kd> !analyze -v FOLLOWUP_IP: nt!ExFreePoolWithTag+1d0 805689c2 e9c8f0ffff jmp SYMBOL_STACK_INDEX: nt!ExFreePoolWithTag+0x1d0 (80567a8f) 2 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 50 PART 2: Professional Crash Dump Analysis FOLLOWUP_NAME: Dynamic memory. .. !sysinfo smbios -v logclose $$ $$ MiniDmp2Txt: End of File $$ Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Minidump Analysis 45 To run WinDbg automatically against each minidump file (.dmp) we can use the following VB script (we need to customize symbol search path after -y parameter to point to our own symbol folders): ' ' MiniDumps2Txt.vbs ' Set fso = CreateObject("Scripting.FileSystemObject")... information to a vendor or a 3rd-party for an automated crash dump analysis Another use is to keep system crash history In this part I discuss the scripting approach to extract information from all minidumps stored on a particular system The script processes all minidump files and creates text log files containing the following information: 1 2 3 Crash dump name and type OS information, crash time and system... + " -c ""$$> . to save the dump. I personally prefer to call these crash dumps just memory dumps to avoid confusion. Some FAQ: Q. How can we get a memory dump if our. ANALYSIS MINIDUMP ANALYSIS SCRIPTS AND WINDBG COMMANDS Small Memory Dumps, also referred to as minidumps because they are stored in %SystemRoot% Minidump folder,