Tài liệu Memory Dump Analysis Anthology- P22 pdf

Dumping Processes Without Breaking Them 631 DUMPING PROCESSES WITHOUT BREAKING THEM We can do it on any Windows system after Windows 2000 without installing any additional tools like Userdump or WinDbg. And a process won’t be interrupted while its memory dump is being saved and will continue to work. We can use the following com- mand: ntsd -pvr -p 'PID' -c ".dump /ma /u process.dmp; q" PID is a decimal process ID we can get from Task Manager, for example. Note: on x64 system to dump a 32-bit process (shown as *32 in Task Manager)we need to use NTSD from \Windows\SysWOW64 folder (page 633). On Windows Vista, NTSD is no longer included but it can be found in Debugging Tools for Windows package. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 632 PART 11: The Origin of Crash Dumps USERDUMP.EXE ON X64 If we install the latest Microsoft user mode process dumper on x64 Windows we would see both x86 and x64 folders. One advice here: do not dump 32-bit applications and services (shown as *32 in Task Manager) using userdump.exe from x64 folder: use userdump.exe from x86 folder. 32-bit application runs in WOW64 emulation layer on x64 Windows and that emulation layer is itself native 64-bit process so x64 userdump.exe saves that emulation layer and not the original 32-bit application. If we open that dump file in WinDbg we would see WOW64 thread stacks and not thread stacks from our original 32-bit application. In summary, on x64 Windows to save a memory dump file of a 64-bit application we can use: x64\userdump.exe \Windows\System32\ntsd.exe 64-bit version of WinDbg.exe to save a memory dump file of a 32-bit application use: x86\userdump.exe \Windows\SysWOW64\ntsd.exe 32-bit WinDbg.exe Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. NTSD on x64 Windows 633 NTSD ON X64 WINDOWS If we need to attach NTSD to a process on x64 Windows and to save a memory dump file we should remember that there are two versions of NTSD: x86 (32-bit) and x64. The former is located in \Windows\SysWOW64 folder and should be used for attaching to 32-bit applications and services. For explanation why you need different versions of NTSD please refer to the first picture in Dumps, Debuggers and Virtualiza- tion (page 516). If we use WinDbg for that purpose we should install both 32-bit and 64-bit ver- sions. If we want to install NTSD or WinDbg as a default postmortem debugger we should use Wow6432Node registry hive: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug Debugger = ntsd -p %ld -e %ld -g -c ".dump /ma /u c:\TEMP\new.dmp; q" Please refer to the following Citrix support articles explaining and describing in more detail how to set NTSD and WinDbg as default postmortem debuggers: How to Set NTSD as a Default Windows Postmortem Debugger (http://support.citrix.com/article/CTX105888) How to Set WinDbg as a Default Windows Postmortem Debugger (http://support.citrix.com/article/CTX107528) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 634 PART 11: The Origin of Crash Dumps NEED A DUMP? COMMON USE CASES The most common scenarios technical support people encounter when facing the need to create a dump file are: Heap corruption http://support.citrix.com/article/CTX104633 the article is applicable to any process. CPU spikes http://support.citrix.com/article/CTX106110 No user dumps saved by Dr. Watson http://support.citrix.com/article/CTX105888 Memory leak http://support.citrix.com/article/CTX106970 the article is applicable to any process. Need a system dump from a remote session? Use SystemDump (page 646) http://support.citrix.com/article/CTX111072 Got correct dump? Use Citrix DumpCheck http://support.citrix.com/article/CTX108825 (Explorer extension) http://support.citrix.com/article/CTX108890 (Command line version) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Memory Dump Analysis Using Excel 635 PART 12: TOOLS MEMORY DUMP ANALYSIS USING EXCEL Some WinDbg commands output data in tabular format so it is possible to save their output into a text file, import it to Excel and do sorting, filtering, and graph visualization. Some commands from WinDbg include: !stacks 1 - Lists all threads with Ticks column so we can sort and filter threads that had been waiting for no more than 100 ticks, for example. !irpfind - Here we can create various histograms, for example, IRP distribution based on [Driver] column. The following graph depicts thread distribution in PID - TID coordinates on a busy multiprocessor system with 25 user sessions and more than 3,000 threads: WinDbg scripts offer the possibility to output various tabulated data via .PRINTF: 0:000> .printf "a\tb\tc" a b c Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 636 PART 12: Tools TESTDEFAULTDEBUGGER.NET Sometimes there are situations when we need to test exception handling to see whether it works and how to get dumps or logs from it. For example, a customer reports infrequent process crashes but no crash dumps are saved. Then we can try some application that crashes immediately to see whether it results in error messages and/or saved crash dumps. This was the motivation behind TestDefaultDebugger pack- age. Unfortunately it contains only native applications and we also needed to test .NET CLR exception handling and see what messages it shows in an environment. This is a simple program in C# that creates an empty Stack object and then calls its Pop method which triggers “Stack empty” exception: The updated package now includes TestDefaultDebugger.NET.exe and can be downloaded from Citrix support web site: http://support.citrix.com/article/CTX111901 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Cons of Symbol Server 637 CONS OF SYMBOL SERVER Symbol servers are great. However I found that in crash dump analysis the ab- sence of automatically loaded symbols sometimes helps to identify a problem or at least gives some directions for further research. It also helps to see which hot fixes or service packs for a product were installed on a problem computer. The scenario I use some- times when I analyze crash dumps from product A is the following: 1. Set up WinDbg to point to Microsoft Symbol Server 2. Load a crash dump and enter various commands based on the issue. Some OS or product A components become visible and their symbols are unresolved. 3. From unresolved OS symbols I’m aware of the latest fixes or privates from Microsoft 4. From unresolved symbols of the product A and PDBFinder tool I determine the base product level and this already gives me some directions. 5. I add the base product A symbols to symbol file path and continue my analysis. 6. If unresolved symbols of the product A continue to come up I use PDBFinder tool again to find corresponding symbols and add them to symbol file path. By doing that I’m aware of the product A hot fix and/or service pack level. 7. Also from PDBFinder tool I know whether there are any updates to the component in question. Of course, all of this works only if we store all PDB files from all our fixes and ser- vice packs in some location(s) with easily identified names or abbreviations, for example, PRODUCTA\VER20\SP31\FIX01. Adding symbols manually helps to be focused on components, gives attention to some threads where they appear. We might think it is a waste of time but it only takes very small percentage of time especially if we look at the memory dump for a couple of hours. What is PDBFinder tool? This is a program I developed to find right symbol files (especially for minidumps). It scans all locations for PDB or DBG files and adds them to a binary database. Next time we run PDBFinder tool it loads that database and we can find PDB or DBG file location by specifying module name and its date. We can also do a fuzzy search by specifying some date interval. If we run it with -update command line option it will build the database automatically, useful for scheduling weekly updates. You can download it from http://support.citrix.com/article/CTX110629 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 638 PART 12: Tools STRESSPRINTERS: STRESSING PRINTER AUTOCREATION Printer drivers are great source of crash dumps especially in Citrix and Microsoft terminal services environments. Bad printer drivers crash or hang spooler service (spoolsv.exe) when multiple users connect to a server. Most of bad drivers were designed and implemented for use in a single user environment without considering multithreading in mind. Some bad drivers display a dialog box every time the printer is created and because this is done on a server side users cannot dismiss it unless spooler service is configured to interact with the desktop and an administrator sees the dialog box. Some drivers are linked to a debug run-time library and every exception brings up a dialog effectively hanging the thread and some- times the whole spooler service if there was heap corruption, for example. Therefore before allowing terminal services users to use certain printers it is good to simulate multiple users trying to create particular printers to determine bad drivers and other printer components. Originally Citrix had very popular command line AddPrinter tool for this purpose and it has been replaced by StressPrinters tool where I designed and implemented GUI to set various options, coordination of multiple AddPrinter command line tools launched simultaneously with different parameters and overall log file management. We can even export settings to a file and import it on another server. The tool also has 64-bit executables to test printer autocreation on x64 Windows. The tool detects spooler crashes (if spoolsv.exe suddenly disappears from a process list) so we can check for crash dumps saved if we set up a default postmortem debugger (Dr. Watson or NTSD). If we see the progress bar hanging for a long time then we can dump the spooler service using Microsoft userdump.exe to check for any stuck threads and resource contention. You can read documentation and download this tool from Citrix support: StressPrinters for 32-bit and 64-bit platforms http://support.citrix.com/article/CTX109374 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. InstantDump (JIT Process Dumper) 639 INSTANTDUMP (JIT PROCESS DUMPER) Techniques utilizing user mode process dumpers and debuggers like Microsoft userdump.exe, NTSD or WinDbg and CDB from Debugging Tools for Windows are too slow to pick up a process and dump it. We need either to attach a debugger ma- nually, run the command line prompt or switch to Task Manager. This deficiency was the primary motivation for me to use JIT (just-in-time) technology for process dumpers. The new tool, InstantDump, will dump a process instantly and non-invasively in a moment when we need it. How does it work? We point to any window and press hot key. InstantDump could be useful to study hang GUI processes or to get several dumps of the same process during some period of time (CPU spiking case or memory leak, for example) or just dump the process for the sake of dumping it (for curiosity). The tool uses tooltips to dynamically display window information. Here is the short user guide: 1. The program will run only on XP/W2K3/Vista (in fact it will not load on Windows 2000). 2. Run InstantDump.exe on 32-bit system or InstantDump64.exe on x64 Win- dows. If we attempt to run InstantDump.exe on x64 Windows it will show this message box and quit: 3. InstantDump puts itself into task bar icon notification area: 4. By default, when we move the mouse pointer over windows, the tooltip follows the cursor describing the process and thread id and process image path (we can disable tips in Options dialog box): Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 640 PART 12: Tools 5. If we hold Ctrl-RightShift-Break for less than a second then the process (which window is under the cursor) will be dumped according to the settings for external process dumper in the options dialog (accessible via task bar icon right mouse click): The saved dump name will be (in our Calculator window case): calc.exe_9f8(2552)_22-17-56_18-Feb-2007.dmp There is no NTSD in Vista so we have to use another user mode dumper, for example, install Microsoft userdump.exe and specify the following command line in Options dialog: userdump.exe %d %s or resort to WinDbg or CDB command line. The tool can be downloaded from here: http://www.dumpanalysis.org/downloads/InstantDump.zip. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... latest memory dump file or copying it to an ftp server All actions are fully configurable and can be enabled/disabled Here is the screenshot of the main window: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 644 PART 12: Tools DUMPDEPENDS There are many cases where we need to dump several processes simultaneously and complete memory dump is not an option DumpDepends tool dumps... userdump.exe) On x64 it will distinguish between 32bit and 64-bit processes and dump them accordingly Command line option will also be available Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Dump Monitor Suite 645 DUMP MONITOR SUITE Following the announced Troubleshooting Tool Ideas database Ramzy Mansour from Citrix Technical Support came up with a brilliant idea about Dump. .. module name where crash happened DumpDepends: - Integrated and enhanced version of SystemDump which allows to dump dependent processes Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 646 PART 12: Tools SYSTEMDUMP It was previously called CtxBSOD v2.1 but was renamed to better show its purpose In addition to renaming I added a command line option to dump a system remotely or from... that time SystemDump can dump a 64-bit server too! You can download it form Citrix support web site: http://support.citrix.com/article/CTX111072 Main features: The tool has both GUI and command line interfaces We can type a message/text (or copy it from clipboard) before forcing a memory dump This message is saved in a dump file and a support engineer can read it after loading the dump file in WinDbg.exe... force a memory dump on both 32-bit and 64-bit platforms Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark SystemDump 647 Before forcing a fatal error on a server, the tool warns about potential damaging consequences: users are disconnected and all the data which is not saved will be lost It asks for a confirmation We can specify a period of time (in minutes) when to force a memory. .. based on severity and specific processes Additionally Dump Monitor Suite includes the following components (some of them already exist): DumpChecks: - Enhanced and improved version of Citrix DumpCheck Explorer extension and its command line version DumpProperties: - New Explorer extension (Properties dialog) which shows various data extracted from a dump, like process name, module list, whether heap checking... a CAB file There are several options: Dump all processes Dump important services (Terminal, IMA, CTXXMLSS, Printing, Spooler, SVCHOST) Dump all processes from the given session ID (additionally including children and important services if needed) Dump an individual process (optionally including children and important services) The tool will use external process dumpers in noninvasive manner (NTSD by... Obviously Waiting Thread can wait only for one completion port Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 656 PART 13: Miscelleneous SYMBOL FILE WARNINGS I started using new WinDbg 6.8.4.0 and found that it prints the following message twice when I open a process dump or a complete memory dump where the current context is from some user mode process: 0:000> *** *** ***... option to attach a postmortem debugger and save a crash dump: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 660 PART 13: Miscelleneous If we choose the recommended option we get the following dialog showing the path where a minidump file was temporarily stored: We need to leave this dialog open if we want to open the crash dump or copy it to another location otherwise report... http://support.citrix.com/article/CTX111901 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark DumpAlerts 643 DUMPALERTS The tool monitors folders where dumps can be saved including Dr Watson, a folder specified when NTSD is set as a default debugger and so on It then alerts a user, an administrator or a software vendor whenever a new dump file is saved: Icon in System Tray changes its color from green to red Popup . Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Memory Dump Analysis Using Excel 635 PART 12: TOOLS MEMORY DUMP ANALYSIS USING. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. InstantDump (JIT Process Dumper) 639 INSTANTDUMP (JIT PROCESS DUMPER) Techniques