Tài liệu Memory Dump Analysis Anthology- P25 docx

692 PART 13: Miscelleneous THE ELEMENTS OF CRASH DUMP ANALYSIS STYLE After looking at multitude of crash dump analysis reports from different compa- nies and engineers I would like to highlight several rules that make them good: Format your debugger output in fixed size font (Courier New or Lucida Console). This is very important for readability. Bold and highlight (using different colors) important addresses or data. Keep the same color for the same address or data consistently. Use red color for bug manifestation points. If we refer to a dump files we should put a link to it. What is considered bad crash dump analysis style? These are: Variable size font (you copy your debugger output to Outlook e-mail as is and it is using the default font) there. Highlight the whole data set (for example, stack trace) in red. Too much irrelevant information. Crash Dump Analysis in Visual Studio 693 CRASH DUMP ANALYSIS IN VISUAL STUDIO If we open a user crash dump as a solution/project, not a as file then we can do crash dump analysis by using Visual Studio debug windows, for example, powerful Watch window. As we can see from the picture below I loaded a crash dump from my test application saved by NTSD and it shows assembly code and source code nicely inter- leaved. If we need to specify additional symbol paths or symbol server settings we can do it in Tools \ Options \ Debugging \ Symbols dialog. 694 PART 13: Miscelleneous This might be good for engineers used to do live debugging in Visual Studio and having no experience in using WinDbg if they need to look at application crash dumps from customers. 32-bit Stack from 64-bit Dump 695 32-BIT STACK FROM 64-BIT DUMP This is a short note that might be useful if we get a 64-bit dump of a 32-bit process from x64 Windows. I wrote already about separate dump types and tools for 32- bit and 64-bit processes (pages 632 - 633). There is a simple solution when we get a 64-bit dump file and we want to look at 32-bit thread stacks, etc. We need to use wow64exts.dll extension in 64-bit WinDbg The whole discussion and solution can be found on Crash Dump Analysis forum: http://www.dumpanalysis.org/forum/viewtopic.php?t=39 Now if by accident a customer sends a 64-bit dump of a 32-bit process then we can look at threads without asking for a new dump and therefore avoid roundtrips and shorten time to problem resolution. 696 PART 13: Miscelleneous ASMPEDIA As a part of my Master’s thesis I founded Wintel assembly language encyclope- dia: http://www.asmpedia.org/ where information is presented from dump analysis and reverse engineering perspective. Currently I created some entries to test and collect comments, for example: MOV instruction http://www.asmpedia.org/index.php?title=MOV Instruction description will include: definition and examples x86 and x64 differences C-style pseudo-code annotated WinDbg disassembly C/C++ compiler translation examples Opcodes and mnemonics are cross-referenced, for example: 0xBB http://www.asmpedia.org/index.php?title=0xBB I use Intel and AMD manuals and disassembly output from WinDbg as reference. Finally I can fulfill my desire to learn all x86 instructions. Further plans are to start with ARM assembly language as soon as I finish most of Wintel part because I do development for Windows Mobile and I’m interested in low level stuff there. How WINE Can Help in Crash Dump Analysis 697 HOW WINE CAN HELP IN CRASH DUMP ANALYSIS You probably already know or have heard about the project WINE: Windows API on top of X and Unix winehq.com I first heard about it more than 10 years ago when it started. Now I rediscovered it again and was really surprised. I was looking for one NT status code I couldn’t find in Microsoft official documentation and found it here: http://cvs.winehq.com/cvsweb/wine/dlls/ntdll/error.c In order to run Win32 programs WINE emulates all API calls including OLE32, USER32, GDI32, KERNEL32, ADVAPI32 and of course, NTDLL: http://cvs.winehq.com/cvsweb/wine/dlls/ntdll/ http://cvs.winehq.com/cvsweb/wine/dlls/ole32/ http://cvs.winehq.com/cvsweb/wine/dlls/user32/ http://cvs.winehq.com/cvsweb/wine/dlls/kernel32/ http://cvs.winehq.com/cvsweb/wine/dlls/gdi32/ http://cvs.winehq.com/cvsweb/wine/dlls/advapi32/ Plus hundreds of other components. All source code is located here: http://cvs.winehq.com/cvsweb/wine/ So if you want to see how particular function or protocol might have been imple- mented hypothetically by Windows OS designers it is a good place to start. 698 PART 13: Miscelleneous HORRORS OF DEBUGGING LEGACY CODE We all know that macro definitions in C and C++ are evil. They cause mainten- ance nightmares by introducing subtle bugs. I never took that seriously until I was debugging my old code written 10 years ago which uses macros written 15 years ago. My Windows Mobile 5.0 application was crashing when I was using POOM COM interfaces (Pocket Outlook Object Model). The crash never pointed to my code. It always happened after pimstore.dll and other Microsoft modules were loaded and COM inter- faces started to return errors. I first suspected that I was using POOM incorrectly and rewrote all code several times and in different ways. No luck. Then I tried PoomMaster sample from Windows Mobile 5.0 SDK and it worked well. So I rewrote my code in ex- actly the same way as in that sample. No luck. My last hope was that moving code from my DLL to EXE (as in that sample SDK project) would eliminate crashes but it didn’t help too. Then I slowly started to realize that the problem might have been in my old code and I also noticed that one old piece of code had never been used before. So I started debugging by elimination (commenting out less and less code) until I found a macro. I had to stare at it for couple of minutes until I realized that one pair of brackets was missing and that caused allocating less memory and worse: the returned pointer to allocated memory was multiplied by 2! So the net result was the pointer pointing to other modules and subsequent string copy was effectively overwriting their memory and eventually causing crashes inside Microsoft dlls. Here is that legacy macro: #define ALLOC(t, p, s) ((p)=(t)GlobalLock(GlobalAlloc(GHND, (s)))) It allocates memory and returns a pointer. It should have been called like this (size parameter is highlighted in bold): if (ALLOC(LPWSTR,lpm->lpszEvents, (lstrlen(lpszMacro)+1)*sizeof(WCHAR))) { lstrcpy(lpm->lpszEvents, lpszMacro); lpm->nEvents=lstrlen(lpm->lpszEvents)+1; } Horrors of Debugging Legacy Code 699 What I found is the missing bracket before lstrlen function and last enclosing bracket (size parameter is highlighted in bold): if (ALLOC(LPWSTR,lpm->lpszEvents, lstrlen(lpszMacro)+1)*sizeof(WCHAR)) { lstrcpy(lpm->lpszEvents, lpszMacro); lpm->nEvents=lstrlen(lpm->lpszEvents)+1; } The resulted code after the macro expansion looks like this if (lpm->lpszEvents=(LPWSTR)GlobalLock(GlobalAlloc(GHND, lstrlen(lpszMacro)+1))*sizeof(WCHAR)) We see that the pointer to allocated memory is multiplied by two and the string copy is performed to a random place in the address space of other loaded dlls corrupt- ing their data and causing the process to crash later. 700 PART 13: Miscelleneous UML AND DEVICE DRIVERS I got the impression after reading numerous books and articles about device driv- ers that UML is almost never used in describing kernel and device driver design and architecture. Everything is described either by words or using proprietary notations. Recently I created some diagrams based on my past experience in using UML to describe and communicate architecture and design: 0. Component diagram depicting major driver interfaces Driver Dispatch IOCTL Driver(Init, Unload) 1. Class and object diagram depicting relationship between drivers and devices +DeviceObject : Device * +DriverName : UNICODE_STRING Driver +DriverObject : Driver * +NextDevice : Device * Device 1 0 * Example driver : Driver Device A : Device Device B : Device 2. Component diagram showing dependencies and interfaces when calling Win32 API function ReadFile UML and Device Drivers 701 Kernel Mode/Space User Mode/Space Application.exekernel32.dllntdll.dll ReadFileNtReadFile ntoskrnl.exe NtReadFile Driver.sys Dispatch I/O Manager is part of ntoskrnl.exe IRP * = IoAllocateIrp(…) IoCallDriver(DEVICE_OBJECT *, IRP *) DEVICE_OBJECT IRP DEVICE_OBJECT DEVICE_OBJECT 3. Component diagram showing IRP flow in a driver stack (driver-to-driver communication) IRP I/O Stack Location 3 I/O Stack Location 2 I/O Stack Location 1 I/O Manager (ntoskrnl.exe) Driver A.sys Dispatch Driver B.sys Dispatch IRP I/O Stack Location 3 I/O Stack Location 2 I/O Stack Location 1 Driver C.sys Dispatch IRP I/O Stack Location 3 I/O Stack Location 2 I/O Stack Location 1 IoCallDriver(…,IRP *) IoCallDriver(…,IRP *) IoCallDriver(…,IRP *) Actually I found that some driver books incorrectly depict the ordering of I/O stack locations in IRP stack corresponding to driver or device stack. The correct layout is depicted above. IRP I/O stack locations grow down (to lower addresses) in memory like any other Wintel stack. We can see it from kernel dumps or the following macro from [...]... KiIdleLoop thread which belongs to Idle process 704 PART 13: Miscelleneous Crash Dump Analysis Portal 705 APPENDIX A CRASH DUMP ANALYSIS PORTAL This all-in-one site features tight integration between: - This book updates and errata Crash Dump Analysis blog Crash Dump Analysis forum Crash Dump Analysis patterns Reviews of debugging and software engineering books Links to various troubleshooting and debugging... http://www.dumpanalysis.org/reference/ReferenceStackTraces_Vista_x86.pdf http://www.dumpanalysis.org/reference/ReferenceStackTraces_Vista_x64.pdf http://www.dumpanalysis.org/reference/ReferenceStackTraces_Windows2003_x86.pdf Useful when trying to spot anomalies in complete and kernel memory dumps coming from problem servers and workstations For updates and future volumes please check DumpAnalysis.org 708 Appendix B 709 INDEX !analyze,... WinDbg Quick Links Crash Dump Analysis checklist and poster E-zine There is no registration unless you want to participate in various forums or subscribe to e-mail notifications 706 Appendix A Reference Stack Traces 707 APPENDIX B REFERENCE STACK TRACES The following volumes contain normal thread stacks and other information from Windows complete memory dumps: http://www.dumpanalysis.org/reference/ReferenceStackTraces_Vista_x86.pdf... 332, 338 Manual Dump (Kernel), 479 Manual Dump (Process), 486 Manual Stack Trace, 293 memory dump, 29, 32, 33, 38, 39, 217, 220, 222, 224, 225, 226, 251, 255, 269, 276, 278, 338, 340, 342, 358, 372, 373, 419, 436, 447, 448, 451, 464, 465, 478, 479, 480, 486, 497, 501, 506, 508, 510, 514, 516, 518, 529, 532, 545, 546, 547, 554, 556, 557, 559, 561, 590,鿠604, 612, 628, 646, 647, 687 memory dump file, 252,... Packet (IRP), 173, 269, 307, 428, 429, 437, 454, 455, 456, 457, 458, 459, 482, 484, 635, 701 import table, 169, 264, 298, 300 Inconsistent Dump, 269, 501 Incorrect Stack Trace, 288 Insufficient Memory, 440 712 Index Insufficient Memory (Committed Memory) , 302 Insufficient Memory (Handle Leak), 327 interrupt, 69, 71, 73, 74, 75, 76, 78, 81, 82, 83, 86, 89, 92, 93, 95, 100, 101, 132, 137, 146, 314, 448, 649... at a memory dump file from a multiprocessor platform we can use ~”p”s command where “p” is a zero-based processor number, for example, ~1s changes the current processor to the second processor Remember that every processor has its own thread and process context If a processor has nothing to do it is looping in KiIdleLoop thread which belongs to Idle process 704 PART 13: Miscelleneous Crash Dump Analysis. .. next IRP I/O stack location is down in memory (1 is subtracted from the current stack location pointer): #define IoGetNextIrpStackLocation( Irp ) (\ (Irp)->Tail.Overlay.CurrentStackLocation - 1 ) Memory dumps (and live debugging) are good in studying component relationships and reconstructing sequence diagrams For example, this edited fragment below is from a crash dump and it shows who calls whom and... 557, 559, 561, 590,鿠604, 612, 628, 646, 647, 687 memory dump file, 252, 269, 340, 501 memory leak, 302, 327, 356, 371, 440 memory visualization, 532, 544, 556 minidump, 43, 423 Missing Thread, 362 module, 310, 367, 389 Module Variety, 310 Multiple Exceptions, 255 multiprocessor, 72, 635, 681, 703 N Need the crash dump, 498 NMI_HARDWARE_FAILURE, 135, 480 No Component Symbols, 298 NTDLL, 170, 276, 277,... 658, 659, 661, 662, 669 Word of Mouth, 495 WOW64, 400, 516, 632 Wrong Dump, 496 Z Zippocricy, 494 715 NOTES 716 Notes 717 718 Notes 719 720 Front cover image is the picture of my personal book library and the back cover image is visualized virtiual process memory generated from a crash dump of TestDefaultDebugger (page 641) from Dump2 Picture (page 532) ... user memory dump, 27, 29, 151, 227, 230, 258, 276, 356, 419, 465, 470, 478, 537, 538, 539, 540, 541, 542, 543, 606, 621, 622, 634, 669, 689 user mode, 71, 76, 83, 93, 104, 105, 108, 181, 182, 223, 230, 257, 258, 267, 305, 307, 308, 358, 387, 460, 465, 620, 632, 639, 640, 656 user space, 83, 86, 109, 141, 220, 236, 257, 267, 268, 273, 274, 278, 302, 409, 414, 430, 461, 475, 604, 664, 684, 685 userdump.exe, . between: - This book updates and errata - Crash Dump Analysis blog - Crash Dump Analysis forum - Crash Dump Analysis patterns - Reviews of debugging and. from Windows complete memory dumps: http://www.dumpanalysis.org/reference/ReferenceStackTraces_Vista_x86.pdf http://www.dumpanalysis.org/reference/ReferenceStackTraces_Vista_x64.pdf