Tài liệu Memory Dump Analysis Anthology- P7 ppt

WinDbg Tips and Tricks 181 SUSPENDING THREADS Suspending threads during live kernel debugging session can be useful for debug- ging or reproducing race condition issues. For example, when we have one thread that depends on another thread finishing its work earlier. Sometimes, very rarely the lat- ter thread finishes after the moment the first thread would expect it. In order to model this race condition we can simply patch the prologue code of the second thread worker function with ret instruction. This has the same effect as suspending the thread so it cannot produce the required data. Note: ~n (suspend) and ~f (freeze) are for user mode live debugging only. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 182 PART 2: Professional Crash Dump Analysis HEAP STACK TRACES If we have user mode stack trace DB enabled on Windows 2003 Server for some service or application and we get a crash dump and try to get saved stack traces using !heap extension command we might get these errors: 0:000> !heap -k -h 000a0000 Heap entries for Segment00 in Heap 000a0000 000a0c50: 00c50 . 00040 [01] - busy (40) 000a0c90: 00040 . 01818 [07] - busy (1800), tail fill - unable to read heap entry extra at 000a24a0 000a24a8: 01818 . 00030 [07] - busy (18), tail fill - unable to read heap entry extra at 000a24d0 000a24d8: 00030 . 005a0 [07] - busy (588), tail fill - unable to read heap entry extra at 000a2a70 The solution is to use old Windows 2000 extension ntsdexts.dll: 0:000> !.\w2kfre\ntsdexts.heap -k -h 000a0000 Stack trace (12) at 1021bfc: 7c85fc22: ntdll!RtlAllocateHeapSlowly+0×00000041 7c81d4df: ntdll!RtlAllocateHeap+0×00000E9F 7c83467a: ntdll!LdrpAllocateUnicodeString+0×00000035 7c8354f4: ntdll!LdrpCopyUnicodeString+0×00000031 7c83517b: ntdll!LdrpResolveDllName+0×00000195 7c834b2a: ntdll!LdrpMapDll+0×0000014F 7c837474: ntdll!LdrpLoadImportModule+0×0000017C 7c837368: ntdll!LdrpHandleOneNewFormatImportDescriptor+0×0000004D 7c837317: ntdll!LdrpHandleNewFormatImportDescriptors+0×0000001D 7c837441: ntdll!LdrpWalkImportDescriptor+0×00000195 7c80f560: ntdll!LdrpInitializeProcess+0×00000E3E 7c80ea0b: ntdll!_LdrpInitialize+0×000000D0 7c82ec2d: ntdll!KiUserApcDispatcher+0×00000025 Note. an example on how to enable user mode stack trace DB: http://support.citrix.com/article/CTX106970 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 183 HYPERTEXT COMMANDS Recent versions of WinDbg have RichEdit command output window that allows syntax highlighting and can simulate hyperlinks. Tooltip from WindowHistory shows its window class: There is also Debugger Markup Language (DML) and new commands that take advantage of it. For documentation please look at dml.doc located in your Debugging Tools for Windows folder. Here is the output of some commands (because WinDbg uses the variant of RichEdit that doesn’t allow copy/paste formatting I put screenshots of the output): Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 184 PART 2: Professional Crash Dump Analysis !dml_proc Here we can click on a process link and get the list of threads: We can click either on “Full details” link or on an individual thread link to see its call stack. If we select “user-mode state” link we switch to process context automatically (useful for complete memory dumps): kd> .process /p /r 0x8342c128 Implicit process is now 8342c128 Loading User Symbols Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 185 We can also navigate frames and local variables very easily: If we click on a thread name (<No name> here) we get its context: Clicking on a number sets the scope and shows local variables (if we have full PDB files): Similar command is kM: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 186 PART 2: Professional Crash Dump Analysis Another useful command is lmD where we can easily inspect modules: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 187 ANALYZING HANGS FASTER Google search shows that the additional parameter (-hang) to the venerable !analyze -v command is rarely used. Here is the command we can use if we get a ma- nually generated dump and there is no exception in it reported by !analyze -v and subsequent visual inspection of ~*kv output doesn’t show anything suspicious, leading to hidden exception(s): !analyze -hang -v Then we should always double check with !locks command because there could be multiple hang conditions in a crash dump. The same parameter can be used for kernel memory dumps too. But double checking ERESOURCE locks (!locks), kernel threads (!stacks) and DPC queues (!dpcs) manually is highly recommended. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 188 PART 2: Professional Crash Dump Analysis TRIPLE DEREFERENCE WinDbg commands like dpp allow us to do double dereference in the following format pointer *pointer **pointer For example: 0:000> dpp 004015a2 004015a2 00405068 7c80929c kernel32!GetTickCount There are cases where we need triple dereference (or even quadruple derefe- rence) done on a range of memory. Here we can utilize WinDbg scripts. The key is to use $p pseudo-register which shows the last value of d* commands (dd, dps, etc): .for (r $t0=00000000`004015a2, $t1=4; @$t1 >= 0; r $t1=$t1-1, $t0=$t0+$ptrsize) { dps @$t0 l1; dps $p l1; dps $p l1; .printf "\n" } where $t0 and $t1 are pseudo-registers holding the starting address of a memory block (we use 64-bit format) and the number of objects to be triple dereferenced and dis- played. $ptrsize is a pointer size. The script is platform independent (can be used on both 32-bit and 64-bit target). For example: 004015a2 00405068 component!_imp__GetTickCount 00405068 7c80929c kernel32!GetTickCount 7c80929c fe0000ba 004015a6 458df033 458df033 ???????? 458df033 ???????? 004015aa 15ff50f0 15ff50f0 ???????? 15ff50f0 ???????? 004015ae 00405064 component!_imp__QueryPerformanceCounter 00405064 7c80a427 kernel32!QueryPerformanceCounter 7c80a427 8b55ff8b 004015b2 33f4458b 33f4458b ???????? 33f4458b ???????? Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg Tips and Tricks 189 If we want quadruple dereferenced memory we just need to add the additional dps @$t0 l1; to .for loop body. With this script even double dereference looks much better because it shows symbol information for the first dereference too whereas dpp command shows symbol name only for the second dereference. Another less “elegant” variation without $p pseudo-register uses poi operator but we need a .catch block to prevent the script termination on invalid memory access: 0:000> .for (r $t0=00000000`004015a2, $t1=4; @$t1 >= 0; r $t1=$t1-1, $t0=$t0+$ptrsize) { .catch { dds $t0 l1; dds poi($t0) l1; dds poi(poi($t0)) l1; }; .printf "\n" } 004015a2 00405068 component!_imp__GetTickCount 00405068 7c80929c kernel32!GetTickCount 7c80929c fe0000ba 004015a6 458df033 458df033 ???????? Memory access error at ') ' 004015aa 15ff50f0 15ff50f0 ???????? Memory access error at ') ' 004015ae 00405064 component!_imp__QueryPerformanceCounter 00405064 7c80a427 kernel32!QueryPerformanceCounter 7c80a427 8b55ff8b 004015b2 33f4458b 33f4458b ???????? Memory access error at ') ' We can also use !list extension but more formatting is necessary: 0:000> .for (r $t0=00000000`004015a2, $t1=4; @$t1 >= 0; r $t1=$t1-1, $t0=$t0+$ptrsize) { .printf "%p:\n--------\n\n", $t0; !list -x "dds @$extret l1" $t0; .printf "\n" } 004015a2: --------- 004015a2 00405068 component!_imp__GetTickCount 00405068 7c80929c kernel32!GetTickCount 7c80929c fe0000ba fe0000ba ???????? Cannot read next element at fe0000ba 004015a6: --------- 004015a6 458df033 458df033 ???????? Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 190 PART 2: Professional Crash Dump Analysis Cannot read next element at 458df033 004015aa: --------- 004015aa 15ff50f0 15ff50f0 ???????? Cannot read next element at 15ff50f0 004015ae: --------- 004015ae 00405064 component!_imp__QueryPerformanceCounter 00405064 7c80a427 kernel32!QueryPerformanceCounter 7c80a427 8b55ff8b 8b55ff8b ???????? Cannot read next element at 8b55ff8b 004015b2: --------- 004015b2 33f4458b 33f4458b ???????? Cannot read next element at 33f4458b The advantage of !list is in unlimited number of pointer dereferences until invalid address is reached. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... third double words we could have just dumped the first 3 double words at TEB address: 0:000> dd 7efdd000 l3 7efdd000 0012fec0 00130000 0011c000 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 204 PART 2: Professional Crash Dump Analysis RESOLVING SYMBOL MESSAGES On one of my debugging workstations I couldn’t analyze kernel and complete memory dumps from Windows 2003 Server R02... symbol prompts on Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 206 PART 2: Professional Crash Dump Analysis THE SEARCH FOR TAGS Sometimes we get pool allocation failures and the driver’s tag is ‘Ddk’: 0: kd> !vm *** Virtual Memory Usage *** Physical Memory: 851775 ( 3407100 Kb) Page File: \??\C:\pagefile.sys Current: 4190208 Kb Free Space: 4175708 Kb Minimum: 4190208 Kb... other stackframe or is called by other stackframe, besides level 0 address - specifies thread address When address is omitted, do stack trace for the current thread For example: Loading Dump File [MEMORY. DMP] Kernel Summary Dump File: Only kernel address space is available Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (8 procs) Free x86 compatible Product: Server, suite: Enterprise TerminalServer... 7.03735e+022 high 0 Double: 8.40769e-315 Note: we push ‘None’ but see ‘enoN’ in memory because of little endian byte ordering Most of the recent drivers use their own tags and it is common not to encounter ‘None’ at all: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 208 PART 2: Professional Crash Dump Analysis kd> !poolused Sorting by Tag Pool Used: Tag None … … NonPaged... 00000000, ); *0 B7A259A8 F713435F Ntfs!ExFreeToNPagedLookasideList(F7150420, 88F93EF8, B7A25ACC, ); Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 196 PART 2: Professional Crash Dump Analysis *0 B7A259D8 8082CBCF nt!KiEspFromTrapFrame(C0001978, 83F251EC, 00000000, ); *0 B7A259F0 80865C32 nt!MiInsertPageInFreeList(C0001978, 00000000, 83F251EC, ); *1 B7A25A30 80A5C456 hal!HalpCheckForSoftwareInterrupt(C0001980,... F37FEC34, ); *0 F37FEC2C 8098AA4A nt!ExpLookupHandleTableEntry(E18D5E38, 00000B55, 89315008, ); Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 198 PART 2: Professional Crash Dump Analysis *2 F37FEC60 808F5E2F afd!AfdFastIoDeviceControl+000003A3(89435340, 00000001, 00ECFDC4, ); *1 F37FEC9C 80933491 nt!ExUnlockHandleTableEntry(E18D5E38, 00000001, 00000000, ); *0 F37FECBC 8081C3DA... files finds one with the bigger size 1,187Kb and we can append it to our symbol search path: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 200 PART 2: Professional Crash Dump Analysis 0:000> sympath+ C:\websymbols\ntdll.pdb\ DCE823FCF71A4BF5AA489994520EA18F2 Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols; C:\websymbols\ntdll.pdb\DCE823FCF71A4BF5AA489994520EA18F2... SystemReserved : [1] 0 +0×034 SpareUlong : 0 +0×038 FreeList : (null) +0×03c TlsExpansionCounter : 0 +0×040 TlsBitmap : 0×7d6a2058 +0×044 TlsBitmapBits : [2] 0xf +0×04c ReadOnlySharedMemoryBase : 0×7efe0000 +0×050 ReadOnlySharedMemoryHeap : 0×7efe0000 +0×054 ReadOnlyStaticServerData : 0×7efe0cd0 -> (null) +0×058 AnsiCodePageData : 0×7efb0000 +0×05c OemCodePageData : 0×7efc1000 +0×060 UnicodeCaseTableData... +0×040 Win32ThreadInfo : (null) +0×044 User32Reserved : [26] 0 +0×0ac UserReserved : [5] 0 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 202 PART 2: Professional Crash Dump Analysis +0×0c0 WOW32Reserved : 0×78b81910 +0×0c4 CurrentLocale : 0×409 +0×0c8 FpSoftwareStatusRegister : 0 +0×0cc SystemReserved1 : [54] (null) +0×1a4 ExceptionCode : 0 +0×1a8 ActivationContextStackPointer... identify all modules that might have been involved in a problem thread we can use the following old Windows 2000 kdex2×86 WinDbg extension command that can even work with Windows 2003 or XP kernel memory dumps: 4: kd> !w2kfre\kdex2x86.stack -? !stack - Do stack trace for specified thread Usage : !stack [-?ha[0|1]] [address] Arguments : -?,-h - display help information -a - specifies display mode This . Dump Analysis HEAP STACK TRACES If we have user mode stack trace DB enabled on Windows 2003 Server for some service or application and we get a crash dump. there could be multiple hang conditions in a crash dump. The same parameter can be used for kernel memory dumps too. But double checking ERESOURCE locks (!locks),