Tài liệu Memory Dump Analysis Anthology- P14 docx

Coincidental Symbolic Information 391 00b1ed18 00720065 00b1ed1c 005b0020 00b1ed20 00500055 00b1ed24 003a0044 00b1ed28 00430050 Application!Print::DocumentLoad+0x5f 00b1ed2c 0035004c 00b1ed30 005d0063 00b1ed34 00630000 . . . However this is the pure coincidence. The data pattern 00NN00NN clearly be- longs to a Unicode string: 0:020> du 00b1ed00 00b1ed00 "ocument Loader [UPD:PCL5c]" It just happens that 00430050 value can be interpreted as an address that falls into Application module address range and its code section: 0:020> lm start end module name 00400000 0044d000 Application In the second example, the crash dump is from some 3rd-party application called AppSql for which we don’t have PDB files. Also we know that myhook.dll is installed as a system wide hook and it had some problems in the past. It is loaded into any address space but is not necessarily used. We want to see if there are traces of it on the problem thread stack. Dumping stack contents shows us the only one reference: . . . 00118cb0 37302f38 00118cb4 00000000 00118cb8 10008e00 myhook!notify_me+0×22c 00118cbc 01400000 00118cc0 00118abc 00118cc4 06a129f0 00118cc8 00118d04 00118ccc 02bc57d0 00118cd0 04ba5d74 00118cd4 00118d30 00118cd8 0000001c 00118cdc 00000010 00118ce0 075922bc 00118ce4 04a732e0 00118ce8 075922bc Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 392 PART 3: Crash Dump Analysis Patterns 00118cec 04a732e0 00118cf0 0066a831 AppSql+0×26a831 00118cf4 04a732d0 00118cf8 02c43190 00118cfc 00000001 00118d00 0000001c 00118d04 00118d14 00118d08 0049e180 AppSql+0×9e180 00118d0c 02c43190 00118d10 0000001c 00118d14 00118d34 … … … 0:020> lm start end module name 00400000 00ba8000 AppSql . . . 10000000 100e0000 myhook The address 10008e00 looks very “round” and it might be the set of bit flags and also, if we disassemble the code at this address backwards, we don’t see the usual call instruction that saved that address on the stack: 0:000> ub 10008e00 myhook!notify_me+0x211 10008de5 81c180000000 add ecx,80h 10008deb 899578ffffff mov dword ptr [ebp-88h],edx 10008df1 89458c mov dword ptr [ebp-74h],eax 10008df4 894d98 mov dword ptr [ebp-68h],ecx 10008df7 6a01 push 1 10008df9 8d45ec lea eax,[ebp-14h] 10008dfc 50 push eax 10008dfd ff75e0 push dword ptr [ebp-20h] In contrast, the other two addresses are return addresses saved on the stack: 0:000> ub 0066a831 AppSql+0x26a81e: 0066a81e 8bfb mov edi,ebx 0066a820 f3a5 rep movs dword ptr es:[edi],dword ptr [esi] 0066a822 8bca mov ecx,edx 0066a824 83e103 and ecx,3 0066a827 f3a4 rep movs byte ptr es:[edi],byte ptr [esi] 0066a829 8b00 mov eax,dword ptr [eax] 0066a82b 50 push eax 0066a82c e8affeffff call AppSql+0×26a6e0 (0066a6e0) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Coincidental Symbolic Information 393 0:000> ub 0049e180 AppSql+0x9e16f: 0049e16f cc int 3 0049e170 55 push ebp 0049e171 8bec mov ebp,esp 0049e173 8b4510 mov eax,dword ptr [ebp+10h] 0049e176 8b4d0c mov ecx,dword ptr [ebp+0Ch] 0049e179 50 push eax 0049e17a 51 push ecx 0049e17b e840c61c00 call AppSql+0×26a7c0 (0066a7c0) Therefore the appearance of myhook!notify_me+0×22c could be a coinci- dence unless it was a pointer to a function. However, if it was the function pointer ad- dress then it wouldn’t have pointed to the middle of the function call sequence that pushes arguments: 0:000> ub 10008e00 myhook!notify_me+0x211 10008de5 81c180000000 add ecx,80h 10008deb 899578ffffff mov dword ptr [ebp-88h],edx 10008df1 89458c mov dword ptr [ebp-74h],eax 10008df4 894d98 mov dword ptr [ebp-68h],ecx 10008df7 6a01 push 1 10008df9 8d45ec lea eax,[ebp-14h] 10008dfc 50 push eax 10008dfd ff75e0 push dword ptr [ebp-20h] 0:000> u 10008e00 myhook!notify_me+0×22c 10008e00 e82ff1ffff call myhook!copy_data (10007f34) 10008e05 8b8578ffffff mov eax,dword ptr [ebp-88h] 10008e0b 3945ac cmp dword ptr [ebp-54h],eax 10008e0e 731f jae myhook!notify_me+0×25b (10008e2f) 10008e10 8b4598 mov eax,dword ptr [ebp-68h] 10008e13 0fbf00 movsx eax,word ptr [eax] 10008e16 8945a8 mov dword ptr [ebp-58h],eax 10008e19 8b45e0 mov eax,dword ptr [ebp-20h] Also, because we have source code and private symbols, we know that if it was a function pointer then it would have been myhook!notify_me address and not notify_me+0×22c address. All this evidence supports the hypothesis that myhook occurrence on the prob- lem stack is just the coincidence and should be ignored. To add, the most coincidental symbolic information I have found so far in one crash dump is accidental correspondence between exported _DebuggerHookData and the location of the postmortem debugger NTSD: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 394 PART 3: Crash Dump Analysis Patterns 002dd434 003a0043 002dd438 0057005c 002dd43c 004e0049 LegacyApp!_DebuggerHookData+0xc4a5 002dd440 004f0044 LegacyApp!_DebuggerHookData+0×1c4a0 002dd444 00530057 002dd448 0073005c 002dd44c 00730079 002dd450 00650074 002dd454 0033006d 002dd458 005c0032 002dd45c 0074006e 002dd460 00640073 002dd464 0065002e 002dd468 00650078 0:000> du 002dd434 002dd434 ―C:\WINDOWS\system32\ntsd.exe‖ Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Stack Trace 395 STACK TRACE The most important pattern that is used for problem identification and resolution is Stack Trace. Consider the following fragment of !analyze -v output from w3wp.exe crash dump: STACK_TEXT: WARNING: Frame IP not in any known module. Following frames may be wrong. 1824f90c 5a39f97e 01057b48 01057bd0 5a3215b4 0x0 1824fa50 5a32cf7c 01057b48 00000000 79e651c0 w3core!ISAPI_REQUEST::SendResponseHeaders+0x5d 1824fa78 5a3218ad 01057bd0 79e651c0 79e64d9c w3isapi!SSFSendResponseHeader+0xe0 1824fae8 79e76127 01057bd0 00000003 79e651c0 w3isapi!ServerSupportFunction+0x351 1824fb0c 79e763a3 80000411 00000000 00000000 aspnet_isapi!HttpCompletion::ReportHttpError+0x3a 1824fd50 79e761c3 34df6cf8 79e8e42f 79e8e442 aspnet_isapi!HttpCompletion::ProcessRequestInManagedCode+0x1d1 1824fd5c 79e8e442 34df6cf8 00000000 00000000 aspnet_isapi!HttpCompletion::ProcessCompletion+0x24 1824fd70 791d6211 34df6cf8 18e60110 793ee0d8 aspnet_isapi!CorThreadPoolWorkitemCallback+0x13 1824fd84 791d616a 18e60110 00000000 791d60fa mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19 1824fda4 791fe95c 00000000 8083d5c7 00000000 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129 1824ffb8 77e64829 17bb9c18 00000000 00000000 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44 1824ffec 00000000 791fe91b 17bb9c18 00000000 kernel32!BaseThreadStart+0x34 Ignoring the first 5 numeric columns gives us the following trace: 0x0 w3core!ISAPI_REQUEST::SendResponseHeaders+0x5d w3isapi!SSFSendResponseHeader+0xe0 w3isapi!ServerSupportFunction+0x351 aspnet_isapi!HttpCompletion::ReportHttpError+0x3a aspnet_isapi!HttpCompletion::ProcessRequestInManagedCode+0x1d1 aspnet_isapi!HttpCompletion::ProcessCompletion+0x24 aspnet_isapi!CorThreadPoolWorkitemCallback+0x13 mscorsvr!ThreadpoolMgr::ExecuteWorkRequest+0x19 mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44 kernel32!BaseThreadStart+0x34 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 396 PART 3: Crash Dump Analysis Patterns In general we have something like this: moduleA!functionX+offsetN moduleB!functionY+offsetM . . . Sometimes function names are not available or offsets are very big like 0×2380. If this is the case then we probably don’t have symbol files for moduleA and moduleB: moduleA+offsetN moduleB+offsetM . . . Usually there is some kind of a database of previous issues we can use to match moduleA!functionX+offsetN against. If there is no such match we can try functionX+offsetN, moduleA!functionX or just functionX. If there is no such match again we can try the next signature, moduleB!functionY+offsetM, and moduleB!functionY, etc. Usually, the further down the trace the less useful the signature is for problem resolu- tion. For example, mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x129 will probably match many issues because this signature is common for many ASP.NET applications. If there is no match in internal databases we can try Google. For our exam- ple, Google search for SendResponseHeaders+0x5d gives the following search results: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Stack Trace 397 Browsing search results reveals the following discussion: http://groups.google.com/group/microsoft.public.inetserver.iis/browse_frm/thr ead/34bc2be635b26531?tvc=1 which can be found directly by searching Google groups: Another example is from BSOD complete memory dump. Analysis command has the following output: MODE_EXCEPTION_NOT_HANDLED (1e) This is a very common bugcheck. Usually the exception address pinpoints the driver/function that caused the problem. Always note this address as well as the link date of the driver/image that contains this address. Arguments: Arg1: c0000005, The exception code that was not handled Arg2: bff90ca3, The address that the exception occurred at Arg3: 00000000, Parameter 0 of the exception Arg4: 00000000, Parameter 1 of the exception TRAP_FRAME: bdf80834 -- (.trap ffffffffbdf80834) ErrCode = 00000000 eax=00000000 ebx=bdf80c34 ecx=89031870 edx=88096928 esi=88096928 edi=8905e7f0 eip=bff90ca3 esp=bdf808a8 ebp=bdf80a44 iopl=0 nv up ei ng nz na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282 tsmlvsa+0xfca3: bff90ca3 8b08 mov ecx,dword ptr [eax] ds:0023:00000000=???????? Resetting default scope Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 398 PART 3: Crash Dump Analysis Patterns STACK_TEXT: bdf807c4 80467a15 bdf807e0 00000000 bdf80834 nt!KiDispatchException+0x30e bdf8082c 804679c6 00000000 bdf80860 804d9f69 nt!CommonDispatchException+0x4d bdf80838 804d9f69 00000000 00000005 e56c6946 nt!KiUnexpectedInterruptTail+0x207 00000000 00000000 00000000 00000000 00000000 nt!ObpAllocateObject+0xe1 Because the crash point tsmlvsa+0xfca3 is not on the stack trace we use .trap command: 1: kd> .trap ffffffffbdf80834 ErrCode = 00000000 eax=00000000 ebx=bdf80c34 ecx=89031870 edx=88096928 esi=88096928 edi=8905e7f0 eip=bff90ca3 esp=bdf808a8 ebp=bdf80a44 iopl=0 nv up ei ng nz na po nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010282 tsmlvsa+0xfca3: bff90ca3 8b08 mov ecx,dword ptr [eax] ds:0023:00000000=???????? 1: kd> k *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr WARNING: Stack unwind information not available. Following frames may be wrong. 00000000 bdf80afc tsmlvsa+0xfca3 89080c00 00000040 nt!ObpLookupObjectName+0x504 00000000 00000001 nt!ObOpenObjectByName+0xc5 c0100080 0012b8d8 nt!IopCreateFile+0x407 c0100080 0012b8d8 nt!IoCreateFile+0x36 c0100080 0012b8d8 nt!NtCreateFile+0x2e c0100080 0012b8d8 nt!KiSystemService+0xc9 c0100080 0012b8d8 ntdll!NtCreateFile+0xb c0000000 00000000 KERNEL32!CreateFileW+0x343 1: kd> lmv m tsmlvsa bff81000 bff987c0 tsmlvsa (no symbols) Loaded symbol image file: tsmlvsa.sys Image path: tsmlvsa.sys Image name: tsmlvsa.sys Timestamp: Thu Mar 18 06:18:51 2004 (40593F4B) CheckSum: 0002D102 ImageSize: 000177C0 Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0 Google search for tsmlvsa+0xfca3 fails but if we search just for tsmlvsa we get the first link towards problem resolution: http://www-1.ibm.com/support/docview.wss?uid=swg1IC40964 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Stack Trace 399 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 400 PART 3: Crash Dump Analysis Patterns VIRTUALIZED PROCESS (WOW64) Sometimes we get a process dump from x64 Windows and when we load it into WinDbg we get the output telling us that an exception or a breakpoint comes from wow64.dll. For example: Loading Dump File [X:\ppid2088.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Userdump generated complete user-mode minidump with Exception Monitor function on SERVER01' Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Server 2003 Version 3790 (Service Pack 2) MP (4 procs) Free x64 Product: Server, suite: TerminalServer Debug session time: Tue Sep 4 13:36:14.000 2007 (GMT+2) System Uptime: 6 days 3:32:26.081 Process Uptime: 0 days 0:01:54.000 WARNING: tsappcmp overlaps ws2_32 WARNING: msvcp60 overlaps oleacc WARNING: tapi32 overlaps rasapi32 WARNING: rtutils overlaps rasman WARNING: dnsapi overlaps rasapi32 WARNING: wldap32 overlaps dnsapi WARNING: ntshrui overlaps userenv WARNING: wtsapi32 overlaps dnsapi WARNING: winsta overlaps setupapi WARNING: activeds overlaps rtutils WARNING: activeds overlaps rasman WARNING: adsldpc overlaps activeds WARNING: drprov overlaps apphelp WARNING: netui1 overlaps netui0 WARNING: davclnt overlaps apphelp . This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (2088.2fe4): Unknown exception - code 000006d9 (first/second chance not available) wow64!Wow64NotifyDebugger+0×9: 00000000`6b006369 b001 mov al,1 Analysis shows that the run-time exception was raised but the stack trace shows only WOW64 CPU simulation code in all process threads: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... various methods to list stack traces Process dumps including various process minidumps: ~*kv command lists all process threads !findstack module[!symbol] 2 command filters all stack traces to show ones containing module or module!symbol !uniqstack command Kernel minidumps: have only one problem thread kv command or its variant is suffice Kernel and complete memory dumps: !process 0 ff command lists all processes... implementing that functionality has crashed or hangs If we know the relationship between processes we can request several user dumps at once or a complete memory dump to analyze the dependency between processes by looking at their stack traces This is an example of the system level crash dump analysis pattern that I call Coupled Processes Process relationship can be implemented via different interprocess communication... remove this watermark 414 PART 3: Crash Dump Analysis Patterns [88692020 csrss.exe] [884dd508 winlogon.exe] [88be1d88 wfshell.exe] [886a7d88 OUTLOOK.EXE] [889baa70 WINWORD.EXE] [8861e3d0 ctfmon.exe] [887bbb68 EXCEL.EXE] [884e4020 csrss.exe] [8889d218 winlogon.exe] [887c8020 wfshell.exe] Threads Processed: 1101 What if we have a list of processes from a complete memory dump by using !process 0 0 command... Split-Merge on www.verypdf.com to remove this watermark 410 PART 3: Crash Dump Analysis Patterns The processor change command is illustrated in this example: 0: kd> ~2s 2: kd> k ChildEBP RetAddr eb42bd58 00000000 nt!KiIdleLoop+0x14 2: kd> ~1s;.reload /user;k Loading User Symbols ChildEBP RetAddr be4f8c30 eb091f43 i8042prt!I8xProcessCrashDump+0x53 be4f8c8c 8046bfe2 i8042prt!I8042KeyboardInterruptService+0x15d... 0x1002f`00000000 0x0 0x0 0x0 0x0 0x0 0x0 This is a clear indication that the process was in fact 32-bit but the dump is 64bit This situation is depicted in the article Dumps, Debuggers and Virtualization (page 516) and we need a debugger plug-in to understand virtualized CPU architecture This crash dump pattern can be called Virtualized Process In our case we need to load wow64exts.dll WinDbg extension and... address) There is also a separate command to reload user space symbol files any time (.reload /user) After switching we can list threads (!process address), dump or search process virtual memory For example: 1: kd> !process 0 0 **** NT ACTIVE PROCESS DUMP **** PROCESS 890a3320 SessionId: 0 Cid: 0008 Peb: 00000000 ParentCid: 0000 DirBase: 00030000 ObjectTable: 890a3e08 TableSize: 405 Image: System PROCESS... (737, 50, 345) (2948KB, 200KB, 1380KB) PeakWorkingSetSize 2764 VirtualSize 46 Mb PeakVirtualSize 52 Mb PageFaultCount 117462 MemoryPriority FOREGROUND BasePriority 13 CommitCharge 1861 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 416 PART 3: Crash Dump Analysis Patterns THREAD 8893dda0 Cid 178.15c Teb: 7ffde000 Win32Thread: a2034908 WAIT: (WrUserRequest) UserMode Non-Alertable... their threads including user space process thread stacks for complete memory dumps This command is valid for Windows XP and later For older systems we can use WinDbg scripts !stacks 2 [module[!symbol]] command shows kernel mode stack traces and we can filter the output based on module or module!symbol Filtering is valid only for crash dumps from Windows XP and later systems ~[ProcessorN]s;.reload /user;kv... logical to take two dumps, one from that application and one from spoolsv.exe Similar example is from Citrix Presentation Server environments related to printer autocreation when there are dependencies between Citrix Printing Service CpSvc.exe and spoolsv.exe Therefore if new user connections hang and restarting both printing services resolves the issue then we might need to analyze memory dumps from both... App!AddNewPrinter+0x816 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 420 PART 3: Crash Dump Analysis Patterns Notice winspool and rpcrt4 modules The application is calling spooler service using RPC to add a new printer and waiting for a reply back Looking at spooler service dump shows several threads displaying message boxes and waiting for user input: 20 Id: 790.5950 Suspend: 1 . example: Loading Dump File [X:ppid2088.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Userdump generated complete. remove this watermark. 400 PART 3: Crash Dump Analysis Patterns VIRTUALIZED PROCESS (WOW64) Sometimes we get a process dump from x64 Windows and when we load