Tài liệu Memory Dump Analysis Anthology- P16 doc

Busy System 451 8088d319 and byte ptr [edi+5Dh],0 8088d31d jmp nt!KiIdleLoop+0xa (8088d262) nt!KiIdleLoop+0xca: 8088d322 cmp byte ptr [ebx+0AA3h],0 8088d329 je nt!KiIdleLoop+0x2 (8088d25a) nt!KiIdleLoop+0xd7: 8088d32f sti 8088d330 lea ecx,[ebx+120h] 8088d336 call nt!KiIdleSchedule (808343e6) 8088d33b test eax,eax 8088d33d mov esi,eax 8088d33f mov edi,dword ptr [ebx+12Ch] 8088d345 jne nt!KiIdleLoop+0x99 (8088d2f1) nt!KiIdleLoop+0xef: 8088d347 jmp nt!KiIdleLoop+0xa (8088d262) In some memory dumps taken when systems or sessions were hanging or very slow for some time we might see Busy System pattern where all processors execute non-idle threads and there are threads in ready queues waiting to be scheduled: 3: kd> !running System Processors f (affinity mask) Idle Processors 0 Prcb Current Next 0 ffdff120 88cef850 1 f7727120 8940b7a0 2 f772f120 8776f020 3 f7737120 87b25360 3: kd> !ready Processor 0: Ready Threads at priority 8 THREAD 88161668 Cid 3d58.43a0 Teb: 7ffdf000 Win32Thread: bc1eba48 READY THREAD 882d0020 Cid 1004.0520 Teb: 7ffdf000 Win32Thread: bc230838 READY THREAD 88716b40 Cid 2034.241c Teb: 7ffdd000 Win32Thread: bc11b388 READY THREAD 88bf7978 Cid 2444.2564 Teb: 7ffde000 Win32Thread: bc1ccc18 READY THREAD 876f7a28 Cid 2308.4bfc Teb: 7ffdd000 Win32Thread: bc1f7b98 READY Processor 0: Ready Threads at priority 0 THREAD 8a3925a8 Cid 0004.0008 Teb: 00000000 Win32Thread: 00000000 READY Processor 1: Ready Threads at priority 9 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 452 PART 3: Crash Dump Analysis Patterns THREAD 87e69db0 Cid 067c.3930 Teb: 7ffdb000 Win32Thread: bc180990 READY Processor 1: Ready Threads at priority 8 THREAD 88398c70 Cid 27cc.15b4 Teb: 7ffde000 Win32Thread: bc159ea8 READY Processor 2: Ready Threads at priority 8 THREAD 8873cdb0 Cid 4c24.4384 Teb: 7ffdd000 Win32Thread: bc1c9838 READY THREAD 89f331e0 Cid 453c.4c68 Teb: 7ffdf000 Win32Thread: bc21dbd0 READY THREAD 889a03f0 Cid 339c.2fcc Teb: 7ffdf000 Win32Thread: bc1cdbe8 READY THREAD 87aacdb0 Cid 3b80.4ed0 Teb: 7ffde000 Win32Thread: bc1c5d10 READY Processor 3: No threads in READY state Here is another example from busy 8-processor system where only one processor was idle at the time of the bugcheck: 5: kd> !ready Processor 0: No threads in READY state Processor 1: No threads in READY state Processor 2: No threads in READY state Processor 3: No threads in READY state Processor 4: No threads in READY state Processor 5: No threads in READY state Processor 6: No threads in READY state Processor 7: No threads in READY state 5: kd> !running System Processors ff (affinity mask) Idle Processors 1 Prcb Current Next 1 f7727120 8713a5a0 2 f772f120 86214750 3 f7737120 86f87020 4 f773f120 86ffe700 5 f7747120 86803a90 6 f774f120 86043db0 7 f7757120 86bcbdb0 5: kd> !thread 8713a5a0 1f THREAD 8713a5a0 Cid 4ef4.4f04 Teb: 7ffdd000 Win32Thread: bc423920 RUNNING on processor 1 Not impersonating DeviceMap e44e9a40 Owning Process 864d1d88 Image: SomeExe.exe Wait Start TickCount 1415535 Ticks: 0 Context Switch Count 7621092 LargeStack Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Busy System 453 UserTime 00:06:59.218 KernelTime 00:19:26.359 Win32 Start Address BROWSEUI!BrowserProtectedThreadProc (0x75ec1c3f) Start Address kernel32!BaseThreadStartThunk (0x77e617ec) Stack Init b68b8a70 Current b68b8c28 Base b68b9000 Limit b68b1000 Call b68b8a7c Priority 13 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr 00c1f4fc 773dc4e4 USER32!DispatchHookA+0x35 00c1f528 7739c9c6 USER32!fnHkINLPCWPRETSTRUCTA+0x60 00c1f550 7c828536 USER32!__fnDWORD+0x24 00c1f550 808308f4 ntdll!KiUserCallbackDispatcher+0x2e b68b8a94 8091d6d1 nt!KiCallUserMode+0x4 b68b8aec bf8a26d3 nt!KeUserModeCallback+0x8f b68b8b70 bf89dd4d win32k!SfnDWORD+0xb4 b68b8be8 bf89d79d win32k!xxxHkCallHook+0x22c b68b8c90 bf89da19 win32k!xxxCallHook2+0x245 b68b8cac bf8a137a win32k!xxxCallHook+0x26 b68b8cec bf85af67 win32k!xxxSendMessageTimeout+0x1e3 b68b8d10 bf8c182c win32k!xxxWrapSendMessage+0x1b b68b8d40 8088978c win32k!NtUserMessageCall+0x9d b68b8d40 7c8285ec nt!KiFastCallEntry+0xfc 00c1f550 7c828536 ntdll!KiFastSystemCallRet 00c1f57c 7739d1ec ntdll!KiUserCallbackDispatcher+0x2e 00c1f5b8 7738cee9 USER32!NtUserMessageCall+0xc 00c1f5d8 01438f73 USER32!SendMessageA+0x7f 5: kd> !thread 86214750 THREAD 86214750 Cid 0b94.1238 Teb: 7ffdb000 Win32Thread: bc2f5ea8 RUNNING on processor 2 Not impersonating DeviceMap e3482310 Owning Process 85790020 Image: SomeExe.exe Wait Start TickCount 1415535 Ticks: 0 Context Switch Count 1745682 LargeStack UserTime 00:01:20.031 KernelTime 00:04:03.484 Win32 Start Address 0x75ec1c3f Start Address kernel32!BaseThreadStartThunk (0x77e617ec) Stack Init b4861000 Current b4860558 Base b4861000 Limit b4856000 Call 0 Priority 13 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr b4860bd8 bf8da699 nt!PsGetThreadProcess b4860bf4 bf89d6e6 win32k!IsRestricted+0x2f b4860c90 bf89da19 win32k!xxxCallHook2+0x12d b4860cac bf8a137a win32k!xxxCallHook+0x26 b4860cec bf85af67 win32k!xxxSendMessageTimeout+0x1e3 b4860d10 bf8c182c win32k!xxxWrapSendMessage+0x1b b4860d40 8088978c win32k!NtUserMessageCall+0x9d b4860d40 7c8285ec nt!KiFastCallEntry+0xfc 00c1f5fc 00000000 ntdll!KiFastSystemCallRet Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 454 PART 3: Crash Dump Analysis Patterns 5: kd> !thread 86f87020 1f THREAD 86f87020 Cid 0238.0ae8 Teb: 7ffa5000 Win32Thread: 00000000 RUNNING on processor 3 IRP List: 86869200: (0006,0094) Flags: 00000900 Mdl: 00000000 85b2a7f0: (0006,0094) Flags: 00000900 Mdl: 00000000 86f80a20: (0006,0094) Flags: 00000800 Mdl: 00000000 85e6af68: (0006,0094) Flags: 00000900 Mdl: 00000000 892a6c78: (0006,0094) Flags: 00000900 Mdl: 00000000 85d06070: (0006,0094) Flags: 00000900 Mdl: 00000000 85da35e0: (0006,0094) Flags: 00000900 Mdl: 00000000 87216340: (0006,0094) Flags: 00000900 Mdl: 00000000 Not impersonating DeviceMap e1003940 Owning Process 8850e020 Image: lsass.exe Wait Start TickCount 1415535 Ticks: 0 Context Switch Count 39608 UserTime 00:00:01.625 KernelTime 00:00:05.437 Win32 Start Address RPCRT4!ThreadStartRoutine (0x77c7b0f5) Start Address kernel32!BaseThreadStartThunk (0x77e617ec) Stack Init f4925000 Current f4924c38 Base f4925000 Limit f4922000 Call 0 Priority 10 BasePriority 9 PriorityDecrement 0 ChildEBP RetAddr f4924640 80972e8e nt!SePrivilegeCheck+0x24 f4924678 80944aa0 nt!SeSinglePrivilegeCheck+0x3a f4924770 8088978c nt!NtOpenProcess+0x13a f4924770 8082eff5 nt!KiFastCallEntry+0xfc f49247f8 f6037bee nt!ZwOpenProcess+0x11 WARNING: Stack unwind information not available. Following frames may be wrong. f4924830 f6002996 SomeDrv+0x48bee 5: kd> !thread 86ffe700 1f THREAD 86ffe700 Cid 1ba4.1ba8 Teb: 7ffdf000 Win32Thread: bc23cea8 RUNNING on processor 4 Not impersonating DeviceMap e44e9a40 Owning Process 87005708 Image: WINWORD.EXE Wait Start TickCount 1415535 Ticks: 0 Context Switch Count 1547251 LargeStack UserTime 00:01:00.750 KernelTime 00:00:45.265 Win32 Start Address WINWORD (0x300019b0) Start Address kernel32!BaseProcessStartThunk (0x77e617f8) Stack Init f3465000 Current f3464c48 Base f3465000 Limit f345e000 Call 0 Priority 8 BasePriority 8 PriorityDecrement 0 ChildEBP RetAddr f3464d64 7c8285eb nt!KiFastCallEntry+0x91 f3464d68 badb0d00 ntdll!KiFastSystemCall+0x3 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Busy System 455 5: kd> !thread 86803a90 1f THREAD 86803a90 Cid 3c20.29f8 Teb: 7ffdf000 Win32Thread: bc295480 RUNNING on processor 5 Not impersonating DeviceMap e518c6b8 Owning Process 857d5500 Image: SystemDump.exe Wait Start TickCount 1415535 Ticks: 0 Context Switch Count 310 LargeStack UserTime 00:00:00.015 KernelTime 00:00:00.046 *** ERROR: Module load completed but symbols could not be loaded for SystemDump.exe Win32 Start Address SystemDump_400000 (0x0040fe92) Start Address kernel32!BaseProcessStartThunk (0x77e617f8) Stack Init b38a4000 Current b38a3c08 Base b38a4000 Limit b389f000 Call 0 Priority 11 BasePriority 8 PriorityDecrement 2 ChildEBP RetAddr Args to Child b38a3bf0 f79e3743 000000e2 cccccccc 866962b0 nt!KeBugCheckEx+0x1b WARNING: Stack unwind information not available. Following frames may be wrong. b38a3c3c 8081df65 SystemDump+0x743 b38a3c50 808f5437 nt!IofCallDriver+0x45 b38a3c64 808f61bf nt!IopSynchronousServiceTail+0x10b b38a3d00 808eed08 nt!IopXxxControlFile+0x5e5 b38a3d34 8088978c nt!NtDeviceIoControlFile+0x2a b38a3d34 7c8285ec nt!KiFastCallEntry+0xfc 0012efc4 7c826fcb ntdll!KiFastSystemCallRet 0012efc8 77e416f5 ntdll!NtDeviceIoControlFile+0xc 0012f02c 00402208 kernel32!DeviceIoControl+0x137 0012f884 00404f8e SystemDump_400000+0x2208 5: kd> !thread 86043db0 1f THREAD 86043db0 Cid 0610.55dc Teb: 7ffa1000 Win32Thread: 00000000 RUNNING on processor 6 IRP List: 86dc99a0: (0006,0094) Flags: 00000a00 Mdl: 00000000 Impersonation token: e7b30030 (Level Impersonation) DeviceMap e4e470a8 Owning Process 891374a8 Image: SomeSvc.exe Wait Start TickCount 1415215 Ticks: 320 (0:00:00:05.000) Context Switch Count 11728 UserTime 00:00:02.546 KernelTime 00:02:57.765 Win32 Start Address 0x0082b983 LPC Server thread working on message Id 82b983 Start Address kernel32!BaseThreadStartThunk (0x77e617ec) Stack Init b49c1000 Current b49c0a7c Base b49c1000 Limit b49be000 Call 0 Priority 8 BasePriority 8 PriorityDecrement 0 ChildEBP RetAddr b49c0b80 8087c9c0 hal!KeReleaseQueuedSpinLock+0x2d b49c0ba0 8087ca95 nt!ExReleaseResourceLite+0xac b49c0ba4 f6faa5ae nt!ExReleaseResourceAndLeaveCriticalRegion+0x5 b49c0bb8 f6faad05 termdd!_IcaCallStack+0x60 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 456 PART 3: Crash Dump Analysis Patterns b49c0bdc f6fa6bda termdd!IcaCallDriver+0x71 b49c0c34 f6fa86dc termdd!IcaWriteChannel+0xd8 b49c0c50 f6fa8cc6 termdd!IcaWrite+0x40 b49c0c68 8081df65 termdd!IcaDispatch+0xd0 b49c0c7c 808f5437 nt!IofCallDriver+0x45 b49c0c90 808f3157 nt!IopSynchronousServiceTail+0x10b b49c0d38 8088978c nt!NtWriteFile+0x663 b49c0d38 7c8285ec nt!KiFastCallEntry+0xfc 0254d814 7c827d3b ntdll!KiFastSystemCallRet 0254d818 77e5b012 ntdll!NtWriteFile+0xc 0254d878 004389f2 kernel32!WriteFile+0xa9 5: kd> !thread 86bcbdb0 1f THREAD 86bcbdb0 Cid 34ac.1b04 Teb: 7ffdd000 Win32Thread: bc3d9a48 RUNNING on processor 7 IRP List: 8581d900: (0006,01fc) Flags: 00000884 Mdl: 00000000 Not impersonating DeviceMap e153fc48 Owning Process 872fb708 Image: SomeExe.exe Wait Start TickCount 1415535 Ticks: 0 Context Switch Count 7655285 LargeStack UserTime 00:10:09.343 KernelTime 00:30:21.296 Win32 Start Address 0x75ec1c3f Start Address 0x77e617ec Stack Init b86cb000 Current b86ca58c Base b86cb000 Limit b86c2000 Call 0 Priority 13 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr b86ca974 f724ffc2 fltmgr!FltpPerformPostCallbacks+0x260 b86ca988 f72504f1 fltmgr!FltpProcessIoCompletion+0x10 b86ca998 f7250b83 fltmgr!FltpPassThroughCompletion+0x89 b86ca9c8 f725e5de fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x269 b86caa04 8081df65 fltmgr!FltpCreate+0x26a b86caa18 f75fa8c7 nt!IofCallDriver+0x45 b86caa40 f75faa5a SomeFlt!PassThrough+0xbb b86caa5c 8081df65 SomeFlt!Create+0xda b86caa70 808f8f71 nt!IofCallDriver+0x45 b86cab58 80937942 nt!IopParseDevice+0xa35 b86cabd8 80933a76 nt!ObpLookupObjectName+0x5b0 b86cac2c 808eae25 nt!ObOpenObjectByName+0xea b86caca8 808ec0bf nt!IopCreateFile+0x447 b86cad04 808efc4f nt!IoCreateFile+0xa3 b86cad44 8088978c nt!NtOpenFile+0x27 b86cad44 7c8285ec nt!KiFastCallEntry+0xfc Running threads have good chance to be Spiking Threads (page 305). Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Historical Information 457 HISTORICAL INFORMATION Although crash dumps are static in nature they contain Historical Informa- tion about past system dynamics that might give clues to a problem and help with troubleshooting and debugging. For example, IRP flow between user processes and drivers is readily available in any kernel or complete memory dump. WinDbg !irpfind command will show the list of currently present I/O request packets. !irp command will give individual packet de- tails. Recent Driver Verifier improvements in Vista and Windows Server 2008 allow to embed stack traces associated with IRP allocation, completion and cancellation. For information please look at the following document: http://www.microsoft.com/whdc/devtools/tools/vistaverifier.mspx Other information that can be included in process, kernel and complete memory dumps may reveal some history of function calls beyond the current snapshot of thread stacks: Heap allocation stack traces that are usually used for debugging memory leaks. Handle traces that are used to debug handle leaks (!htrace command). Raw stack data interpreted symbolically. Some examples include dumping stack. data from all process threads and dumping kernel mode stack data. LPC messages (!lpc thread). Waiting Thread Time pattern (page 343). Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 458 PART 3: Crash Dump Analysis Patterns IRP DISTRIBUTION ANOMALY In kernel or complete memory dumps coming from hanging or slow workstations and servers !irpfind WinDbg command may show IRP Distribution Anomaly pattern when certain drivers have excessive count of active IRPs not observed under normal circumstances. I created two IRP distribution graphs from two problem kernel dumps by preprocessing command output using Visual Studio keyboard macros to eliminate completed IRPs and then using Excel. In one case it was a big number of I/O request packets from 3rd-party antivirus filter driver: \Driver\3rdPartyAvFilter In the second case it was the huge number of active IRPs targeted to kernel socket ancillary function driver: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. IRP Distribution Anomaly 459 \Driver\AFD Two other peaks on both graphs are related to NTPS and NTFS, pipes and file system and usually normal. Here is IRP distribution graph from my Vista workstation captured while I was writing this post: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 460 PART 3: Crash Dump Analysis Patterns LOCAL BUFFER OVERFLOW Local Buffer Overflow pattern It is observed on x86 platforms when a local varia- ble and a function return address and/or saved frame pointer EBP are overwritten with some data. As a result, the instruction pointer EIP becomes Wild Pointer (see Volume 2) and we have a process crash in user mode or a bugcheck in kernel mode. Sometimes this pattern is diagnosed by looking at mismatched EBP and ESP values and in the case of ASCII or UNICODE buffer overflow EIP register may contain 4-char or 2-wchar_t value and ESP or EBP or both registers might point at some string fragment like in the example below: 0:000> r eax=000fa101 ebx=0000c026 ecx=01010001 edx=bd43a010 esi=000003e0 edi=00000000 eip=0048004a esp=0012f158 ebp=00510044 iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000202 0048004a 0000 add byte ptr [eax],al ds:0023:000fa101=?? 0:000> kL ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 0012f154 00420047 0x48004a 0012f158 00440077 0x420047 0012f15c 00420043 0x440077 0012f160 00510076 0x420043 0012f164 00420049 0x510076 0012f168 00540041 0x420049 0012f16c 00540041 0x540041 . . . Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... purchase PDF Split-Merge on www.verypdf.com to remove this watermark Manual Dump (Kernel) 479 MANUAL DUMP (KERNEL) Some memory dumps are generated on purpose to troubleshoot process and system hangs They are usually called Manual Dumps, manual crash dumps or manual memory dumps Kernel, complete and kernel mini dumps can be generated using the famous keyboard method described in the following Microsoft article... PART 3: Crash Dump Analysis Patterns SPECIAL STACK TRACE Sometimes we encounter thread stacks related to debugger events like Exit a Process, Load or Unload a Module These thread stacks are not normally encountered in healthy process dumps and, statistically speaking, when a process terminates or unloads a library the chances to save a memory dump manually using process dumpers like userdump.exe or Task... the dump saved? Someone advised to attach NTSD to that process, hit ‘g’ and then save the memory dump when the process breaks into the debugger again So the problem was already gone by that time and the better way would have been to create the manual user dump of that process when it was displaying the error message Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Manual Dump. .. (Unconfigured First Chance Exceptions option is set to Full Userdump): Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 466 PART 3: Crash Dump Analysis Patterns When we push the big crash button in TestDefaultDebugger dialog box two crash dumps are saved, with the first and second-chance exceptions pointing to the same code: Loading Dump File [C:\Program Files (x86)\DebugDiag\Logs\Crash... Split-Merge on www.verypdf.com to remove this watermark 468 PART 3: Crash Dump Analysis Patterns HOOKED FUNCTIONS Hooking functions using trampoline method is so common on Windows that sometimes we need to check Hooked Functions in specific modules and determine which module hooked them for troubleshooting or memory forensic analysis needs If original unhooked modules are available (via symbol server,... byte ptr [eax],al byte ptr [eax],al byte ptr [eax],al Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 470 PART 3: Crash Dump Analysis Patterns CUSTOM EXCEPTION HANDLER As discussed in Early Crash Dump pattern (page 465) saving crash dumps on firstchance exceptions helps to diagnose components that might have caused corruption and later crashes, hangs or CPU spikes by ignoring... violation exception code (c0000005) to avoid False Positive Dumps (page 259) During application execution various 1st-chance exception crash dumps were saved pointing to numerous access violations including function calls into unloaded modules, for example: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 472 PART 3: Crash Dump Analysis Patterns 0:000> kL 100 ChildEBP RetAddr WARNING:... default thread exception handler usually saves a postmortem user dump We can get first-chance exception memory dumps with: Debug Diagnostics (http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941c458-46f1-b24d-f60151d875a3&displaylang=en) ADPlus in crash mode from Debugging Tools for Windows Exception Monitor from User Mode Process Dumper package (http://www.microsoft.com/downloads/details.aspx?FamilyI... srv!WorkerThread+0×90 nt!PspSystemThreadStartup+0×5b nt!KiStartSystemThread+0×16 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 464 PART 3: Crash Dump Analysis Patterns Any deviations in a memory dump can raise suspicion like in the stack below for driver.sys nt!KiSwapContext+0x26 nt!KiSwapThread+0x284 nt!KeWaitForSingleObject+0×346 nt!ExpWaitForResource+0xd5 nt!ExAcquireResourceExclusiveLite+0×8d... mov dword ptr ds:[0],0 ds:002b:00000000=???????? Loading Dump File [C:\Program Files (x86)\DebugDiag\Logs\Crash rule for all instances of TestDefaultDebugger.exe\TestDefaultDebugger PID 4316 Date 11_21_2007 Time_04_28_34PM 693 Second_Chance_Exception_C0000005.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Dump created by DbgHost Second_Chance_Exception_C0000005‗ . remove this watermark. 458 PART 3: Crash Dump Analysis Patterns IRP DISTRIBUTION ANOMALY In kernel or complete memory dumps coming from hanging or slow workstations. www.verypdf.com to remove this watermark. 464 PART 3: Crash Dump Analysis Patterns Any deviations in a memory dump can raise suspicion like in the stack below for