Tài liệu Memory Dump Analysis Anthology- P17 doc

Wait Chain (General) 481 WAIT CHAIN (GENERAL) Wait Chain pattern is simply a sequence of causal relations between events: thread A is waiting for an event E to happen that threads B, C or D are supposed to signal at some time in the future but they are all waiting for an event F to happen that a thread G is about to signal as soon as it finishes processing some critical task: Thread A Event E Thread B Thread C Thread D Event F Thread G This subsumes various deadlock patterns too which are causal loops where a thread A is waiting for an event AB that a thread B will signal as soon as the thread A signals an event BA the thread B is waiting for: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 482 PART 3: Crash Dump Analysis Patterns Thread B Event AB Event BA Thread A In this context “Event” means any type of synchronization object, critical section, LPC/RPC reply or data arrival through some IPC channel and not only Win32 event ob- ject or kernel _KEVENT. As the first example of Wait Chain pattern I show a process being terminated and waiting for another thread to finish or in other words, considering thread termination as an event itself, the main process thread is waiting for the second thread object to be signaled. The second thread tries to cancel previous I/O request directed to some de- vice. However that IRP is not cancellable and process hangs. This can be depicted on the following diagram: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Wait Chain (General) 483 Thread B (Event A) Event B Thread A where Thread A is our main thread waiting for Event A which is the thread B itself wait- ing for I/O cancellation (Event B). Their stack traces are: THREAD 8a3178d0 Cid 04bc.01cc Teb: 7ffdf000 Win32Thread: bc1b6e70 WAIT: (Unknown) KernelMode Non-Alertable 8af2c920 Thread Not impersonating DeviceMap e1032530 Owning Process 89ff8d88 Image: processA.exe Wait Start TickCount 80444 Ticks: 873 (0:00:00:13.640) Context Switch Count 122 LargeStack UserTime 00:00:00.015 KernelTime 00:00:00.156 Win32 Start Address 0x010148a4 Start Address 0x77e617f8 Stack Init f3f29000 Current f3f28be8 Base f3f29000 Limit f3f25000 Call 0 Priority 15 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr f3f28c00 80833465 nt!KiSwapContext+0x26 f3f28c2c 80829a62 nt!KiSwapThread+0x2e5 f3f28c74 8094c0ea nt!KeWaitForSingleObject+0x346 ; stack trace with arguments shows the first parameter as 8af2c920 f3f28d0c 8094c63f nt!PspExitThread+0×1f0 f3f28d24 8094c839 nt!PspTerminateThreadByPointer+0×4b f3f28d54 8088978c nt!NtTerminateProcess+0×125 f3f28d54 7c8285ec nt!KiFastCallEntry+0xfc Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 484 PART 3: Crash Dump Analysis Patterns THREAD 8af2c920 Cid 04bc.079c Teb: 7ffd7000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 8af2c998 NotificationTimer IRP List: 8ad26260: (0006,0220) Flags: 00000000 Mdl: 00000000 Not impersonating DeviceMap e1032530 Owning Process 89ff8d88 Image: processA.exe Wait Start TickCount 81312 Ticks: 5 (0:00:00:00.078) Context Switch Count 169 LargeStack UserTime 00:00:00.000 KernelTime 00:00:00.000 Win32 Start Address 0×77da3ea5 Start Address 0×77e617ec Stack Init f3e09000 Current f3e08bac Base f3e09000 Limit f3e05000 Call 0 Priority 13 BasePriority 13 PriorityDecrement 0 ChildEBP RetAddr f3e08bc4 80833465 nt!KiSwapContext+0×26 f3e08bf0 80828f0b nt!KiSwapThread+0×2e5 f3e08c38 808ea7a4 nt!KeDelayExecutionThread+0×2ab f3e08c68 8094c360 nt!IoCancelThreadIo+0×62 f3e08cf0 8094c569 nt!PspExitThread+0×466 f3e08cfc 8082e0b6 nt!PsExitSpecialApc+0×1d f3e08d4c 80889837 nt!KiDeliverApc+0×1ae f3e08d4c 7c8285ec nt!KiServiceExit+0×56 By inspecting IRP we can see a device it was directed to, see that it has the cancel bit but doesn’t have a cancel routine: 0: kd> !irp 8ad26260 1 Irp is active with 5 stacks 4 is current (= 0x8ad2633c) No Mdl: No System Buffer: Thread 8af2c920: Irp stack trace. Flags = 00000000 ThreadListEntry.Flink = 8af2cb28 ThreadListEntry.Blink = 8af2cb28 IoStatus.Status = 00000000 IoStatus.Information = 00000000 RequestorMode = 00000001 Cancel = 01 CancelIrql = 0 ApcEnvironment = 00 UserIosb = 77ecb700 UserEvent = 00000000 Overlay.AsynchronousParameters.UserApcRoutine = 00000000 Overlay.AsynchronousParameters.UserApcContext = 00000000 Overlay.AllocationSize = 00000000 - 00000000 CancelRoutine = 00000000 UserBuffer = 77ecb720 &Tail.Overlay.DeviceQueueEntry = 8ad262a0 Tail.Overlay.Thread = 8af2c920 Tail.Overlay.AuxiliaryBuffer = 00000000 Tail.Overlay.ListEntry.Flink = 00000000 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Wait Chain (General) 485 Tail.Overlay.ListEntry.Blink = 00000000 Tail.Overlay.CurrentStackLocation = 8ad2633c Tail.Overlay.OriginalFileObject = 89ff8920 Tail.Apc = 00000000 Tail.CompletionKey = 00000000 cmd flg cl Device File Completion-Context [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 [ 0, 0] 0 0 00000000 00000000 00000000-00000000 Args: 00000000 00000000 00000000 00000000 >[ c, 2] 0 1 8ab20388 89ff8920 00000000-00000000 pending \Device\DeviceA Args: 00000020 00000017 00000000 00000000 [ c, 2] 0 0 8affa4b8 89ff8920 00000000-00000000 \Device\DeviceB Args: 00000020 00000017 00000000 00000000 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 486 PART 3: Crash Dump Analysis Patterns MANUAL DUMP (PROCESS) Now I discuss Manual Dump pattern as seen from process memory dumps. It is not possible to reliably identify manual dumps here because a debugger or another process dumper might have been attached to a process noninvasively and not leaving traces of intervention so we can only rely on the following information: Comment field Loading Dump File [C:\kktools\userdump8.1\x64\notepad.dmp] User Mini Dump File with Full Memory: Only application data is available Comment: 'Userdump generated complete user-mode minidump with Standalone function on COMPUTER-NAME' Absence of exceptions Loading Dump File [C:\UserDumps\notepad.dmp] User Mini Dump File with Full Memory: Only application data is available Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Vista Version 6000 MP (2 procs) Free x64 Product: WinNt, suite: SingleUserTS Debug session time: Mon Dec 17 16:31:31.000 2007 (GMT+0) System Uptime: 0 days 0:45:11.148 Process Uptime: 0 days 0:00:36.000 user32!ZwUserGetMessage+0xa: 00000000`76c8e6aa c3 ret 0:000> ~*kL . 0 Id: 1b8.ed4 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen Child-SP RetAddr Call Site 00000000`0029f618 00000000`76c8e6ea user32!ZwUserGetMessage+0xa 00000000`0029f620 00000000`ff2b6eca user32!GetMessageW+0x34 00000000`0029f650 00000000`ff2bcf8b notepad!WinMain+0x176 00000000`0029f6d0 00000000`76d7cdcd notepad!IsTextUTF8+0x24f 00000000`0029f790 00000000`76ecc6e1 kernel32!BaseThreadInitThunk+0xd 00000000`0029f7c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Manual Dump (Process) 487 Wake debugger exception Loading Dump File [C:\UserDumps\notepad2.dmp] User Mini Dump File with Full Memory: Only application data is available Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Vista Version 6000 MP (2 procs) Free x64 Product: WinNt, suite: SingleUserTS Debug session time: Mon Dec 17 16:35:37.000 2007 (GMT+0) System Uptime: 0 days 0:49:13.806 Process Uptime: 0 days 0:02:54.000 This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (314.1b4): Wake debugger - code 80000007 (first/second chance not available)” user32!ZwUserGetMessage+0xa: 00000000`76c8e6aa c3 ret Break instruction exception Loading Dump File [C:\UserDumps\notepad3.dmp] User Mini Dump File with Full Memory: Only application data is available Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Vista Version 6000 MP (2 procs) Free x64 Product: WinNt, suite: SingleUserTS Debug session time: Mon Dec 17 16:45:15.000 2007 (GMT+0) System Uptime: 0 days 0:58:52.699 Process Uptime: 0 days 0:14:20.000 This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. ntdll!DbgBreakPoint: 00000000`76ecfdf0 cc int 3 0:001> ~*kL 0 Id: 1b8.ed4 Suspend: 1 Teb: 000007ff`fffdc000 Unfrozen Child-SP RetAddr Call Site 00000000`0029f618 00000000`76c8e6ea user32!ZwUserGetMessage+0xa 00000000`0029f620 00000000`ff2b6eca user32!GetMessageW+0x34 00000000`0029f650 00000000`ff2bcf8b notepad!WinMain+0x176 00000000`0029f6d0 00000000`76d7cdcd notepad!IsTextUTF8+0x24f 00000000`0029f790 00000000`76ecc6e1 kernel32!BaseThreadInitThunk+0xd 00000000`0029f7c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 488 PART 3: Crash Dump Analysis Patterns # 1 Id: 1b8.ec4 Suspend: 1 Teb: 000007ff`fffda000 Unfrozen Child-SP RetAddr Call Site 00000000`030df798 00000000`76f633e8 ntdll!DbgBreakPoint 00000000`030df7a0 00000000`76d7cdcd ntdll!DbgUiRemoteBreakin+0×38 00000000`030df7d0 00000000`76ecc6e1 kernel32!BaseThreadInitThunk+0xd 00000000`030df800 00000000`00000000 ntdll!RtlUserThreadStart+0×1d The latter might also be some assertion statement in the code leading to a process crash like in the following instance of Dynamic Memory Corruption pattern (heap corruption, page 257): FAULTING_IP: ntdll!DbgBreakPoint+0 77f813b1 cc int 3 EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff) ExceptionAddress: 77f813b1 (ntdll!DbgBreakPoint) ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 3 Parameter[0]: 00000000 Parameter[1]: 09aef2ac Parameter[2]: 09aeeee8 STACK_TEXT: 09aef0bc 77fb76aa ntdll!DbgBreakPoint 09aef0c4 77fa65c2 ntdll!RtlpBreakPointHeap+0×26 09aef2bc 77fb5367 ntdll!RtlAllocateHeapSlowly+0×212 09aef340 77fa64f6 ntdll!RtlDebugAllocateHeap+0xcb 09aef540 77fcc9e3 ntdll!RtlAllocateHeapSlowly+0×5a 09aef720 786f3f11 ntdll!RtlAllocateHeap+0×954 09aef730 786fd10e rpcrt4!operator new+0×12 09aef748 786fc042 rpcrt4!OSF_CCONNECTION::OSF_CCONNECTION+0×174 09aef79c 786fbe0d rpcrt4!OSF_CASSOCIATION::AllocateCCall+0xfa 09aef808 786fbd53 rpcrt4!OSF_BINDING_HANDLE::AllocateCCall+0×1cd 09aef83c 786f1f2f rpcrt4!OSF_BINDING_HANDLE::GetBuffer+0×28 09aef854 786f1ee4 rpcrt4!I_RpcGetBufferWithObject+0×6e 09aef860 786f1ea4 rpcrt4!I_RpcGetBuffer+0xb 09aef86c 78754762 rpcrt4!NdrGetBuffer+0×2b 09aefab8 796d78b5 rpcrt4!NdrClientCall2+0×3f9 09aefac8 796d7821 advapi32!LsarOpenPolicy2+0×14 09aefb1c 796d8b04 advapi32!LsaOpenPolicy+0xaf 09aefb84 796d8aa9 advapi32!LookupAccountSidInternal+0×63 09aefbac 0aaf5d8b advapi32!LookupAccountSidW+0×1f WARNING: Stack unwind information not available. Following frames may be wrong. 09aeff40 0aad1665 ComponentDLL+0×35d8b 09aeff5c 3f69264c ComponentDLL+0×11665 09aeff7c 780085bc ComponentDLL+0×264c 09aeffb4 77e5438b msvcrt!_endthreadex+0xc1 09aeffec 00000000 kernel32!BaseThreadStart+0×52 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Manual Dump (Process) 489 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 490 PART 3: Crash Dump Analysis Patterns WAIT CHAIN (CRITICAL SECTIONS) Here is another example of Wait Chain pattern (page 481) where objects are critical sections. WinDbg can detect them if we use !analyze -v -hang command but it detects only one and not necessarily the longest or widest chain in cases with multiple wait chains: DERIVED_WAIT_CHAIN: Dl Eid Cid WaitType -- --- ------- -------------------------- 2 8d8.90c Critical Section --> 4 8d8.914 Critical Section --> 66 8d8.f9c Unknown Looking at threads we see this chain and we also see that the final thread is blocked waiting for a socket (shown in smaller font for visual clarity). 0:167> ~~[90c]kvL ChildEBP RetAddr Args to Child 00bbfd9c 7c942124 7c95970f 00000ea0 00000000 ntdll!KiFastSystemCallRet 00bbfda0 7c95970f 00000ea0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 00bbfddc 7c959620 00000000 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x19c 00bbfdfc 6748d2f9 06018b50 00000000 00000000 ntdll!RtlEnterCriticalSection+0xa8 … … … 00bbffb8 7c82608b 00315218 00000000 00000000 msvcrt!_endthreadex+0xa3 00bbffec 00000000 77b9b4bc 00315218 00000000 kernel32!BaseThreadStart+0×34 0:167> ~~[914]kvL 100 ChildEBP RetAddr Args to Child 00dbf1cc 7c942124 7c95970f 000004b0 00000000 ntdll!KiFastSystemCallRet 00dbf1d0 7c95970f 000004b0 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 00dbf20c 7c959620 00000000 00000004 0031abcc ntdll!RtlpWaitOnCriticalSection+0x19c 00dbf22c 6748d244 0031abd8 003174e0 00dbf254 ntdll!RtlEnterCriticalSection+0xa8 … … … 00dbffb8 7c82608b 00315218 00000000 00000000 msvcrt!_endthreadex+0xa3 00dbffec 00000000 77b9b4bc 00315218 00000000 kernel32!BaseThreadStart+0×34 0:167> ~~[f9c]kvL 100 ChildEBP RetAddr Args to Child 0fe2a09c 7c942124 71933a09 00000b50 00000001 ntdll!KiFastSystemCallRet 0fe2a0a0 71933a09 00000b50 00000001 0fe2a0c8 ntdll!NtWaitForSingleObject+0xc 0fe2a0dc 7194576e 00000b50 00000234 00000000 mswsock!SockWaitForSingleObject+0x19d 0fe2a154 71a12679 00000234 0fe2a1b4 00000001 mswsock!WSPRecv+0x203 0fe2a190 62985408 00000234 0fe2a1b4 00000001 WS2_32!WSARecv+0x77 0fe2a1d0 6298326b 00000234 0274ebc6 00000810 component!wait+0x338 . . . 0fe2ffb8 7c82608b 060cfc70 00000000 00000000 msvcrt!_endthreadex+0xa3 0fe2ffec 00000000 77b9b4bc 060cfc70 00000000 kernel32!BaseThreadStart+0x34 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... www.verypdf.com to remove this watermark 496 PART 4: Crash Dump Analysis AntiPatterns WRONG DUMP A customer reports application.exe crashes and we ask for a dump file We get a dump, open it and see that the dump is not from our application.exe We ask for print spooler crash dump and we get mplayer.exe crash dump I originally thought about calling it Wrong Dump pattern and place it into the patterns category... of Science WHAT IS MEMORY DUMP ANALYSIS? From a computer system we get a memory dump composed from fixed size observable values called bit or byte values Then we impose some structure on it in order to extract various derived objects like threads and processes, build some organization and understand what had happened This activity is called modeling and memory, crash or core dump analysis is all about... support background will most likely look for hang patterns inside the dump An engineer with experience writing unmanaged applications in C and C++ will open the memory dump and check an exception stored in it and if it is a breakpoint the suspicion might arise that the memory dump was taken manually because of the hanging process Based on the analysis the engineer might even correct the problem description... partition the memory into input, output states for particular computation and other states Saving a memory dump takes certain amount of time Suppose that it takes 3 discrete time events (ticks) During the first tick we save memory up to (…, s i-1, si) and that memory has some relationship to sj state During the second tick sj state changes its value and during the 3rd tick we copy the rest of the memory (s... we copy the rest of the memory (s i+1, …, sj-1, sj, sj+1, …) Now we see that the final memory dump is inconsistent: (…, si-1, si, si+1, …, sj-1, sj, sj+1, …) This is explained earlier in plain words in Inconsistent Dump pattern (page 269) Therefore we might consider a real memory dump as a direct sum of disjoint memory areas Mt taken during some time interval (t0, …, tn) M = ∑t Mt where Mt = ∑k stk... ask for a complete memory dump but we get a kernel dump or various mini-dumps Fortunately Citrix DumpCheck Explorer extension can warn users before they submit a dump file Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Fooled by Description 497 FOOLED BY DESCRIPTION From my observation an engineer with software development background opens a crash dump after glancing at a... when general is presupposed instead of particular, for example, ”Mms” prefix is read as Memory Management Subsystem” but belongs to a multimedia system vendor Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Memory Dump - A Mathematical Definition 501 PART 5: A BIT OF SCIENCE MEMORY DUMP - A MATHEMATICAL DEFINITION This post was inspired after reading “Life Itself” book... obvious from the memory dump file the engineer will read the problem description thoroughly On the contrary, an engineer with technical support or system administration background will thoroughly read the problem description first In the latter case the description might influence the direction of analysis Here is an example The description says: slow application start and we have a memory dump from a process... a dump file before sending it for analysis: Load the user process dump in WinDbg Type command symfix; reload; !analyze -v and wait until WinDbg is not busy analyzing Find PROCESS_NAME: in the output We get something like: PROCESS_NAME: spoolsv.exe We can also use dumpchk.exe from Debugging Tools for Windows (http:/support.citrix.com/article/CTX108825) Another example is when we ask for a complete memory. .. between what is seen in the dump and what users perceive Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 498 PART 4: Crash Dump Analysis AntiPatterns NEED THE CRASH DUMP This is might be the first thought when an engineer gets a stack trace fragment without symbolic information It is usually based on the following presupposition: We need an actual dump file to suggest further . watermark. 486 PART 3: Crash Dump Analysis Patterns MANUAL DUMP (PROCESS) Now I discuss Manual Dump pattern as seen from process memory dumps. It is not possible. Crash Dump Analysis AntiPatterns WRONG DUMP A customer reports application.exe crashes and we ask for a dump file. We get a dump, open it and see that the dump