Tài liệu hạn chế xem trước, để xem đầy đủ mời bạn chọn Tải xuống
1
/ 30 trang
THÔNG TIN TÀI LIỆU
Thông tin cơ bản
Định dạng
Số trang
30
Dung lượng
683,05 KB
Nội dung
Interrupts and Exceptions Explained 91 So finally we get our stack trace: 0: kd> k L=0012f968 0012f934 00469583 100 ChildEBP RetAddr 0012f930 00469a16 processA!LPtoDP+0x19 0012f968 0046915d processA!GetColumnWidth+0x45 0012f998 0047cb72 processA!CalculateClientSizeFromPoint+0x5f 0012f9bc 0047cc1d processA!CalculateFromPoint+0x30 0012fa64 0047de83 processA!DrawUsingMemDC+0x1b9 0012fac0 0099fb43 processA!OnDraw+0x13 0012fb5c 7c17332d processA!OnPaint+0x56 0012fbe8 7c16e0b0 MFC71!CWnd::OnWndMsg+0x340 0012fc08 00c6253a MFC71!CWnd::WindowProc+0x22 0012fc24 0096cf9d processA!WindowProc+0x38 0012fcb8 7c16e1b8 MFC71!AfxCallWndProc+0x91 0012fcd8 7c16e1f6 MFC71!AfxWndProc+0x46 0012fd04 7739b6e3 MFC71!AfxWndProcBase+0x39 0012fd30 7739b874 USER32!InternalCallWinProc+0x28 0012fda8 7739c8b8 USER32!UserCallWinProcCheckWow+0x151 0012fe04 7739c9c6 USER32!DispatchClientMessage+0xd9 0012fe2c 7c828536 USER32!__fnDWORD+0x24 0012fe2c 80832dee ntdll!KiUserCallbackDispatcher+0x2e f44dcbf0 8092d605 nt!KiCallUserMode+0x4 f44dcc48 bf8a26d3 nt!KeUserModeCallback+0x8f f44dcccc bf89e985 win32k!SfnDWORD+0xb4 f44dcd0c bf89eb27 win32k!xxxDispatchMessage+0x223 f44dcd58 80833bdf win32k!NtUserDispatchMessage+0x4c f44dcd58 7c8285ec nt!KiFastCallEntry+0xfc 0012fe2c 7c828536 ntdll!KiFastSystemCallRet 0012fe58 7739c57b ntdll!KiUserCallbackDispatcher+0x2e 0012fea8 773a16e5 USER32!NtUserDispatchMessage+0xc 0012feb8 7c169076 USER32!DispatchMessageA+0xf 0012fec8 7c16913e MFC71!AfxInternalPumpMessage+0x3e 0012fee4 0041cb0b MFC71!CWinThread::Run+0x54 0012ff08 7c172fc5 processA!CMain::Run+0x3b 0012ff18 00c5364d MFC71!AfxWinMain+0x68 0012ffc0 77e6f23b processA!WinMainCRTStartup+0x185 0012fff0 00000000 kernel32!BaseProcessStart+0x23 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 92 PART 2: Professional Crash DumpAnalysis TRAP COMMAND ON X86 Now I explain WinDbg .trap command and show how to simulate it manually. Upon an interrupt a processor saves the current instruction pointer and transfers execution to an interrupt handler as explained in x86 Interrupts article (page 69). This interrupt handler has to save full thread context before calling other functions to do complex interrupt processing. For example, if we disassemble KiTrap0E handler from x86 Windows 2003 crash dump we would see that it saves a lot of registers including segment registers: 3: kd> uf nt!KiTrap0E . . . nt!KiTrap0E: e088bb2c mov word ptr [esp+2],0 e088bb33 push ebp e088bb34 push ebx e088bb35 push esi e088bb36 push edi e088bb37 push fs e088bb39 mov ebx,30h e088bb3e mov fs,bx e088bb41 mov ebx,dword ptr fs:[0] e088bb48 push ebx e088bb49 sub esp,4 e088bb4c push eax e088bb4d push ecx e088bb4e push edx e088bb4f push ds e088bb50 push es e088bb51 push gs e088bb53 mov ax,23h e088bb57 sub esp,30h e088bb5a mov ds,ax e088bb5d mov es,ax e088bb60 mov ebp,esp e088bb62 test dword ptr [esp+70h],20000h e088bb6a jne nt!V86_kite_a (e088bb04) . . . Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Interrupts and Exceptions Explained 93 The saved processor state information (context) forms the so called Windows kernel trap frame: 3: kd> dt _KTRAP_FRAME +0x000 DbgEbp : Uint4B +0x004 DbgEip : Uint4B +0x008 DbgArgMark : Uint4B +0x00c DbgArgPointer : Uint4B +0x010 TempSegCs : Uint4B +0x014 TempEsp : Uint4B +0x018 Dr0 : Uint4B +0x01c Dr1 : Uint4B +0x020 Dr2 : Uint4B +0x024 Dr3 : Uint4B +0x028 Dr6 : Uint4B +0x02c Dr7 : Uint4B +0x030 SegGs : Uint4B +0x034 SegEs : Uint4B +0x038 SegDs : Uint4B +0x03c Edx : Uint4B +0x040 Ecx : Uint4B +0x044 Eax : Uint4B +0x048 PreviousPreviousMode : Uint4B +0x04c ExceptionList : Ptr32 _EXCEPTION_REGISTRATION_RECORD +0x050 SegFs : Uint4B +0x054 Edi : Uint4B +0x058 Esi : Uint4B +0x05c Ebx : Uint4B +0x060 Ebp : Uint4B +0x064 ErrCode : Uint4B +0x068 Eip : Uint4B +0x06c SegCs : Uint4B +0x070 EFlags : Uint4B +0x074 HardwareEsp : Uint4B +0x078 HardwareSegSs : Uint4B +0x07c V86Es : Uint4B +0x080 V86Ds : Uint4B +0x084 V86Fs : Uint4B +0x088 V86Gs : Uint4B This Windows trap frame is not the same as an interrupt frame a processor saves on the current thread stack when an interrupt occurs in kernel mode. The latter frame is very small and consists only of EIP, CS, EFLAGS and ErrorCode. When an interrupt occurs in user mode an x86 processor additionally saves the current stack pointer SS:ESP. The .trap command finds the trap frame on the current thread stack and sets the current thread register context using the values from that saved structure. We can see that command in action for certain bugchecks when we use !analyze –v command: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 94 PART 2: Professional Crash DumpAnalysis 3: kd> !analyze -v KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e) . . . Arguments: Arg1: c0000005, The exception code that was not handled Arg2: de65190c, The address that the exception occurred at Arg3: f24f8a74, Trap Frame Arg4: 00000000 … … … TRAP_FRAME: f24f8a74 — (.trap fffffffff24f8a74) .trap fffffffff24f8a74 ErrCode = 00000000 eax=dbc128c0 ebx=dbe4a010 ecx=f24f8ac4 edx=00000001 esi=46525356 edi=00000000 eip=de65190c esp=f24f8ae8 ebp=f24f8b18 iopl=0 nv up ei pl nz na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010206 driver!foo+0×16: de65190c 837e1c00 cmp dword ptr [esi+1Ch],0 ds:0023:46525372=???????? … … … If we look at the trap frame we would see the same register values that WinDbg reports above: 3: kd> dt _KTRAP_FRAME f24f8a74 +0x000 DbgEbp : 0xf24f8b18 +0x004 DbgEip : 0xde65190c +0x008 DbgArgMark : 0xbadb0d00 +0x00c DbgArgPointer : 1 +0x010 TempSegCs : 0xb0501cd +0x014 TempEsp : 0xdcc01cd0 +0x018 Dr0 : 0xf24f8aa8 +0x01c Dr1 : 0xde46c90a +0x020 Dr2 : 0 +0x024 Dr3 : 0 +0x028 Dr6 : 0xdbe4a000 +0x02c Dr7 : 0 +0x030 SegGs : 0 +0x034 SegEs : 0x23 +0x038 SegDs : 0x23 +0x03c Edx : 1 +0x040 Ecx : 0xf24f8ac4 +0x044 Eax : 0xdbc128c0 +0x048 PreviousPreviousMode : 0xdbe4a010 +0x04c ExceptionList : 0xffffffff _EXCEPTION_REGISTRATION_RECORD +0x050 SegFs : 0x30 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Interrupts and Exceptions Explained 95 +0x054 Edi : 0 +0x058 Esi : 0x46525356 +0x05c Ebx : 0xdbe4a010 +0x060 Ebp : 0xf24f8b18 +0x064 ErrCode : 0 +0x068 Eip : 0xde65190c ; driver!foo+0x16 +0x06c SegCs : 8 +0x070 EFlags : 0x10206 +0x074 HardwareEsp : 0xdbc171b0 +0x078 HardwareSegSs : 0xde667677 +0x07c V86Es : 0xdbc128c0 +0x080 V86Ds : 0xdbc171c4 +0x084 V86Fs : 0xf24f8bc4 +0x088 V86Gs : 0 It is good to know how to find a trap frame manually when the stack is corrupt or WinDbg cannot find a trap frame automatically. In this case we can take the advantage of the fact that DS and ES segment registers have the same value in Windows flat mem- ory model: +0x034 SegEs : 0x23 +0x038 SegDs : 0x23 We need to find 2 consecutive 0×23 values on the stack. There may be several such places but usually the correct one comes between KiTrapXX address on the stack and the initial processor trap frame shown below in bold. This is because KiTrapXX ob- viously calls other functions to further process an interrupt so its return address is saved on the stack. 3: kd> r eax=f535713c ebx=de65190c ecx=00000000 edx=e088e1d2 esi=f5357120 edi=00000000 eip=e0827451 esp=f24f8628 ebp=f24f8640 iopl=0 nv up ei ng nz na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00000286 nt!KeBugCheckEx+0×1b: e0827451 5d pop ebp 3: kd> dds f24f8628 f24f8628+1000 . . . f24f8784 de4b2995 win32k!NtUserQueryWindow f24f8788 00000000 f24f878c fe76a324 f24f8790 f24f8d64 f24f8794 0006e43c f24f8798 e087c041 nt!ExReleaseResourceAndLeaveCriticalRegion+0x5 f24f879c 83f3b801 f24f87a0 f24f8a58 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 96 PART 2: Professional Crash DumpAnalysis f24f87a4 0000003b f24f87a8 00000000 f24f87ac 00000030 f24f87b0 00000023 f24f87b4 00000023 f24f87b8 00000000 … … … f24f8a58 00000111 f24f8a5c f24f8a74 f24f8a60 e088bc08 nt!KiTrap0E+0xdc f24f8a64 00000000 f24f8a68 46525372 f24f8a6c 00000000 f24f8a70 e0889686 nt!Kei386EoiHelper+0×186 f24f8a74 f24f8b18 f24f8a78 de65190c driver!foo+0×16 f24f8a7c badb0d00 f24f8a80 00000001 f24f8a84 0b0501cd f24f8a88 dcc01cd0 f24f8a8c f24f8aa8 f24f8a90 de46c90a win32k!HANDLELOCK::vLockHandle+0×80 f24f8a94 00000000 f24f8a98 00000000 f24f8a9c dbe4a000 f24f8aa0 00000000 f24f8aa4 00000000 f24f8aa8 00000023 f24f8aac 00000023 f24f8ab0 00000001 f24f8ab4 f24f8ac4 f24f8ab8 dbc128c0 f24f8abc dbe4a010 f24f8ac0 ffffffff f24f8ac4 00000030 f24f8ac8 00000000 f24f8acc 46525356 f24f8ad0 dbe4a010 f24f8ad4 f24f8b18 f24f8ad8 00000000 f24f8adc de65190c driver!foo+0×16 f24f8ae0 00000008 f24f8ae4 00010206 f24f8ae8 dbc171b0 f24f8aec de667677 driver!bar+0×173 f24f8af0 dbc128c0 f24f8af4 dbc171c4 f24f8af8 f24f8bc4 f24f8afc 00000000 … … … Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Interrupts and Exceptions Explained 97 Subtracting the offset 0×38 from the address of the 00000023 value (f24f8aac) and using dt command we can check _KTRAP_FRAME structure and apply .trap com- mand afterwards: 3: kd> dt _KTRAP_FRAME f24f8aac-38 +0x000 DbgEbp : 0xf24f8b18 +0x004 DbgEip : 0xde65190c +0x008 DbgArgMark : 0xbadb0d00 +0x00c DbgArgPointer : 1 +0x010 TempSegCs : 0xb0501cd +0x014 TempEsp : 0xdcc01cd0 +0x018 Dr0 : 0xf24f8aa8 +0x01c Dr1 : 0xde46c90a +0x020 Dr2 : 0 +0x024 Dr3 : 0 +0x028 Dr6 : 0xdbe4a000 +0x02c Dr7 : 0 +0x030 SegGs : 0 +0x034 SegEs : 0x23 +0x038 SegDs : 0x23 +0x03c Edx : 1 +0x040 Ecx : 0xf24f8ac4 +0x044 Eax : 0xdbc128c0 +0x048 PreviousPreviousMode : 0xdbe4a010 +0x04c ExceptionList : 0xffffffff _EXCEPTION_REGISTRATION_RECORD +0x050 SegFs : 0x30 +0x054 Edi : 0 +0x058 Esi : 0x46525356 +0x05c Ebx : 0xdbe4a010 +0x060 Ebp : 0xf24f8b18 +0x064 ErrCode : 0 +0x068 Eip : 0xde65190c +0x06c SegCs : 8 +0x070 EFlags : 0x10206 +0x074 HardwareEsp : 0xdbc171b0 +0x078 HardwareSegSs : 0xde667677 +0x07c V86Es : 0xdbc128c0 +0x080 V86Ds : 0xdbc171c4 +0x084 V86Fs : 0xf24f8bc4 +0x088 V86Gs : 0 3: kd> ? f24f8aac-38 Evaluate expression: -229668236 = f24f8a74 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 98 PART 2: Professional Crash DumpAnalysis 3: kd> .trap f24f8a74 ErrCode = 00000000 eax=dbc128c0 ebx=dbe4a010 ecx=f24f8ac4 edx=00000001 esi=46525356 edi=00000000 eip=de65190c esp=f24f8ae8 ebp=f24f8b18 iopl=0 nv up ei pl nz na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010206 driver!foo+0x16: de65190c 837e1c00 cmp dword ptr [esi+1Ch],0 ds:0023:46525372=???????? In complete memory dumps we can see that _KTRAP_FRAME is saved system ser- vices are called too: 3: kd> kL ChildEBP RetAddr f24f8ae8 de667677 driver!foo+0x16 f24f8b18 de667799 driver!bar+0x173 f24f8b90 de4a853e win32k!GreSaveScreenBits+0x69 f24f8bd8 de4922bd win32k!CreateSpb+0x167 f24f8c40 de490bb8 win32k!zzzChangeStates+0x448 f24f8c88 de4912de win32k!zzzBltValidBits+0xe2 f24f8ce0 de4926c6 win32k!xxxEndDeferWindowPosEx+0x13a f24f8cfc de49aa8f win32k!xxxSetWindowPos+0xb1 f24f8d34 de4acf4d win32k!xxxShowWindow+0x201 f24f8d54 e0888c6c win32k!NtUserShowWindow+0x79 f24f8d54 7c94ed54 nt!KiFastCallEntry+0xfc (TrapFrame @ f24f8d64) 0006e48c 77e34f1d ntdll!KiFastSystemCallRet 0006e53c 77e2f12f USER32!NtUserShowWindow+0xc 0006e570 77e2b0fe USER32!InternalDialogBox+0xa9 0006e590 77e29005 USER32!DialogBoxIndirectParamAorW+0×37 0006e5b4 0103d569 USER32!DialogBoxParamW+0×3f 0006e5d8 0102d2f5 winlogon!Fusion_DialogBoxParam+0×24 We can get the current thread context before its transition to kernel mode: 3: kd> .trap f24f8d64 ErrCode = 00000000 eax=7ffff000 ebx=00000000 ecx=00000000 edx=7c94ed54 esi=00532e68 edi=0002002c eip=7c94ed54 esp=0006e490 ebp=0006e53c iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246 ntdll!KiFastSystemCallRet: 001b:7c94ed54 c3 ret Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Interrupts and Exceptions Explained 99 3: kd> kL ChildEBP RetAddr 0006e48c 77e34f1d ntdll!KiFastSystemCallRet 0006e53c 77e2f12f USER32!NtUserShowWindow+0xc 0006e570 77e2b0fe USER32!InternalDialogBox+0xa9 0006e590 77e29005 USER32!DialogBoxIndirectParamAorW+0x37 0006e5b4 0103d569 USER32!DialogBoxParamW+0x3f 0006e5d8 0102d2f5 winlogon!Fusion_DialogBoxParam+0x24 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 100 PART 2: Professional Crash DumpAnalysis TRAP COMMAND ON X64 Now I show how to simulate .trap WinDbg command when we have x64 Win- dows kernel and complete memory dumps. When we have a fault an x64 processor saves some registers on the current thread stack as explained in x64 Interrupts article (page 76). Then an interrupt handler saves _KTRAP_FRAME on the stack: 6: kd> uf nt!KiPageFault nt!KiPageFault: fffff800`0102d400 push rbp fffff800`0102d401 sub rsp,158h fffff800`0102d408 lea rbp,[rsp+80h] fffff800`0102d410 mov byte ptr [rbp-55h],1 fffff800`0102d414 mov qword ptr [rbp-50h],rax fffff800`0102d418 mov qword ptr [rbp-48h],rcx fffff800`0102d41c mov qword ptr [rbp-40h],rdx fffff800`0102d420 mov qword ptr [rbp-38h],r8 fffff800`0102d424 mov qword ptr [rbp-30h],r9 fffff800`0102d428 mov qword ptr [rbp-28h],r10 fffff800`0102d42c mov qword ptr [rbp-20h],r11 . . . 6: kd> dt _KTRAP_FRAME +0x000 P1Home : Uint8B +0x008 P2Home : Uint8B +0x010 P3Home : Uint8B +0x018 P4Home : Uint8B +0x020 P5 : Uint8B +0x028 PreviousMode : Char +0x029 PreviousIrql : UChar +0x02a FaultIndicator : UChar +0x02b ExceptionActive : UChar +0x02c MxCsr : Uint4B +0x030 Rax : Uint8B +0x038 Rcx : Uint8B +0x040 Rdx : Uint8B +0x048 R8 : Uint8B +0x050 R9 : Uint8B +0x058 R10 : Uint8B +0x060 R11 : Uint8B +0x068 GsBase : Uint8B +0x068 GsSwap : Uint8B +0x070 Xmm0 : _M128A +0x080 Xmm1 : _M128A +0x090 Xmm2 : _M128A +0x0a0 Xmm3 : _M128A Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... set) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Interrupts and Exceptions Explained 109 HOW TO DISTINGUISH BETWEEN 1ST AND 2ND CHANCES Sometimes we look for Early Crash Dump pattern (page 465) but information about whether an exception was first-chance or second-chance is missing from a crash dump file name or in a crash dump itself, for example: This dump file has an exception... see it on a raw stack in a second-chance exception crash dump: 0:000> s -d 000000000012b000 0000000000130000 c0000005 00000000`0012f000 c0000005 00000000 00000000 00000000 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 110 PART 2: Professional Crash DumpAnalysis From raw stack data we can even tell when a crash dump was saved from a debugger handling a second-chance exception... ntdll!RtlUserThreadStart+0×95 00000000`0012e3c0 00000000`0012e420 00000000`0012e3c8 00000000`7a8b477b 00000000`0012e3d0 00000000`00000000 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 112 PART 2: Professional Crash DumpAnalysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Interrupts and Exceptions Explained 113 WHO CALLS THE POSTMORTEM DEBUGGER? I was trying to understand... rsp,158h rbp,[rsp+80h] byte ptr [rbp-55h],1 qword ptr [rbp-50h],rax qword ptr [rbp-48h],rcx qword ptr [rbp-40h],rdx qword ptr [rbp-38h],r8 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 108 PART 2: Professional Crash DumpAnalysis Error code 6 is 110 in binary and volume 3A of Intel manual tells us that “the fault was caused by a non-present page” (bit 0 is cleared), “the... if values occupy only 16 or 32-bit Therefore we can try to find CS and SS on the stack because they have the following constant values: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 102 PART 2: Professional Crash DumpAnalysis 6: kd> r cs cs=0010 6: kd> r ss ss=0018 6: kd> k Child-SP fffffadc`6e02b9e8 … … … fffffadc`6e02cd70 fffffadc`6e02cdd0 RetAddr Call Site fffff800`013731b1... nt!KiExceptionExit nt!KiPageFault+0x1e1 (TrapFrame @ driver+0×44ed3 driver+0×465f7 nt!PspSystemThreadStartup+0×3e nt!KxStartSystemThread+0×16 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 104 PART 2: Professional Crash DumpAnalysis EXCEPTIONS IN USER MODE Previous articles were dealing with exceptions in kernel mode Now I’m going to investigate the flow of exception processing... Win32Thread: 0000000000000000 WAIT: (Unknown) KernelMode Non-Alertable SuspendCount 1 FreezeCount 1 fffffadfe734c670 Semaphore Limit 0x2 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 106 PART 2: Professional Crash DumpAnalysis kd> thread /r /p fffffadfe78f2bf0 Implicit thread is now fffffadf`e78f2bf0 Implicit process is now fffffadf`e7055c20 Loading User Symbols kd> kL... faultrep!ReportFault+0×533 kernel32!UnhandledExceptionFilter+0×587 TestDefaultDebugger+0×11eaa TestDefaultDebugger+0xe879 kernel32!BaseProcessStart+0×23 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 114 PART 2: Professional Crash DumpAnalysis The combination of StartDWException and WaitForMultipleObjects suggests that dwwin.exe process is started there Indeed, when I disassembled StartDWException... to our TestDefaultDebugger process we would no longer see our default unhandled exception filter waiting for the error reporting process: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 118 PART 2: Professional Crash DumpAnalysis Windows XP 0:000> k ChildEBP RetAddr 0012d318 7c90e9ab ntdll!KiFastSystemCallRet 0012d31c 7c8094e2 ntdll!ZwWaitForMultipleObjects+0xc 0012d3b8 7c80a075... HardErrorMode: 0012f9e8 00130000 0012d000 00000000 00001e00 00000000 7ffdf000 00000000 0000120c 0000148c 00000000 7ffdf02c 7ffda000 0 c000008a 0 0 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 120 PART 2: Professional Crash DumpAnalysis 0:000>dds 0012d000 00130000 … … … 0012f368 0012f3c0 0012f36c 7760fb01 kernel32!GetApplicationRecoveryCallback+0×33 0012f370 ffffffff 0012f374 0012f380 . kernel32!BaseProcessStart+0x23 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 92 PART 2: Professional Crash Dump Analysis TRAP COMMAND ON X86. –v command: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 94 PART 2: Professional Crash Dump Analysis 3: kd> !analyze