Tài liệu Memory Dump Analysis Anthology- P21 docx

WinDbg is Privacy-Aware 601 002dfc38 7d9472d8 00580a9e 00000000 00000000 Button_WndProc 002dfc64 7d9475c3 7dbfa313 00580a9e 00000000 InternalCallWinProc 002dfcdc 7d9477f6 00000000 7dbfa313 00580a9e UserCallWinProcCheckWow 002dfd54 7d947838 00000000 00000000 002dfd90 DispatchMessageWorker 002dfd64 7d956ca0 00000000 00000000 002dfe90 DispatchMessageW 002dfd90 0040568b 00000000 00000000 002dfe90 IsDialogMessageW 002dfda0 004065d8 00000000 00402a07 00000000 IsDialogMessageW 002dfda8 00402a07 00000000 00000000 00000000 PreTranslateInput 002dfdb8 00408041 00000000 00000000 002dfe90 PreTranslateMessage 002dfdc8 00403ae3 00000000 00000000 00000000 WalkPreTranslateTree 002dfddc 00403c1e 00000000 00403b29 00000000 AfxInternalPreTranslateMessage 002dfde4 00403b29 00000000 00403c68 00000000 PreTranslateMessage 002dfdec 00403c68 00000000 00000000 002dfe90 AfxPreTranslateMessage 002dfdfc 00407920 00000000 002dfe90 002dfe6c AfxInternalPumpMessage 002dfe20 004030a1 00000000 00000000 0042ec18 CWnd::RunModalLoop 002dfe6c 0040110d 00000000 0042ec18 0042ec18 CDialog::DoModal 002dff18 004206fb 00000000 00000000 00000000 InitInstance 002dff28 0040e852 00400000 00000000 00000000 AfxWinMain 002dffc0 7d4e992a 00000000 00000000 00000000 __tmainCRTStartup 002dfff0 00000000 0040e8bb 00000000 00000000 BaseProcessStart We can see that most arguments are zeroes. Those that are not, either do not point to valid data or correspond to function return addresses and frame pointers. This can be seen from the raw stack data as well: 0:000> dds esp 002df86c 00403263 TestDefaultDebugger!_AfxDispatchCmdMsg+0x43 002df870 00425ae8 TestDefaultDebugger!CTestDefaultDebuggerApp::`vftable'+0x154 002df874 00000000 002df878 002df8a8 002df87c 00403470 TestDefaultDebugger!CCmdTarget::OnCmdMsg+0x118 002df880 002dfe90 002df884 00000000 002df888 00000000 002df88c 004014f0 TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1 002df890 00000000 002df894 00000000 002df898 00000000 002df89c 002dfe90 002df8a0 00000000 002df8a4 00000000 002df8a8 002df8cc 002df8ac 00402a27 TestDefaultDebugger!CDialog::OnCmdMsg+0x1b 002df8b0 00000000 002df8b4 00000000 002df8b8 00000000 002df8bc 00000000 002df8c0 00000000 002df8c4 002dfe90 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 602 PART 10: Security 002df8c8 00000000 002df8cc 002df91c 002df8d0 00408e69 TestDefaultDebugger!CWnd::OnCommand+0x90 002df8d4 00000000 002df8d8 00000000 002df8dc 00000000 002df8e0 00000000 002df8e4 002dfe90 002df8e8 002dfe90 We can compare it with the normal full or minidump saved with other /m op- tions. The data zeroed when we use /mr option is shown in bold (module names and function offsets are removed for visual clarity): 0:000> kvL 100 ChildEBP RetAddr Args to Child 002df868 00403263 00425ae8 00000111 002df8a8 OnBnClickedButton1 002df878 00403470 002dfe90 000003e8 00000000 _AfxDispatchCmdMsg 002df8a8 00402a27 000003e8 00000000 00000000 OnCmdMsg 002df8cc 00408e69 000003e8 00000000 00000000 OnCmdMsg 002df91c 004098d9 00000000 00271876 d5b6c7f7 OnCommand 002df9b8 00406258 00000111 000003e8 00271876 OnWndMsg 002df9d8 0040836d 00000111 000003e8 00271876 WindowProc 002dfa40 004083f4 00000000 00561878 00000111 AfxCallWndProc 002dfa60 7d9472d8 00561878 00000111 000003e8 AfxWndProc 002dfa8c 7d9475c3 004083c0 00561878 00000111 InternalCallWinProc 002dfb04 7d948626 00000000 004083c0 00561878 UserCallWinProcCheckWow 002dfb48 7d94868d 00aec860 00000000 00000111 SendMessageWorker 002dfb6c 7dbf87b3 00561878 00000111 000003e8 SendMessageW 002dfb8c 7dbf8895 002ec9e0 00000000 0023002c Button_NotifyParent 002dfba8 7dbfab9a 002ec9e0 00000001 002dfcb0 Button_ReleaseCapture 002dfc38 7d9472d8 00271876 00000202 00000000 Button_WndProc 002dfc64 7d9475c3 7dbfa313 00271876 00000202 InternalCallWinProc 002dfcdc 7d9477f6 00000000 7dbfa313 00271876 UserCallWinProcCheckWow 002dfd54 7d947838 002e77f8 00000000 002dfd90 DispatchMessageWorker 002dfd64 7d956ca0 002e77f8 00000000 002dfe90 DispatchMessageW 002dfd90 0040568b 00561878 00000000 002dfe90 IsDialogMessageW 002dfda0 004065d8 002e77f8 00402a07 002e77f8 IsDialogMessageW 002dfda8 00402a07 002e77f8 002e77f8 00561878 PreTranslateInput 002dfdb8 00408041 002e77f8 002e77f8 002dfe90 PreTranslateMessage 002dfdc8 00403ae3 00561878 002e77f8 002e77f8 WalkPreTranslateTree 002dfddc 00403c1e 002e77f8 00403b29 002e77f8 AfxInternalPreTranslateMessage 002dfde4 00403b29 002e77f8 00403c68 002e77f8 PreTranslateMessage 002dfdec 00403c68 002e77f8 00000000 002dfe90 AfxPreTranslateMessage 002dfdfc 00407920 00000004 002dfe90 002dfe6c AfxInternalPumpMessage 002dfe20 004030a1 00000004 d5b6c023 0042ec18 RunModalLoop 002dfe6c 0040110d d5b6c037 0042ec18 0042ec18 DoModal 002dff18 004206fb 00000ece 00000002 00000001 InitInstance 002dff28 0040e852 00400000 00000000 001d083e AfxWinMain 002dffc0 7d4e992a 00000000 00000000 7efdf000 __tmainCRTStartup 002dfff0 00000000 0040e8bb 00000000 000000c8 BaseProcessStart Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. WinDbg is Privacy-Aware 603 0:000> dds esp 002df86c 00403263 TestDefaultDebugger!_AfxDispatchCmdMsg+0x43 002df870 00425ae8 TestDefaultDebugger!CTestDefaultDebuggerApp::`vftable'+0x154 002df874 00000111 002df878 002df8a8 002df87c 00403470 TestDefaultDebugger!CCmdTarget::OnCmdMsg+0×118 002df880 002dfe90 002df884 000003e8 002df888 00000000 002df88c 004014f0 TestDefaultDebugger!CTestDefaultDebuggerDlg::OnBnClickedButton1 002df890 00000000 002df894 00000038 002df898 00000000 002df89c 002dfe90 002df8a0 000003e8 002df8a4 00000000 002df8a8 002df8cc 002df8ac 00402a27 TestDefaultDebugger!CDialog::OnCmdMsg+0×1b 002df8b0 000003e8 002df8b4 00000000 002df8b8 00000000 002df8bc 00000000 002df8c0 000003e8 002df8c4 002dfe90 002df8c8 00000000 002df8cc 002df91c 002df8d0 00408e69 TestDefaultDebugger!CWnd::OnCommand+0×90 002df8d4 000003e8 002df8d8 00000000 002df8dc 00000000 002df8e0 00000000 002df8e4 002dfe90 002df8e8 002dfe90 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 604 PART 10: Security CRASH DUMPS AND SECURITY Suppose you work in a banking industry or for any company that has sensitive information. Is it secure to send a crash dump outside for analysis? One semi-anonym- ous person asked this question on Crash Dump Analysis forum and here is my answer based on my experience in crash dump analysis and kernel level development: "It depends on credit card transaction software design and architecture and what type of memory dump is configured in Control Panel\System\ Advanced\Startup and Recovery applet: Small, Kernel or Complete. Software usually encrypts data before sending it down TCP/IP stack or other network protocol. If a credit card transaction software doesn't have any kernel space encryption drivers and doesn't rely on any Microsoft or other third-party encryption API that might send data to kernel, communicate to KSECDD or to a user-space component like LSASS via LPC/RPC, we can safely assume that kernel memory dumps will not have unencrypted data. If encryption is done entirely in user space Small memory dump and Kernel memory dump will only have encrypted fragments. Otherwise there is a probability that BSOD happens just before encryption or after decryption or when a secure protocol is being handled. This exposure can even happen in Small memory dumps if BSOD happens in the thread that handles sensitive information in kernel mode. The same applies if software stores credit data on any medium. If it stores only encrypted data and decrypts entirely in user space without any transition to kernel it should be safe to enable kernel memory dump. If our goal is ultimate security then even Small memory dump (64Kb) should not be allowed. But in reality as we consider probabilities sending a small memory dump is equivalent to no more than exposing just one credit card number or just one password. What we must avoid at any cost is to enable complete memory dump option in Control Panel. In this case all credit card transaction software code and data including file system cache will be exposed. Contrary to complete memory dump, kernel memory dump will not have much data even if some potion of it is being communicated during the crash time." If you are interested too you can participate in that discussion: http://www.dumpanalysis.org/forum/viewtopic.php?t=56 or see the solution from WinDbg (page 600). Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. JIT Service Debugging 605 PART 11: THE ORIGIN OF CRASH DUMPS JIT SERVICE DEBUGGING If we have services running under network service account (prior to Vista) and they crash we can use NTSD from recent Debugging Tools for Windows and -noio switch as described in the following article: NTSD as a better Dr. Watson http://www.debuginfo.com/articles/ntsdwatson.html We need to copy the latest ntsd.exe, dbghelp.dll and dbgeng.dll to some folder if we don’t want to install Debugging Tools for Windows in a production environment. The example of AeDebug key we can use for 64-bit JIT debugging is C:\ntsd\ntsd -p %ld -e %ld -g -noio -c ".dump /ma /u c:\TEMP\new.dmp; q" It is always good to double check these settings with TestDefaultDebugger tool (page 641). Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 606 PART 11: The Origin of Crash Dumps LOCAL CRASH DUMPS IN VISTA It appears that Microsoft decided to help customers to save full user dumps locally for later postmortem analysis. According to MSDN this can be done with using LocalDumps registry key starting from Vista SP1 and Windows Server 2008: http://msdn2.microsoft.com/en-us/library/bb787181.aspx This is a quote from the article above: […] Prior to application termination, the system will check the registry settings to determine whether a local dump is to be collected. The registry settings control whether a full dump is collected versus a minidump. The custom flags specified also determine which information is collected in the dump. […] You can make use of the local dump collection even if WER is disabled. The local dumps are collected even if the user cancels WER reporting at any point. […] From my understanding it is independent from the default postmortem debugger mechanism via AeDebug registry key. If it works then full user dump collection might be easier in production environments because of no need to install Debugging Tools for Windows to set up a postmortem debugger. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. COM+ Crash Dumps 607 COM+ CRASH DUMPS If we have problems with COM+ components we can configure Component Ser- vices in Control Panel to save a crash dump: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 608 PART 11: The Origin of Crash Dumps Refer to the following article for details: http://msdn.microsoft.com/msdnmag/issues/01/08/ComXP/ If we want to use userdump.exe to save a crash dump when a failing COM+ application displays an error dialog box the following article might help: http://support.microsoft.com/kb/287643 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. COM+ Crash Dumps 609 If we want crash dumps to be automatically collected after some timeout value refer to the following article for details: http://support.microsoft.com/kb/910904/ If we have an exception the following article describes how to get a stack trace from a saved process dump: http://support.microsoft.com/kb/317317 The following article explains how COM+ handles application faults: Fault Isolation and Failfast Policy http://msdn2.microsoft.com/en- us/library/ms679253.aspx Now I show how to get an error message that was written to event log when COM+ application was terminated due to a different error code than an access violation. If we get a crash dump from COM+ process we need to look at all threads and find the one that runs through comsvcs.dll (shown in small font for visual clarity): 0:000> ~*kL 6 Id: 8d4.1254 Suspend: 0 Teb: 7ffd9000 Unfrozen ChildEBP RetAddr Args to Child 0072ee30 7c822124 77e6baa8 00000394 00000000 ntdll!KiFastSystemCallRet 0072ee34 77e6baa8 00000394 00000000 00000000 ntdll!NtWaitForSingleObject+0xc 0072eea4 77e6ba12 00000394 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac 0072eeb8 75c2b250 00000394 ffffffff 0072f640 kernel32!WaitForSingleObject+0x12 0072f340 75c2bb91 75b8e7fc 75b8e810 000008d4 comsvcs!FF_RunCmd+0xa2 0072f60c 75c2bc76 0072f640 75c6c5c0 0072fe44 comsvcs!FF_DumpProcess_MD+0x21a 0072f850 75c2be83 00000000 77ce21ce 0bd5f0f0 comsvcs!FF_DumpProcess+0×39 0072fdc0 75c2c351 75c6c5c0 75b8b008 00000142 comsvcs!FailFastStr+0×2ce 0072fe20 75bf31fa 0072fe44 75b8b008 00000142 comsvcs!CError::WriteToLog+0×198 0072fe8c 75bf3d48 0bcf5d0c 00000000 0bcf5cf8 comsvcs!CSurrogateServices::FireApplicationLaunch+0×13b 0072fee0 75bf3e19 75bf3e01 0072ff44 7c81a3c5 comsvcs!CApplication::AsyncApplicationLaunch+0×101 0072feec 7c81a3c5 0bcf5cf8 7c889880 0bcf5d50 comsvcs!CApplication::AppLaunchThreadProc+0×18 0072ff44 7c8200fc 75bf3e01 0bcf5cf8 00000000 ntdll!RtlpWorkerCallout+0×71 0072ff64 7c81a3fa 00000000 0bcf5cf8 0bcf5d50 ntdll!RtlpExecuteWorkerRequest+0×4f 0072ff78 7c82017f 7c8200bb 00000000 0bcf5cf8 ntdll!RtlpApcCallout+0×11 0072ffb8 77e66063 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0×61 0072ffec 00000000 7c83ad38 00000000 00000000 kernel32!BaseThreadStart+0×34 … … … FF_DumpProcess function is an indication that the process was being dumped. There is no ComSvcsExceptionFilter function on the thread stack but we can still get an error message if we look at FailFastStr function arguments: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 610 PART 11: The Origin of Crash Dumps 0:000> du 75c6c5c0 75c6c5c0+400 75c6c5c0 ―{646F1874-46B6-4149-BD55-8C317FB‖ 75c6c600 ―71CC0}….Server Application ID:‖ 75c6c640 ‖ {646F1874-46B6-4149-BD55-8C317F‖ 75c6c680 ―B71CC0} Server Application Inst‖ 75c6c6c0 ―ance ID: {7A39BC48-78DA-4FBB-A7″ 75c6c700 ―46-EEA7E42CDAC7} Server Applica‖ 75c6c740 ―tion Name: My Server‖ 75c6c780 ― The serious nature of this err‖ 75c6c7c0 ―or has caused the process to ter‖ 75c6c800 ―minate…Error Code = 0×80131600″ 75c6c840 ‖ : COM+ Services Internals Inf‖ 75c6c880 ―ormation: File: d:\nt\com\compl‖ 75c6c8c0 ―us\src\comsvcs\srgtapi\csrgtserv‖ 75c6c900 ―.cpp, Line: 322 Comsvcs.dll fil‖ 75c6c940 ―e version: ENU 2001.12.4720.2517″ 75c6c980 ‖ shp‖ Also if we examine parameters of FF_RunCmd call we would see what applica- tion was used to dump the process: ChildEBP RetAddr Args to Child 0072f340 75c2bb91 75b8e7fc 75b8e810 000008d4 comsvcs!FF_RunCmd+0xa2 0:000> du 75b8e7fc 75b8e7fc ―%s %d %s‖ 0:000> du 75b8e810 75b8e810 ―RunDll32 comsvcs.dll,MiniDump‖ We can guess that the first parameter is a format string, the second one is a com- mand line for a process dumper, the third one is PID and the fourth one should be the name of a dump file to save. We can double check this from the raw stack: ChildEBP RetAddr Args to Child 0072f340 75c2bb91 75b8e7fc 75b8e810 000008d4 comsvcs!FF_RunCmd+0xa2 0:000> dd 0072f340 0072f340 0072f60c 75c2bb91 75b8e7fc 75b8e810 ; saved EBP, return EIP, 1st param, 2nd param 0072f350 000008d4 0072f640 0072f84a 00000000 ; 3rd param, 4th param 0:000> du 0072f640 0072f640 ―C:\WINDOWS\system32\com\dmp\{646″ 0072f680 ―F1874-46B6-4149-BD55-8C317FB71CC‖ 0072f6c0 ―0}_2007_07_16_12_05_08.dmp‖ Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... Manager and use userdump.exe to save that process dump manually Then inside the dump it is possible to see that error Therefore in the case when a default postmortem debugger wasn’t configured in the registry we can still get a memory dump for postmortem crash dump analysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Correcting Microsoft Article about Userdump.exe 613 Here... to save dumps and logs Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 628 PART 11: The Origin of Crash Dumps SAVEDUMP.EXE AND PAGEFILE I was curious about what savedump.exe does And after some research I found that on Windows 2000 it is a part of the logon process where it copies the crash dump section from the pagefile to a memory dump and also creates a mini dump file... Create the folder where userdump will save our dump files I use c:\UserDumps in my example 5 Copy dbghelp.dll and userdump.exe from x86 or x64 folder depending on the version of Windows we use to system32\kktools folder created in step 3 6 Run the elevated command prompt and enter the following command: C:\Windows\System32\kktools>userdump -I -d c:\UserDumps User Mode Process Dumper (Version 8.1.2929.5)... Origin of Crash Dumps PROCESS CRASH - GETTING THE DUMP MANUALLY Sometimes we have process crashes with exception dialogs but no memory dumps are saved due to some reason, for example, Dr Watson limitation or NTSD doesn’t save dumps on Windows 2000, etc Then one solution is to dump the process manually while it displays an error message Customers and support engineers can use Microsoft userdump.exe for... message box: If we save TestDefaultDebugger process dump manually using userdump.exe when this message box is shown: C:\kktools\userdump8.1\x64>userdump.exe 5264 c:\tdd.dmp User Mode Process Dumper (Version 8.1.2929.4) Copyright (c) Microsoft Corp All rights reserved Dumping process 5264 (TestDefaultDebugger64.exe) to c:\tdd.dmp The process was dumped successfully and open it in WinDbg we can see... have a crash userdump.exe will show a window on top of our screen while saving the dump file: Of course, we can setup userdump.exe as a postmortem debugger on other Windows platforms The problem with userdump.exe is that it overwrites the previous process dump file because it uses the module name for the file name, for example, TestDefaultDebugger.dmp, so we need to rename or save the dump file if we... C:\W2K3\system32\DbgHelp.dll Version: 5.2.3790.1830 C:\kktools\userdump8.1\x64> For most customers running setup.exe and configuring the default rules in Exception Monitor creates the significant amount of False Positive Dumps (page 259) If we want to manually dump a process we don’t need automatically generated memory dumps or fine tune Exception Monitor rules to reduce the number of dump files Just an additional note: if we... to remove this watermark 612 PART 11: The Origin of Crash Dumps CORRECTING MICROSOFT ARTICLE ABOUT USERDUMP.EXE There is much confusion among Microsoft and Citrix customers on how to use userdump.exe to save a process dump Microsoft published an article about this tool and it has the following title: How to use the Userdump.exe tool to create a dump file: http://support.microsoft.com/kb/241215/ Unfortunately... to remove this watermark Dumping Vista 629 DUMPING VISTA 32-bit Vista If we need to dump a running 32-bit Vista system we can do it with Citrix SystemDump tool (page 646) We just need to run it with elevated administrator rights: Right click SystemDump.exe in appropriate Computer explorer folder and choose “Run as administrator If we want to use command line options run SystemDump.exe from elevated command... Command Prompt, and then select “Run as administrator”) Here is a screenshot before dumping my Vista system and WinDbg output from the saved kernel dump: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 630 PART 11: The Origin of Crash Dumps Loading Dump File [C:\Windows \MEMORY. DMP] Kernel Summary Dump File: Only kernel address space is available Windows Vista Kernel Version 6000 . that kernel memory dumps will not have unencrypted data. If encryption is done entirely in user space Small memory dump and Kernel memory dump will only. for analysis? One semi-anonym- ous person asked this question on Crash Dump Analysis forum and here is my answer based on my experience in crash dump analysis