Tài liệu Memory Dump Analysis Anthology- P23 ppt

Windows Service Crash Dumps in Vista 661 Loading Dump File [C:\ProgramData\Microsoft\Windows\WER\ReportQueue\Report19527353 \WER7346.tmp.mdmp] User Mini Dump File: Only registers, stack and portions of memory are available Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Vista Version 6000 MP (2 procs) Free x64 Product: WinNt, suite: SingleUserTS Debug session time: Fri Sep 28 16:36:38.000 2007 (GMT+1) System Uptime: 2 days 1:42:22.810 Process Uptime: 0 days 0:00:10.000 This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (13b0.d54): Access violation - code c0000005 (first/second chance not available) simple!service_ctrl+0x19: 00000000`00401ab9 c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=???????? 0:000> k Child-SP RetAddr Call Site 00000000`0012fab0 000007fe`fe276cee simple!service_ctrl+0x19 00000000`0012faf0 000007fe`fe2cea5d advapi32!ScDispatcherLoop+0x54c 00000000`0012fbf0 00000000`004019f5 advapi32!StartServiceCtrlDispatcherA+0x8d 00000000`0012fe70 00000000`00408b8c simple!main+0x155 00000000`0012fec0 00000000`0040895e simple!__tmainCRTStartup+0x21c 00000000`0012ff30 00000000`7792cdcd simple!mainCRTStartup+0xe 00000000`0012ff60 00000000`77a7c6e1 kernel32!BaseThreadInitThunk+0xd 00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d Fault in any other service thread, for example, the one that SCM starts per every SERVICE_TABLE_ENTRY in dispatch table results in a default postmortem debugger sav- ing a crash dump on Windows Server 2003 x64 but not on Vista x64 or Vista x86 (32-bit): void __cdecl main(int argc, char **argv) { SERVICE_TABLE_ENTRY dispatchTable[] = { { TEXT(SZSERVICENAME), (LPSERVICE_MAIN_FUNCTION)service_main}, { NULL, NULL} }; if (!StartServiceCtrlDispatcher(dispatchTable)) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 662 PART 13: Miscelleneous AddToMessageLog(TEXT("StartServiceCtrlDispatcher failed.")); } void WINAPI service_main(DWORD dwArgc, LPTSTR *lpszArgv) { // register our service control handler: // sshStatusHandle = RegisterServiceCtrlHandler( TEXT(SZSERVICENAME), service_ctrl); if (!sshStatusHandle) goto cleanup; // SERVICE_STATUS members that don't change in example // ssStatus.dwServiceType = SERVICE_WIN32_OWN_PROCESS; ssStatus.dwServiceSpecificExitCode = 0; // report the status to the service control manager. // if (!ReportStatusToSCMgr( SERVICE_START_PENDING, // service state NO_ERROR, // exit code 3000)) // wait hint goto cleanup; *(int *)NULL = 0; … … … } Seems the only way to get a crash minidump for analysis is to copy it from the re- port data like I explained above: Loading Dump File [C:\ProgramData\Microsoft\Windows\WER\ReportQueue\Report0fa05f9d \WER5F42.tmp.mdmp] User Mini Dump File: Only registers, stack and portions of memory are available Symbol search path is: srv*c:\mss*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Vista Version 6000 MP (2 procs) Free x64 Product: WinNt, suite: SingleUserTS Debug session time: Fri Sep 28 17:50:06.000 2007 (GMT+1) System Uptime: 0 days 0:30:59.495 Process Uptime: 0 days 0:00:04.000 This dump file has an exception of interest stored in it. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Windows Service Crash Dumps in Vista 663 The stored exception information can be accessed via .ecxr. (d6c.fcc): Access violation - code c0000005 (first/second chance not available) simple!service_main+0x60: 00000000`00401aa0 c704250000000000000000 mov dword ptr [0],0 ds:00000000`00000000=???????? 0:001> ~*k 0 Id: d6c.cf4 Suspend: 0 Teb: 000007ff`fffdd000 Unfrozen Child-SP RetAddr Call Site 00000000`0012f978 00000000`777026da ntdll!NtReadFile+0xa 00000000`0012f980 000007fe`feb265aa kernel32!ReadFile+0x8a 00000000`0012fa10 000007fe`feb262e3 advapi32!ScGetPipeInput+0x4a 00000000`0012faf0 000007fe`feb7ea5d advapi32!ScDispatcherLoop+0x9a 00000000`0012fbf0 00000000`004019f5 advapi32!StartServiceCtrlDispatcherA+0x8d 00000000`0012fe70 00000000`00408bac simple!main+0x155 00000000`0012fec0 00000000`0040897e simple!__tmainCRTStartup+0x21c 00000000`0012ff30 00000000`7770cdcd simple!mainCRTStartup+0xe 00000000`0012ff60 00000000`7792c6e1 kernel32!BaseThreadInitThunk+0xd 00000000`0012ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d # 1 Id: d6c.fcc Suspend: 0 Teb: 000007ff`fffdb000 Unfrozen Child-SP RetAddr Call Site 00000000`008eff00 000007fe`feb24bf5 simple!service_main+0x60 00000000`008eff30 00000000`7770cdcd advapi32!ScSvcctrlThreadW+0x25 00000000`008eff60 00000000`7792c6e1 kernel32!BaseThreadInitThunk+0xd 00000000`008eff90 00000000`00000000 ntdll!RtlUserThreadStart+0x1d Spawning a custom thread with NULL pointer access violation doesn’t result in a crash dump on my Vista x86 and x64 too. Therefore it appears that there are no auto- matic postmortem crash dumps saved for native Window services in Vista unless there is some setting that I missed. This might create some problems for traditional 3rd party Technical Support procedures. However it appears that there is a possible solution with Vista SP1 and Windows Server 2003 (page 606). Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 664 PART 13: Miscelleneous THE ROAD TO KERNEL SPACE If you are developing and debugging user space applications (and/or doing crash dump analysis in user space) and you want to understand Windows kernel dumps and device drivers better (and probably start writing your own kernel tools) here is the read- ing list I found the most effective: 0. Read and re-read Windows Internals book in parallel while reading all other books. It shows you the big picture and some useful WinDbg commands and techniques but you need to read device driver books to fill the gaps and be confident in kernel space. 1. Start with “The Windows 2000 Device Driver Book: A Guide for Programmers (2nd Edition)”. This short book shows the basics and you can start writing drivers and kernel tools immediately. 2. Next read “Windows NT Device Driver Development” book to consolidate your knowledge. 3. Don’t stop here. Read “Developing Windows NT Device Drivers: A Programmer’s Handbook”. This very good book explains everything in great detail and good pictures. 4. Continue with WDM drivers and modern presentation: “Programming the Microsoft Windows Driver Model, Second Edition”. Must read even if your drivers are not WDM. 5. Finally read “Developing Drivers with the Windows Driver Foundation” book as this is the future and it also covers ETW (event tracing for Windows), WinDbg exten- sions, PREfast and static driver verifier. Additional reading (not including DDK Help which you will use anyway) can be done in parallel after finishing “Windows NT Device Driver Development” book: 1. OSR NT Insider articles: http://www.osronline.com 2. “Windows NT File System Internals”. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. The Road to Kernel Space 665 3. “Rootkits: Subverting the Windows Kernel” book shows Windows kernel from a hacker perspective. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 666 PART 13: Miscelleneous MEMORY DUMP ANALYSIS INTERVIEW QUESTIONS The following interview questions might be useful to assess the skill level in crash dump analysis on Windows platforms. These could be useful for debugging interviews as well. 1. What is FPO? 2. How many exceptions can be found in a crash dump? 3. You see the following message from WinDbg: WARNING: Stack unwind information not available. Following frames may be wrong. What would you do? 4. How would you find spinlock implementation if you have a kernel dump? 5. What is OMAP? 6. What is full page heap? 7. Company name is missing from module information. How would you try to find it? 8. What is IDT? 9. How does a postmortem debugger work? 10. You’ve got a mini dump of your application. How would you disassemble the code? 11. Memory consumption is growing for an application. How would you discover the leaking component? 12. What is IRQL? 13. When do you use TEB? 14. You’ve got 200 process dumps from a server. You need to find a deadlock. How would you do it? 15. You’ve got a complete memory dump from a server. You need to find a dead- lock. How would you do it? 16. What is GC heap? 17. Your customer is reluctant to send a dump file due to security policies. What is your next step? 18. What is a first chance exception? Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Music for Debugging 667 MUSIC FOR DEBUGGING Debugging and understanding multithreaded programs is hard and sometimes it requires running several execution paths mentally. Here listening to composers who use multithreading in music can help. My favorite is J.S. Bach. Virtuoso and heroic music helps me in live debugging too and here my favorites are Chopin, Liszt and Beethoven. Many software engineers listen to music when writing code and I’m not the exception. However, I have found that not all music suitable for programming helps me during debugging sessions. Music for relaxation, quiet classical or modern music helps me to think about program design and write solid code. Music with several melodies played simulta- neously, heroic and virtuoso works help me to achieve breakthrough and find a bug. The latter kind of music also suits me for listening when doing crash dump analysis or prob- lem troubleshooting. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 668 PART 13: Miscelleneous PDBFINDER Version 3.5 uses the new binary database format and achieves the following results compare to the previous version 3.0.1: 2 times smaller database size 5 times faster database load time on startup! It is fully backwards compatible with 3.0.1 and 2.x database formats and silently converts your old database to the new format on the first load. Additionally the new version fixes the bug in version 3.0.1 sometimes manifested when removing and then adding folders before building the new database which resulted in incorrectly built database. The next version 4.0 is currently under development and it will have the following features: The ability to open multiple databases The ability to exclude certain folders during build to avoid excessive search results output Fully configurable OS and language search options (which are currently disabled for public version) PDBFinder upgrade is available for download from Citrix support: http://support.citrix.com/article/CTX110629. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. When a Process Dies Silently 669 WHEN A PROCESS DIES SILENTLY There are cases when a default postmortem debugger doesn’t save a dump file. This is because the default postmortem debugger is called from the crashed application thread on Windows prior to Vista and if a thread stack is exhausted or critical thread data is corrupt there is no user dump. On Vista the default postmorten debugger is called from WER (Windows Error Reporting) process WerFault.exe so there is a chance that it can save a user dump. During my experiments today on Windows 2003 (x64) I found that if we have a stack overflow inside a 64-bit process then the process silently dies. This doesn’t happen for 32-bit processes on the same server or on a native 32-bit OS. Here is the added code from the modified default Win32 API project created in Vis- ual Studio 2005: volatile DWORD dwSupressOptimization; void SoFunction(); LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam) { case WM_PAINT: hdc = BeginPaint(hWnd, &ps); SoFunction(); EndPaint(hWnd, &ps); break; } void SoFunction() { if (++dwSupressOptimization) { SoFunction(); WndProc(0,0,0,0); } } Adding WndProc call to SoFunction is done to eliminate an optimization in Re- lease build when a recursion call is transformed into a loop: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 670 PART 13: Miscelleneous void SoFunction() { if (++dwSupressOptimization) { SoFunction(); } } 0:001> uf SoFunction 00401300 mov eax,1 00401305 jmp StackOverflow!SoFunction+0x10 (00401310) 00401310 add dword ptr [StackOverflow!dwSupressOptimization (00403374)],eax 00401316 mov ecx,dword ptr [StackOverflow!dwSupressOptimization (00403374)] 0040131c jne StackOverflow!SoFunction+0x10 (00401310) 0040131e ret Therefore without WndProc added or more complicated SoFunction there is no stack overflow but a loop with 4294967295 (0xFFFFFFFF) iterations. If we compile an x64 project with WndProc call included in SoFunction and run it we would never get a dump file from any default postmortem debugger although TestDefaultDebugger64 (page 641) tool crashes with a dump. We can also observe a strange behavior that the application disappears only during the second window repaint although it shall crash immediately when we launch it and the main window is shown. What we see is when we launch the application it is running and the main window is visible. When we force it to repaint by minimizing and then maximizing, for example, only then it disappears from the screen and the process list. If we launch 64-bit WinDbg, load and run our application we would hit the first chance exception: 0:000> g (159c.fc4): Stack overflow - code c00000fd (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. StackOverflow!SoFunction+0x22: 00000001`40001322 e8d9ffffff call StackOverflow!SoFunction (00000001`40001300) Stack trace looks like normal stack overflow: 0:000> k Child-SP RetAddr Call Site 00000000`00033fe0 00000001`40001327 StackOverflow!SoFunction+0x22 00000000`00034020 00000001`40001327 StackOverflow!SoFunction+0x27 00000000`00034060 00000001`40001327 StackOverflow!SoFunction+0x27 00000000`000340a0 00000001`40001327 StackOverflow!SoFunction+0x27 00000000`000340e0 00000001`40001327 StackOverflow!SoFunction+0x27 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... everything from GUI down to machine language instructions or up Certainly understanding how software works at every level is very helpful in memory dump analysis Seeing thread stacks in memory dumps helps in understanding software The more we know the better we are at dump analysis and debugging Debugging is not about stepping through the code This is a very narrow view of a specialist programmer Programmer... this watermark Dr Watson Logs Analysis 689 We see that when the log entry was saved there were no symbols available and this is the most common case If we have such a log and no corresponding user dump (perhaps it was overwritten) then we can still reconstruct stack trace To do this we run WinDbg, set path to our application symbol files and load our application as a crash dump: Microsoft (R) Windows... silently There was no stack space left for exception dispatch handler functions and therefore for the default unhandled exception filter that launches the default postmortem debugger to save a process dump So it looks like on x64 Windows when our process had first chance stack overflow exception there was no second chance exception afterwards and after handling the first chance stack overflow exception... debugging at every possible level and therefore can write any possible software layer Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 688 PART 13: Miscelleneous DR WATSON LOGS ANALYSIS The main problem with Dr Watson logs is the lack of symbol information but this can be alleviated by using WinDbg if we have the same binary that crashed and produced the log entry I’m going to... Environment pattern (page 283) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Process and Thread Startup in Vista 679 PROCESS AND THREAD STARTUP IN VISTA If we looked at process dumps from Vista or did live debugging we might have noticed that there are no longer kernel32 functions BaseProcessStart on the main thread stack and BaseThreadStart on subsequent thread stacks In Vista... WinDbg, set path to our application symbol files and load our application as a crash dump: Microsoft (R) Windows Debugger Version 6.6.0007.5 Copyright (c) Microsoft Corporation All rights reserved Loading Dump File [C:\Work\TestDefaultDebugger.exe] Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols;c:\work Executable search path is: ModLoad: 00400000 00435000 C:\Work\TestDefaultDebugger.exe . Miscelleneous MEMORY DUMP ANALYSIS INTERVIEW QUESTIONS The following interview questions might be useful to assess the skill level in crash dump analysis. the only way to get a crash minidump for analysis is to copy it from the re- port data like I explained above: Loading Dump File [C:ProgramDataMicrosoftWindowsWERReportQueueReport0fa05f9d