Tài liệu Memory Dump Analysis Anthology- P9 pdf

Case Study 241 CASE STUDY Consider the following legacy C++/Win32 code fragment highlighted in WinDbg after opening a crash dump: 1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); 2: if (hFile != INVALID_HANDLE_VALUE) 3: { 4: DWORD dwSize = GetFileSize(hFile, NULL); 5: DWORD dwRead = 0; 6: CHAR *bufferA = new CHAR[dwSize+2]; 7: memset(bufferA, 0, dwSize+2); 8: if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL)) 9: { 10: DWORD i = 0, j = 0; 11: for (; i < dwSize+2-7; ++i) 12: { 13: if (bufferA[i] == 0xD && bufferA[i+1] != 0xA) At the first glance the code seems to be right: we open a file, get its size and allocate a buffer to read. All loop indexes are within array bounds too. Let’s look at disassembly and crash point: 0:000> uf component!CMyDlg::OnTimer … … … 004021bc push 0 004021be push esi 004021bf call dword ptr [component!_imp__GetFileSize (0042e26c)] 004021c5 mov edi,eax ; dwSize 004021c7 lea ebx,[edi+2] ; dwSize+2 004021ca push ebx 004021cb mov dword ptr [esp+34h],0 004021d3 call component!operator new[] (00408e35) 004021d8 push ebx 004021d9 mov ebp,eax ; bufferA 004021db push 0 004021dd push ebp 004021de call component!memset (00418500) 004021e3 add esp,10h 004021e6 push 0 004021e8 lea edx,[esp+34h] 004021ec push edx 004021ed push edi 004021ee push ebp 004021ef push esi 004021f0 call dword ptr [component!_imp__ReadFile (0042e264)] 004021f6 test eax,eax Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 242 PART 2: Professional Crash Dump Analysis 004021f8 jne component!CMyDlg::OnTimer+0×3b1 (00402331) … … … 00402331 xor esi,esi ; i 00402333 add edi,0FFFFFFFBh ; +2-7 (edi contains dwSize) 00402336 cmp edi,esi ; loop condition 00402338 mov dword ptr [esp+24h],esi 0040233c jbe component!CMyDlg::OnTimer+0×43e (004023be) 00402342 mov al,byte ptr [esi+ebp] ; bufferA[i] 0:000> r eax=00002b00 ebx=00000002 ecx=00431000 edx=00000000 esi=00002b28 edi=fffffffb eip=00402342 esp=0012efd4 ebp=0095b4d8 iopl=0 nv up ei pl nz ac pe cy cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000217 component!CMyDlg::OnTimer+0×3c2: 00402342 8a042e mov al,byte ptr [esi+ebp] ds:0023:0095e000=?? If we look at ebx (dwSize+2) and edi registers (array upper bound, dwSize+2-7) we can easily see that dwSize was zero. Clearly we had buffer overrun because upper array bound was calculated as 0+2-7 = FFFFFFFB (the loop index was unsigned integer, DWORD). Were the index signed integer variable (int) we wouldn’t have had any prob- lem because the condition 0 < 0+2-7 is always false and the loop body would have never been executed. Based on that the following fix was proposed: 1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); 2: if (hFile != INVALID_HANDLE_VALUE) 3: { 4: DWORD dwSize = GetFileSize(hFile, NULL); 5: DWORD dwRead = 0; 6: CHAR *bufferA = new CHAR[dwSize+2]; 7: memset(bufferA, 0, dwSize+2); 8: if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL)) 9: { 10: DWORD i = 0, j = 0; 10: int i = 0, j = 0; 11: for (; i < dwSize+2-7; ++i) 11: for (; i < (int)dwSize+2-7; ++i) 12: { Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Case Study 243 GetFileSize can return INVALID_FILE_SIZE (0xFFFFFFFF) and operator new can fail theoretically (if the size is too big) so we can correct the code even further: 1: HANDLE hFile = CreateFile(str.GetBuffer(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL); 2: if (hFile != INVALID_HANDLE_VALUE) 3: { 4: DWORD dwSize = GetFileSize(hFile, NULL); 4a: if (dwSize != INVALID_FILE_SIZE) 4b: { 5: DWORD dwRead = 0; 6: CHAR *bufferA = new CHAR[dwSize+2]; 6a: if (bufferA) 6b: { 7: memset(bufferA, 0, dwSize+2); 8: if (ReadFile(hFile, bufferA, dwSize, &dwRead, NULL)) 9: { 10: int i = 0, j = 0; 11: for (; i < (int)dwSize+2-7; ++i) 12: { Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 244 PART 2: Professional Crash Dump Analysis DETECTING LOOPS IN CODE Sometimes when we look at a stack trace and disassembled code we see that a crash couldn’t have happened if the code path was linear. In such cases we need to see if there is any loop that changes some variables. This is greatly simplified if we have source code but in cases where we don’t have access to source code it is still possible to detect loops. We just need to find a direct (JMP) or conditional jump instruction (Jxxx, for example, JE) after the crash point branching to the beginning of the loop before the crash point as shown in the following pseudo code: set the pointer value … label: … >>> crash when dereferencing the pointer … change the pointer value … jmp label Let’s look at one example I found very interesting because it also shows __thiscall calling convention for C++ code generated by Visual С++ compiler. Before we look at the dump I quickly remind you about how C++ non-static class methods are called. Let’s first look at non-virtual method call. class A { public: int foo() { return i; } virtual int bar() { return i; } private: int i; }; Internally class members are accessed via implicit this pointer (passed via ECX): int A::foo() { return this->i; } Suppose we have an object instance of class A and we call its foo method: A obj; obj.foo(); Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Detecting Loops in Code 245 The compiler has to generate code which calls foo function and the code inside the function has to know which object it is associated with. So internally the compiler passes implicit parameter - a pointer to that object. In pseudo code: int foo_impl(A *this) { return this->i; } A obj; foo_impl(&obj); In x86 assembly language it should be similar to this code: lea ecx, obj call foo_impl If we have obj declared as a local variable the code is similar: lea ecx, [ebp-N] call foo_impl If we have a pointer to an obj then the compiler usually generates MOV instruc- tion instead of LEA instruction: A *pobj; pobj->foo(); mov ecx, [ebp-N] call foo_impl If we have other function parameters they are pushed on the stack from right to left. This is __thiscall calling convention. For virtual function call we have an indirect call through a virtual function table. The pointer to it is the first object layout member and in the latter case where the pointer to obj is declared as the local variable we have the following x86 code: A *pobj; pobj->bar(); mov ecx, [ebp-N] mov eax, [ecx] call [eax] Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 246 PART 2: Professional Crash Dump Analysis Now let’s look at the crash point and stack trace: 0:021> r eax=020864ee ebx=00000000 ecx=0000005c edx=7518005c esi=020864dc edi=00000000 eip=67dc5dda esp=075de820 ebp=075dea78 iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010202 component!CDirectory::GetDirectory+0×8a: 67dc5dda 8b03 mov eax,dword ptr [ebx] ds:0023:00000000=???????? 0:021> k ChildEBP RetAddr 075dea78 004074f0 component!CDirectory::GetDirectory+0x8a 075deaac 0040e4fc component!CDirectory::FindFirstFileW+0xd0 075dffb8 77e64829 component!MonitorThread+0x13 075dffec 00000000 kernel32!BaseThreadStart+0x34 If we look at GetDirectory code we would see: 0:021> .asm no_code_bytes Assembly options: no_code_bytes 0:021> uf component!CDirectory::GetDirectory component!CDirectory::GetDirectory: 67dc5d50 push ebp 67dc5d51 mov ebp,esp 67dc5d53 push 0FFFFFFFFh 67dc5d55 push offset component!CreateErrorInfo+0x553 (67ded93b) 67dc5d5a mov eax,dword ptr fs:[00000000h] 67dc5d60 push eax 67dc5d61 mov dword ptr fs:[0],esp 67dc5d68 sub esp,240h 67dc5d6e mov eax,dword ptr [component!__security_cookie (67e0113c)] 67dc5d73 mov dword ptr [ebp-10h],eax 67dc5d76 mov eax,dword ptr [ebp+8] 67dc5d79 test eax,eax 67dc5d7b push ebx 67dc5d7c mov ebx,ecx 67dc5d7e mov dword ptr [ebp-238h],ebx 67dc5d84 je component!CDirectory::GetDirectory+0×2a1 (67dc5ff1) component!CDirectory::GetDirectory+0x3a: 67dc5d8a cmp word ptr [eax],0 67dc5d8e je component!CDirectory::GetDirectory+0x2a1 (67dc5ff1) component!CDirectory::GetDirectory+0x44: 67dc5d94 push esi 67dc5d95 push eax 67dc5d96 call dword ptr [component!_imp__wcsdup (67df050c)] 67dc5d9c add esp,4 67dc5d9f mov dword ptr [ebp-244h],eax Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Detecting Loops in Code 247 67dc5da5 mov dword ptr [ebp-240h],eax 67dc5dab push 5Ch 67dc5dad lea ecx,[ebp-244h] 67dc5db3 mov dword ptr [ebp-4],0 67dc5dba call component!CStrToken::Next (67dc4f80) 67dc5dbf mov esi,eax 67dc5dc1 test esi,esi 67dc5dc3 je component!CDirectory::GetDirectory+0x28c (67dc5fdc) component!CDirectory::GetDirectory+0x79: 67dc5dc9 push edi 67dc5dca lea ebx,[ebx] component!CDirectory::GetDirectory+0x80: 67dc5dd0 cmp word ptr [esi],0 67dc5dd4 je component!CDirectory::GetDirectory+0x28b (67dc5fdb) component!CDirectory::GetDirectory+0x8a: >>> 67dc5dda mov eax,dword ptr [ebx] 67dc5ddc mov ecx,ebx … If we trace EBX backwards we would see that it comes from ECX so ECX could be considered as an implicit this pointer according to __thiscall calling convention. There- fore it looks like the caller passed NULL this pointer via ECX. Let’s look at the caller. To see the code we can either disassemble FindFirstFileW or disassemble backwards at the GetDirectory return address. We’ll do the latter: 0:021> k ChildEBP RetAddr 075dea78 004074f0 component!CDirectory::GetDirectory+0×8a 075deaac 0040e4fc component!CDirectory::FindFirstFileW+0xd0 075dffb8 77e64829 component!MonitorThread+0×13 075dffec 00000000 kernel32!BaseThreadStart+0×34 0:021> ub 004074f0 component!CDirectory::FindFirstFileW+0xbe: 004074de pop ebp 004074df clc 004074e0 mov ecx,dword ptr [esi+8E4h] 004074e6 mov eax,dword ptr [ecx] 004074e8 push 0 004074ea push 0 004074ec push edx 004074ed call dword ptr [eax+10h] Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 248 PART 2: Professional Crash Dump Analysis We see that ECX is our this pointer. However the virtual table pointer is taken from the memory it references: 004074e6 mov eax,dword ptr [ecx] … … 004074ed call dword ptr [eax+10h] Were ECX a NULL we would have had our crash at this point. However we have our crash in the called function. So it couldn’t be NULL. There is a contradiction here. The only plausible explanation is that in GetDirectory function there is a loop that changes EBX (shown in bold in GetDirectory function code above). If we have a second look at the code we would see that EBX is saved in [ebp-238h] local variable before it is used: 0:021> uf component!CDirectory::GetDirectory component!CDirectory::GetDirectory: 67dc5d50 push ebp 67dc5d51 mov ebp,esp 67dc5d53 push 0FFFFFFFFh 67dc5d55 push offset component!CreateErrorInfo+0x553 (67ded93b) 67dc5d5a mov eax,dword ptr fs:[00000000h] 67dc5d60 push eax 67dc5d61 mov dword ptr fs:[0],esp 67dc5d68 sub esp,240h 67dc5d6e mov eax,dword ptr [component!__security_cookie (67e0113c)] 67dc5d73 mov dword ptr [ebp-10h],eax 67dc5d76 mov eax,dword ptr [ebp+8] 67dc5d79 test eax,eax 67dc5d7b push ebx 67dc5d7c mov ebx,ecx 67dc5d7e mov dword ptr [ebp-238h],ebx 67dc5d84 je component!CDirectory::GetDirectory+0×2a1 (67dc5ff1) component!CDirectory::GetDirectory+0x3a: 67dc5d8a cmp word ptr [eax],0 67dc5d8e je component!CDirectory::GetDirectory+0x2a1 (67dc5ff1) component!CDirectory::GetDirectory+0x44: 67dc5d94 push esi 67dc5d95 push eax 67dc5d96 call dword ptr [component!_imp__wcsdup (67df050c)] 67dc5d9c add esp,4 67dc5d9f mov dword ptr [ebp-244h],eax 67dc5da5 mov dword ptr [ebp-240h],eax 67dc5dab push 5Ch 67dc5dad lea ecx,[ebp-244h] 67dc5db3 mov dword ptr [ebp-4],0 67dc5dba call component!CStrToken::Next (67dc4f80) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Detecting Loops in Code 249 67dc5dbf mov esi,eax 67dc5dc1 test esi,esi 67dc5dc3 je component!CDirectory::GetDirectory+0x28c (67dc5fdc) component!CDirectory::GetDirectory+0x79: 67dc5dc9 push edi 67dc5dca lea ebx,[ebx] component!CDirectory::GetDirectory+0x80: 67dc5dd0 cmp word ptr [esi],0 67dc5dd4 je component!CDirectory::GetDirectory+0x28b (67dc5fdb) component!CDirectory::GetDirectory+0x8a: >>> 67dc5dda mov eax,dword ptr [ebx] 67dc5ddc mov ecx,ebx … If we look further past the crash point we would see that [ebp-238h] value is changed and then used again to change EBX: component!CDirectory::GetDirectory+0x80: 67dc5dd0 cmp word ptr [esi],0 67dc5dd4 je component!CDirectory::GetDirectory+0×28b (67dc5fdb) component!CDirectory::GetDirectory+0x8a: >>> 67dc5dda mov eax,dword ptr [ebx] 67dc5ddc mov ecx,ebx … … … component!CDirectory::GetDirectory+0×11e: 67dc5e6e mov eax,dword ptr [ebp-23Ch] 67dc5e74 mov ecx,dword ptr [eax] 67dc5e76 mov dword ptr [ebp-238h],ecx 67dc5e7c jmp component!CDirectory::GetDirectory+0×20e (67dc5f5e) … … … component!CDirectory::GetDirectory+0×23e: 67dc5f8e cmp esi,edi 67dc5f90 mov ebx,dword ptr [ebp-238h] 67dc5f96 jne component!CDirectory::GetDirectory+0×80 (67dc5dd0) We see that after changing EBX the code jumps to 67dc5dd0 address and this ad- dress is just before our crash point. It looks like a loop. Therefore there is no contradic- tion. ECX as this pointer was passed as non-NULL and valid pointer. Before the loop started its value was passed to EBX. In the loop body EBX was changed and after some loop iterations the new value became NULL. It could be the case that there were no checks for NULL pointers in the loop code. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 250 PART 2: Professional Crash Dump Analysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... purchase PDF Split-Merge on www.verypdf.com to remove this watermark 254 PART 2: Professional Crash Dump Analysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Multiple Exceptions 255 PART 3: CRASH DUMP ANALYSIS PATTERNS MULTIPLE EXCEPTIONS After doing crash dump analysis for some time I decided to organize my knowledge into a set of patterns (so to speak in a memory dump analysis. ..Crash Dump Analysis Checklist 251 CRASH DUMP ANALYSIS CHECKLIST Often the root cause of a problem is not obvious from a memory dump Here is the first version of crash dump analysis checklist to help experienced engineers not to miss any important information The check list doesn’t prescribe any specific steps, just lists all possible points to double check when looking at a memory dump General:... BSOD: • Default analysis (!analyze -v) • Pool address (!pool) • Component timestamps • Processes and virtual memory (!vm 4) • Current threads on other processors • Raw stack • Bugcheck description (including ln exception address for corrupt or truncated dumps) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Crash Dump Analysis Poster (HTML version) 253 CRASH DUMP ANALYSIS POSTER... (http://support.microsoft.com/kb/188831) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark False Positive Dump 259 FALSE POSITIVE DUMP Another frequent pattern is False Positive Dump Here we get crash dump files pointing to a wrong direction or not useful for analysis and this usually happens when a wrong tool was selected or the right one was not properly configured for capturing crash dumps Here is one example... Process Dumper will catch it and save the dump The customer should have used NTSD as a default postmortem debugger Then if a crash happened we would have seen the real offending component or could have applied other patterns and requested additional dumps Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark False Positive Dump 263 Please purchase PDF Split-Merge on www.verypdf.com... commands display “strange” output Easy and painless (for customers) crash dump generation using such “Live” tools means that it is widely used and we have to analyze memory dumps saved by these tools and sent from customers This brings us to the next crash dump analysis pattern called Inconsistent Dump If we have such a memory dump we should look at it in order to extract maximum useful information... dump under discussion WinDbg automatic analysis command recognized only the last breakpoint exception (shown as # 16) In conclusion we shouldn’t rely on ”automatic analysis often anyway Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Dynamic Memory Corruption 257 DYNAMIC MEMORY CORRUPTION Next pattern I would like to discuss is Dynamic Memory Corruption (and its user and kernel... during crash dump analysis and inspect any inconsistencies closer Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Invalid Pointer 267 INVALID POINTER Here I want to ”introduce” Invalid Pointer pattern It’s just a number saved in a register or in a memory location and when we try to interpret it as a memory address itself and follow it (dereference) to fetch memory contents... located at 0×00000010 memory address When we get a crash dump and we see an invalid pointer pattern the next step is to interpret the pointer value which should help in understanding possible steps that led to the crash Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Inconsistent Dump 269 INCONSISTENT DUMP We have to live with tools that produce inconsistent dumps For example,... save complete memory dumps without server reboot There is an article for Citrix customers: Using LiveKD to Save a Complete Memory Dump for Session or System Hangs (http://support.citrix.com/article/CTX107717) If we read it we will find the important note which is reproduced here: LiveKd.exe-generated dumps are always inconsistent and cannot be a reliable source for certain types of dump analysis, for . truncated dumps) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Crash Dump Analysis Poster (HTML version) 253 CRASH DUMP ANALYSIS. watermark. Crash Dump Analysis Checklist 251 CRASH DUMP ANALYSIS CHECKLIST Often the root cause of a problem is not obvious from a memory dump. Here is the