reverse engineering for beginners

Praise for Reverse Engineering for Beginners • “It’s very well done and for free amazing.”1 Daniel Bilar, Siege Technologies, LLC • “ excellent and free”2 Pete Finnigan, Oracle RDBMS security guru • “ book is interesting, great job!” Michael Sikorski, author of Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software • “ my compliments for the very nice tutorial!” Herbert Bos, full professor at the Vrije Universiteit Amsterdam, co-author of Modern Operating Systems (4th Edition) • “ It is amazing and unbelievable.” Luis Rocha, CISSP / ISSAP, Technical Manager, Network & Information Security at Verizon Business • “Thanks for the great work and your book.” Joris van de Vis, SAP Netweaver & Security specialist • “ reasonable intro to some of the techniques.”3 (Mike Stay, teacher at the Federal Law Enforcement Training Center, Georgia, US.) • “I love this book! I have several students reading it at the moment, plan to use it in graduate course.”4 (Sergey Bratus, Research Assistant Professor at the Computer Science Department at Dartmouth College) • “Dennis @Yurichev has published an impressive (and free!) book on reverse engineering”5 Tanel Poder, Oracle RDBMS performance tuning expert twitter twitter reddit twitter twitter Reverse Engineering for Beginners Dennis Yurichev ii Reverse Engineering for Beginners Dennis Yurichev cb n d ©2013-2014, Dennis Yurichev This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/ Text version (January 11, 2015) There is probably a newer version of this text, and Russian language version also accessible at beginners.re E-book reader version is also available on the page You may also subscribe to my twitter, to get information about updates of this text, etc: @yurichev or to subscribe to mailing list The cover was made by Andy Nechaevsky: facebook twitter yurichev.com iii Please donate! I worked more than one year and half on this book, here are more than 900 pages, and it’s free Same level books has price tag from $20 to $50 More about it: 0.0.1 I am also looking for a publisher who may want to translate and publish my “Reverse Engineering for Beginners” book to a language other than English/Russian, under the condition that English/Russian version will remain freely available in open-source form Interested? dennis(a)yurichev.com i SHORT CONTENTS SHORT CONTENTS Short contents I Code patterns II Important fundamentals 438 III Slightly more advanced examples 445 IV Finding important/interesting stuff in the code 585 V OS-specific 607 VI Tools 660 VII More examples 666 VIII Examples of reversing proprietary file formats 778 IX Other things 803 X Books/blogs worth reading 821 XI Exercises 825 Afterword 861 Appendix 863 Acronyms used 901 ii CONTENTS CONTENTS Contents 0.0.1 I Donate Code patterns v 1 A short introduction to the CPU 1.1 A couple of words about difference between ISA8 3 Simplest possible function 2.1 x86 2.2 ARM 2.3 MIPS 2.3.1 Note about MIPS instruction/register names 4 4 Hello, world! 3.1 x86 3.1.1 MSVC 3.1.2 GCC 3.1.3 GCC: AT&T syntax 3.2 x86-64 3.2.1 MSVC—x86-64 3.2.2 GCC—x86-64 3.3 GCC—one more thing 3.4 ARM 3.4.1 Non-optimizing Keil 6/2013 (ARM mode) 3.4.2 Non-optimizing Keil 6/2013 (thumb mode) 3.4.3 Optimizing Xcode 4.6.3 (LLVM) (ARM mode) 3.4.4 Optimizing Xcode 4.6.3 (LLVM) (thumb-2 mode) 3.4.5 ARM64 3.5 MIPS 3.5.1 Word about “global pointer” 3.5.2 Optimizing GCC 3.5.3 Non-optimizing GCC 3.5.4 Role of the stack frame in this example 3.5.5 Optimizing GCC: load it into GDB 3.6 Conclusion 3.7 Exercises 3.7.1 Exercise #1 6 9 10 10 11 11 13 13 14 14 16 16 16 17 19 19 20 20 20 Function prologue and epilogue 21 4.1 Recursion 21 Stack 5.1 Why does the stack grow backwards? 5.2 What is the stack used for? 5.2.1 Save the return address where a function must return control after execution 5.2.2 Passing function arguments 5.2.3 Local variable storage 5.2.4 x86: alloca() function 5.2.5 (Windows) SEH 5.2.6 Buffer overflow protection 5.3 Typical stack layout Instruction Set Architecture iii 22 22 23 23 24 24 24 26 26 26 CONTENTS 5.4 Noise in stack 5.5 Exercises 5.5.1 Exercise #1 5.5.2 Exercise #2 27 30 30 30 printf() with several arguments 6.1 x86 6.1.1 x86: arguments 6.1.2 x64: arguments 6.2 ARM 6.2.1 ARM: arguments 6.2.2 ARM: arguments 6.3 MIPS 6.3.1 arguments 6.3.2 arguments 6.4 Conclusion 6.5 By the way 33 33 33 39 42 42 44 47 47 50 53 54 scanf() 7.1 Simple example 7.1.1 About pointers 7.1.2 x86 7.1.3 MSVC + OllyDbg 7.1.4 x64 7.1.5 ARM 7.1.6 MIPS 7.2 Global variables 7.2.1 MSVC: x86 7.2.2 MSVC: x86 + OllyDbg 7.2.3 GCC: x86 7.2.4 MSVC: x64 7.2.5 ARM: Optimizing Keil 6/2013 (thumb mode) 7.2.6 ARM64 7.2.7 MIPS 7.3 scanf() result checking 7.3.1 MSVC: x86 7.3.2 MSVC: x86: IDA 7.3.3 MSVC: x86 + OllyDbg 7.3.4 MSVC: x86 + Hiew 7.3.5 MSVC: x64 7.3.6 ARM 7.3.7 MIPS 7.3.8 Exercise 55 55 55 55 58 60 61 62 64 64 66 67 67 68 69 69 72 73 74 78 80 81 82 83 84 Accessing passed arguments 8.1 x86 8.1.1 MSVC 8.1.2 MSVC + OllyDbg 8.1.3 GCC 8.2 x64 8.2.1 MSVC 8.2.2 GCC 8.2.3 GCC: uint64_t instead of int 8.3 ARM 8.3.1 Non-optimizing Keil 6/2013 (ARM mode) 8.3.2 Optimizing Keil 6/2013 (ARM mode) 8.3.3 Optimizing Keil 6/2013 (thumb mode) 8.3.4 ARM64 8.4 MIPS 85 85 85 86 86 87 87 88 89 90 90 90 91 91 92 More about results returning 9.1 Attempt to use the result of a function returning void 9.2 What if we not use the function result? 9.3 Returning a structure 94 94 95 95 iv CONTENTS 10 Pointers 10.1 Global variables example 10.2 Local variables example 10.3 Conclusion 97 97 103 106 11 GOTO 107 11.1 Dead code 109 11.2 Exercise 109 12 Conditional jumps 12.1 Simple example 12.1.1 x86 12.1.2 ARM 12.1.3 MIPS 12.2 Calculating absolute value 12.2.1 Optimizing MSVC 12.2.2 Optimizing Keil 6/2013: thumb mode 12.2.3 Optimizing Keil 6/2013: ARM mode 12.2.4 Non-optimizing GCC 4.9 (ARM64) 12.2.5 MIPS 12.2.6 Branchless version? 12.3 Conditional operator 12.3.1 x86 12.3.2 ARM 12.3.3 ARM64 12.3.4 MIPS 12.3.5 Let’s rewrite it in an if/else way 12.3.6 Conclusion 12.3.7 Exercise 12.4 Getting minimal and maximal values 12.4.1 32-bit 12.4.2 64-bit 12.4.3 MIPS 12.5 Conclusion 12.5.1 x86 12.5.2 ARM 12.5.3 MIPS 12.5.4 Branchless 110 110 110 121 124 126 126 127 127 127 128 128 128 128 129 130 130 131 131 131 131 131 133 135 136 136 136 136 136 13 switch()/case/default 13.1 Small number of cases 13.1.1 x86 13.1.2 ARM: Optimizing Keil 6/2013 (ARM mode) 13.1.3 ARM: Optimizing Keil 6/2013 (thumb mode) 13.1.4 ARM64: Non-optimizing GCC (Linaro) 4.9 13.1.5 ARM64: Optimizing GCC (Linaro) 4.9 13.1.6 MIPS 13.1.7 Conclusion 13.2 A lot of cases 13.2.1 x86 13.2.2 ARM: Optimizing Keil 6/2013 (ARM mode) 13.2.3 ARM: Optimizing Keil 6/2013 (thumb mode) 13.2.4 MIPS 13.2.5 Conclusion 13.3 When there are several case statements in one block 13.3.1 MSVC 13.3.2 GCC 13.3.3 ARM64: Optimizing GCC 4.9.1 13.4 Fall-through 13.4.1 MSVC x86 13.4.2 ARM64 13.5 Exercises 13.5.1 Exercise #1 138 138 138 148 148 149 150 150 151 151 151 157 158 159 161 161 162 163 163 165 166 166 167 167 v CONTENTS 14 Loops 14.1 Simple example 14.1.1 x86 14.1.2 x86: OllyDbg 14.1.3 x86: tracer 14.1.4 ARM 14.1.5 MIPS 14.1.6 One more thing 14.2 Memory blocks copying routine 14.2.1 Straight-forward implementation 14.2.2 ARM in ARM mode 14.2.3 MIPS 14.2.4 Vectorization 14.3 Conclusion 14.4 Exercises 14.4.1 Exercise #1 14.4.2 Exercise #2 14.4.3 Exercise #3 14.4.4 Exercise #4 168 168 168 172 172 174 177 178 178 178 179 180 180 180 182 182 182 182 184 15 Simple C-strings processing 15.1 strlen() 15.1.1 x86 15.1.2 ARM 15.1.3 MIPS 15.2 Exercises 15.2.1 Exercise #1 187 187 187 194 196 197 197 16 Replacing arithmetic instructions to other ones 16.1 Multiplication 16.1.1 Multiplication using addition 16.1.2 Multiplication using shifting 16.1.3 Multiplication using shifting/subtracting/adding 16.2 Division 16.2.1 Division using shifts 16.3 Exercises 16.3.1 Exercise #2 200 200 200 200 201 204 204 205 205 17 Floating-point unit 17.1 IEEE 754 17.2 x86 17.3 ARM, MIPS, x86/x64 SIMD 17.4 C/C++ 17.5 Simple example 17.5.1 x86 17.5.2 ARM: Optimizing Xcode 4.6.3 (LLVM) (ARM mode) 17.5.3 ARM: Optimizing Keil 6/2013 (thumb mode) 17.5.4 ARM64: Optimizing GCC (Linaro) 4.9 17.5.5 ARM64: Non-optimizing GCC (Linaro) 4.9 17.5.6 MIPS 17.6 Passing floating point numbers via arguments 17.6.1 x86 17.6.2 ARM + Non-optimizing Xcode 4.6.3 (LLVM) (thumb-2 mode) 17.6.3 ARM + Non-optimizing Keil 6/2013 (ARM mode) 17.6.4 ARM64 + Optimizing GCC (Linaro) 4.9 17.6.5 MIPS 17.7 Comparison example 17.7.1 x86 17.7.2 ARM 17.7.3 ARM64 17.7.4 MIPS 17.8 x64 17.9 Exercises 17.9.1 Exercise #1 17.9.2 Exercise #2 207 207 207 207 207 208 208 215 215 216 216 217 218 218 219 219 220 220 221 222 249 252 253 253 254 254 254 vi CONTENTS 18 Arrays 18.1 Simple example 18.1.1 x86 18.1.2 ARM 18.1.3 MIPS 18.2 Buffer overflow 18.2.1 Reading outside array bounds 18.2.2 Writing beyond array bounds 18.3 Buffer overflow protection methods 18.3.1 Optimizing Xcode 4.6.3 (LLVM) (thumb-2 mode) 18.4 One more word about arrays 18.5 Array of pointers to strings 18.5.1 x64 18.5.2 32-bit ARM 18.5.3 ARM64 18.5.4 MIPS 18.5.5 Array overflow 18.6 Multidimensional arrays 18.6.1 Two-dimensional array example 18.6.2 Access two-dimensional array as one-dimensional 18.6.3 Three-dimensional array example 18.6.4 More examples 18.7 Pack of strings as a two-dimensional array 18.7.1 32-bit ARM 18.7.2 ARM64 18.7.3 MIPS 18.7.4 Conclusion 18.8 Conclusion 18.9 Exercises 18.9.1 Exercise #1 18.9.2 Exercise #2 18.9.3 Exercise #3 18.9.4 Exercise #4 18.9.5 Exercise #5 256 256 256 259 261 263 263 266 270 272 274 274 275 276 277 277 278 280 281 282 284 286 286 288 289 289 290 290 290 290 293 297 298 299 19 Manipulating specific bit(s) 19.1 Specific bit checking 19.1.1 x86 19.1.2 ARM 19.2 Specific bit setting/clearing 19.2.1 x86 19.2.2 ARM + Optimizing Keil 6/2013 (ARM mode) 19.2.3 ARM + Optimizing Keil 6/2013 (thumb mode) 19.2.4 ARM + Optimizing Xcode 4.6.3 (LLVM) (ARM mode) 19.2.5 ARM: more about BIC instruction 19.2.6 ARM64: Optimizing GCC (Linaro) 4.9 19.2.7 ARM64: Non-optimizing GCC (Linaro) 4.9 19.2.8 MIPS 19.3 Shifts 19.4 Specific bit setting/clearing: FPU9 example 19.4.1 A word about XOR operation 19.4.2 x86 19.4.3 MIPS 19.4.4 ARM 19.5 Counting bits set to 19.5.1 x86 19.5.2 x64 19.5.3 ARM + Optimizing Xcode 4.6.3 (LLVM) (ARM mode) 19.5.4 ARM + Optimizing Xcode 4.6.3 (LLVM) (thumb-2 mode) 19.5.5 ARM64 + Optimizing GCC 4.9 19.5.6 ARM64 + Non-optimizing GCC 4.9 19.5.7 MIPS 19.6 Conclusion 304 304 304 306 307 308 313 314 314 314 314 315 315 315 315 316 316 317 318 320 321 329 331 331 332 332 333 334 Floating-point unit vii z 900 APPENDIX G EXERCISE SOLUTIONS G.5 OTHER Acronyms used OS Operating System OOP Object-Oriented Programming 524 PL Programming language PRNG Pseudorandom number generator 383 ROM Read-only memory 598 ALU Arithmetic logic unit 17 RA Return Address PE Portable Executable: 65.2 631 SP stack pointer SP/ESP/RSP in x86/x64 SP in ARM 12 DLL Dynamic-link library 631 PC Program Counter IP/EIP/RIP in x86/64 PC in ARM 12 LR Link Register IDA Interactive Disassembler IAT Import Address Table 631 INT Import Name Table 631 RVA Relative Virtual Address 631 VA Virtual Address 631 OEP Original Entry Point 620 MSVC Microsoft Visual C++ MSVS Microsoft Visual Studio 876 ASLR Address Space Layout Randomization 632 MFC Microsoft Foundation Classes 634 TLS Thread Local Storage iv AKA Also Known As 901 APPENDIX G EXERCISE SOLUTIONS G.5 OTHER CRT C runtime library: 65.1 CPU Central processing unit iv FPU Floating-point unit 207 CISC Complex instruction set computing 12 RISC Reduced instruction set computing GUI Graphical user interface 628 RTTI Run-time type information 538 BSS Block Started by Symbol 16 SIMD Single instruction, multiple data 178 BSOD Black Screen of Death 621 DBMS Database management systems iv ISA Instruction Set Architecture CGI Common Gateway Interface 858 HPC High-Performance Computing 498 SEH Structured Exception Handling: 65.3 26 ELF Executable file format widely used in *NIX system including Linux iv TIB Thread Information Block 272 TEA Tiny Encryption Algorithm 897 PIC Position Independent Code: 64.1 iv NAN Not a Number 868 NOP No OPeration 18 BEQ (PowerPC, ARM) Branch if Equal 82 BNE (PowerPC, ARM) Branch if Not Equal 195 BLR (PowerPC) Branch to Link Register 689 902 APPENDIX G EXERCISE SOLUTIONS G.5 OTHER XOR eXclusive OR 874 MCU Microcontroller unit 473 RAM Random-access memory 68 EGA Enhanced Graphics Adapter 818 VGA Video Graphics Array 818 API Application programming interface 590 ASCII American Standard Code for Information Interchange 481 ASCIIZ ASCII Zero (null-terminated ASCII string) 80 IA64 Intel Architecture 64 (Itanium): 90 442 EPIC Explicitly parallel instruction computing 815 OOE Out-of-order execution 444 MSDN Microsoft Developer Network 340 MSB Most significant bit/byte 315 LSB Least significant bit/byte STL (C++) Standard Template Library: 48.4 544 PODT (C++) Plain Old Data Type 554 HDD Hard disk drive 565 VM Virtual Memory WRK Windows Research Kernel 603 GPR General Purpose Registers SSDT System Service Dispatch Table 621 RE Reverse Engineering 824 BCD Binary-coded decimal 766 BOM Byte order mark 593 903 APPENDIX G EXERCISE SOLUTIONS G.5 OTHER GDB GNU debugger 37 FP Frame Pointer 15 MBR Master Boot Record 598 JPE Jump Parity Even (x86 instruction) 227 CIDR Classless Inter-Domain Routing 461 STMFD Store Multiple Full Descending (ARM instruction) LDMFD Load Multiple Full Descending (ARM instruction) STMED Store Multiple Empty Descending (ARM instruction) 22 LDMED Load Multiple Empty Descending (ARM instruction) 22 STMFA Store Multiple Full Ascending (ARM instruction) 22 LDMFA Load Multiple Full Ascending (ARM instruction) 22 STMEA Store Multiple Empty Ascending (ARM instruction) 22 LDMEA Load Multiple Empty Ascending (ARM instruction) 22 APSR (ARM) Application Program Status Register 250 FPSCR (ARM) Floating-Point Status and Control Register 250 PID Program/process ID 679 LF Line feed (10 or’\n’ in C/C++) 506 CR Carriage return (13 or’\r’ in C/C++) 506 RFC Request for Comments 596 904 Glossary Glossary decrement Decrease by 12, 168, 189, 433, 606, 729, 870, 872, 876 increment Increase by 12, 168, 172, 189, 194, 325, 328, 433, 708, 726, 870 integral data type usual numbers, but not floating point ones 220 product Multiplication result 86, 213, 215, 412, 477, 894 arithmetic mean a sum of all values divided by its count 501 stack pointer A register pointing to the place in the stack 7, 8, 12, 22, 24, 33, 43, 45, 62, 87, 526, 575, 608–610, 865, 871, 881, 901 tail call It is when compiler (or interpreter) transforms recursion (with which it is possible: tail recursion) into iteration for efficiency: wikipedia 21 quotient Division result 205, 209, 211, 212, 215, 475, 502, 683 anti-pattern Generally considered as bad practice 23, 64, 443 atomic operation “ατ oµoς” mean “indivisible” in Greek, so atomic operation is what guaranteed not to be broke up during operation by other threads 658, 812 basic block a group of instructions not having jump/branch instructions, and also not having jumps inside block from the outside In IDA it looks just like as a list of instructions without breaking empty lines 819, 820 callee A function being called by another 21, 24, 36, 55, 73, 85, 87, 89, 140, 425, 526, 575, 608–610, 612, 613, 884 caller A function calling another 4, 5, 7, 36, 73, 85, 86, 88, 95, 140, 425, 446, 526, 608, 610, 613 compiler intrinsic A function specific to a compiler which is not usual library function Compiler generate a specific machine code instead of call to it Often, it’s a pseudofunction for specific CPU instruction Read more: (87) 876 CP/M Control Program for Microcomputers: a very basic disk OS used before MS-DOS 764 dongle Dongle is a small piece of hardware connected to LPT printer port (in past) or to USB Its function was akin to security token, it has some memory and, sometimes, secret (crypto-)hashing algorithm 688 endianness Byte order: 30 14, 66, 350, 874, 893 GiB Gibibyte: 230 or 1024 mebibytes or 1073741824 bytes 10 heap usually, a big chunk of memory provided by OS so that applications can divide it by themselves as they wish malloc()/free() works with heap 22, 24, 351, 540, 543, 554, 556, 630, 631 jump offset a part of JMP or Jcc instruction opcode, it just to be added to the address of the next instruction, and thus is how new PC is calculated May be negative as well 81, 119, 870 kernel mode A restrictions-free CPU mode in which it executes OS kernel and drivers cf user mode 906 keygenme A program which imitates fictional software protection, for which one needs to make a keys/licenses generator 859 leaf function A function, which is not calling any other function 19, 23 link register (RISC) A register where return address is usually stored This makes calling leaf functions without stack usage, i.e., faster 23, 689, 881, 882 905 Glossary Glossary loop unwinding It is when a compiler instead of generation loop code of n iteration, generates just n copies of the loop body, in order to get rid of loop maintenance instructions 170 name mangling used at least in C++, where compiler need to encode name of class, method and argument types in the one string, which will become internal name of the function read more here: 48.1.1 524, 587, 588 NaN not a number: special cases of floating point numbers, usually signaling about errors 223, 245, 817 NEON AKA “Advanced SIMD”—SIMD from ARM 882 NOP “no operation”, idle instruction 606 NTAPI API available only in Windows NT line Largely, not documented by Microsoft 667 PDB (Win32) Debugging information file, usually just function names, but sometimes also function arguments and local variables names 586, 633, 667, 668, 675, 679, 746 POKE BASIC language instruction writing byte on specific address 606 register allocator Compiler’s function assigning local variables to CPU registers 188, 306, 425 reverse engineering act of understanding, how the thing works, sometimes, in order to clone it iv, 876 security cookie A random value, different at each execution Read more about it: 18.3 649 stack frame Part of stack containing information specific to the current functions: local variables, function arguments, RA, etc 56, 86, 454, 649 stdout standard output 14, 25, 140, 855, 856 thunk function Tiny function with a single role: call another function 14, 398, 689, 697 tracer My own simple debugging tool Read more about it: 67.1 172–174, 591, 600, 604, 645, 654, 748, 754, 758, 760, 806, 857 user mode A restricted CPU mode in which it executes all applied software code cf kernel mode 704, 905 Windows NT Windows NT, 2000, XP, Vista, 7, 280, 423, 574, 594, 621, 632, 658, 767, 876 word data type fitting in GPR In the computers older than personal, memory size was often measured in words rather then bytes 546 xoring often used in English language, meaning applying XOR operation 649, 699, 702 906 Index NET, 636 AT&T syntax, 9, 26 Buffer Overflow, 263, 649 C language elements Pointers, 55, 62, 97, 390, 425 Post-decrement, 433 Post-increment, 433 Pre-decrement, 433 Pre-increment, 433 C99, 96 bool, 304 restrict, 496 variable length arrays, 274 const, 6, 69 for, 168, 459 if, 110, 139 return, 7, 73, 95 switch, 138, 139, 148 while, 187 C standard library alloca(), 24, 274, 443, 641 assert(), 279, 597 atexit(), 545 atoi(), 481, 737 calloc(), 719 close(), 625 exit(), 446 free(), 443 fseek(), 719 ftell(), 719 getenv(), 738 localtime(), 583 localtime_r(), 359 longjmp(), 140 malloc(), 352, 443 memchr(), 872 memcmp(), 494, 598, 874 memcpy(), 9, 55, 491, 871 memset(), 490, 758, 873 open(), 625 pow(), 218 puts(), 13 qsort(), 390 rand(), 343, 590, 672, 675, 707 read(), 625 realloc(), 443 scanf(), 55 strcmp(), 488, 625 strcpy(), 9, 490, 708 strlen(), 187, 421, 490, 507, 872 strstr(), 896 time(), 583 tolower(), 724 va_arg, 501 va_list, 504 vprintf, 504 Compiler’s anomalies, 132, 218, 288, 314, 331, 469, 512, 808 C++, 749 exceptions, 641 C++11, 554, 616 ostream, 538 References, 539 RTTI, 538 STL, 586 std::forward_list, 554 std::list, 546 std::map, 561 std::set, 561 std::string, 540 std::vector, 554 grep usage, 174, 251, 586, 600, 604, 747 Intel syntax, 9, 11 Mac OS X, 663 position-independent code, 12, 623 RAM, 68 ROM, 68, 69 Base address, 631 Global variables, 64 Binary tree, 561 Dynamically loaded libraries, 14 Linker, 68, 524 RISC pipeline, 122 Non-a-numbers (NaNs), 245 OOP Polymorphism, 524 Buffer overflow, 269 Hash functions, 680 Recursion, 21, 23, 457 Tail recursion, 21 Stack, 22, 85, 140 Stack overflow, 23 Stack frame, 56 Syntactic Sugar, 139, 360 thumb-2 mode, 14 iPod/iPhone/iPad, 11 OllyDbg, 34, 58, 66, 86, 98, 113, 153, 172, 190, 210, 224, 235, 258, 265, 268, 281, 323, 350, 366, 367, 371, 374, 393, 633, 662, 888 Oracle RDBMS, 7, 412, 595, 638, 750, 757, 758, 788, 797, 808, 819 Ada, 93 Alpha AXP, Angry Birds, 250, 251 ARM, 194, 518, 688, 881 ARM mode, Instructions 907 INDEX INDEX ADC, 405 ADD, 13, 92, 122, 175, 319, 331, 476, 883 ADDAL, 122 ADDCC, 158 ADDS, 91, 405, 883 ADR, 12, 122 ADRcc, 122, 148, 444 ADRP/ADD pair, 15, 44, 69, 277, 289, 435 ASR, 334 ASRS, 314, 477 B, 43, 122, 123 Bcc, 83, 84, 133 BCS, 123, 252 BEQ, 82, 148 BGE, 123 BIC, 313, 314, 318, 336 BL, 12–15, 122, 436 BLcc, 122 BLE, 123 BLS, 123 BLT, 175 BLX, 14 BNE, 123 BX, 90, 159 CLZ, 839, 840 CMP, 82, 83, 122, 148, 158, 175, 331, 883 CSEL, 130, 135, 136, 332 EOR, 318 FCMPE, 252 FCSEL, 252 FMOV, 435 FMRS, 319 IT, 137, 250, 274 LDMccFD, 122 LDMEA, 22 LDMED, 22 LDMFA, 22 LDMFD, 12, 22, 122 LDP, 15 LDR, 45, 62, 69, 260, 276, 433 LDR.W, 286 LDRB, 367 LDRB.W, 195 LDRSB, 194 LSL, 331, 334 LSL.W, 331 LSLS, 260, 319 LSR, 334 LSRS, 319 MADD, 91 MLA, 90 MOV, 4, 12, 13, 331, 476 MOVcc, 133, 136 MOVK, 434 MOVT, 13, 476 MOVT.W, 14 MOVW, 14 MUL, 92 MULS, 91 MVNS, 195 NEG, 486 ORR, 313 POP, 12, 13, 22, 23 PUSH, 13, 22, 23 RET, 15 RSB, 127, 286, 331, 486 SBC, 405 SMMUL, 476 STMEA, 22 STMED, 22 STMFA, 22, 46 STMFD, 12, 22 STMIA, 45 STMIB, 46 STP, 15, 44 STR, 45, 260 SUB, 45, 286, 331 SUBEQ, 195 SUBS, 405 SXTB, 368 SXTW, 289 TEST, 188 TST, 307, 331 VADD, 215 VDIV, 215 VLDR, 215 VMOV, 215, 250 VMOVGT, 250 VMRS, 250 VMUL, 215 XOR, 127, 319 Pipeline, 158 Mode switching, 90, 159 Addressing modes, 433 mode switching, 14 Registers APSR, 250 FPSCR, 250 Link Register, 12, 23, 43, 159, 881 R0, 94, 881 scratch registers, 195, 881 X0, 882 Z, 82, 882 thumb mode, 3, 123, 159 thumb-2 mode, 3, 159, 250, 251 armel, 216 armhf, 216 Condition codes, 122 D-registers, 215, 882 Data processing instructions, 476 DCB, 12 hard float, 216 if-then block, 250 Leaf function, 23 Optional operators ASR, 331, 476 LSL, 260, 286, 331, 434 LSR, 331, 476 ROR, 331 RRX, 331 S-registers, 215, 882 soft float, 216 ASLR, 632 AWK, 602 Base64, 595 base64, 596 bash, 95 908 INDEX BASIC POKE, 606 binary grep, 599, 665 BIND.EXE, 635 binutils, 387 Bitcoin, 809 Borland C++Builder, 588 Borland Delphi, 588, 592, 806 BSoD, 621 BSS, 632 C11, 616 Callbacks, 390 Canary, 271 cdecl, 33, 608 COFF, 695 column-major order, 281 Compiler intrinsic, 25, 807 CRC32, 458, 680 CRT, 628, 646 Cygwin, 587, 591, 636, 663 DES, 412, 425 dlopen(), 625 dlsym(), 625 DOSBox, 767 DosBox, 604 double, 207, 613 dtruss, 663 Duff’s device, 472 EICAR, 763 ELF, 67 Error messages, 596 fastcall, 9, 54, 305, 609 float, 207, 613 FORTRAN, 281, 496, 571, 587 Function epilogue, 21, 43, 45, 122, 367, 602 Function prologue, 8, 21, 23, 45, 271, 602 Fused multiply–add, 90, 91 GCC, 587, 886, 889 GDB, 19, 37, 41, 270, 398, 399, 662, 889 Glibc, 398 Hex-Rays, 680 Hiew, 80, 119, 592, 633, 636, 806 IDA, 74, 387, 495, 594, 794, 888 var_?, 45, 62 IEEE 754, 207, 315, 383, 863 Inline code, 176, 313, 487, 529, 558 Integer overflow, 93 Intel 8080, 194 8086, 194, 313, 703 Memory model, 581, 818 8253, 766 80286, 703, 818 80386, 313, 818 80486, 207 FPU, 207 Intel C++, 7, 413, 808, 819, 871 Itanium, 815 INDEX jumptable, 152, 159 Keil, 11 kernel panic, 621 kernel space, 621 LD_PRELOAD, 625 Linux, 306, 623, 750 libc.so.6, 305, 398 LLVM, 11 long double, 207 Loop unwinding, 170 Mac OS Classic, 688 MD5, 598, 680 MFC, 634, 738 MIDI, 598 MinGW, 587 minifloat, 435 MIPS, 3, 519, 632, 688 Global Pointer, 16 Instructions ADD, 93 ADD.D, 218 ADDIU, 16, 72 ADDU, 93 AND, 315 BC1F, 253 BC1T, 253 BEQ, 84, 124 BLTZ, 128 BNE, 124 BNEZ, 161 BREAK, 477 C.LT.D, 253 DIV.D, 218 J, 5, 17 JAL, 93 JALR, 16, 93 JR, 151 LB, 180 LBU, 180 LI, 437 LUI, 16, 72, 218, 318, 437 LW, 16, 62, 72, 151 LWC1, 218 MFC1, 221 MFHI, 93, 477, 884 MFLO, 93, 477, 884 MTC1, 388 MUL.D, 218 MULT, 93 NOR, 197 OR, 19 ORI, 315, 437 SB, 180 SLL, 161, 201, 334 SLLV, 334 SLT, 124 SLTIU, 161 SLTU, 124, 125, 161 SRL, 205 SUBU, 128 SW, 50 Pseudoinstructions 909 INDEX INDEX B, 178 BEQZ, 125 L.D, 218 LA, 19 LI, MOVE, 16, 71 NEGU, 128 NOP, 19, 71 NOT, 197 Registers FCCR, 253 HI, 477 LO, 477 Branch delay slot, Global Pointer, 286 Load delay slot, 151 O32, 50, 54, 884 MS-DOS, 272, 578, 598, 604, 606, 631, 703, 763, 765, 781, 806, 818, 863, 872, 876 DOS extenders, 818 MSVC, 887, 889 Name mangling, 524 NEC V20, 767 objdump, 387, 624, 636 OEP, 631, 636 opaque predicate, 521 OpenMP, 589, 809 OpenWatcom, 588, 610 Page (memory), 423 Pascal, 592 PDP-11, 433 PowerPC, 3, 16, 688 puts() instead of printf(), 13, 60, 94, 120 Raspberry Pi, 11 ReactOS, 643 Register allocation, 425 Relocation, 14 row-major order, 281 RVA, 631 tracer, 172, 395, 397, 591, 600, 604, 645, 654, 662, 748, 754, 758, 760, 806, 857 Unicode, 593 Unrolled loop, 176, 274, 491 uptime, 625 USB, 690 user space, 621 UTF-16LE, 593 UTF-8, 593 VA, 631 Watcom, 588 Windows, 658 API, 863 IAT, 631 INT, 631 KERNEL32.DLL, 304 MSVCR80.DLL, 391 NTAPI, 667 ntoskrnl.exe, 750 PDB, 586, 633, 667, 675, 746 Structured Exception Handling, 26, 636 TIB, 272, 636, 866 Win32, 304, 593, 625, 631, 818 GetProcAddress, 635 LoadLibrary, 635 Ordinal, 634 RaiseException(), 636 SetUnhandledExceptionFilter(), 638 Windows 2000, 632 Windows 3.x, 574, 818 Windows NT4, 632 Windows Vista, 631, 667 Windows XP, 632, 636, 675 Wine, 643 Wolfram Mathematica, 479, 682 x86 Flags CF, 870, 872, 874, 877 DF, 874, 877 IF, 874, 877 Instructions AAA, 880 AAS, 880 ADC, 404, 578, 870 ADD, 7, 33, 86, 482, 578, 870 ADRcc, 129 AND, 8, 304, 305, 308, 321, 335, 373, 870, 873 BSF, 424, 839, 874 BSR, 874 BSWAP, 442, 874 BT, 874 BTC, 317, 874 BTR, 317, 658, 874 BTS, 317, 874 CALL, 7, 23, 516, 635, 870 CBW, 440, 874 CDQ, 411, 440, 874 CDQE, 440, 874 CLD, 874 CLI, 874 CMC, 874 SAP, 586, 746 SCO OpenServer, 694 Scratch space, 611 Security cookie, 271, 649 Security through obscurity, 596 SHA1, 680 SHA512, 809 Shadow space, 88, 89 Shellcode, 520, 621, 632, 764, 879 Signed numbers, 111, 439 SIMD, 494 stdcall, 608, 806 strace, 625, 663 syscall, 305, 621, 663 TCP/IP, 442 thiscall, 524, 526, 610 thunk-functions, 14, 635, 689, 697 TLS, 272, 616, 632, 636, 866 Callbacks, 619 Callbacks, 636 910 INDEX INDEX CMOVcc, 122, 129, 131, 133, 136, 444, 874 CMP, 73, 870, 880 CMPSB, 598, 874 CMPSD, 874 CMPSQ, 874 CMPSW, 874 CPUID, 371, 876 CWD, 440, 578, 775, 874 CWDE, 440, 874 DEC, 189, 870, 880 DIV, 440, 876 DIVSD, 601 FABS, 878 FADD, 878 FADDP, 209, 215, 878 FATRET, 330, 331 FCHS, 878 FCMOVcc, 246 FCOM, 234, 245, 878 FCOMP, 222, 878 FCOMPP, 878 FDIV, 208, 600, 878 FDIVP, 209, 878 FDIVR, 215, 878 FDIVRP, 878 FILD, 878 FIST, 878 FISTP, 878 FLD, 219, 222, 878 FLD1, 878 FLDCW, 878 FLDZ, 878 FMUL, 209, 878 FMULP, 878 FNSTCW, 878 FNSTSW, 222, 245, 878 FSINCOS, 878 FSQRT, 878 FST, 878 FSTCW, 878 FSTP, 219, 878 FSTSW, 878 FSUB, 879 FSUBP, 879 FSUBR, 878 FSUBRP, 878 FUCOM, 245, 879 FUCOMI, 246 FUCOMP, 879 FUCOMPP, 245, 879 FWAIT, 207 FXCH, 879 IDIV, 440, 475, 876 IMUL, 86, 288, 440, 870, 880 IN, 516, 704, 766, 876 INC, 189, 806, 870, 880 INT, 764, 876 INT3, 591 IRET, 876 JA, 111, 246, 439, 870, 880 JAE, 111, 870, 880 JB, 111, 439, 870, 880 JBE, 111, 870, 880 JC, 870 Jcc, 84, 132 JCXZ, 870 JE, 139, 870, 880 JECXZ, 870 JG, 111, 439, 870 JGE, 111, 870 JL, 111, 439, 870 JLE, 111, 870 JMP, 23, 43, 635, 806, 870 JNA, 870 JNAE, 870 JNB, 870 JNBE, 245, 870 JNC, 870 JNE, 73, 111, 870, 880 JNG, 870 JNGE, 870 JNL, 870 JNLE, 870 JNO, 870, 880 JNS, 870, 880 JNZ, 870 JO, 870, 880 JP, 223, 767, 870, 880 JPO, 870 JRCXZ, 870 JS, 870, 880 JZ, 82, 139, 808, 870 LAHF, 871 LEA, 56, 87, 354, 447, 460, 482, 612, 670, 871 LEAVE, 8, 871 LES, 708, 775 LOCK, 658 LODSB, 766 LOOP, 168, 181, 182, 602, 775, 876 MOV, 4, 7, 9, 491, 516, 633, 806, 872 MOVDQA, 415 MOVDQU, 415 MOVSB, 871 MOVSD, 492, 723, 871 MOVSQ, 871 MOVSW, 871 MOVSX, 188, 194, 366–368, 440, 872 MOVSXD, 275 MOVZX, 188, 352, 688, 872 MUL, 440, 872 NEG, 485, 872 NOP, 460, 804, 806, 872 NOT, 193, 195, 727, 872 OR, 308, 507, 872 OUT, 516, 704, 876 PADDD, 415 PCMPEQB, 423 PLMULHW, 412 PLMULLD, 412 PMOVMSKB, 424 POP, 7, 22, 23, 872, 880 POPA, 876, 880 POPCNT, 876 POPF, 766, 876 PUSH, 7, 8, 22, 23, 56, 516, 872, 880 PUSHA, 876, 880 PUSHF, 877 PXOR, 423 911 INDEX RCL, 602, 877 RCR, 877 RET, 4, 7, 23, 271, 526, 575, 806, 872 ROL, 330, 807, 877 ROR, 807, 877 SAHF, 245, 872 SAL, 877 SALC, 767 SAR, 334, 440, 499, 775, 877 SBB, 404, 872 SCASB, 766, 767, 872 SCASD, 872 SCASQ, 872 SCASW, 872 SETALC, 767 SETcc, 124, 188, 245, 877 SHL, 201, 257, 334, 873 SHR, 205, 334, 373, 873 SHRD, 410, 873 STC, 877 STD, 877 STI, 877 STOSB, 873 STOSD, 873 STOSQ, 491, 873 STOSW, 873 SUB, 7, 8, 73, 139, 482, 870, 873 SYSCALL, 876, 877 SYSENTER, 622, 876, 877 TEST, 188, 304, 307, 335, 873 UD2, 877 XADD, 659 XCHG, 872, 874 XOR, 7, 73, 193, 499, 602, 699, 806, 874, 880 Prefixes LOCK, 658, 869 REP, 869, 872, 873 REPE/REPNE, 869 REPNE, 873 Registers Flags, 73, 113, 866 AH, 871, 872 CS, 818 DR6, 868 DR7, 869 DS, 818 EAX, 73, 94 EBP, 56, 86 ECX, 524 ES, 775, 818 ESP, 33, 56 FS, 617 GS, 271, 617, 620 JMP, 157 RIP, 624 SS, 818 ZF, 73, 305 AVX, 412 FPU, 867 Instructions STOSB, 474 MMX, 412 SSE, 412 SSE2, 412 INDEX x86-64, 9, 10, 39, 55, 60, 81, 87, 425, 517, 610, 624, 863, 868 Xcode, 11 Z3, 680, 683 912 BIBLIOGRAPHY Bibliography [al12] Nick Montfort et al 10 PRINT CHR$(205.5+RND(1)); : GOTO 10 Also available as http://go.yurichev.com/ 17286 The MIT Press, 2012 [AMD13a] AMD AMD64 Architecture Programmer’s Manual Also available as http://go.yurichev.com/17284 2013 [AMD13b] AMD Software Optimization Guide for AMD Family 16h Processors Also available as http://go.yurichev com/17285 2013 [App10] Apple iOS ABI Function Call Guide Also available as http://go.yurichev.com/17276 2010 [ARM12] ARM ARM® Architecture Reference Manual, ARMv7-A and ARMv7-R edition 2012 [ARM13a] ARM ARM Architecture Reference Manual, ARMv8, for ARMv8-A architecture profile 2013 [ARM13b] ARM ELF for the ARM 64-bit Architecture (AArch64) Also available as http://go.yurichev.com/17288 2013 [ARM13c] ARM Procedure Call Standard for the ARM 64-bit Architecture (AArch64) Also available as http://go.yurichev com/17287 2013 [ASS96] Harold Abelson, Gerald Jay Sussman, and Julie Sussman Structure and Interpretation of Computer Programs 1996 [Bro] Ralf Brown The x86 Interrupt List Also available as http://go.yurichev.com/17292 [Bur] Mike Burrell “Writing Effcient Itanium Assembly Code” In: () Also available as http://go.yurichev com/17265 [Cli] Marshall Cline C++ FAQ Also available as http://go.yurichev.com/17291 [Cor+09] Thomas H Cormen et al Introduction to Algorithms, Third Edition 3rd The MIT Press, 2009 ISBN: 0262033844, 9780262033848 [Dij68] Edsger W Dijkstra “Letters to the editor: go to statement considered harmful” In: Commun ACM 11.3 (Mar 1968), pp 147–148 ISSN: 0001-0782 DOI: 10 1145 / 362929 362947 URL: http : / / go yurichev com / 17299 [Dol13] Stephen Dolan “mov is Turing-complete” In: (2013) Also available as http://go.yurichev.com/17269 [Dre07] Ulrich Drepper What Every Programmer Should Know About Memory Also available as http://go.yurichev com/17341 2007 [Dre13] Ulrich Drepper “ELF Handling For Thread-Local Storage” In: (2013) Also available as http://go.yurichev com/17272 [Eic11] Jens Eickhoff Onboard Computers, Onboard Software and Satellite Operations: An Introduction 2011 [Fog13a] Agner Fog Optimizing software in C++: An optimization guide for Windows, Linux and Mac platforms http://go yurichev.com/17279 2013 [Fog13b] Agner Fog The microarchitecture of Intel, AMD and VIA CPUs / An optimization guide for assembly programmers and compiler makers http://go.yurichev.com/17278 2013 [Fog14] Agner Fog Calling conventions http://go.yurichev.com/17280 2014 [haq] papasutra of haquebright “WRITING SHELLCODE FOR IA-64” In: () Also available as http://go.yurichev com/17340 [IBM00] IBM PowerPC(tm) Microprocessor Family: The Programming Environments for 32-Bit Microprocessors Also available as http://go.yurichev.com/17281 2000 [Int13] Intel Intel® 64 and IA-32 Architectures Software Developer’s Manual Combined Volumes:1, 2A, 2B, 2C, 3A, 3B, and 3C Also available as http://go.yurichev.com/17283 2013 [Int14] Intel Intel® 64 and IA-32 Architectures Optimization Reference Manual Also available as http://go.yurichev com/17342 September 2014 [ISO07] ISO ISO/IEC 9899:TC3 (C C99 standard) Also available as http://go.yurichev.com/17274 2007 [ISO13] ISO ISO/IEC 14882:2011 (C++ 11 standard) Also available as http://go.yurichev.com/17275 2013 [Ker88] Brian W Kernighan The C Programming Language Ed by Dennis M Ritchie 2nd Prentice Hall Professional Technical Reference, 1988 ISBN: 0131103709 913 BIBLIOGRAPHY BIBLIOGRAPHY [Knu74] Donald E Knuth “Structured Programming with go to Statements” In: ACM Comput Surv 6.4 (Dec 1974) Also available as http://go.yurichev.com/17271, pp 261–301 ISSN: 0360-0300 DOI: 10.1145/356635 356640 URL: http://go.yurichev.com/17300 [Knu98] Donald E Knuth The Art of Computer Programming Volumes 1-3 Boxed Set 2nd Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 1998 ISBN: 0201485419 [Loh10] Eugene Loh “The Ideal HPC Programming Language” In: Queue 8.6 (June 2010), 30:30–30:38 ISSN: 1542-7730 DOI: 10.1145/1810226.1820518 URL: http://go.yurichev.com/17298 [Ltd94] Advanced RISC Machines Ltd The ARM Cookbook Also available as http://go.yurichev.com/17273 1994 [Mit13] Michael Matz / Jan Hubicka / Andreas Jaeger / Mark Mitchell System V Application Binary Interface AMD64 Architecture Processor Supplement Also available as http://go.yurichev.com/17295 2013 [One96] Aleph One “Smashing The Stack For Fun And Profit” In: Phrack (1996) Also available as http://go.yurichev com/17266 [Pie] Matt Pietrek “A Crash Course on the Depths of Win32™ Structured Exception Handling” In: MSDN magazine () URL: http://go.yurichev.com/17293 [Pie02] Matt Pietrek “An In-Depth Look into the Win32 Portable Executable File Format” In: MSDN magazine (2002) URL: http://go.yurichev.com/17318 [Pre+07] William H Press et al Numerical Recipes 2007 [RA09] Mark E Russinovich and David A Solomon with Alex Ionescu Windows® Internals: Including Windows Server 2008 and Windows Vista, Fifth Edition 2009 [Ray03] Eric S Raymond The Art of UNIX Programming Also available as http://go.yurichev.com/17277 Pearson Education, 2003 ISBN: 0131429019 [Rit79] Dennis M Ritchie “The Evolution of the Unix Time-sharing System” In: (1979) [Rit86] Dennis M Ritchie Where did ++ come from? (net.lang.c) http://go.yurichev.com/17296 [Online; accessed 2013] 1986 [Rit93] Dennis M Ritchie “The development of the C language” In: SIGPLAN Not 28.3 (Mar 1993) Also available as http://go.yurichev.com/17264, pp 201–208 ISSN: 0362-1340 DOI: 10.1145/155360.155580 URL: http://go.yurichev.com/17297 [RT74] D M Ritchie and K Thompson “The UNIX Time Sharing System” In: (1974) Also available as http://go yurichev.com/17270 [Sch94] Bruce Schneier Applied Cryptography: Protocols, Algorithms, and Source Code in C 1994 [SK95] SunSoft Steve Zucker and IBM Kari Karhi SYSTEM V APPLICATION BINARY INTERFACE: PowerPC Processor Supplement Also available as http://go.yurichev.com/17282 1995 [Sko12] Igor Skochinsky Compiler Internals: Exceptions and RTTI Also available as http : / / go yurichev com / 17294 2012 [Str13] Bjarne Stroustrup The C++ Programming Language, 4th Edition 2013 [Swe10] Dominic Sweetman See MIPS Run, Second Edition 2010 [War02] Henry S Warren Hacker’s Delight Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2002 ISBN: 0201914654 [Yur12] Dennis Yurichev “Finding unknown algorithm using only input/output pairs and Z3 SMT solver” In: (2012) Also available as http://go.yurichev.com/17268 [Yur13] Dennis Yurichev C/C++ programming language notes Also available as http://go.yurichev.com/17289 2013 914