Mac OS x and iOS internals

This book aims to do for XNU what Bovet & Cesati’s Understanding the Linux Kernel does for Linux, and ich’s Windows Internals does for Windows.. Darwin Libraries & syscallsChapter 2,3,4

www.it-ebooks.info

www.it-ebooks.info

MAC OS® X AND iOS INTERNALS

INTRODUCTION xxv

 PART I FOR POWER USERS CHAPTER 1 Darwinism: The Evolution of OS X 3

CHAPTER 2 E Pluribus Unum: Architecture of OS X and iOS 17

CHAPTER 3 On the Shoulders of Giants: OS X and iOS Technologies 55

CHAPTER 4 Parts of the Process: Mach-O, Process, and Thread Internals 91

CHAPTER 5 Non Sequitur: Process Tracing and Debugging 147

CHAPTER 6 Alone in the Dark: The Boot Process: EFI and iBoot 183

CHAPTER 7 The Alpha and the Omega — launchd 227

 PART II THE KERNEL CHAPTER 8 Some Assembly Required: Kernel Architectures 261

CHAPTER 9 From the Cradle to the Grave — Kernel Boot and Panics 299

CHAPTER 10 The Medium Is the Message: Mach Primitives 343

CHAPTER 11 Tempus Fugit — Mach Scheduling 389

CHAPTER 12 Commit to Memory: Mach Virtual Memory 447

CHAPTER 13 BS”D — The BSD Layer 501

CHAPTER 14 Something Old, Something New: Advanced BSD Aspects 539

CHAPTER 15 Fee, FI-FO, File: File Systems and the VFS 565

CHAPTER 16 To B (-Tree) or Not to Be — The HFS+ File Systems 607

CHAPTER 17 Adhere to Protocol: The Networking Stack 649

CHAPTER 18 Modu(lu)s Operandi — Kernel Extensions 711

CHAPTER 19 Driving Force — I/O Kit 737

APPENDIX Welcome to the Machine 773

INDEX 793

www.it-ebooks.info

www.it-ebooks.info

Mac OS® X and iOS Internals

TO THE APPLE’S CORE

Jonathan Levin

www.it-ebooks.info

Mac OS® X and iOS Internal

Copyright © 2013 by Jonathan Levin

Published by John Wiley & Sons, Inc., Indianapolis, Indiana

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means,

electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Sections 107 or 108

of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization

through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA

01923, (978) 750-8400, fax (978) 646-8600 Requests to the Publisher for permission should be addressed to the

Permis-sions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008,

or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with

respect to the accuracy or completeness of the contents of this work and specifi cally disclaim all warranties, including

without limitation warranties of fi tness for a particular purpose No warranty may be created or extended by sales or

promotional materials The advice and strategies contained herein may not be suitable for every situation This work is

sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional

services If professional assistance is required, the services of a competent professional person should be sought Neither

the publisher nor the author shall be liable for damages arising herefrom The fact that an organization or Web site is

referred to in this work as a citation and/or a potential source of further information does not mean that the author or the

publisher endorses the information the organization or Web site may provide or recommendations it may make Further,

readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this

work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the

United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley publishes in a variety of print and electronic formats and by print-on-demand Some material included with

standard print versions of this book may not be included in e-books or in print-on-demand If this book refers to media

such as a CD or DVD that is not included in the version you purchased, you may download this material at

http://booksupport.wiley.com For more information about Wiley products, visit www.wiley.com.

Library of Congress Control Number: 2011945020

Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Wrox Programmer to Programmer, and related trade dress are

trademarks or registered trademarks of John Wiley & Sons, Inc and/or its affi liates, in the United States and other

coun-tries, and may not be used without written permission Mac OS is a registered trademark of Apple, Inc All other

trade-marks are the property of their respective owners John Wiley & Sons, Inc., is not associated with any product or vendor

mentioned in this book.

www.it-ebooks.info

To Steven Paul Jobs: From Mac OS’s very fi rst incarnation, to the present one, wherein the legacy of NeXTSTEP still lives, his relationship with Apple is forever entrenched in OS X (and iOS) People focus on his effect on Apple as a company No less of an effect, though hidden to the naked eye, is on its architecture

I resisted the pixie dust for 25 years, but he

fi nally made me love Mac OS Just as soon as I got

my shell prompt.

— Jonathan Levin

www.it-ebooks.info

ACQUISITIONS EDITOR

Mary James

SENIOR PROJECT EDITOR

Adaobi Obi Tulton

DEVELOPMENT EDITOR

Sydney Argenta

TECHNICAL EDITORS

Arie HaenelDwight Spivey

PRODUCTION EDITOR

Christine Mugnolo

COPY EDITORS

Paula Lowell Nancy Rapoport

EDITORIAL MANAGER

Mary Beth Wakefi eld

FREELANCER EDITORIAL MANAGER

ABOUT THE AUTHOR

JONATHAN LEVIN is a seasoned technical trainer and consultant focusing on the internals of the

“Big Three” (Windows, Linux, and Mac OS) as well as their mobile derivatives (Android and iOS) Jonathan has been spreading the gospel of kernel engineering and hacking for 15 years, and has given technical talks at DefCON as well as other technical conferences He is the founder and CTO

of Technologeeks.com, a partnership of expert like-minded individuals, devoted to propagating knowledge through technical training, and solving tough technical challenges through consulting Their areas of expertise cover real-time and other critical aspects of software architectures, system/kernel-level programming, debugging, reverse engineering, and performance optimizations

ABOUT THE TECHNICAL EDITORS

ARIE HAENEL is a security and internals expert at NDS Ltd (now part of Cisco) Mr Haenel has vast experience in data and device security across the board He holds a Bachelor of Science Engi-neering in Computer Science from the Jerusalem College of Technology, Israel and an MBA from the University of Poitiers, France His hobbies include learning Talmud, judo, and solving riddles He lives in Jerusalem, Israel

DWIGHT SPIVEY is the author of several Mac books, including OS X Mountain Lion Portable

Genius and OS X Lion Portable Genius He is also a product manager for Konica Minolta, where

he has specialized in working with Mac operating systems, applications, and hardware, as well as color and monochrome laser printers He teaches classes on Mac usage, writes training and support materials for Konica Minolta, and is a member of the Apple Developer Program Dwight lives on the Gulf Coast of Alabama with his beautiful wife Cindy and their four amazing children, Victoria, Devyn, Emi, and Reid He studies theology, draws comic strips, and roots for the Auburn Tigers (“War Eagle!”) in his ever-decreasing spare time

www.it-ebooks.info

www.it-ebooks.info

“Y’KNOW, JOHNNY,” said my friend Yoav, taking a puff from his cigarette on a warm summer night

in Shanghai, “Why don’t you write a book?”

And that’s how it started It was Yoav (Yobo) Chernitz who planted the seed to write my own book, for a change, after years of reading others’ From that moment, in the Far, Middle, and US East (and the countless fl ights in between), the idea began to germinate, and this book took form I had little idea it would turn into the magnum opus it has become, at times taking on a life of its own, and becoming quite the endeavor With so many unforeseen complications and delays, it’s hard to believe

it is now done I tried to illuminate the darkest reaches of this monumental edifi ce, to delineate them, and leave no stone unturned Whether or not I have succeeded, you be the judge But know, I couldn’t have done it without the following people:

Arie Haenel, my longtime friend — a natural born hacker, and no small genius Always among my harshest critics, and an obvious choice for a technical reviewer

Moshe Kravchik — whose insights and challenging questions as the book’s fi rst reader fully made it a lot more readable for all those who follow

hope-Yuval Navon — from down under in Melbourne, Australia, who has shown me that ship knows no geographical bounds

friend-And last, but hardly least, to my darling Amy, who was patient enough to endure my quent travels, more than understanding enough to support me to no end, and infi nitely wise enough

all-too-fre-to constantly remind me not only of the important deadlines and obligations I had with this book, but of the things that are truly the most important in life

— Jonathan Levin

www.it-ebooks.info

www.it-ebooks.info

INTRODUCTION xxv

PART I: FOR POWER USERS

Summary 16 References 16CHAPTER 2: E PLURIBUS UNUM: ARCHITECTURE OF OS X AND IOS 17

Aqua 19Quicklook 20Spotlight 21

www.it-ebooks.info

CONTENTS

Libraries 44

POSIX 48

sysctl 56kqueues 57

www.it-ebooks.info

Summary 90 References 90CHAPTER 4: PARTS OF THE PROCESS: MACH-O,

CONTENTS

CHAPTER 5: NON SEQUITUR:

DTrace 147

dtruss 150

sysctl 156proc_info 156

sysdiagnose(1) 159allmemory(1) 160stackshot(1) 160

kdebug 165

heap(1) 177leaks(1) 177malloc_history(1) 178

CONTENTS

CHAPTER 6: ALONE IN THE DARK:

Summary 225

launchd 227

CONTENTS

PART II: THE KERNEL

CHAPTER 8: SOME ASSEMBLY REQUIRED:

Threads 390Tasks 395

www.it-ebooks.info

www.it-ebooks.info

CONTENTS

Summary 499 References 500

CONTENTS

Sysctl 552Kqueues 555

Summary 563 References 563CHAPTER 15: FEE, FI-FO, FILE: FILE SYSTEMS AND THE VFS 565

Summary 605

CHAPTER 16: TO B (-TREE) OR NOT TO BE —

Timestamps 607

www.it-ebooks.info

CONTENTS

Forks 611Compression 612

CONTENTS

Summary 707

Kernelcaches 719Multi-Kexts 723

Summary 735 References 735

www.it-ebooks.info

CONTENTS

INDEX 793

www.it-ebooks.info

www.it-ebooks.info

EVEN MORE THAN TEN YEARS AFTER ITS INCEPTION, there is a dearth of books discussing the ture of OS X, and virtually none about iOS While there is plentiful documentation on Objective-C, the frameworks, and Cocoa APIs of OS X, it often stops short of the system-call level and implemen-tation specifi cs There is some documentation on the kernel (mostly by Apple), but it, too, focuses on building drivers (with I/O Kit), and shows only the more elegant parts, and virtually nothing on the Mach core that is foundation of XNU XNU is open source, granted, but with over a million lines of source (and comments) with some dating as far back to 1987, it’s not exactly a fun read

architec-This is not the case with other operating systems Linux, being fully open source, has no shortage of books, including the excellent series by O’Reilly Windows, though closed, is exceptionally well docu-mented by Microsoft (and its source has been “liberated” on more than one occasion) This book aims

to do for XNU what Bovet & Cesati’s Understanding the Linux Kernel does for Linux, and ich’s Windows Internals does for Windows Both are superb books, clearly explaining the architectures

Russinov-of these incredibly complex operating systems With any luck, the book you are holding (or downloaded

as a PDF) will do the same to expound on the inner workings of Apple’s operating systems

A previous book on Mac OS — Amit Singh’s excellent OS X Internals: A Systems Approach is an

amazing reference, and provides a vast wealth of valuable information Unfortunately, it is PowerPC oriented, and is only updated up until Tiger, circa 2006 Since then, some six years have passed Six long years, in which OS X has abandoned PowerPC, has been fully ported to Intel, and has progressed

by almost four versions Through Leopard, Snow Leopard, Lion and, most recently Mountain Lion, the wild cat family is expanding, and many more features have been added Additionally, OS X has been ported anew This time to the ARM architecture, as iOS, (which is, by some counts, the world’s leading operating system in the mobile environments) This book, therefore, aims to pick up where its predeces-sor left off, and discuss the new felines in the Apple ecosystem, as well as the various iOS versions

Apple’s operating systems have proven to be moving targets This book was originally written to target iOS 5 and Lion, but both have gone on evolving iOS is, at the time this book goes to print,

at 5.1.1 with hints of iOS 6 OS X is still at Lion (10.7.4), but Mountain Lion (10.8) is in advanced developer previews, and this book will hit the shelves coinciding with its release Every attempt has been made to keep the information as updated as possible to refl ect all the versions, and remain rel-evant going forward

OVERVIEW AND READING SUGGESTION

This is a pretty large book Initially, it was not designed to be this big and detailed, but the more I delved into OS X I uncovered more of the abstruse, for which I could fi nd no detailed explanation

or documentation I therefore found myself writing about more and more aspects An operating tem is a full eco-system with its own geography (hardware), atmosphere (virtual memory), fl ora and fauna (processes) This book tries to methodically document as much as it can, while not sacrifi cing clarity for detail (or vice versa) No mere feat

sys-www.it-ebooks.info

INTRODUCTION

Architecture at a Glance

OS X and iOS are have a complex architecture, which is a hybrid of several very different

technolo-gies: The UI and APIs of the legacy OS 9 (for OS X) with NextSTEP’s Cocoa, the system calls and

kernel layer of BSD, and the kernel structure of NeXTSTEP Though an amalgam, it still maintains

a relatively clean separation between its components Figure I-1 shows a bird’s eye view of the

archi-tecture, and maps the components to the corresponding chapters in this book

Darwin Libraries & syscalls(Chapter 2,3,4)

Hardware

Application FrameworksUser Experience

Mach Abstractions (Chapter 10)

VFS (15)

Networking (17)

VM (14)

VM (11)

FIGURE I-1: OS X Architecture, and its mapping to chapters in this book

This book additionally contains chapters on non-architectural, yet very important topics, such as

debugging (5), fi rmware (6) and user mode startup (7), kernel-mode startup (9), and kernel modules

(18) Lastly, there are two appendices: The fi rst, providing a quick reference for POSIX system calls

and Mach traps, and the second, providing a gentle high-level introduction to the assembly of both

Intel and ARM architectures

Target Audience

There are generally four types of people who might fi nd this tome, or its parts, interesting:

‰ Power users and system administrators who want to get a better idea of how OS X works

Mac OS adoption grows steadily by the day, as market claws back market share that was, for

www.it-ebooks.info

Choose your own adventure

While this book can be read cover to cover, let’s not forget it is a technical book, after all The ters are therefore designed to be read individually, as a detailed explanation or as a quick reference You have the option of reading chapters in sequential or random access, skimming or even skipping over some chapters, and coming back to them later for a more thorough read If a chapter refers to a concept or function discussed in a previous chapter, it is clearly noted

chap-You are also welcome to employ a reading strategy which refl ects the type of target reader you sify yourself as For example, the chapters of the fi rst part of this book can therefore be broken into the fl ow shown in Figure I-2:

User mode 4: Process Internals

5: Process Tracing and Debugging

PowerUser UserDev Kernel Dev

INTRODUCTION

In Figure I-2, a full bar implies the chapter contents are of interest to the target reader, and a partial

bar implies at least some interest Naturally, every reader’s interest will vary This is why every

chap-ter starts with a brief introduction, discussing what the chapchap-ter is about Likewise, just by looking

at the section headers in the table of contents you can fi gure out if the section merits a read or just a

quick skim

The second part of this book could actually have been a volume by itself It focuses on the XNU

kernel architecture, and is considerably more complicated than the fi rst This cannot be avoided; by

their very nature, kernels are subject to a more complicated, real-time, and hardware constrained

environment This part shows many more code listings, and (thankfully, rarely) even has to go into

snippets of code implemented in assembly Reading suggestions for this part of the book are shown

com-fi guration Normally, the results of these experiments are demonstrated in detail, but you are more than encouraged to try the experiments on your own system, and witness the results Like UNIX, which it implements, Mac OS X can truly be experienced and absorbed through the fi ngers, not the eyes or ears

In some cases, some parts of the experiments have been left out as an exercise for the reader

Even though the book’s companion website will have the solutions — i.e fully working versions of the exercises in question — you are encouraged to try to complete those parts yourself Careful reading of the book, with a modicum of common sense, should provide you with everything you need to do so

TOOLS

The book also makes use of a few tools, which were developed by the author to accompany the book The tools, true to the UNIX heritage, are command line tools, and are meant to be both easily readable as well as grep(1)-able, making them useful not just for manual usage, but also in scripts

fi lemon

Chapter 3 presents a tool called “fi lemon,” to display real time fi le system activity on OS X and iOS

An homage to Russinovich’s tool of the same name, this simple utility relies on the FSEvents device, present in OS X and iOS 5, to follow fi le system related events, such as creation and deletion of fi les

psx

Chapter 4 presents a tool called psx, an extended ps-like command which can display pretty much any tidbit of information one could possibly require about processes and threads in OS X It is particularly useful for this chapter, which deals with process internals, and demonstrates using an undocumented system call, proc_info The tool requires no special permissions if you are viewing your own processes, but will require root permissions otherwise The tool can be freely downloaded from the book’s companion website, with full source code

jtool

While for most binary function one can use the OS X built-in otool(1), it leaves much to be desired

in analyzing data section and can get confused when displaying ARM binaries due to the two modes of assembly in the ARM architecture jtool aims to improve on otool, by addressing these

www.it-ebooks.info

INTRODUCTION

shortcomings, and offering useful new features for static binary analysis The tool comes in handy

in Chapter 4, which details the Mach-O fi le format, as well as later in this book, due to its many

useful features, like fi nding references in fi les and limited disassembly skills The tool can be freely

downloaded from the book’s companion website, but is closed source

dEFI

This is a simple program to dump the fi rmware (EFI) variables on an Intel Mac and to display

reg-istered EFI providers This tool demonstrates the basics of EFI programming — interfacing with the

boot and runtime services This tool can be freely downloaded, along with its source code It is

pre-sented in Chapter 6

joker

The joker tool, presented in Chapter 8, is a simple tool created to play with the kernel (specifi cally,

in iOS) The tool can fi nd and display the system call and Mach trap tables of iOS and OS X kernels,

show sysctl structures, and look for particular patterns in the binary This tool is highly useful for

reverse engineers and hackers alike, as the trap and system call symbols are no longer exported

corerupt

Chapter 11 discusses the low-level APIs of the Mach virtual memory manager To demonstrate just

how powerful (and dangerous) these APIs are, the book provides the corerupt tool This tool enables

you to dump any process’s virtual memory map to a fi le in a core-compatible format, similar to

Windows’ Create Dump File option, and much like the gcore tool in this book’s predecessor It

fur-ther improves on its precursor, by providing support for ARM and allowing invasive operations on

the vm map, such as modifying its pages

HFSleuth

A key tool used in the book is HFSleuth, a command line all-in-one utility for viewing the

support-ing structures of HFS+ fi le systems, which are the native OS X fi le system type The tool was

devel-oped because there really are no alternative ways to demonstrate the inner workings of this rather

complicated fi le system Singh’s book, Mac Os X Internals: A Systems Approach (Addison-Wesley;

2006) also included a similar, though less feature-ful tool called hfsdebug, but the tool was only

provided for PowerPC, and was discontinued in favor of a commercial tool, fi leXRay

To use HFSleuth on an actual fi le system, you must be able to read the fi le system One option is to

simply be root HFSleuth’s functions are nearly all read-only, so rest assured it is perfectly safe But

access permissions to the underlying block (and sometimes, character) devices on which the fi le

sys-tems are usually rw-r -, meaning the devices are not readable by plebes If you generally distrust

root and adhere to least privilege (a wise choice!), an equally potent alternative is to chmod(1) the

permissions on the HFS+ partition devices, making them readable to your user (usually, this involves

an o+r) Advanced functions (such as repair, or HFS+/HFSX conversion) will require write access

www.it-ebooks.info

sock-in Chapter 17, uses an undocumented kernel control protocol called com.apple.network.statistics to obtain real-time notifi cations of sockets as they are created The tool is especially easy to incorporate into scripts, making it handy for use as a connection event handler

as output to XML format

All the tools mentioned here are made available for free, and will remain free, whether you buy (or copy) the book This is because they are generally useful, and fi ll many advanced functions, which are either lacking, or present but well hidden, in Apple’s own tools

CONVENTIONS USED IN THIS BOOK

To make it easier to follow along the book and not be bogged down by reiterating specifi c ground for example code and programs, this book adopts a few conventions, which are meant to subtly remind you of the context of the given listings

back-Dramatis Personae

The demos and listings in this book have naturally been produced and tested on various versions of Apple computers and i-Devices As is in the habit of sysadmins to name their boxes, each host has his or her own “personality” and name Rather than repeatedly specifying which demo is based on which device and OS, the shell command prompt has been left as is, and by the hostname you can easily fi gure out which version of OS X or iOS the demo can be reproduced on (See Table I-1.)

www.it-ebooks.info

INTRODUCTION

TABLE I-1: Host Name and Version Information for the Book’s Demos

Ergo MacBook Air,

2010

Snow Leopard , 10.6.8 Generic OS X feature demonstration

Tested in Snow Leopard and later

iPhonoclast iPhone 4S iOS 5.1.1 iOS 5 and later features on an A5 (ARM

multi-core)

Minion Mac Mini, 2010 Lion, 10.7.4 Lion specifi c feature demonstration

Simulacrum VMWare image Mountain Lion, 10.8.0

DP3

Mountain Lion (Developer Preview) specifi c feature demonstration

Padishah iPad 2 iOS 4.3.3 iOS 4 and later features

Podicum iPod Touch, 4G iOS 5.0.1 iOS 5 specifi c features, on A4 or A5

Further, shell prompts of root@ demonstrate a command runnable only by the root user This makes

it easy to see which examples will run on which system, with what privileges

Code Excerpts and Samples

This book contains a considerable number of code samples of two types:

‰ Example programs, which are found mostly in the fi rst part These usually demonstrate simple

concepts and principles that hold in user mode, or specifi c APIs or libraries The example

pro-grams were all devised by the author, are well commented, and are free for you to try

your-self, modify in any way you see fi t, or just leave on the page In an effort to promote the lazy,

all these programs are available on the book’s website, in both open source and binary form

‰ Darwin code excerpts, which are found mostly in the second part These are almost entirely

snippets of XNU’s code, taken from the latest open source version, i.e 1699.26.8

(cor-responding to Lion 10.7.4) All code is open source, but subject to Apple’s Public Source

License The excerpts are provided here for demonstration of the relevant parts in XNU’s

architecture While natural language is potentially prone to some ambiguities, code is context

free and precise (though unfortunately sometimes less readable), and so at times the most

precise explanation comes from reading the code When code references are provided, they

are usually either to the header fi les (denoted by the standard C < > notation, e.g <mach/

mach-o.h>) in /usr/include Other times, they may refer to the Darwin sources, either of

XNU or some related package In those cases, the relative path is used (e.g osfmk/kern/

spl.c, relating to where the XNU kernel source is extracted) The related package will

always be specifi ed in the section, and in Part II of the book nearly all references are to the

XNU kernel source

www.it-ebooks.info

INTRODUCTION

XNU and Darwin components are fairly well documented, but this book tries to go the extra step, and sometimes provide additional explanations inline, as comments To be clear, such annotations, which are not part of the original source code, can be clearly marked by their C++ style comment, rather than the C style comment which is typical in Darwin as in this sample listing:

LISTING I-1: SAMPLE LISTING

/* This is a Darwin comment, as it appears in the original source */

// This is an annotation provided by the author, elaborating or explaining // something which the documentation may or may not leave wanting

// Where the source code is long and tedious, or just obvious, some parts may // be omitted, and this is denoted by a comment marking ellipsis ( ), i.e:

//

important parts of a listing or output may be shown in bold

The book distinguishes between outputs and listings Listings are verbatim references from fi les,

either program source code or system fi les Outputs, on the other hand, are textual captures of user commands, shown for demonstration on OS X, iOS, or — sometimes — both The book aims to compare and contrast the two systems, so it is not uncommon to fi nd the same sequence of com-mands shown on both systems In an output, you will see the user commands that were typed marked in bold, and are encouraged to follow along and try them on your own systems

In general, the code listings are provided to elucidate, not to confuse Natural language is not out its ambiguities, but code can only be interpreted one way (even if sometimes that way is not entirely clear) Whenever possible, clear descriptions aided by detailed fi gures will hopefully enable you to just skim through the code Fluency in C (and sometimes a little assembly) is naturally helpful for reading the code samples, but is not necessary The comments — especially the extra annota-tions — help you understand the gist of the code More commonly, block diagrams and fl ow charts are presented, leaving the functions as black boxes This enables to choose between remaining at an overview level, or delving deeper and seeing the actual variables and functions of the implementa-tions Be warned, however, that the complexity of the code, being the product of many people and many coding styles, varies greatly throughout XNU

with-In the case of iOS, XNU remains closed iOS versions actually use a version of XNU many sions ahead of the publicly released versions Naturally, code samples cannot be shown, but in some

revi-cases disassembly (mostly of iOS 5.x) is provided The assembly in question is ARM, and comments

there — all provided by the author — aim to explicate its inner workings For all things assembly, you can refer to the appendix in this book for a quick overview

www.it-ebooks.info

INTRODUCTION

Typographic Conventions

Every effort has been made to ensure that these conventions are followed throughout this book:

‰ Words in courier font denote commands, fi le names, function names, or variable names

from the Darwin sources

‰ Commands are further specifi ed by their man section (if applicable) in parentheses Example:

ls(1) for a user command, write(2) for a system call, printf(3) for a library call, and

ipfw(8) for a system administration command Most commands and system calls shown in

this book are usually well documented in the manual page, and the book does not attempt to

upstage the fi ne manual (i.e RTFM, fi rst) Occasionally, however, the documentation may

leave some aspects wanting — or, rarely, undocumented at all — and this is where further

information is provided

THE COMPANION WEBSITE(S)

Both OS X and iOS have rapidly evolved, and continue to do so I will try to play catch up, and

keep an updated companion website for this book at http://newosxbook.com My company,

(http://technologeeks.com), also maintains the OS X and iOS Kernel developers group on

LinkedIn (alongside those of Windows and Android), with its website of http://darwin

kerneldevelopers.com (the name chosen in a forward-compatible view of a post OS X era The

latter site includes a questions and answers forum, which will hopefully become a bustling arena for

OS X and iOS related discussions

On the book’s companion website you can fi nd:

‰ An appendix that lists the various POSIX and Mach system calls

‰ The sample programs included in experiments throughout this book — for the enthusiastic

to try, yet lazy to code The programs are provided in source form, but also as binaries (for

those even lazier to compile(!) or devoid of XCode)

‰ The tools introduced in this book, and discussed in this introduction freely downloadable in

binary form for both OS X and iOS, and often times with source

‰ Updated references and links to other web resources, as they become available

‰ Updated articles about new features or enhancements, as time goes by

‰ Errata — Errare est humanum, and — especially in iOS, where most of the details were eked

out by painful disassembly, there may be inaccuracies or version differences that need to be

fi xed

This book has been an unbelievable journey, through the looking glass (while playing with kittens),

unraveling the very fabric of the reality presented to user mode applications I truly hope that you,

the reader, will fi nd it as illuminating as I have, drawing ideas not just on OS X and iOS, but on

operating system architecture and software design in general

Read on then, ye devout Apple-lyte, and learn

www.it-ebooks.info

Levin c01 V4 - 05/11/2012

PART I

For Power Users

 CHAPTER 1: Darwinism: The Evolution of OS X

 CHAPTER 2: E Pluribus Unum: Architecture of OS X and iOS

 CHAPTER 3: On the Shoulders of Giants: OS X and iOS Technologies

 CHAPTER 4: Parts of the Process: Mach-O, Process, and Thread

Internals

 CHAPTER 5: Non Sequitur: Process Tracing and Debugging

 CHAPTER 6: Alone in the Dark: The Boot Process: EFI and iBoot

 CHAPTER 7: The Alpha and the Omega — launchd

www.it-ebooks.info

www.it-ebooks.info

by some accounts the mobile operating system with the largest market share, head-to-head with Linux’s derivative, Android

The growth, however, did not happen overnight In fact, it was a long and excruciating cess, which saw Mac OS come close to extinction, before it was reborn as “OS X.” Simply

pro-“reborn” is an understatement, as Mac OS underwent a total reincarnation, with its ture torn down and rebuilt anew Even then, Mac OS still faced signifi cant hardship before the big breakthrough — which came with Apple’s transition to Intel-based architecture, leaving behind its long history with PowerPC architectures

architec-The latest and greatest version, OS X 10.7, or Lion, occurred shortly before the release of this

book, as did the release of iOS 5.x, the most recent version of iOS To understand their

fea-tures and the relationship between the two, however, it makes sense to take a few steps back and understand how the architecture unifying both came to be

The following is by no means a complete listing of features, but rather a high-level tive Apple has been known to add hundreds of features between releases, mostly in GUI and application support frameworks Rather, more emphasis is placed on design and engineering features For a comprehensive treatise on Mac OS versions to date, see Amit Singh’s work on the subject[1], or check Ars Technica’s comprehensive reviews[2] Wikipedia also maintains a fairly complete list of changes[3]

perspec-THE PRE-DARWIN ERA: MAC OS CLASSIC

Mac OS Classic is the name given the pre-OS X era of Mac OS The operating system then was nothing much to boast about True, it was novel in that it was an all-GUI system (earlier versions did not have a command line like today’s “Terminal” app) Memory management was

www.it-ebooks.info

Levin c01 V4 - 05/11/2012

poor, however, and multitasking was cooperative, which — by today’s standards — is considered

primitive Cooperative multitasking involves processes voluntarily yielding their CPU timeslice, and

works reasonably well when processes are well behaved If even one process refuses to cooperate,

however, the entire system screeches to a halt Nonetheless, Mac OS Classic laid some of the

foun-dations for the contemporary Mac OS, or OS X Primarily, those founfoun-dations include the “Finder”

GUI, and the fi le system support for “forks” in the fi rst generation HFS fi le system These affect OS

X to this very day

THE PRODIGAL SON: NEXTSTEP

While Mac OS experienced its growing pains in the face of the gargantuan PC, its founder Steve

Jobs left Apple (by some accounts was ousted) to get busy with a new and radically different

com-pany The company, NeXT, manufactured specialized hardware, the NeXT computer and

NeXTsta-tion, with a dedicated operating system called NeXTSTEP

NeXTSTEP boasted some avant-garde features for the time:

‰ NeXTSTEP was based on the Mach microkernel, a little-known kernel developed by

Carne-gie Mellon University (CMU) The concept of a microkernel was, itself, considered a novelty, and remains rarely implemented even today

‰ The development language used was Objective-C, a superset of C, which — unlike C++ — is

heavily object-oriented

‰ The same object-orientation was prevalent all throughout the operating system The system

offered frameworks and kits, which allowed for rapid GUI development using a rich object library, based on the NSObject

‰ The device driver environment was an object-oriented framework as well, known as

DriverKit Drivers could subclass other drivers, inheriting from them and extending their functionality

‰ Applications and libraries were distributed in self-contained bundles Bundles consisted of a

fi xed directory structure, which was used to package software, along with its dependencies and related fi les, so installing and uninstalling could be as easy as moving around a folder

‰ PostScript was heavily used in the system, including a variant called “display postscript,”

which enabled the rendering of display images as postscript Printing support was thus 1:1, unlike other operating systems, which needed to convert to a printer-friendly format

NeXTSTEP went down the road of better operating systems (remember OS/2?), and is nowadays

extinct, save for a GNUStep port Yet, its legacy lives on to the present day One winter day in 1997,

Apple — with an OS that wasn’t going anywhere — ended up acquiring NeXT, bringing its

intellec-tual property into Apple, along with Steve Jobs And the rest, as they say, is history

ENTER: OS X

As a result of the acquisition of NeXT, Apple gained access to Mach, Objective-C, and the other

aspects of the NeXTSTEP architecture While NeXTSTEP was discontinued as a result, these

components live on in OS X In fact, OS X can be considered as a fusion of Mac OS Classic and

www.it-ebooks.info