OS X and iOS Kernel Programming pptx

COMPANION eBOOK Shelve in Programming / Mac / Mobile User level: Intermediate–Advanced www.apress.com BOOKS FOR PROFESSIONALS BY PROFESSIONALS ® O S X and iOS Kernel Programming combines essential operating system and kernel architecture knowledge with a highly practical approach that will help you write effective kernel-level code. You’ll learn fundamental concepts such as memory management and thread synchronization, as well as the I/O Kit framework. You’ll also learn how to write your own kernel-level extensions, such as device drivers for USB and Thunderbolt devices, including networking, storage and audio drivers. OS X and iOS Kernel Programming provides an incisive and complete introduc- tion to the XNU kernel, which runs iPhones, iPads, iPods, and Mac OS X servers and clients. Then, you’ll expand your horizons to examine Mac OS X and iOS system architecture. Understanding Apple’s operating systems will allow you to write efficient device drivers, such as those covered in the book, using I/O Kit. With OS X and iOS Kernel Programming, you’ll: • Discover classical kernel architecture topics such as memory management and thread synchronization • Become well-versed in the intricacies of the kernel development process by applying kernel debugging and profiling tools • Learn how to deploy your kernel-level projects and how to successfully package them • Write code that interacts with hardware devices • Examine easy to understand example code that can also be used in your own projects • Create network filters Whether you’re a hobbyist, student, or professional engineer, turn to OS X and iOS Kernel Programming and find the knowledge you need to start developing your own device drivers and applications that control hardware devices. Companion eBook Available Master kernel programming for efficiency and performance OS X and iOS Kernel Programming Ole Henry Halvorsen | Douglas Clarke OS X and iOS Kernel Programming Halvorsen Clarke SOURCE CODE ONLINE For your convenience Apress has placed some of the front matter material after the index. Please use the Bookmarks and Contents at a Glance links to access them. iv Contents at a Glance  About the Authors xiv  About the Technical Reviewers xv  Acknowledgments xvi  Introduction xvii  Chapter 1: Operating System Fundamentals 1  Chapter 2: Mac OS X and iOS 15  Chapter 3: Xcode and the Kernel Development Environment 39  Chapter 4: The I/O Kit Framework 51  Chapter 5: Interacting with Drivers from Applications 69  Chapter 6: Memory Management 99  Chapter 7: Synchronization and Threading 119  Chapter 8: Universal Serial Bus 141  Chapter 9: PCI Express and Thunderbolt 173  Chapter 10: Power Management 205  Chapter 11: Serial Port Drivers 223  Chapter 12: Audio Drivers 249  Chapter 13: Networking 275  Chapter 14: Storage Systems 319  Chapter 15: User-Space USB Drivers 357  Chapter 16: Debugging 381  Chapter 17: Advanced Kernel Programming 411  Chapter 18: Deployment 429  Index 443 xvii Introduction Kernel development can be a daunting task and is very different from programming traditional user applications. The kernel environment is more volatile and complex. Extraordinary care must be taken to ensure that kernel code is free of bugs because any issue may have serious consequences to the stability, security, and performance of the system. This book covers the fundamentals necessary to begin programming in the kernel. We cover kernel development from a theoretical and practical point of view. We cover concepts fundamental to kernel development such as virtual memory and synchronization, as well as more practical knowledge. The book primarily focuses on Mac OS X, however the XNU kernel is also used by iOS, and hence the theoretical material in this book will also apply to it. By far the most common reason for doing development within the kernel’s execution environment is to implement a device driver for controlling internal or external hardware devices. Because of this, much of the focus of this book is centred on the development of device drivers. The primary framework for device driver development in the XNU kernel is I/O Kit, which we cover extensively. As theory becomes boring quickly we have provided working code samples which you can play with to learn more or use as a starting point for your own drivers. We hope you have as much fun reading this book as we have enjoyed writing it. Who Is This Book For? The book was written for anyone interested in Apple’s iOS and Mac OS X operating systems, with a focus on practical kernel development, especially driver devel. Regardless of whether you are a hobbyist, student, or professional engineer, we hope to provide you with material of interest. While the focus is on kernel programming and development, we will cover many theoretical aspects of OS technology and provide a detailed overview of the OS X and iOS kernel environments. The aim of the book is to provide the knowledge necessary to start developing your own kernel extensions and drivers. We will focus in particular on the I/O Kit framework for writing device drivers and extensions, but we will also cover general knowledge that will give you a deeper understanding of how I/O Kit interacts with the OS. If you are mainly interested in developing OS X or iOS user applications, this book may not be for you. We will not cover Cocoa or any other framework used for developing end-user applications. This book covers kernel-programming topics such as driver and kernel extension development on Apple’s OS X and iOS platform. Some knowledge of operating system internals will be useful in understanding the concepts discussed in this book. Having completed an introductory computer science or engineering course will be a helpful starting point. Additionally, knowledge of at least one programming language will be required in order to understand examples throughout the book. Since we focus on I/O Kit, which is written in a subset of C++ called Embedded C++, it would be highly beneficial to have some experience with C++ (or at least C) to make the most of this book. The book does not cover general programming topics or theory. We will briefly cover some fundamentals of OS theory to provide a context for further discussions.  INTRODUCTION xviii Book Structure The following is a brief description of each chapter in this book: Chapter 1, Operating System Fundamentals. Details the functionality of an operating system and its role in managing the computer’s hardware resources. We describe the purpose of device drivers and when they are needed, and introduce the differences between programming in the kernel environment as compared to standard application development. Chapter 2, Mac OS X and iOS. Provides a brief overview of the technical structure of XNU, the kernel used by Mac OS X and iOS. Chapter 3, Xcode and the Kernel Development Environment. Provides an overview of the development tools provided by Apple for Mac OS X and iOS development. The chapter ends with a short “Hello world” kernel extension. Chapter 4, The I/O Kit Framework. Introduces the I/O Kit framework that provides the driver model for Mac OS X and its object-oriented architecture. We explain how the I/O Kit finds the appropriate device driver to manage a hardware device. We demonstrate a generic device driver to illustrate the basic structure of any I/O Kit driver. Chapter 5, Interacting with Drivers from Applications. Explains how application code can access a kernel driver. We demonstrate how to search and match against a specific driver as well as how to install a notification to wait for the arrival of a driver or a particular device. We will show how an application can send commands to a driver and watch for events sent by the driver. Chapter 6, Memory Management. Provides an overview of kernel memory management and the different types of memory that a driver needs to work with. We describe the differences between physical and kernel virtual addresses and user-space memory. We also introduce the reader to the concepts such as memory descriptors and memory mapping. Chapter 7, Synchronization and Threading. Describes the fundamentals of synchronization and why it is a necessity for every kernel driver. We discuss the usage of kernel locking mechanisms such as IOLock and IOCommandGate and their appropriate use. We explain how a typical driver requires synchronization between its own threads, user-space threads, and hardware interrupts. We discuss the kernel facilities for creating kernel threads and asynchronous timers. Chapter 8, USB Drivers. Introduces the reader to the architecture of USB and how a driver interfaces with them. We provide an overview of the I/O Kit USB API and the classes it provides for enumerating devices and transferring data to or from a USB device. We also discuss steps needed to support device removal and provide an example to show how a driver can enumerate resources such as pipes. Chapter 9, PCI and Thunderbolt. Provides an overview of the PCI architecture. We also describe the concepts that are unique to PCI drivers, such as memory-mapped I/O, high-speed data transfer through Direct Memory Access (DMA), and handling of device interrupts. We give an overview of the IOPCIDevice class that the I/O Kit provides for accessing and configuring PCI devices. We also discuss the related and more recent Thunderbolt technology. Chapter 10, Power Management. Describes the methods that drivers need to implement in order to allow the system to enter low power states such as machine sleep. We also describe advanced power management that a driver can implement if it wishes to place its hardware into a low power state after a period of inactivity. Chapter 11, Serial Port Drivers. Describes how to implement a serial port driver on Mac OS X. We introduce relevant data structures such as circular queues and techniques for managing data flow through blocking I/O and notification events. We show how a user application can enumerate and access a serial port driver.  INTRODUCTION xix Chapter 12, Audo Drivers. Discusses how system-wide audio input and output devices can be developed using the IOAudioFamily framework. We demonstrate a simple virtual audio device that copies audio output to its input. Chapter 13, Network Drivers. Describes how a network interface can be implemented using the IONetworkingFamily. We also cover how to write network filters to filter, block, and modify network packets. The chapter concludes with an example of how to write an Ethernet driver. Chapter 14, Storage Drivers. Covers the storage driver stack on Mac OS X that provides support for storage devices such as disks and CDs. We describe the drivers at each layer of the storage stack, including how to write a RAM disk, a partition scheme, and a filter driver that provides disk encryption. Chapter 15, User space USB Drivers. Describes how certain drivers can be implemented entirely inside a user application. We describe the advantages to this approach and also when this may not be applicable. Chapter 16, Debugging. Contains practical information on how to debug drivers, as well as common problems and pitfalls. It will enable a reader to work backwards from a kernel crash report to a location in their code, a common scenario facing a kernel developer. We will discuss the tools OS X provides to enable this, such as the GNU debugger (GDB). Chapter 17, Advanced Kernel Programming. Explores some of the more advanced topics in kernel programming, such as utilizing SSE and floating point or implementing advanced driver architectures. Chapter 18, Deployment. Concludes the book by describing how to distribute a driver to the end user. We cover the use of the Apple installation system for both first-time installation and upgrades. The chapter includes practical tips on how to avoid common driver installation problems. C H A P T E R 1 1 Operating System Fundamentals The role of an operating system is to provide an environment in which the user is able to run application software. The applications that users run rely on services provided by the operating system to perform tasks while they execute, in many cases without the user—or even the programmer—giving much thought to them. For an application to read a file from disk, for example, the programmer simply needs to call a function that the operating system provides. The operating system handles the specific steps required to perform that read. This frees the application programmer from having to worry about the differences between reading a file that resides on the computer’s internal hard disk or a file on an external USB flash drive; the operating system takes care of such matters. Most programmers are familiar with developing code that is run by the user and perhaps uses a framework such as Cocoa to provide a graphical user interface with which to interact with the user. All of the applications available on the Mac or iPhone App Store fit into this category. This book is not about writing application software, but rather about writing kernel extensions—that is, code that provides services to applications. Two possible situations in which a kernel extension is necessary are allowing the operating system to work with custom hardware devices and adding support for new file systems. For example, a kernel extension could allow a new USB audio device to be used by iTunes or allow an Ethernet card to provide an interface for networking applications, as shown in Figure 1-1. A file system kernel extension could allow a hard disk formatted on a Windows computer to mount on a Mac as if it were a standard Mac drive. CHAPTER 1  OPERATING SYSTEM FUNDAMENTALS 2 Figure 1-1. The network interfaces listed in the Mac OS X system preferences represent network kernel extensions. An important role of the operating system is to manage the computer’s hardware resources, such as memory and the CPU, and peripherals, such as disk storage and the keyboard. The collection of hardware devices that the operating system needs to support varies greatly from machine to machine. The hardware configuration of a MacBook Air is very different to that of a Mac Pro, although they both run the same operating system. To allow the operating system to support multiple hardware configurations without becoming bloated, the code required to support each hardware component is packaged into a special type of kernel extension known as a driver. This modularity allows the operating system to load drivers on demand, depending on the hardware that is present on the system. This approach also allows for drivers to be installed into the system by vendors to support their custom hardware. The standard installation of Mac OS X comes with over one hundred drivers, of which only a subset is needed to run a particular system. Developing a kernel extension is very different from writing an application. The execution of an application tends to be driven by events originating from the user. The application runs when the user launches it; it may then wait for the user to click a button or select a menu item, at which point the application handles that request. Kernel extensions, on the other hand, have no user interface and do not interact with the user. They are loaded by the operating system, and are called by the operating system to perform tasks that it could not perform by itself, such as when the operating system needs to access a hardware device that the kernel extension is driving. CHAPTER 1  OPERATING SYSTEM FUNDAMENTALS 3 To help with the security and stability of the system, modern operating systems, such as Mac OS X, isolate the core operating system code (the kernel) from the applications and services that are run by the user. Any code that runs as part of the kernel, such as driver code, is said to run in “kernel space.” Code that runs in kernel space is granted privileges that standard user applications do not have, such as the ability to directly read and write to hardware devices connected to the computer. In contrast, the standard application code that users work with are said to run in “user space.” Software that runs in user space has no direct access to hardware. Therefore, to access hardware, user code must send a request to the kernel, such as a disk read request, to request that the kernel perform a task on behalf of the application. There is a strict barrier between code that runs in user space and code that runs in the kernel. Applications can only access the kernel by calling functions that the operating system publishes to user space code. Similarly, code that executes in kernel space runs in a separate environment to user space code. Rather than using the same rich programming APIs that are available to user space code, the kernel provides its own set of APIs that developers of kernel extensions must use. If you are accustomed to user space programming, these APIs may appear restrictive at first, since operations such as user interaction and file system access are typically not available to kernel extensions. Figure 1-2 shows the separation of user space code and kernel space code, and the interaction between each layer. Figure 1-2. The separate layers of responsibility in a modern operating system An advantage of forcing applications to make a request to the kernel to access hardware is that the kernel (and kernel driver) becomes the central arbiter of a hardware device. Consider the case of a sound card. There may be multiple applications on the system that are playing audio at any one time, but because their requests are funneled through to a single audio driver, that driver is able to mix the audio streams from all applications and provide the sound card with the resulting mixed stream. In the remainder of this chapter, we provide an overview of the functionality provided by the operating system kernel, with a focus on its importance in providing user applications with access to hardware. We begin at the highest level, looking at application software, and then digging down into the operating system kernel level, and finally down into the deepest level, the hardware driver. If you are already familiar with these concepts, you can safely proceed to Chapter 2. CHAPTER 1  OPERATING SYSTEM FUNDAMENTALS 4 The Role of the Operating System As part of the boot sequence, the operating system determines the hardware configuration of the system, finds any external devices connected to USB ports or plugged into PCI expansion slots, and initializes them, loading drivers along the way, if necessary. Once the operating system has completed loading, the user is able to run application software. Application software may need to allocate memory or write a file to disk, and it is the operating system that handles these requests. To the user, the involvement of the operating system is largely transparent. The operating system provides a layer of abstraction between running applications and the physical hardware. Applications typically communicate with hardware by issuing high-level requests to the operating system. Because the operating system handles these requests, the application can be completely unaware of the hardware configuration on which it is running, such as the amount of RAM installed and whether the disk storage is an internal SSD or an external USB drive. This abstraction allows application software to be run on a wide variety of different hardware configurations without the programmer having to add support for each one, even if new hardware devices are created after the program has been released. Application developers can often ignore many of the details of the workings of a computer system, because the operating system abstracts away the intricacies of the hardware platform on which the application is running. As a driver developer, however, the code that you write becomes part of the operating system and will interface directly with the computer’s hardware; you are not immune to the inner-workings of a system. For this reason, a basic understanding of how the operating system performs its duties is necessary. Process Management A user typically has many applications installed on his or her computer. These are purely passive entities. The programs on disk contain data that is needed only when the program is run, consisting of the executable code and application data. When the user launches an application, the operating system loads the program’s code and data into memory from disk and begins executing its code. A program being executed is known as a “process.” Unlike a program, a process is an active entity, and consists of a snapshot of the state of the program at a single instance during execution. This includes the program’s code, the memory that the program has allocated, and the current state of its execution, such as the CPU instruction of the function that the program is currently executing, and the contents of its variables and memory allocations. There are typically many processes running on a system at once. These include applications that the user has launched (such as iTunes or Safari), as well as processes that are started automatically by the operating system and that run with no indication to the user. For example, the Time Machine backup service will automatically run a background process every hour to perform a backup of your data. There may even be multiple instances of the same program being executed at any one time, each of which is considered a distinct process by the operating system. Figure 1-3 shows the Activity Monitor utility that is included with Mac OS X, which allows all of the processes running on the system to be examined. [...]... products A high-level view of the Mac OS X architecture is shown in Figure 2-1 16 CHAPTER 2  MAC OS X AND IOS Figure 2-1 Mac OS X architecture The core of Mac OS X and iOS is POSIX compliant and has since Mac OS X 10.5 (Leopard) complied with the Unix 03 Certification The core of OS X and iOS, which includes the kernel and the Unix base of the OS, is known as Darwin, and it is an open source operating... developer website The Darwin OS (and therefore OS X and iOS) runs the XNU kernel, which is based on code from the Mach kernel, as well as parts of the FreeBSD operating system Figure 2-2 shows the Mac OS X desktop 17 4 CHAPTER 2  MAC OS X AND IOS Figure 2-2 The Mac OS X desktop Programming APIs As you can see from Figure 2-1, OS X has a layered architecture Between the Darwin core and the user application... the interfaces provided by Mac OS X to allow drivers to work with virtual and physical memory addresses, respond to requests from user applications, and communicate with PCI and USB devices 13 CHAPTER 2 Mac OS X and iOS Mac OS X is a modern Unix-based operating system developed by Apple Inc for their Macintosh computer series OS X is the tenth incarnation of Mac OS OS X features a graphical user interface... MAC OS X AND IOS Mac OS X comes with a range of tools for developers, including Xcode, which allow the development of a wide range of applications, including the major topic of this book kernel extensions For the end-user, OS X usually comes bundled with the iLife suite, which contains software for photo, audio, and video editing, as well as software for authoring web pages NEXTSTEP OS X and iOS are... a0952000-a0a00000 [ Stack bf800000-bffff000 [ Stack bffff000-c0000000 [ 4K] 264K] 80K] 1696K] 56.0M] r /rwx r -x/ rwx r /rwx r -x/ r -x -/rwx SM=PRV SM=COW SM=COW SM=COW SM=NUL 4K] 44K] 1024K] 8192K] 180K] 4K] 1256K] 96K] 696K] 8188K] 4K] rw-/rwx rw-/rwx rw-/rwx rw-/rwx rw-/rwx rwx/rwx rw-/rwx rw-/rwx rw-/rwx rw-/rwx rw-/rwx SM=PRV SM=PRV SM=PRV SM=PRV SM=PRV SM=COW SM=COW SM=COW SM=COW SM=ZER SM=COW /usr/lib/dyld... invented in the early 1980s by Brad Cox and Tom Love Objective-C is still the de-facto standard language for application development on both OS X and iOS, although driver or system level programming is typically done in C or C++ Many core frameworks still use the NS (for NeXTSTEP) prefix in their class names, such as NSString and NSArray 18 CHAPTER 2  MAC OS X AND IOS Other programming APIs include the BSD... of use and visual appeal Apple has gained a cult-like following for their products, and any new feature addition to either OS X or iOS receives widespread attention In addition to the regular edition of OS X, Apple also provided a server edition of OS X called Mac OS X Server The server version was later merged with the regular version in Mac OS X 10.7 (Lion) OS X was the successor to Mac OS 9, and represented... between address spaces (context switching) is an expensive operation Because the Mach layer is still, to some degree, an isolated component, many refer to XNU as a hybrid kernel, as opposed to a microkernel or a monolithic kernel, where all OS services run in the same context Figure 2-3 shows a simplified view of XNU’s architecture 21 CHAPTER 2  MAC OS X AND IOS Figure 2-3 The XNU kernel architecture The... XNU kernel is the core of Mac OS X and iOS XNU has a layered architecture consisting of three major components The inner ring of the kernel is referred to as the Mach layer, derived from the Mach 3.0 kernel developed at Carnegie Mellon University References to Mach throughout the book will refer to Mach as it is implemented in OS X and iOS and not the original project Mach was developed as a microkernel,... “Jailbreak” iOS and gain access to the underlying Unix and kernel environment, but this voids the warranty Due to concerns about battery life, the iPhone was not able to properly multitask third-party applications until the release of iOS 4.0 iOS now supports the iPhone, iPod Touch, and iPad, and also runs on the latest generation of Apple TVs, which were previously based on OS X, running on Intel x8 6 CPUs . Companion eBook Available Master kernel programming for efficiency and performance OS X and iOS Kernel Programming Ole Henry Halvorsen | Douglas Clarke OS X and iOS Kernel Programming Halvorsen Clarke SOURCE. to the XNU kernel, which runs iPhones, iPads, iPods, and Mac OS X servers and clients. Then, you’ll expand your horizons to examine Mac OS X and iOS system