Mac OS X and IOS Internals

Jonathan Levin is a longtime trainer and consultant focusing on the system and kernel levels of the 'Big Three' Windows, Linux, and OS X, as well as their mobile derivatives. He is the founder and CTO of Technologeeks.com, a partnership of experts delivering advanced training on systems/kernel programming, debugging, and profiling.

INTRODUCTION xxv PART I: FOR POWER USERS CHAPTER 1: DARWINISM: THE EVOLUTION OF OS X 3 The Pre-Darwin Era: Mac OS Classic 3 The Prodigal Son: NeXTSTEP 4 Enter: OS X 4 OS X Versions, to Date 5 10.0 Cheetah and the First Foray 5 10.1 Puma a Stronger Feline, but ... 6 10.2 Jaguar Getting Better 6 10.3 Panther and Safari 6 10.4 Tiger and Intel Transition 6 10.5 Leopard and UNIX 7 10.6 Snow Leopard 7 10.7 Lion 8 10.8 Mountain Lion 9 iOS OS X Goes Mobile 10 1.x Heavenly and the First iPhone 11 2.x App Store, 3G and Corporate Features 11 3.x Farewell, 1st gen, Hello iPad 11 4.x iPhone 4, Apple TV, and the iPad 2 11 5.x To the iPhone 4S and Beyond 12 iOS vs. OS X 12 The Future of OS X 15 Summary 16 References 16 CHAPTER 2: E PLURIBUS UNUM: ARCHITECTURE OF OS X AND IOS 17 OS X Architectural Overview 17 The User Experience Layer 19 Aqua 19 Quicklook 20 Spotlight 21 Darwin The UNIX Core 22 The Shell 22 The File System 23 UNIX System Directories 24 OS X Specifi c Directories 25 iOS File System Idiosyncrasies 25 Interlude: Bundles 26 Applications and Apps 26 Info.plist 28 Resources 30 NIB Files 30 Internationalization with .lproj Files 31 Icons (.icns) 31 CodeResources 31 Frameworks 34 Framework Bundle Format 34 List of OS X and iOS Public Frameworks 37 Libraries 44 Other Application Types 46 System Calls 48 POSIX 48 Mach System Calls 48 A High-Level View of XNU 51 Mach 51 The BSD Layer 51 libkern 52 I/O Kit 52 Summary 52 References 53 CHAPTER 3: ON THE SHOULDERS OF GIANTS: OS X AND IOS TECHNOLOGIES 55 BSD Heirlooms 55 sysctl 56 kqueues 57 Auditing (OS X) 59 Mandatory Access Control 62 OS X- and iOS-Specifi c Technologies 65 User and Group Management (OS X) 65 System Confi guration 67 Logging 69 Apple Events and AppleScript 72 FSEvents 74 Notifi cations 78 Additional APIs of interest 79 OS X and iOS Security Mechanisms 79 Code Signing 80 Compartmentalization (Sandboxing) 81 Entitlements: Making the Sandbox Tighter Still 83 Enforcing the Sandbox 89 Summary 90 References 90 CHAPTER 4: PARTS OF THE PROCESS: MACH-O, PROCESS, AND THREAD INTERNALS 91 A Nomenclature Refresher 91 Processes and Threads 91 The Process Lifecycle 92 UNIX Signals 95 Executables 98 Universal Binaries 99 Mach-O Binaries 102 Load Commands 106 Dynamic Libraries 111 Launch-Time Loading of Libraries 111 Runtime Loading of Libraries 122 dyld Features 124 Process Address Space 130 The Process Entry Point 130 Address Space Layout Randomization 131 32-Bit (Intel) 132 64-Bit 132 32-Bit (iOS) 133 Experiment: Using vmmap(1) to Peek Inside a Process s Address Space 135 Process Memory Allocation (User Mode) 138 Heap Allocations 139 Virtual Memory The sysadmin Perspective 140 Threads 143 Unraveling Threads 143 References 146 CHAPTER 5: NON SEQUITUR: PROCESS TRACING AND DEBUGGING 147 DTrace 147 The D Language 147 dtruss 150 How DTrace Works 152 Other Profi ling mechanisms 154 The Decline and Fall of CHUD 154 AppleProfi leFamily: The Heir Apparent 155 Process Information 156 sysctl 156 proc-info 156 Process and System Snapshots 159 system-profi ler(8) 159 sysdiagnose(1) 159 allmemory(1) 160 stackshot(1) 160 The stack-snapshot System Call 162 kdebug 165 kdebug-based Utilities 165 kdebug codes 166 Writing kdebug messages 168 Reading kdebug messages 169 Application Crashes 170 Application Hangs and Sampling 173 Memory Corruption Bugs 174 Memory Leaks 176 heap(1) 177 leaks(1) 177 malloc-history(1) 178 Standard UNIX Tools 178 Process listing with ps(1) 179 System-Wide View with top(1) 179 File Diagnostics with lsof(1) and fuser(1) 180 Using GDB 181 GDB Darwin Extensions 181 GDB on iOS 182 LLDB 182 Summary 182 References and Further Reading 182 CHAPTER 6: ALONE IN THE DARK: THE BOOT PROCESS: EFI AND IBOOT 183 Traditional Forms of Boot 183 EFI Demystifi ed 185 Basic Concepts of EFI 186 The EFI Services 188 NVRAM Variables 192 OS X and boot.efi 194 Flow of boot.efi 195 Booting the Kernel 201 Kernel Callbacks into EFI 203 Boot.efi Changes in Lion 204 Boot Camp 204 Count Your Blessings 204 Experiment: Running EFI Programs on a Mac 206 iOS and iBoot 210 Precursor: The Boot ROM 210 Normal Boot 211 Recovery Mode 212 Device Firmware Update (DFU) Mode 213 Downgrade and Replay Attacks 213 Installation Images 214 OS X Installation Process 214 iOS File System Images (.ipsw) 219 Summary 225 References and Further Reading 225 CHAPTER 7: THE ALPHA AND THE OMEGA LAUNCHD 227 launchd 227 Starting launchd 227 System-Wide Versus Per-User launchd 228 Daemons and Agents 229 The Many Faces of launchd 229 Lists of LaunchDaemons 241 GUI Shells 246 Finder (OS X) 247 SpringBoard (iOS) 248 XPC (Lion and iOS) 253 Summary 257 References and Further Reading 258 PART II: THE KERNEL CHAPTER 8: SOME ASSEMBLY REQUIRED: KERNEL ARCHITECTURES 261 Kernel Basics 261 Kernel Architectures 262 User Mode versus Kernel Mode 266 Intel Architecture Rings 266 ARM Architecture: CPSR 267 Kernel/User Transition Mechanisms 268 Trap Handlers on Intel 269 Voluntary kernel transition 278 System Call Processing 283 POSIX/BSD System calls 284 Mach Traps 287 Machine Dependent Calls 292 Diagnostic calls 292 XNU and hardware abstraction 295 Summary 297 References 297 CHAPTER 9: FROM THE CRADLE TO THE GRAVE KERNEL BOOT AND PANICS 299 The XNU Sources 299 Getting the Sources 299 Making XNU 300 One Kernel, Multiple Architectures 302 The XNU Source Tree 305 Booting XNU 308 The Bird s Eye View 309 OS X: vstart 310 iOS: start 310 [i386|arm]-init 311 i386-init-slave() 313 machine-startup 314 kernel-bootstrap 314 kernel-bootstrap-thread 318 bsd-init 320 bsdinit-task 325 Sleeping and Waking Up 328 Boot Arguments 329 Kernel Debugging 332 Don t Panic 333 Implementation of Panic 334 Panic Reports 336 Summary 340 References 341 CHAPTER 10: THE MEDIUM IS THE MESSAGE: MACH PRIMITIVES 343 Introducing: Mach 344 The Mach Design Philosophy 344 Mach Design Goals 345 Mach Messages 346 Simple Messages 346 Complex messages 347 Sending Messages 348 Ports 349 The Mach Interface Generator (MIG) 351 IPC, in Depth 357 Behind the Scenes of Message Passing 359 Synchronization Primitives 360 Lock Group Objects 361 Mutex Object 362 Read-Write Lock Object 363 Spinlock Object 364 Semaphore Object 364 Lock Set Object 366 Machine Primitives 367 Clock Object 378 Processor Object 380 Processor Set Object 384 Summary 388 References 388 CHAPTER 11: TEMPUS FUGIT MACH SCHEDULING 389 Scheduling Primitives 389 Threads 390 Tasks 395 Task and Thread APIs 399 Task APIs 399 Thread APIs 404 Scheduling 408 The High-Level View 408 Priorities 409 Run Queues 412 Mach Scheduler Specifi cs 415 Asynchronous Software Traps (ASTs) 423 Scheduling Algorithms 427 Timer Interrupts 431 Interrupt-Driven Scheduling 431 Timer Interrupt Processing in XNU 432 Exceptions 436 The Mach Exception Model 436 Implementation Details 437 Experiment: Mach Exception Handling 440 Summary 446 References 446 CHAPTER 12: COMMIT TO MEMORY: MACH VIRTUAL MEMORY 447 Virtual Memory Architecture 447 The 30,000-Foot View of Virtual Memory 448 The Bird s Eye View 449 The User Mode View 452 Physical Memory Management 462 Mach Zones 467 The Mach Zone Structure 468 Zone Setup During Boot 470 Zone Garbage Collection 471 Zone Debugging 473 Kernel Memory Allocators 473 kernel-memory-allocate() 473 kmem-alloc() and Friends 477 kalloc 477 OSMalloc 479 Mach Pagers 480 The Mach Pager interface 480 Universal Page Lists 484 Pager Types 486 Paging Policy Management 494 The Pageout Daemon 495 Handling Page Faults 497 The dynamic-pager(8) (OS X) 498 Summary 499 References 500 CHAPTER 13: BS D THE BSD LAYER 501 Introducing BSD 501 One Ring to Bind Them 502 What s in the POSIX Standard? 503 Implementing BSD 503 XNU Is Not Fully BSD 504 Processes and Threads 504 BSD Process Structs 504 Process Lists and Groups 507 Threads 508 Mapping to Mach 510 Process Creation 512 The User Mode Perspective 512 The Kernel Mode Perspective 513 Loading and Executing Binaries 516 Mach-O Binaries 522 Process Control and Tracing 525 ptrace (#26) 525 proc-info (#336) 527 Policies 527 Process Suspension/Resumption 529 Signals 529 The UNIX Exception Handler 529 Hardware-Generated Signals 534 Software-Generated Signals 535 Signal Handling by the Victim 536 Summary 536 References 537 CHAPTER 14: SOMETHING OLD, SOMETHING NEW: ADVANCED BSD ASPECTS 539 Memory Management 539 POSIX Memory and Page Management System Calls 540 BSD Internal Memory Functions 541 Memory Pressure 545 Jetsam (iOS) 546 Kernel Address Space Layout Randomization 548 Work Queues 550 BSD Heirlooms Revisited 552 Sysctl 552 Kqueues 555 Auditing (OS X) 556 Mandatory Access Control 558 Apple s Policy Modules 560 Summary 563 References 563 CHAPTER 15: FEE, FI-FO, FILE: FILE SYSTEMS AND THE VFS 565 Prelude: Disk Devices and Partitions 565 Partitioning Schemes 567 Generic File System Concepts 577 Files 577 Extended Attributes 577 Permissions 577 Timestamps 578 Shortcuts and Links 578 File Systems in the Apple Ecosystem 579 Native Apple File Systems 579 DOS/Windows File Systems 580 CD/DVD File Systems 581 Network-Based File Systems 582 Pseudo File Systems 583 Mounting File Systems (OS X only) 587 Disk Image Files 589 Booting from a Disk Image (Lion) 590 The Virtual File System Switch 591 The File System Entry 591 The Mount Entry 592 The vnode Object 595 FUSE File Systems in USEr Space 597 File I/O from Processes 600 Summary 605 References and Further Reading 605 CHAPTER 16: TO B (-TREE) OR NOT TO BE THE HFS+ FILE SYSTEMS 607 HFS+ File System Concepts 607 Timestamps 607 Access Control Lists 608 Extended Attributes 608 Forks 611 Compression 612 Unicode Support 617 Finder integration 617 Case Sensitivity (HFSX) 619 Journaling 619 Dynamic Resizing 620 Metadata Zone 620 Hot Files 621 Dynamic Defragmentation 622 HFS+ Design Concepts 624 B-Trees: The Basics 624 Components 630 The HFS+ Volume Header 631 The Catalog File 633 The Extent Overfl ow 640 The Attribute B-Tree 640 The Hot File B-Tree 641 The Allocation File 642 HFS Journaling 642 VFS and Kernel Integration 645 fsctl(2) integration 645 sysctl(2) integration 646 File System Status Notifi cations 647 Summary 647 References 648 CHAPTER 17: ADHERE TO PROTOCOL: THE NETWORKING STACK 649 User Mode Revisited 650 UNIX Domain Sockets 651 IPv4 Networking 651 Routing Sockets 652 Network Driver Sockets 652 IPSec Key Management Sockets 654 IPv6 Networking 654 System Sockets 655 Socket and Protocol Statistics 658 Layer V: Sockets 660 Socket Descriptors 660 mbufs 661 Sockets in Kernel Mode 667 Layer IV: Transport Protocols 668 Domains and Protosws 669 Initializing Domains 673 Layer III: Network Protocols 676 Layer II: Interfaces 678 Interfaces in OS X and iOS 678 The Data Link Interface Layer 680 The ifnet Structure 680 Case Study: utun 682 Putting It All Together: The Stack 686 Receiving Data 686 Sending Data 690 Packet Filtering 693 Socket Filters 694 ipfw(8) 696 The PF Packet Filter (Lion and iOS) 697 IP Filters 698 Interface Filters 701 The Berkeley Packet Filter 701 Traffi c Shaping and QoS 705 The Integrated Services Model 706 The Diff erentiated Services Model 706 Implementing dummynet 706 Controlling Parameters from User Mode 707 Summary 707 References and Further Reading 708 CHAPTER 18: MODU(LU)S OPERANDI KERNEL EXTENSIONS 711 Extending the Kernel 711 Securing Modular Architecture 712 Kernel Extensions (Kexts) 713 Kext Structure 717 Kext Security Requirements 718 Working with Kernel Extensions 719 Kernelcaches 719 Multi-Kexts 723 A Programmer s View of Kexts 724 Kernel Kext Support 725 Summary 735 References 735 CHAPTER 19: DRIVING FORCE I/O KIT 737 Introducing I/O Kit 738 Device Driver Programming Constraints 738 What I/O Kit Is 738 What I/O Kit Isn t 741 LibKern: The I/O Kit Base Classes 742 The I/O Registry 743 I/O Kit from User Mode 746 I/O Registry Access 747 Getting/Setting Driver Properties 749 Plug and Play (Notifi cation Ports) 750 I/O Kit Power Management 751 Other I/O Kit Subsystems 753 I/O Kit Diagnostics 753 I/O Kit Kernel Drivers 755 Driver Matching 755 The I/O Kit Families 757 The I/O Kit Driver Model 761 The IOWorkLoop 764 Interrupt Handling 765 I/O Kit Memory Management 769 BSD Integration 769 Summary 771 References and Further Reading 771 APPENDIX: WELCOME TO THE MACHINE 773 INDEX 793