Understanding Apple's Binary Protection in Mac OS X

© Amit Singh. All Rights Reserved. Written in October 2006

Introduction

With the advent of Intel-based Macintosh computers, Apple was faced with a new requirement: to make it non-trivial to run Mac OS X on non-Apple hardware. The "solution" to this "problem" is multifaceted. One important aspect of the solution involves the use of encrypted executables for a few key applications like the Finder and the Dock. Apple calls such executables apple-protected binaries. In this document, we will see how apple-protected binaries work in Mac OS X.

Relax. Please don't send me a note telling me about your "friend" who has been "easily" running Mac OS X on the laundry machine. When I say "non-trivial," we're not talking about mathematical impossibility, etc.

Note that besides hindering software piracy, there are other scenarios in which encrypted binaries could be desirable. For example, one could turn the requirement around and say that a given system must not run any binaries unless they are from a certain source (or set of sources). This could be used to create an admission-control mechanism for executables, which in turn could be used in defending against malware. In a draconian managed environment, it might be desired to limit program execution on managed systems to a predefined set of programs—nothing else will execute. In general, a set of one or more binaries could be arbitrarily mapped (in terms of runnability) to a set of one or more machines, possibly taking users, groups, and other attributes into account. I must point out that to create such a mechanism, one doesn't have to use encrypted binaries.

Further details on various concepts and terminology used in this document can be found in the book Mac OS X Internals: A Systems Approach. Specific references are provided within "Further Reading" boxes throughout this page.

The Structure of an Apple-Protected Binary

A typical Mach-O file contains a header at its beginning, followed by a sequence of load commands, followed by data for one or more segments. Load commands, which can have variable size, specify how the file is laid out and how it is to be linked. A segment, which contains a sequence of zero or more sections, is described by an LC_SEGMENT load command. Figure 1 shows the relevant data structure.

struct segment_command { unsigned long cmd; /* LC_SEGMENT */ unsigned long cmdsize; /* includes sizeof section structs */ char segname[16]; /* segment name */ unsigned long vmaddr; /* memory address of this segment */ unsigned long vmsize; /* memory size of this segment */ unsigned long fileoff; /* file offset of this segment */ unsigned long filesize; /* amount to map from the file */ vm_prot_t maxprot; /* maximum VM protection */ vm_prot_t initprot; /* initial VM protection */ unsigned long nsects; /* number of sections in segment */ unsigned long flags; /* flags */ };

Figure 1. The data structure representing the LC_SEGMENT command


Further Reading

In Mac OS X 10.4.x, an apple-protected binary is a Mach-O file containing one or more AES-encrypted segments. The LC_SEGMENT load command for an encrypted segment has a special bit (0x8) set. Actually, the first few (3 in Mac OS X 10.4.x) pages of the segment are not encrypted—the rest are.

We can use the otool command-line program to view the load commands in a Mach-O file. Figure 2 shows excerpts from the output of running otool on the binaries for ls and the Finder. We see that the flags field has the 0x8 bit set in the case of the Finder, which is an apple-protected binary, but not in the case of ls.

$ otool -l /bin/ls /bin/ls: Load command 0 ... Load command 1 cmd LC_SEGMENT cmdsize 600 segname __TEXT vmaddr 0x00001000 vmsize 0x00004000 fileoff 0 filesize 16384 maxprot 0x00000007 initprot 0x00000005 nsects 8 flags 0x0 Section ... $ cd /System/Library/CoreServices/Finder.app/Contents/MacOS $ otool -l Finder Finder: Load command 0 ... Load command 1 cmd LC_SEGMENT cmdsize 872 segname __TEXT vmaddr 0x00001000 vmsize 0x002e8000 fileoff 0 filesize 3047424 maxprot 0x00000007 initprot 0x00000005 nsects 12 flags 0x8 Section ...

Figure 2. Using otool to view Mach-O load commands


One can write a simple program that trawls the entire system looking for Mach-O files matching the apple-protected criteria. The following is a list of (possibly non-exhaustive) apple-protected binaries in Mac OS X 10.4.x.

Before we see how apple-protected binaries are executed, let us look at an overview of how the Intel version of Mac OS X executes program in general.

Mac OS X Program Execution: A Simplified Description

In the first version of UNIX, the exec system call was used to execute a file by overlaying the calling process with the file and transferring control to beginning of the file's core image. The basic idea is the same in Mac OS X. The execve system call ultimately executes programs from the kernel's standpoint, regardless of the user-level API used to initiate the execution attempt.

#include <unitstd.h> int execve(const char *path, char *const argv[], char *const envp[]);

Figure 3. The execve system call


As Figure 3 indicates, execve() begins with the path to an executable. The system call handler uses the namei() kernel function to convert a path into a vnode. Given the vnode, the corresponding file can be read from.

Further Reading

execve() reads the first page (4096 bytes) from the file to examine what kind of a program it is. The x86 version of the Mac OS X kernel can handle the execution of the following file types.

The following additional points are noteworthy:

Given the first page of the file, execve() attempts to determine the file type using a multipass algorithm. For example, in the case of a fat binary, it calculates the location of the appropriate Mach-O binary (if any) within the fat binary, reads the first page of the new target binary, and reexamines its header. Here we are interested in apple-protected binaries, so we will not discuss how the special cases of Rosetta and interpreter scripts are handled.

Upon identifying a Mach-O binary, execve() calls the Mach-O image activator (fancy name for a handler function) to take care of the actual execution. The activator performs a variety of sanity checks, sets various flags, handles task-working-set caching details, and eventually commences to load the Mach-O file. This loading step includes setting up an appropriate virtual memory map and parsing the Mach-O commands in the binary.

Of particular interest to us is the LC_SEGMENT command, which, as we saw earlier, describes a segment. For every LC_SEGMENT command, the kernel checks the flags field to see if the apple-protected bit is set, and if so, it proceeds to unprotect that segment as an additional step before deeming the load a success. Figure 4 shows a relevant kernel code excerpt illustrating the control flow that occurs during the handling of execve().

// The execve system call handler int execve(...) { ... // The case of a Mach-O file error = exec_mach_imgact(...); ... } // Mach-O image activator static int exec_mach_imgact(struct image_params *imgp) { ... lret = load_machfile(...); ... } load_return_t load_machfile(...) { ... lret = parse_machfile(...); ... } // Parse and handle Mach-O load commands static load_return_t parse_machfile(...) { ... switch (lcp->cmd) { case LC_SEGMENT: ... ret = load_segment(...); ... } ... } static load_return_t load_segment(...) { ... if (scp->flags & 0x8) { // This is an apple-protected segment ret = unprotect_segment_64((uint64_t)scp->fileoff, (uint64_t)scp->filesize, map, (vm_map_offset_t)map_addr, (vm_map_size_t)map_size); } else { ret = LOAD_SUCCESS; } return ret; }

Figure 4. Unprotecting a protected Mach-O segment during execution


Let us now look at how the kernel handles "unprotection".

Unprotecting Apple-Protected Binaries

The "unprotect" operation determines the offset at which to begin (recall that the first few pages of an apple-protected segment are not encrypted) and calls a special memory-mapping function, vm_map_apple_protected().

In a Mach-based VM subsystem, vm_map() maps the specified memory object to a region of virtual memory. Crudely put, a memory object represents a range of pages all of which are accessible through a given pager (also called a memory manager). The pager handles page-ins and page-outs over the memory region(s) it is responsible for. For example, when the kernel calls a pager to handle a page-in operation, the pager can retrieve the page's data from a backing store such as a disk. When a file is being executed, it is the vnode pager that retrieves pages from the file.


Further Reading

vm_map_apple_protected() does a rather simple thing: it essentially interposes another pager, the apple-protect pager, between the kernel and the vnode pager. Normally, the kernel would communicate with the vnode pager to request data for a page. With this interposition in place, the kernel communicates with the apple-protect pager instead, which takes the encrypted mapping backed by the vnode pager and transforms (decrypts) each page as it is requested by the kernel. Figure 5 shows a code excerpt.

// Called to handle page-ins kern_return_t apple_protect_pager_data_request( memory_object_t mem_obj, memory_object_offset_t offset, vm_size_t length, vm_prot_t protection_required) { ... dsmos_page_transform((const void *)src_vaddr, (void *)dst_vaddr); ... }

Figure 5. Page-transformation performed by the apple-protect pager


Note that the apple-protect pager doesn't support page-outs. In fact, its implementation of the routine for handling page-outs (the "data return" routine in Mach pager parlance) panics the system because it should never be called under normal circumstances (unlike vm_map(), vm_map_apple_protected() is not an exposed interface).

Amusingly, the "dsmos" prefix in Figure 5 stands for "Don't Steal Mac OS X". In fact, the Don't Steal Mac OS X.kext kernel extension plays a critical role in the working of the apple-protect pager. The dsmos_page_transform() function, which takes as arguments a source (encrypted) page and a destination (decrypted) page, simply calls the function pointed to by the dsmos_hook function pointer. The "dsmos" kernel extension sets this function pointer to point to page_transform(), a function implemented within the kernel extension. page_transform() performs the actual AES decryption.

Using some means of peeking at kernel memory, you can catch the apple-protect pager in action. For example, one of the counters this pager maintains is an integer named apple_protect_pager_count_mapped. We can retrieve the value of this integer from the kernel and see how the value changes as we run programs. Since the Rosetta binaries are apple-protected, we can run a trivial PowerPC binary that sleeps for a little while, and examine the counter's value before, during, and after the execution of the binary. Figure 6 shows the experiment, which uses a script (readksym.sh) for reading kernel symbols via /dev/kmem.

Further Reading

$ cat sleeper.c main() { sleep(30); } $ gcc -arch ppc -o sleeper sleeper.c $ sudo readksym.sh _apple_protect_pager_count_mapped 4 0000000 0008 0000 $ ./sleeper & [1] 7173 $ sudo readksym.sh _apple_protect_pager_count_mapped 4 0000000 0009 0000 [1] + done ./sleeper $ sudo readksym.sh _apple_protect_pager_count_mapped 4 0000000 0008 0000

Figure 6. Viewing the number of times the apple-protect pager is mapped


Referring to the list of apple-protected binaries we came across earlier, we should be able to account for each mapped instance of the apple-protect pager.

Don't Steal Mac OS X (In Your Address Space)

On Mac OS X 10.4.x, the "dsmos" kernel extension also stashes a 256-byte blob of integrity data in the commpage. This data is visible at an offset of 0x1600 bytes from the start of the commpage in every user task's virtual address space. In the x86 version of Mac OS X 10.4.x, the commpage starts at -16 x 4096 bytes. Therefore, the integrity data is at the address -16 x 4096 + 0x1600. Figure 7 shows a trivial program that demonstrates this by printing the "string" at the virtual address of interest.

$ cat dsmos.c main() { puts(-16 * 4096 + 0x1600); } $ gcc -o dsmos dsmos.c $ ./dsmos Your karma check for today: There once was was a user that whined his existing OS was so blind, he'd do better to pirate an OS that ran great but found his hardware declined. Please don't steal Mac OS! Really, that's way uncool. (C) Apple Computer, Inc.U??VWS?5P

Figure 7. Dumping the commpage-resident "dsmos" data


Further Reading