Earlier this year, I released GrabFS, a MacFUSE file system that shows “live” screenshots of Mac OS X applications. If you wish to understand how GrabFS works, you can now browse its source.

Enjoy.

MacFUSE has a new install/update mechanism that greatly simplifies and improves things both for end users and developers who use MacFUSE in their software.

The relevant wiki page has all the details.

Note that instead of Tiger- and Leopard-specific downloads, now there’s a single downloadable disk image containing a single installable package. The package, which third parties can choose to include within their metapackages, knows how to install the latest version of MacFUSE for your platform.

Recently, I had a need to know if any files or folders had been modified or created on an HFS+ volume in the past N seconds. There are many ways you could generate this type of information on Mac OS X.

To begin with, you could try asking Spotlight.

Besides Spotlight, Mac OS X has a rich variety of mechanisms and APIs for learning about file system changes.

On Leopard, you could write a program that uses the FSEvents API to learn of directory-level changes that occur on a volume. The FSEvents API is part of CoreServices.framework. "Directory-level" means that this API is best suited for monitoring large directory trees—it will not tell you when a particular file changes.

To monitor specific files, you could use the kqueues mechanism. (See kqueue(2).) Being file-level, kqueues don’t scale like the FSEvents API as you will need to monitor each file system object separately. Therefore, it’s better suited for situations where you need to monitor only a few specific objects.

You could also directly use the low-level fsevents mechanism (/dev/fsevents) that underlies the FSEvents API and Spotlight—but only if your need is experimental in nature. The fslogger program is an example of directly using the fsevents mechanism. fslogger will tell you—in pretty much real time—when file system objects change. (Make sure to see caveats.)

Then there is the kauth mechanism that was introduced in Tiger, primarily to help creators of virus scanning software. Kauth allows for extremely fine-grained file system activity monitoring—you can see vnode-level operations. In fact, monitoring is sort of a side effect of using the kauth mechanism. You can actually allow and deny individual operations, as virus scanning software might need to. However, kauth is not easy to use. (Not that the other APIs mentioned necessarily are!) To use kauth, you need to write a kernel extension. You also need to be extremely careful in what you do so as not to deadlock the operating system.

There also exist tools like fs_usage and dtrace on Mac OS X. fs_usage uses the kernel’s kdebug facility to perform fine-grained tracing of kernel events. In particular, it allows you to trace file system activity. Beginning with Leopard, the DTrace facility, to which dtrace is a front-end, lets you trace all kinds of activity at both the kernel and user levels. You could do some very imaginative things with dtrace.

So, we see that there is no dearth of ways to monitor file system activity on Mac OS X. However, there are caveats associated with each way we looked at so far. Consider Spotlight. To use it, we would be assuming that Spotlight indexing was enabled on the volume in question. Spotlight also doesn’t look everywhere: your areas of interest on the file system might be outside of Spotlight’s default or configured search scope. Moreover, to use or the other APIs we talked about, you will need to have the volume mounted—usually a reasonable requirement, except it may not be an option if, for example, you are trying to recover valuable data from a volume that has been through an accident. Or you could be performing file system forensics. Or the volume could be damaged enough to not be in a mountable state—at least not without repair. In these situations, you can’t or wouldn’t want to mount the volume. That aside, in my case, I didn’t know until after the volume had been modified that I wanted to know what had changed. That is, I didn’t happen to be conveniently running any monitoring programs and such.

You can always old plain old Unix-style find and walk the entire file system, examining each file and folder. This still needs the volume to be mounted, but it is exhaustive. Of course, if you have a large volume, exhaustively examining each file and folder through a brute-force find or other programs could take "forever." (In my case, I had over 4 million files on the volume. I also had little patience.)

Fortunately, Mac OS X lets you exploit the fact that the HFS+ volume format uses a central catalog B-Tree for storing hierarchy: the searchfs() system call can be used to "quickly" search HFS+ volumes. (It is much, much quicker than a typical portable user-space file-tree-walk.) In my case, I could use searchfs() to search for files and folders with creation or modification dates that match my criteria. Well, almost. I actually did require the volume to be unmounted. I also felt more inclined to do something general purpose.

hfsdebug is a tool that can walk the catalog tree even on unmounted volumes. I decided to add filtering capability to hfsdebug. "Filtering" means that hfsdebug can walk the HFS+ catalog B-Tree, examining each file and folder, and produce output based on some matching criteria. The new version of hfsdebug contains two built-in filters: mtime and crtime. You can use these filters to look for files and folders that have been modified or created, respectively, in the past N seconds. The number of seconds is passed as an argument to these filters. For example, to look for file system objects modified within the past 60 seconds, you would run hfsdebug as follows:

$ sudo hfsdebug ––filter=builtin:crtime ––filter_args=60
1216795688 [Tue Jul 22 23:48:08 2008]: Macintosh HD:/private/var/log/asl.db
1216795688 [Tue Jul 22 23:48:08 2008]: Macintosh HD:/private/var/log/system.log
...

Better still, you can write your own filters that hfsdebug can use. A filter is implemented as a dynamic library that implements up to 3 functions: one of them mandatory (hfsdebug_filter_callback()) and two of them optional (hfsdebug_filter_init() and hfsdebug_filter_fini().) To use your own filter, you would run hfsdebug the same way as in the case of built-in filters:

$ sudo hfsdebug ––filter=/path/to/myfilter.dylib ––filter_args=string
...

If your filter implements the hfsdebug_filter_init() function, hfsdebug would call it with the filter argument string, if any, as the argument. You could parse the argument string in the init function and initialize your filter’s state, if necessary.

int
hfsdebug_filter_init(const char *filter_args);

If you return a non-zero value from the init function, hfsdebug will terminate. If your filter doesn’t have any arguments, you could choose not to implement the init function.

After you return 0 from the init function, hfsdebug will invoke your filter’s callback function once for each file and folder record in the HFS+ catalog.

typedef char*(*hfsdebug_filter_path_retriever_t)(void);

int
hfsdebug_filter_callback(
    void *info, hfsdebug_filter_path_retriever_t pathRetriever);

The info argument is a pointer to either an HFSPlusCatalogFile structure or an HFSPlusCatalogFolder structure. (See the xnu kernel source for details of these structures.) You can determine which structure it is based on the first int16_t within the structure: it’s either kHFSPlusFileRecord or kHFSPlusFolderRecord. Given these structures, your filter can examine various attributes of the file system object.

Note that hfsdebug does not pass you the path to the file system object in question. This is because path computation is expensive. Instead, hfsdebug passes you a pointer to a path retriever function. You can invoke this function to make hfsdebug compute the path on demand and return a C string pointer. This pointer is valid for the given file system object only until your callback returns. You should only call the path retriever function if you truly need the path—doing so for each file system object would be quite time consuming. Note that hfsdebug filters are not multithreaded.

Again, you must return 0 from the callback for hfsdebug to keep calling you as long as there are more file system objects. If you return a non-zero value, hfsdebug will terminate.

Finally, once hfsdebug is done with all file system objects, it will call your filter’s fini function if one is implemented.

void
hfsdebug_filter_fini(void);

The following is a complete example of an hfsdebug filter. It does the same things as the built-in mtime filter, that is, it looks for files and folders that were modified within the last N seconds.

/*
 * myfilter.c
 *
 * HFSDebug Filter for "mtime"
 *
 * Look for file system objects that have been modified
 * within the past N seconds.
 *
 * gcc -I/path/to/xnu/bsd/ -Wall -o myfilter.dylib myfilter.c
 */

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <errno.h>
#include <time.h>

#include <hfs/hfs_format.h>

#define MAC_GMT_FACTOR 2082844800UL

typedef char*(*hfsdebug_filter_path_retriever_t)(void);

static uint32_t mtime_seconds = 0;

int
hfsdebug_filter_init(const char *filter_args)
{
    mtime_seconds = strtoul(filter_args, NULL, 10);
    if ((errno == ERANGE) || (errno == EINVAL)) {
        fprintf(stderr,
                "invalid argument (%s) to mtime filter\n", filter_args);
        return errno;
    }

    time_t now = time(NULL);

    mtime_seconds = now - mtime_seconds + MAC_GMT_FACTOR;

    return 0;
}

int
hfsdebug_filter_callback(void *info,
                         hfsdebug_filter_path_retriever_t pathRetriever)
{
    int16_t recordType = *(int16_t*)info;
    uint32_t modDate;

    if (recordType == kHFSPlusFileRecord) {
        HFSPlusCatalogFile *file = (HFSPlusCatalogFile*)info;
        modDate = file->contentModDate;
    } else if (recordType == kHFSPlusFolderRecord) {
        HFSPlusCatalogFolder *folder = (HFSPlusCatalogFolder*)info;
        modDate = folder->contentModDate;
    } else {
        /* ignore */
        return 0;
    }

    if (modDate > mtime_seconds) {
        modDate -= MAC_GMT_FACTOR;
        char *tmpTime = asctime(localtime((time_t*)&modDate));
        *(tmpTime + 24) = 0;
        fprintf(stdout,
                "%u [%s]: %s\n", modDate, tmpTime, pathRetriever());
    }

    return 0;
}

void
hfsdebug_filter_fini(void)
{
    return;
}

Download HFSDebug 3.20

1200 Baud Archaeology

Here is a unique computer puzzle: the audio file (MP3 encoding) contains something that could well be music to many a hacker’s ears. What is it? Can you "prove" that it is what you say it is?

Version 1.5 of MacFUSE is out.

The CHANGELOG has details of what’s new.

I discovered a bug in hfsdebug. It causes hfsdebug to crash while printing Access Control Entry (ACE) details for certain files or folders. For example, consider the standard ~/Library/Preferences/ folder on Leopard.

$ ls -lde ~/Library/Preferences
drwx------@ 167 singh  staff ... /Users/singh/Library/Preferences
 0: group:everyone deny delete

This folder has an ACE for the group everyone. In particular, the ACE applies to no specific user (or you could say it applies to the wildcard user). HFSDebug was not dealing with this situation well. See what happens.

$ sudo hfsdebug ~/Library/Preferences/
  <Catalog B-Tree node = 15028 (sector 0x49080)>
  path                 = Macintosh HD:/Users/singh/Library/Preferences
# Catalog Folder Record
...
        # ACL Entry
        ace_applicable     = ab cd ef ab cd ef ab cd ef ab cd ef 0 0 0 c
zsh: bus error  sudo ./hfsdebug ~/Library/Preferences

I’ve released a bugfix version of HFSDebug to take care of this. The correct behavior should be as follows.

$ sudo hfsdebug ~/Library/Preferences/
  <Catalog B-Tree node = 15028 (sector 0x49080)>
  path                 = Macintosh HD:/Users/singh/Library/Preferences
# Catalog Folder Record
...
        # ACL Entry
        ace_applicable     = ab cd ef ab cd ef ab cd ef ab cd ef 0 0 0 c
          user             = *
          group            = everyone
          gid              = 12
        ace_flags          = 00000000000000000000000000000010 (0x000002)
                             . KAUTH_ACE_DENY
        ace_rights         = 00000000000000000000000000010000 (0x000010)
                             . KAUTH_VNODE_DELETE

Download HFSDebug 3.10

A Myth That Won’t Die

Quoting my Google Mac Blog post in its entirety:

A while ago, I wrote about procfs for Mac OS X, a MacFUSE-based file system. Subsequently, I added more cool features to my procfs implementation. Recently, I had reason to demonstrate procfs again and realized that I needed still more cool features. That need led to GrabFS.

In a pinch, GrabFS is a file system that shows you a live view of the window contents of currently running applications. In a GrabFS volume, folders represent running applications and image files represent instant screenshots (”grabs”) of the applications’ windows. You simply copy a file or just open it in place, and you have a screenshot. Open it again, and you have a new screenshot!

Go here to read more about GrabFS and to download it. GrabFS requires Mac OS X "Leopard" and MacFUSE.

I found some time today to make a certain feature of HFSDebug work on Leopard. The new version is available for download here. The new version should run on both Leopard and Tiger, but there are no visible changes whatsoever for Tiger users.

If you use HFSDebug, you might have realized that the -m option doesn’t work on Leopard any more. This option is used to retrieve and display the in-kernel mount data for a currently mounted HFS+ volume. This is what you would see if you ran the now deprecated version 2.56 of HFSDebug on Leopard:


$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.5.1
BuildVersion: 9B18
$ sudo hfsdebug
populateHFSPlusMount(222): failed to retrieve symbol information.
hfsdebug: failed to locate mount data (perhaps the volume is not mounted)
$


The updated version should work correctly as follows. As you can see, if you did care about this information, this is a rather useful feature that needed fixing for Leopard.

$ sudo ./hfsdebug -m
  Volume name                             = Macintosh HD (volfs_id=234881026)
  block device number                     = { major=14, minor=2 }
  HFS+ flags                              = 000...0000000000000010001100
                                            + HFS_WRITEABLE_MEDIA
                                            + HFS_CLEANED_ORPHANS
                                            + HFS_METADATA_ZONE
  default owner                           = { uid=99, gid=99 }
  directory protection bits mask          = 755
  file protection bits mask               = 755
# Key Data Structures
  struct mount *                          = 0x41ebb90
  block device vnode                      = 0x4333f40
  Extents file vnode                      = 0x4333eb0
  Catalog file vnode                      = 0x4333e20
  Allocation file vnode                   = 0x4333d90
  Attributes file vnode                   = 0x4333d00
# Statistics
  physical block size                     = 512
  physical block count                    = 0x12975e60
  alternate volume header location        = 0x12975e5e
  size of a buffer cache buffer           = 4096
  number of files in file system          = 1047391
  number of directories in file system    = 156694
  free allocation blocks                  = 0x1967df5
  start block for next allocation search  = 0xdfd404
  next unused catalog node ID             = 2011304
  file system write count                 = 84130726
  free block reserve                      = 64000
  blocks on loan for delayed allocations  = 0
  encodings in use                        = 00...010000000000000000001001011
# Notification Variables
  notification conditions bits            = 0
  freespace warning limit                 = 64000
  freespace desired level                 = 96000
# Times
  last mounted time                       = Sun Dec 30 21:36:21 2007
  last mounted modification time          = Sun Dec 30 21:35:51 2007
  last modification time                  = Sun Dec 30 22:08:31 2007
  cache of largest known free extents     =
# Journal
  journal for this volume                 = 0x4338f00
  vnode for journal device                = 0x4333f40
  start block of journal                  = 0x4a8
  journal size                            = 16777216
  journal file ID                         = 16
  journal info block file ID              = 17
# Hot File Clustering
  clustering stage                        = HFC_RECORDING
  recording period start time             = Thu Dec 20 07:41:40 2007
  recording period stop time              = Wed Jan  2 13:17:52 2008
  opaque recording data                   = 0x24189004
  maximum files to track                  = 1000
  vnode of Hot Files B-Tree               = 0x0
# Metadata Zone
  metadata zone start block               = 0x1
  metadata zone end block                 = 0x67fff
  hotfile start block                     = 0x45be2
  hotfile end block                       = 0x67fff
  hotfile free blocks                     = 0x20491
  hotfile maximum blocks                  = 0x2241e
  overflow maximum blocks                 = 0x800
  catalog maximum blocks                  = 0x43f3b
# Other
  maximum inline attribute size           = 3802

A new release of MacFUSE is here. There is a new version for Leopard, a new version for Tiger, and a new version of sshfs.app that runs on both Tiger and Leopard.

Downloads: http://code.google.com/p/macfuse/downloads/list

Documentation: http://code.google.com/p/macfuse/w/list

I don’t have an iPhone (and don’t really intend to get one), but for the iPhone-equipped curious operating system investigators, the iPhone Restore image downloadable from Apple’s web site has plenty of interesting details about the hardware and software composition of the iPhone, how some of the things work, and so on. Enjoy.

(Hint: Looks like several components of the image might have inadvertently escaped encryption before it was put up for download.)

Several years ago, while I was working at the IBM Almaden Research Center, we came up with a security mechanism called the Assured Execution Environment (AxE). We had implementations for Windows XP and Mac OS X. (Although AxE supports code signing as a feature, it’s not the same—in any case, this was long before code signing was known as a forthcoming feature in Mac OS X "Leopard".)

An evolved version of the Windows implementation is now available for download from the IBM alphaWorks web site.

A few weeks ago, I released as open source a MacFUSE-based process file system for Mac OS X.

I recently added several new features to this procfs implementation. Some of these features are "cool" in that they put a new twist on certain types of visual information.

For example, there’s a folder /proc/system/hardware/displays/ that contains a subfolder each for connected displays. Subfolder 0 represents the first display, 1 is the second display (if any), and so on. Within each such subfolder, there’s a file called info that contains information about that particular display: its resolution, bits-per-pixel, bytes-per-row, whether the display is built-in, whether it supports OpenGL acceleration, and so on. There’s another file called screenshot.tiff that contains a TIFF rendition of what’s on that display at that moment—an always-live screenshot, if you will. You copy this file and you get a screenshot. Copy it again, and you get a new screenshot. You can just open it in place too.

Along similar lines, there’s another folder /proc/system/hardware/camera/ and a file screenshot.tiff within it. When you open this file, procfs activates the camera momentarily, takes a picture, deactivates the camera, and makes the picture available as a TIFF file. You can copy the file and you get an image of what the camera’s seeing at that moment. Copy it again and you get another "live" image.

Besides these, the updated procfs has other (not-so-visual) interesting features.

More details, source code, and a precompiled binary available here:

Making procfs Cooler

We did a MacFUSE talk at Google last week:

Video on YouTube