Extending HFSDebug

Recently, I had a need to know if any files or folders had been modified or created on an HFS+ volume in the past N seconds. There are many ways you could generate this type of information on Mac OS X.

To begin with, you could try asking Spotlight.

Besides Spotlight, Mac OS X has a rich variety of mechanisms and APIs for learning about file system changes.

On Leopard, you could write a program that uses the FSEvents API to learn of directory-level changes that occur on a volume. The FSEvents API is part of CoreServices.framework. "Directory-level" means that this API is best suited for monitoring large directory trees—it will not tell you when a particular file changes.

To monitor specific files, you could use the kqueues mechanism. (See kqueue(2).) Being file-level, kqueues don’t scale like the FSEvents API as you will need to monitor each file system object separately. Therefore, it’s better suited for situations where you need to monitor only a few specific objects.

You could also directly use the low-level fsevents mechanism (/dev/fsevents) that underlies the FSEvents API and Spotlight—but only if your need is experimental in nature. The fslogger program is an example of directly using the fsevents mechanism. fslogger will tell you—in pretty much real time—when file system objects change. (Make sure to see caveats.)

Then there is the kauth mechanism that was introduced in Tiger, primarily to help creators of virus scanning software. Kauth allows for extremely fine-grained file system activity monitoring—you can see vnode-level operations. In fact, monitoring is sort of a side effect of using the kauth mechanism. You can actually allow and deny individual operations, as virus scanning software might need to. However, kauth is not easy to use. (Not that the other APIs mentioned necessarily are!) To use kauth, you need to write a kernel extension. You also need to be extremely careful in what you do so as not to deadlock the operating system.

There also exist tools like fs_usage and dtrace on Mac OS X. fs_usage uses the kernel’s kdebug facility to perform fine-grained tracing of kernel events. In particular, it allows you to trace file system activity. Beginning with Leopard, the DTrace facility, to which dtrace is a front-end, lets you trace all kinds of activity at both the kernel and user levels. You could do some very imaginative things with dtrace.

So, we see that there is no dearth of ways to monitor file system activity on Mac OS X. However, there are caveats associated with each way we looked at so far. Consider Spotlight. To use it, we would be assuming that Spotlight indexing was enabled on the volume in question. Spotlight also doesn’t look everywhere: your areas of interest on the file system might be outside of Spotlight’s default or configured search scope. Moreover, to use or the other APIs we talked about, you will need to have the volume mounted—usually a reasonable requirement, except it may not be an option if, for example, you are trying to recover valuable data from a volume that has been through an accident. Or you could be performing file system forensics. Or the volume could be damaged enough to not be in a mountable state—at least not without repair. In these situations, you can’t or wouldn’t want to mount the volume. That aside, in my case, I didn’t know until after the volume had been modified that I wanted to know what had changed. That is, I didn’t happen to be conveniently running any monitoring programs and such.

You can always old plain old Unix-style find and walk the entire file system, examining each file and folder. This still needs the volume to be mounted, but it is exhaustive. Of course, if you have a large volume, exhaustively examining each file and folder through a brute-force find or other programs could take "forever." (In my case, I had over 4 million files on the volume. I also had little patience.)

Fortunately, Mac OS X lets you exploit the fact that the HFS+ volume format uses a central catalog B-Tree for storing hierarchy: the searchfs() system call can be used to "quickly" search HFS+ volumes. (It is much, much quicker than a typical portable user-space file-tree-walk.) In my case, I could use searchfs() to search for files and folders with creation or modification dates that match my criteria. Well, almost. I actually did require the volume to be unmounted. I also felt more inclined to do something general purpose.

hfsdebug is a tool that can walk the catalog tree even on unmounted volumes. I decided to add filtering capability to hfsdebug. "Filtering" means that hfsdebug can walk the HFS+ catalog B-Tree, examining each file and folder, and produce output based on some matching criteria. The new version of hfsdebug contains two built-in filters: mtime and crtime. You can use these filters to look for files and folders that have been modified or created, respectively, in the past N seconds. The number of seconds is passed as an argument to these filters. For example, to look for file system objects modified within the past 60 seconds, you would run hfsdebug as follows:

$ sudo hfsdebug --filter=builtin:crtime --filter_args=60
1216795688 [Tue Jul 22 23:48:08 2008]: Macintosh HD:/private/var/log/asl.db
1216795688 [Tue Jul 22 23:48:08 2008]: Macintosh HD:/private/var/log/system.log
...

Better still, you can write your own filters that hfsdebug can use. A filter is implemented as a dynamic library that implements up to 3 functions: one of them mandatory (hfsdebug_filter_callback()) and two of them optional (hfsdebug_filter_init() and hfsdebug_filter_fini().) To use your own filter, you would run hfsdebug the same way as in the case of built-in filters:

$ sudo hfsdebug --filter=/path/to/myfilter.dylib --filter_args=string
...

If your filter implements the hfsdebug_filter_init() function, hfsdebug would call it with the filter argument string, if any, as the argument. You could parse the argument string in the init function and initialize your filter’s state, if necessary.

int
hfsdebug_filter_init(const char *filter_args);

If you return a non-zero value from the init function, hfsdebug will terminate. If your filter doesn’t have any arguments, you could choose not to implement the init function.

After you return 0 from the init function, hfsdebug will invoke your filter’s callback function once for each file and folder record in the HFS+ catalog.


typedef char*(*hfsdebug_filter_path_retriever_t)(void);

int
hfsdebug_filter_callback(
    void *info, hfsdebug_filter_path_retriever_t pathRetriever);

The info argument is a pointer to either an HFSPlusCatalogFile structure or an HFSPlusCatalogFolder structure. (See the xnu kernel source for details of these structures.) You can determine which structure it is based on the first int16_t within the structure: it’s either kHFSPlusFileRecord or kHFSPlusFolderRecord. Given these structures, your filter can examine various attributes of the file system object.

Note that hfsdebug does not pass you the path to the file system object in question. This is because path computation is expensive. Instead, hfsdebug passes you a pointer to a path retriever function. You can invoke this function to make hfsdebug compute the path on demand and return a C string pointer. This pointer is valid for the given file system object only until your callback returns. You should only call the path retriever function if you truly need the path—doing so for each file system object would be quite time consuming. Note that hfsdebug filters are not multithreaded.

Again, you must return 0 from the callback for hfsdebug to keep calling you as long as there are more file system objects. If you return a non-zero value, hfsdebug will terminate.

Finally, once hfsdebug is done with all file system objects, it will call your filter’s fini function if one is implemented.


void
hfsdebug_filter_fini(void);

The following is a complete example of an hfsdebug filter. It does the same things as the built-in mtime filter, that is, it looks for files and folders that were modified within the last N seconds.

/*
 * myfilter.c
 *
 * HFSDebug Filter for "mtime"
 *
 * Look for file system objects that have been modified
 * within the past N seconds.
 *
 * gcc -arch ppc -dynamiclib -I/path/to/xnu/bsd/ -Wall -o myfilter.dylib myfilter.c
 */

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <errno.h>
#include <time.h>

#include <hfs/hfs_format.h>

#define MAC_GMT_FACTOR 2082844800UL

typedef char*(*hfsdebug_filter_path_retriever_t)(void);

static uint32_t mtime_seconds = 0;

int
hfsdebug_filter_init(const char *filter_args)
{
    mtime_seconds = strtoul(filter_args, NULL, 10);
    if ((errno == ERANGE) || (errno == EINVAL)) {
        fprintf(stderr,
                "invalid argument (%s) to mtime filter\n", filter_args);
        return errno;
    }

    time_t now = time(NULL);

    mtime_seconds = now - mtime_seconds + MAC_GMT_FACTOR;

    return 0;
}

int
hfsdebug_filter_callback(void *info,
                         hfsdebug_filter_path_retriever_t pathRetriever)
{
    int16_t recordType = *(int16_t*)info;
    uint32_t modDate;

    if (recordType == kHFSPlusFileRecord) {
        HFSPlusCatalogFile *file = (HFSPlusCatalogFile*)info;
        modDate = file->contentModDate;
    } else if (recordType == kHFSPlusFolderRecord) {
        HFSPlusCatalogFolder *folder = (HFSPlusCatalogFolder*)info;
        modDate = folder->contentModDate;
    } else {
        /* ignore */
        return 0;
    }

    if (modDate > mtime_seconds) {
        modDate -= MAC_GMT_FACTOR;
        char *tmpTime = asctime(localtime((time_t*)&modDate));
        *(tmpTime + 24) = 0;
        fprintf(stdout,
                "%u [%s]: %s\n", modDate, tmpTime, pathRetriever());
    }

    return 0;
}

void
hfsdebug_filter_fini(void)
{
    return;
}


Download HFSDebug 3.20

Comments are closed.


All contents of this site, unless otherwise noted, are ©1994-2014 Amit Singh. All Rights Reserved.