GrabFS Source Code

August 19th, 2008

Earlier this year, I released GrabFS, a MacFUSE file system that shows “live” screenshots of Mac OS X applications. If you wish to understand how GrabFS works, you can now browse its source.

Enjoy.

New Install/Update Capabilities in MacFUSE

July 25th, 2008

MacFUSE has a new install/update mechanism that greatly simplifies and improves things both for end users and developers who use MacFUSE in their software.

The relevant wiki page has all the details.

Note that instead of Tiger- and Leopard-specific downloads, now there’s a single downloadable disk image containing a single installable package. The package, which third parties can choose to include within their metapackages, knows how to install the latest version of MacFUSE for your platform.

Extending HFSDebug

July 23rd, 2008

Recently, I had a need to know if any files or folders had been modified or created on an HFS+ volume in the past N seconds. There are many ways you could generate this type of information on Mac OS X.

To begin with, you could try asking Spotlight.

Besides Spotlight, Mac OS X has a rich variety of mechanisms and APIs for learning about file system changes.

On Leopard, you could write a program that uses the FSEvents API to learn of directory-level changes that occur on a volume. The FSEvents API is part of CoreServices.framework. "Directory-level" means that this API is best suited for monitoring large directory trees—it will not tell you when a particular file changes.

To monitor specific files, you could use the kqueues mechanism. (See kqueue(2).) Being file-level, kqueues don’t scale like the FSEvents API as you will need to monitor each file system object separately. Therefore, it’s better suited for situations where you need to monitor only a few specific objects.

You could also directly use the low-level fsevents mechanism (/dev/fsevents) that underlies the FSEvents API and Spotlight—but only if your need is experimental in nature. The fslogger program is an example of directly using the fsevents mechanism. fslogger will tell you—in pretty much real time—when file system objects change. (Make sure to see caveats.)

Then there is the kauth mechanism that was introduced in Tiger, primarily to help creators of virus scanning software. Kauth allows for extremely fine-grained file system activity monitoring—you can see vnode-level operations. In fact, monitoring is sort of a side effect of using the kauth mechanism. You can actually allow and deny individual operations, as virus scanning software might need to. However, kauth is not easy to use. (Not that the other APIs mentioned necessarily are!) To use kauth, you need to write a kernel extension. You also need to be extremely careful in what you do so as not to deadlock the operating system.

There also exist tools like fs_usage and dtrace on Mac OS X. fs_usage uses the kernel’s kdebug facility to perform fine-grained tracing of kernel events. In particular, it allows you to trace file system activity. Beginning with Leopard, the DTrace facility, to which dtrace is a front-end, lets you trace all kinds of activity at both the kernel and user levels. You could do some very imaginative things with dtrace.

So, we see that there is no dearth of ways to monitor file system activity on Mac OS X. However, there are caveats associated with each way we looked at so far. Consider Spotlight. To use it, we would be assuming that Spotlight indexing was enabled on the volume in question. Spotlight also doesn’t look everywhere: your areas of interest on the file system might be outside of Spotlight’s default or configured search scope. Moreover, to use or the other APIs we talked about, you will need to have the volume mounted—usually a reasonable requirement, except it may not be an option if, for example, you are trying to recover valuable data from a volume that has been through an accident. Or you could be performing file system forensics. Or the volume could be damaged enough to not be in a mountable state—at least not without repair. In these situations, you can’t or wouldn’t want to mount the volume. That aside, in my case, I didn’t know until after the volume had been modified that I wanted to know what had changed. That is, I didn’t happen to be conveniently running any monitoring programs and such.

You can always old plain old Unix-style find and walk the entire file system, examining each file and folder. This still needs the volume to be mounted, but it is exhaustive. Of course, if you have a large volume, exhaustively examining each file and folder through a brute-force find or other programs could take "forever." (In my case, I had over 4 million files on the volume. I also had little patience.)

Fortunately, Mac OS X lets you exploit the fact that the HFS+ volume format uses a central catalog B-Tree for storing hierarchy: the searchfs() system call can be used to "quickly" search HFS+ volumes. (It is much, much quicker than a typical portable user-space file-tree-walk.) In my case, I could use searchfs() to search for files and folders with creation or modification dates that match my criteria. Well, almost. I actually did require the volume to be unmounted. I also felt more inclined to do something general purpose.

hfsdebug is a tool that can walk the catalog tree even on unmounted volumes. I decided to add filtering capability to hfsdebug. "Filtering" means that hfsdebug can walk the HFS+ catalog B-Tree, examining each file and folder, and produce output based on some matching criteria. The new version of hfsdebug contains two built-in filters: mtime and crtime. You can use these filters to look for files and folders that have been modified or created, respectively, in the past N seconds. The number of seconds is passed as an argument to these filters. For example, to look for file system objects modified within the past 60 seconds, you would run hfsdebug as follows:

$ sudo hfsdebug --filter=builtin:crtime --filter_args=60
1216795688 [Tue Jul 22 23:48:08 2008]: Macintosh HD:/private/var/log/asl.db
1216795688 [Tue Jul 22 23:48:08 2008]: Macintosh HD:/private/var/log/system.log
...

Better still, you can write your own filters that hfsdebug can use. A filter is implemented as a dynamic library that implements up to 3 functions: one of them mandatory (hfsdebug_filter_callback()) and two of them optional (hfsdebug_filter_init() and hfsdebug_filter_fini().) To use your own filter, you would run hfsdebug the same way as in the case of built-in filters:

$ sudo hfsdebug --filter=/path/to/myfilter.dylib --filter_args=string
...

If your filter implements the hfsdebug_filter_init() function, hfsdebug would call it with the filter argument string, if any, as the argument. You could parse the argument string in the init function and initialize your filter’s state, if necessary.

int
hfsdebug_filter_init(const char *filter_args);

If you return a non-zero value from the init function, hfsdebug will terminate. If your filter doesn’t have any arguments, you could choose not to implement the init function.

After you return 0 from the init function, hfsdebug will invoke your filter’s callback function once for each file and folder record in the HFS+ catalog.


typedef char*(*hfsdebug_filter_path_retriever_t)(void);

int
hfsdebug_filter_callback(
    void *info, hfsdebug_filter_path_retriever_t pathRetriever);

The info argument is a pointer to either an HFSPlusCatalogFile structure or an HFSPlusCatalogFolder structure. (See the xnu kernel source for details of these structures.) You can determine which structure it is based on the first int16_t within the structure: it’s either kHFSPlusFileRecord or kHFSPlusFolderRecord. Given these structures, your filter can examine various attributes of the file system object.

Note that hfsdebug does not pass you the path to the file system object in question. This is because path computation is expensive. Instead, hfsdebug passes you a pointer to a path retriever function. You can invoke this function to make hfsdebug compute the path on demand and return a C string pointer. This pointer is valid for the given file system object only until your callback returns. You should only call the path retriever function if you truly need the path—doing so for each file system object would be quite time consuming. Note that hfsdebug filters are not multithreaded.

Again, you must return 0 from the callback for hfsdebug to keep calling you as long as there are more file system objects. If you return a non-zero value, hfsdebug will terminate.

Finally, once hfsdebug is done with all file system objects, it will call your filter’s fini function if one is implemented.


void
hfsdebug_filter_fini(void);

The following is a complete example of an hfsdebug filter. It does the same things as the built-in mtime filter, that is, it looks for files and folders that were modified within the last N seconds.

/*
 * myfilter.c
 *
 * HFSDebug Filter for "mtime"
 *
 * Look for file system objects that have been modified
 * within the past N seconds.
 *
 * gcc -arch ppc -dynamiclib -I/path/to/xnu/bsd/ -Wall -o myfilter.dylib myfilter.c
 */

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <errno.h>
#include <time.h>

#include <hfs/hfs_format.h>

#define MAC_GMT_FACTOR 2082844800UL

typedef char*(*hfsdebug_filter_path_retriever_t)(void);

static uint32_t mtime_seconds = 0;

int
hfsdebug_filter_init(const char *filter_args)
{
    mtime_seconds = strtoul(filter_args, NULL, 10);
    if ((errno == ERANGE) || (errno == EINVAL)) {
        fprintf(stderr,
                "invalid argument (%s) to mtime filter\n", filter_args);
        return errno;
    }

    time_t now = time(NULL);

    mtime_seconds = now - mtime_seconds + MAC_GMT_FACTOR;

    return 0;
}

int
hfsdebug_filter_callback(void *info,
                         hfsdebug_filter_path_retriever_t pathRetriever)
{
    int16_t recordType = *(int16_t*)info;
    uint32_t modDate;

    if (recordType == kHFSPlusFileRecord) {
        HFSPlusCatalogFile *file = (HFSPlusCatalogFile*)info;
        modDate = file->contentModDate;
    } else if (recordType == kHFSPlusFolderRecord) {
        HFSPlusCatalogFolder *folder = (HFSPlusCatalogFolder*)info;
        modDate = folder->contentModDate;
    } else {
        /* ignore */
        return 0;
    }

    if (modDate > mtime_seconds) {
        modDate -= MAC_GMT_FACTOR;
        char *tmpTime = asctime(localtime((time_t*)&modDate));
        *(tmpTime + 24) = 0;
        fprintf(stdout,
                "%u [%s]: %s\n", modDate, tmpTime, pathRetriever());
    }

    return 0;
}

void
hfsdebug_filter_fini(void)
{
    return;
}


Download HFSDebug 3.20

/bin/aural: The Solution

July 14th, 2008

1200 Baud Archaeology

/bin/aural

June 30th, 2008

Here is a unique computer puzzle: the audio file (MP3 encoding) contains something that could well be music to many a hacker’s ears. What is it? Can you "prove" that it is what you say it is?

New Version of MacFUSE

April 28th, 2008

Version 1.5 of MacFUSE is out.

The CHANGELOG has details of what’s new.

HFSDebug Bugfix Release 3.10

February 26th, 2008

I discovered a bug in hfsdebug. It causes hfsdebug to crash while printing Access Control Entry (ACE) details for certain files or folders. For example, consider the standard ~/Library/Preferences/ folder on Leopard.

$ ls -lde ~/Library/Preferences
drwx------@ 167 singh  staff ... /Users/singh/Library/Preferences
 0: group:everyone deny delete

This folder has an ACE for the group everyone. In particular, the ACE applies to no specific user (or you could say it applies to the wildcard user). HFSDebug was not dealing with this situation well. See what happens.

$ sudo hfsdebug ~/Library/Preferences/
  <Catalog B-Tree node = 15028 (sector 0x49080)>
  path                 = Macintosh HD:/Users/singh/Library/Preferences
# Catalog Folder Record
...
        # ACL Entry
        ace_applicable     = ab cd ef ab cd ef ab cd ef ab cd ef 0 0 0 c
zsh: bus error  sudo ./hfsdebug ~/Library/Preferences

I’ve released a bugfix version of HFSDebug to take care of this. The correct behavior should be as follows.

$ sudo hfsdebug ~/Library/Preferences/
  <Catalog B-Tree node = 15028 (sector 0x49080)>
  path                 = Macintosh HD:/Users/singh/Library/Preferences
# Catalog Folder Record
...
        # ACL Entry
        ace_applicable     = ab cd ef ab cd ef ab cd ef ab cd ef 0 0 0 c
          user             = *
          group            = everyone
          gid              = 12
        ace_flags          = 00000000000000000000000000000010 (0x000002)
                             . KAUTH_ACE_DENY
        ace_rights         = 00000000000000000000000000010000 (0x000010)
                             . KAUTH_VNODE_DELETE

Download HFSDebug 3.10

"TPM DRM" In Mac OS X

January 31st, 2008

A Myth That Won’t Die

MacFUSE Now Friendlier with Objective-C

January 9th, 2008

Quoting my Google Mac Blog post in its entirety:

A new version of MacFUSE is now available. As always, you can download a ready-to-install prebuilt package, or browse the ready-to-build source. Besides bug fixes and other minor improvements, there is a major new developer feature in this release: an Objective-C framework is now part of the core MacFUSE distribution! MacFUSE.framework will make developing user-space file systems in Objective-C easier than ever before. We look forward to seeing lots of interesting new applications based on MacFUSE.

Ted Bonkenburg, one of the engineers behind MacFUSE.framework, will give a talk this Thursday, January 10, during the next Silicon Valley Cocoaheads meeting at the Apple campus in Cupertino. His talk will focus on using the MacFUSE Objective-C API, but much of it will carry over to using other programming languages with MacFUSE. We’ll also show some very cool file system demos. So, if you’re interested in MacFUSE and are in the area, be there! It will be a hands-on talk, so please bring your laptops if you want to follow along. (Xcode 2.5 or newer required.)

GrabFS: The Screenshot File System

January 2nd, 2008


A while ago, I wrote about procfs for Mac OS X, a MacFUSE-based file system. Subsequently, I added more cool features to my procfs implementation. Recently, I had reason to demonstrate procfs again and realized that I needed still more cool features. That need led to GrabFS.

In a pinch, GrabFS is a file system that shows you a live view of the window contents of currently running applications. In a GrabFS volume, folders represent running applications and image files represent instant screenshots (“grabs”) of the applications’ windows. You simply copy a file or just open it in place, and you have a screenshot. Open it again, and you have a new screenshot!

Go here to read more about GrabFS and to download it. GrabFS requires Mac OS X "Leopard" and MacFUSE.


All contents of this site, unless otherwise noted, are ©1994-2014 Amit Singh. All Rights Reserved.