The Book

The Book — Extras

Software

Meta

About

elsewhere

The Apple Motion Sensor As A Human Interface Device

© Amit Singh. All Rights Reserved. Written in March 2005

Background

I first described the working of Apple's Sudden Motion Sensor (SMS, also known as AMS — in this chapter, we will use "SMS" and "AMS" as interchangeable terms and prefixes), initially a feature of the PowerBook line of computers, in "The Sudden Motion Sensor". My original reason for investigating the sensor was to fabricate some interesting programming examples for my book on Mac OS X internals. The discussion of the SMS generated a substantial amount of interest — perhaps more than I had expected. It is quite interesting to observe how one's fantasy is tickled by abnormal ways of doing the normal.

Early Sudden Motion Sensor Tools

I initially provided the following Mac OS X applications that use the orientation data retrieved from the SMS:

• AMSVisualizer — a graphical application that displays a 3-D picture of a PowerBook. The picture's orientation is a real-time approximation of the PowerBook's physical orientation. Thus, the on-screen picture moves with the movement of the SMS-equipped PowerBook.
• StableWindow — a graphical application that creates a window displaying a bicycle wheel. The window is "stable" in the sense that if you rotate the SMS-equipped PowerBook left or right, the window compensates by rotating itself by an equal amount in the opposite direction.
• AMSTracker — a command-line program that retrieves raw orientation data from the SMS hardware.

AMSTracker proved to be the most interesting application. Whereas the others are software "toys", AMSTracker can be used as a fundamental building block to create arbitrary applications that do something interesting based on the computer's orientation.

Still More Interesting Uses

I had also mentioned other ways in which the SMS could be employed, such as for scrolling in applications and for input in video games. The latter was constantly brought up in the feedback I received from "the Internet". Apparently many owners of SMS-equipped PowerBooks wish to experience physical motion-based gaming.

The games most suitable for being played using the SMS would be those that wholly or primarily require directional input. The x and y components of the SMS orientation data can be mapped to the motion of a mouse in a 2D plane, or to specific keys (say, the arrow keys) on a keyboard. Thus, it is conceptually straightforward, both for the programmer and the end-user, to use the SMS as a human interface device (HID).

Using the SMS as a Human Interface Device

While one could modify existing applications, including games, to accept SMS orientation data as input, it would be unnecessary and sub-optimal to do so in most cases. A surprisingly large number of people seemed to think that applications and games must be specially written to be "tilt-sensitive". While such an approach may be justifiable in certain special cases (such as when you need to process the orientation data for a purpose other than human input), doing so would typically be an impulsive reinvention of the proverbial wheel. If the application of interest already receives mouse or keyboard input, it is the SMS that should masquerade as a mouse or a keyboard. After all, one does not create games from scratch for specific models of keyboards, joysticks, or mice. In most cases, the transparent approach would allow the SMS to be used with target applications without any modifications to the latter. This is especially desirable when you do not have access to the source of a target application.

An SMS-based HID could be implemented in two broad ways, as described below.

The Ideal Solution

A particularly seamless solution would be to create a HID driver for a pseudo-device whose input "signals" are the motion sensor's orientation data. Such a software mouse or joystick would then be available to applications simply as yet another input device.

While an SMS-based HID driver is feasible, it is somewhat easier to adopt a relatively "Quicker and Dirtier" approach.

The Pragmatic Solution

Given my utter lack of free time, I had promised myself (and some other people) that I would not experiment with motion sensors beyond what I had already done. There are several justifiable reasons for such a vow:

• It is admittedly frivolous to attempt to connect everything in the Universe to a motion sensor. Sensibility ought to curb motion sensing after a limit.
• I do not have an SMS-equipped PowerBook, which makes it harder to validate ideas. Earlier, I had been simulating the SMS using the mouse pointer's movement. If I were to use a mouse-based or keyboard-based "fake" SMS to masquerade as a fake mouse or a keyboard, it begins to get somewhat messy and cumbersome.
• I have to finish writing a book with an impossible schedule, which leaves no time for arbitrary endeavors.

Now, I had played a game (on real hardware) that was modified (hardwired, for experimentation) to use the motion sensor's data, and it was a very entertaining and unique experience. Moreover, I had received numerous requests for an SMS-based game. In light of all this, I decided to create a user-space "mapper" of SMS data to a HID. While not as clean a solution as a HID driver, it is more flexible and easier to use in many cases. These properties make it particularly suited for experimentation by the end-user.

Let us now look at the program itself, which I call AMS2HID.

AMS2HID

AMS2HID is a prototype application that translates raw SMS orientation data into input data from a keyboard or a mouse. It presents the translated data to other applications through synthesized keyboard and mouse events. Thus, you can use an SMS-equipped PowerBook as an orientation-driven mouse or keyboard. Perhaps the most interesting and entertaining demonstration of AMS2HID's capabilities is its use for playing a variety of existing games.

AMS2HID performs the SMS-to-HID channeling for a single application at a time. Whenever its target application is active (that is, it is the "front" application), AMS2HID activates its event synthesis. Conversely, when the active application is not AMS2HID's target application, AMS2HID is dormant.

In a pinch, the use of AMS2HID could be described as the following (refer to Figure 1): you select a HID personality (mouse or keyboard), and a target application from a list of currently running applications. The target application will thereafter receive HID events based on the PowerBook's orientation and movement.

The AMS2HID User Interface

Let us look at the tunable parameters available in AMS2HID for mouse-event synthesis, with reference to Figure 2. Corresponding parameters and their semantics for keyboard-event synthesis are identical except when otherwise noted.

1. Select a HID personality (such as "Keyboard" or "Mouse") using this pop-up button. SMS data will be translated to events corresponding to the selected device.
2. Select a target application. HID events will only be synthesized for this application, and only when it is the "front" application.
3. Select whether to run in "Normal" or "Test" mode. Although unlikely in practice, it is possible under certain circumstances for the user-interface to become unresponsive to events from the physical keyboard or mouse. This may happen if the combination of AMS2HID settings, the PowerBook's orientation, and available resources is such that more events than what can be handled are being generated. In such a case, you may not be able to interact with AMS2HID, its target application, or perhaps even the rest of the user interface. If you have a sane "cold zone" defined (see item 11), keeping the PowerBook level and steady will quiesce AMS2HID. Alternatively, logging in from another computer, say via SSH, should allow you to terminate AMS2HID from the command line. Finally, Test mode allows you to use AMS2HID with a 60-second auto-reset timer. Regardless of what state AMS2HID and its target application, if any, are in, AMS2HID's state will reset to no HID personality being selected. It is recommended that you test AMS2HID in each new scenario before running it normally.
4. When AMS2HID is running in Test mode, a countdown timer appears in this area.
5. This area shows the raw orientation data retrieved from the SMS.
6. Select this button to cause the mouse pointer to visually (in addition to logically) move on the screen in response to SMS-caused events. De-selecting the button causes the pointer to remain visually stationary. The corresponding setting for the keyboard personality is "repeat keys", which simulates a rapid sequence of key presses and depresses, as opposed to a key being held continuously.
7. This specifies the frequency at which the SMS is sampled, and thus represents the maximum number of events that can be generated in a given time interval. 60 Hz is an appropriate frequency for a mouse.
8. This setting is used to reverse the logical direction of the x-axis. It can be used when AMS2HID's sense of direction along the x-axis is the opposite of that of the SMS.
9. This slider, initially at zero, can be moved left or right to correct the "zero-error" of the SMS, if it has any. Zero error can be determined by keeping the PowerBook on an approximately perfectly level surface, and noting the raw orientation values. If the x and y values are not zero, then correction values with the same magnitudes but opposite signs can be specified. However, even if you do have a minor zero-error, you should try AMS2HID with its default settings before correcting the error.
10. The sensitivity value represents the minimum "delta" for which SMS2HID would generate an event. The maximum sensitivity value corresponds to the minimum delta.
11. A "cold zone" along an axis is a line segment that defines a range of SMS values for which AMS2HID will not generate any events. It corresponds to the center of the display screen in case of the mouse, and a no-keypress zone in case of the keyboard. For example, an x-axis cold zone specification of 10 means that unless the SMS x value is more than 10, or less than -10, no events will be generated for that x value. Thus, the combined x and y cold zone values define a rectangle. This feature is useful since it is normal for the SMS values to fluctuate by minor amounts even when the PowerBook is perceptibly still.
12. Settings 8 through 11 are y-axis settings corresponding to those discussed above for the x-axis.
13. Since the ranges of possible SMS x and y values are different from typical screen dimensions, transforms must be applied to scale SMS values to arrive at the locations of the synthesized events. If you have multiple displays, only the resolution of the primary display (the one with the menu bar) is used in these calculations. This area shows the resolution in use.
14. This area shows the currently active application.

Experimenting With AMS2HID

Let us briefly discuss two uses of AMS2HID: first, to play video games, and second, to scroll in an application.

Playing Games

AMS2HID allows you to play games using the PowerBook itself as a controller — the computer's motion in physical space provides input to games. As noted earlier, any game whose primary controls are directional should work well with SMS2HID. This includes many car-racing or driving games, where "left"/"right", "up", and "down" can be used for steering, acceleration and braking, respectively. Flight simulators and maze-based puzzle games are other likely candidates.

A game extremely well-suited to AMS2HID is Neverball, in which you "tilt the floor to roll the ball through an obstacle course before time runs out". In this case, the physical tilting of the computer closely matches the virtual tilting required in the game to result in a very immersive and enjoyable gaming experience.

The following is a rough sequence of steps needed to play Neverball through AMS2HID:

• Ensure that the PowerBook is stationary and level, ideally on a flat surface. It is desirable that the PowerBook is stationary and level whenever you need to generate events from a physical mouse or a keyboard while AMS2HID's target application is the "front" application. Once you get used to AMS2HID, you may find that this is not a requirement, and you can normally use the physical HIDs in most cases.
• Launch Neverball.
• Launch AMS2HID. Note that Neverball automatically suspends gameplay when it is not the current application.
• Select the keyboard personality using the "Select HID" pop-up button.

• Select Neverball as the target application using the "Select An Application" pop-up-button.
• Optionally, adjust any keyboard settings. If this is the first time that you are attempting to use AMS2HID with Neverball, select "Test Mode" using the pop-up button that says "Normal Mode" by default. At this point, switching to Neverball will activate AMS2HID's event generation core.
• Switch to Neverball, and select any game options if necessary (as you normally would). Finally, start a new game.
• Pick the PowerBook up and start playing, using the PowerBook itself as the game controller.

Scrolling In Safari

You may use the following steps to scroll horizontally or vertically in an application, say Safari, by tilting the PowerBook appropriately:

• Ensure that the PowerBook is stationary and level, ideally on a flat surface.
• Launch Safari and have it display a page that can be scrolled.
• Launch AMS2HID, and select the "Keyboard" personality using the "Select HID" pop-up button.
• Select Safari as the target application using the "Select An Application" pop-up button.
• Depending on the content being displayed, you may need to enable the "repeat keys" option to be able to scroll the content.
• Switch to Safari. Pick up the PowerBook and start scrolling.

Tips and Miscellaneous Notes

The out-of-the-box experience while using AMS2HID will differ based on various factors such as the calibration and sensitivity of the specific SMS unit, the nature and behavior of the target application, and individual user skills and preferences. This section contains information that may be helpful while experimenting with AMS2HID.

• For a game that accepts directional from either the mouse of the keyboard, you may find that one or the other personality provides a better experience. In particular, some games may not work well with the mouse personality. This has to do with the mouse pointer warping behavior of the game. In cases where warping the physical mouse would "work", warping the SMS-based mouse may not, because unlike a physical mouse, the SMS-based mouse is dependent on the absolute physical orientation of the PowerBook. To put it differently, logically warping the SMS-based mouse would desynchronize it with the PowerBook's physical orientation.
• You can experiment by making the keyboard sampling frequency higher. While the default of 60 Hz should be appropriate in most cases, a much higher frequency (for example, 300 Hz) may work better for certain games. The maximum possible frequency is 1 kHz (999 Hz to be precise).
• Observe the "raw" readings when your PowerBook is still and level (preferably placed on a flat surface). It is normal for the readings to fluctuate in tiny amounts. Ensure that the x and y cold zones are somewhat larger than the fluctuations along the corresponding axes. In most cases, the default cold zone values should suffice. An appropriately sized cold zone rectangle will cause AMS2HID to stop generating events when the PowerBook is placed in a level position. This is particularly useful when you wish to use the physical keyboard or mouse, say, to switch to an application other than the AMS2HID target application, to perform mouse clicks, and so on.
• Try decreasing the sensitivity values, perhaps in decrements of 1, if you find that mouse is "jumping" around when you expect it to be stationary.
• In general, if you must change any AMS2HID settings from their default values, do so in small amounts, rather than changing one or more settings drastically.
• Before trying any combination of AMS2HID settings (including default settings) with an application, run AMS2HID in its Test mode so that you can recover from potential problems that may prevent you from using the physical keyboard or mouse. Alternatively, enable remote login on the PowerBook and have another machine available from which you can log on to the PowerBook in case of a problem.

Some Limitations

• AMS2HID does not currently synthesize mouse clicks. An option could be to have a certain amount of change in the z value trigger a mouse click.
• When the keyboard personality is in effect, the tilting of the PowerBook along the x and y axes is translated only to the arrow keys. If a game insists upon using some other keys for directional control, the current version of AMS2HID will not work. However, it would be trivial to make this translation configurable.
• The current mechanism used by AMS2HID to detect application switching is not foolproof.

Download

Download AMS2HID.

Acknowledgments

I would like to thank Ted Bonkenburg for his invaluable feedback on AMS2HID. Ted, a longtime friend, is one of the finest minds I know in Computer Science. I have never had a fruitless discussion on a technology-related matter with him, and this time was no different.

All contents of this site, unless otherwise noted, are ©1994-2008 Amit Singh. All Rights Reserved. Terms of Use