ARWin - A Desktop Augmented Reality Window Manager


ARWin - A Desktop Augmented Reality Window Manager

Stephen DiVerdi, Daniel Nurmi, Tobias Höllerer

An example ARWin session.


The long term goal of the ARWin project is to create a usable 3D replacement for the traditional 2D window-icon-menu-pointer interface computers currently use. Short term, ARWin will serve as a useful testbed for 3D interfaces and interaction techniques.


Stage 1

As a window manager, ARWin's main responsibilities are processing user input into events, keeping track of running applications, and distributing events among those applications. User input comes from keyboard and mouse devices, along with a firewire camera used for vision based tracking. The camera input is processed by the ARToolkit, to register a global coordinate space above the user's desk, and to provide a tangible interface via markers to the environment. Applications can be set to float at some position in the global coordinate space, or they can be linked to a floating marker, so the user can manipulate the tile to act on the application.

ARWin provides an API for application development to take advantage of the 3D nature of the environment, as well as the interaction enhancements we've introduced. Custom 3D apps can be easily developed by subclassing an App3d superclass that includes callbacks for the different events. To provide legacy support, there's also a wrapper app called App2d that allows traditional X apps to be run inside ARWin, mapped onto a quad. To accomplish this, the X app is run in a virtual VNC server, and a custom VNC client connects, passing events through and drawing window updates to an OpenGL texture map.

Stage 2

We've developed some basic applications to showcase the environment, including a file browser that uses the ConeTree metaphor, apps that tag physical objects to allow virtual iteractions with them, and hybrid apps that involve a physical componented augmented by a virtual component.

The first 3D interaction technique we integrated was proximity based, as it's very intuitive in a 3D environment. When two apps are brought within a distance threshold, they're each notified of their neighbor, and allowed the opportunity to exchange metadata, so they may act on it. Metadata can be any information about the application - its name, the file its operating on, its status, etc. In our example, the business card app contains information about what's printed on the card, and the phone app processes it to initiate some action.

Stage 3

Later, we went on to provide a framework for inter-application interaction, similar to UNIX's pipe metaphor, but extended to GUI applications. We provide each application with an abritrary number of input and output data streams (like stdin and stdout), and allow the user to connect them at will.

There are two demos of this interaction. The first involves an MPEG encoded movie file being selected from the file browser. It is streamed into an MPEG decoder app, which has two outputs, raw audio and raw video. The raw video output is then streamed to a simple movie player app, which displays the video stream on the screen. The second demo allows for more interaction - this time an image file is selected in the file browser and streamed to a rotation image manipulator. The user has a slider to control the rotation of the image that gets displayed. Then the rotated image is streamed to a levels image manipulater, which provides sliders to adjust brightness and contrast. The user can tweak these sliders, and can go back to modify the rotation angle and see the changes propogate through the network.

And Beyond...

We have many avenues for future work we'd like to pursue with the ARWin environment

    Data Flow Visualization - our inter-application interaction visualization technique is cumbersome and unappealing. We'd like to find more useful visualizations that convey information about the stream and look nicer. Display Devices - HMD technology simply isn't where we need it to be for a daily-use system like ARWin. We'd like to explore other display technologies that are easier on the user, such as passive stereo on a wall screen. Input Devices - we'd like to forego the mouse and keyboard except when necessary for traditional apps, and so are considering integrating the hand tracking and gesture recognition work of Mathias Kolsch. Usability - the holy grail of AR research is a usable system, so we've taken measures to make ours as easy as possible, but a more formal approach including a user study would be appropriate.


S. DiVerdi, T. Höllerer and R. Schreyer.
Level of detail interfaces.
In ISMAR, Nov. 2004.

S. DiVerdi, D. Nurmi, T. Höllerer.
ARWin - A Desktop Augmented Reality Window Manager.
In ISMAR, pages 123-124, Oct. 2003.

S. DiVerdi, D. Nurmi, T. Höllerer.
ARWin - A Desktop Augmented Reality Window Manager.
UCSB-TR CSD-03-12, Department of Computer Science, University of California, Santa Barbara.

S. DiVerdi, D. Nurmi, T. Höllerer.
An Interface for Generic Inter-Application Interaction in a 3D AR Environment.
In ARToolkit Workshop, pages 123-124, Oct. 2003.

Related Projects

Original Project Page
Hand Gesture Input for Battuta