The last days have been somewhat crazy as we saw a lot of interest in Virtual Projection (VP), a project by Steve Feiner, Sebastian Boring and myself that will be published at this year’s CHI in Austin. First Golem.de, then Engadget and finally The Verge all picked up the story and the corresponding video. I want to use this blog post to provide a little more information on the project.
In case you haven’t seen the video, take three minutes of your time and watch it:
When we started with Virtual Projection the initial idea was just to artificially replicate the workings of an optical projector. Just pointing at a wall and pressing a button to make an enlarged image appear is a very easy and powerful concept. To transfer information to a wall display is comparably difficult (fiddling with cables, struggling with software issues), so we wanted to make that just as quick and painless. But once we had reached that goal it became clear that while we’re simulating the whole thing anyway we can even improve on that metaphor. While Virtual Projection has the clear downside that it requires a suitable display and does not work on any regular surface, we can at least “fix” some of the downsides of its real-world model.
One of the first things was getting rid of distortion: When projectors are aimed towards a wall at an angle, keystone distortion can arise, warping the resulting image. Virtual projections can be freely adjusted when it comes to distortion and transformations of the resulting image: It’s possible to either (a) fully recreate a projector’s image, (b) remove at least the distortion, (c) ignore the orientation of the handheld (e.g., to have a photo always upright), (d) ignore both orientation and scaling (e.g., to have the photo in its original size), (e) ignore all transformations and just use the technique for selecting a display that’s used fullscreen (e.g., to show a video or a presentation).
In addition to this control over transformation and distortion, virtual projections can also be “decoupled” from the handheld. While an optical projection is always fixed to its light source, we can fix virtual projections to displays and also create more of them at the same time. The above image shows a typical VP workflow: We first (a) start an application (for that, we implemented something very similar to Apple’s Springboard for the iPhone) and (b) interact with it. Once we (c) point at a display a preview of the VP appears. By (d) long-pressing we can control the VP’s placement on the display and (e) fix it by lifting the finger. Both the handheld view and the VP are now synchronized, so (f) interacting on the handheld changes the VP (in this example: the currently highlighted section of the photo. Note that the VP shows the whole photo, while the handheld is only able to display a part of it). By pointing at the VP, we can (g) select a different part of the photo to be shown on the handheld by tapping. Once we’re done, we can (h) remove the VP by long-pressing again and dragging it off the display. In case you’re wondering where the transformations/distortions went: They are predefined for each application, so in this photo-viewer example we take everything except the distortion (so type (b) from above).
This also works with multiple VPs: By pointing at an inactive VP on the display and tapping, the respective view immediately becomes active and visible on the handheld (the VP before that still stays on the display and continues running in the background on the handheld). Another option is to shake the handheld to bring up the menu (see (a)) and switch to a background app or start a new one. It’s no coincidence that starting and working with apps in VP is similar to the regular application management on a smartphone: We wanted to show that the VP interaction technique can easily be included into existing smartphone operating systems. Of course, due to security restrictions we weren’t able to completely integrate it (shaking the device, for example, was our replacement for pressing the Home-button on the iPhone). However, our VP implementation works on a regular iPhone, not even jailbroken, and without any private libraries.
With this interaction and tracking framework we built several applications. While the video has much more, here are three interesting ones. (a) is the example from above, so a photo-viewer that shows whole photos and parts of them on the handheld. It’s possible to quickly switch the visible part by pointing and tapping. The handheld’s perspective frustum determines which part that is, but it’s always possible to see that from the preview on the display (grey-ish border). (b) shows an image filter that can be applied to photos (in this example greyscale). Image filters work like regular virtual projections, only they show nothing when on their own. But placed on another VP that displays a photo they filter it. It’s also possible to combine multiple of them by stacking them on top of each other. (c) finally demonstrates multiple maps next to each other. Map viewers work similar to photo viewers in that they show a larger section of the map than possible on the handheld. In this example, the handheld also works as a “magic lens”: It shows a satellite image for the current section, while the stationary display shows the road map. By moving the handheld in front of the display that image changes correspondingly in real-time.
To sum up: In Virtual Projection we did interesting things with simulated projections and tried to keep it as close to a real-world scenario as possible. Our prototype works with regular, unmodified iPhones, the corresponding server runs on a regular Windows-PC (for the video we used an i7 machine) and everything happens via Wifi (so no cables needed). Imagine having a VP server running on every display that you encounter in your daily life and being able to “borrow” the display space for a while (e.g., to look something up on a map). Give it a few more years (and a friendly industry consortium ;)) and this could become reality.