Collaborative Mixed Reality

A basic premise of collaborative mixed reality systems is that they enable mixed presence (collocated and remote) collaboration by effectively blending physical and virtual spaces. In this project we consider what it means to 'effectively blend' real and virtual for collaboration. We take a multi-pronged approach, combining infrastructure development, physical-virtual co-design, iterative prototyping and controlled experimentation.

TwinSpace

TwinSpace is our infrastructure for collaborative mixed reality [2]. It provides a robust platform for rapid prototyping of applications that span physical and virtual spaces, allowing for comparative evaluation of designs. The system is applicable to a wide range of physical settings, virtual environments, and collaborative activities. TwinSpace provides the following four key features:

  1. A communications layer that seamlessly links the event-notification and transactional mechanisms in the virtual space with those in the physical space.
  2. A common model for both physical and virtual spaces, promoting the interoperability of physical and virtual devices, services and representations.
  3. A mapping capability, managing how physical and virtual spaces are connected and synchronized.
  4. Specialized virtual world clients that participate fully in the larger ecology of collaborative mixed reality devices and services.

TwinSpace has been used in several research projects to date; most are outlined below.



inSpace Lab

Video

The inSpace Lab is an experimental environment modelled after the project room, a room that provides a physical home for a project (its members, activities, tools, and products) over its lifespan [1]. The lab is designed to utilize the spatial layout and ‘collaboration affordances’ of the physical space in order to connect with a complementary virtual space [2].

The inSpace lab is composed of three main regions (see figure). Region A supports brainstorming and generating new content on a whiteboard, region B supports extended group discussions and sharing content on wall displays, and region C supports composing, modifying, and discussing content over an interactive tabletop display.

inSpace serves as our primary testing ground for mixed reality interaction techniques, physical-virtual co-design [1], policies for privacy, sharing mechanisms, and physical-virtual mappings.

Or de l'Acadie

Video

Or de l’Acadie is a prototype collaborative game built using TwinSpace [2,3]. Set in an abandoned 18th Century French colonial fort, two pairs (French and British sides) compete for valuables distributed throughout the town, with the goal to collect the most valuables within a time limit.

The game emphasizes asymmetry in both game controls and team dynamics. Each British player controls a single soldier using keyboard controls. Players are not collocated, and can communicate with each other only when their soldiers are within earshot. Each soldier is equipped with a static top-down map of the fort, and can collect a limited number of riches before needing to return to a location at the edge of the fort to stash them.

The French side consists of a Mi’kmaq mystic who is able to communicate with spirits, and a deceased French Gendarme. The French players are physically collocated, and use a range of physical controls to compete for the valuables (see figure).

Virtual Supermarket

Video

This project implemented a mixed reality solution to the design contest held at the 3DUI conference in 2010 [4]. Items were selected from virtual supermarket shelves by moving a physical trolley around a carpet that was mapped to the supermarket floor. The touchscreen on the trolley presented the first-person perspective of the shopper.

Each selected item was dynamically associated with a physical token. When it was time to 'check out', the physical tokens were carried to a physical table/display. The display presented the surface of a corresponding virtual table. When tokens were placed on the physical table, the associated items appear on the virtual table at teh same position and orientation.

Publications

  1. Reilly, D., Tang A., Wu, A., Echenique, A., Massey, J., Mathiasen, N., Mazalek, A., and Edwards, W.K. Organic UIs in Cross-Reality Spaces. Second International Workshop on Organic User Interfaces, TEI 2011.
  2. Wu, A., Reilly, D., Tang, A., and Mazalek, A. Tangible Navigation and Object Manipulation in Virtual Environments. Fifth International Conference on Tangible, Embedded and Embodied Interaction (TEI '11), Funchal, Portugal, January 2011.
  3. Reilly, D., Voida, S., McKeon, M., Le Dantec, C., Edwards, W. K., Mynatt, E. and Mazalek, A. Space Matters: Physical-Digital and Physical-Virtual Co-Design in the Inspace Project. IEEE Pervasive Computing, 9(3):54-63, 2010.
  4. Reilly, D., Rouzati, H., Wu, A., Brudvik, J., Hwang, J. Y., and Edwards, K. (2010) TwinSpace: an Infrastructure for Cross-Reality Team Spaces. ACM Symposium on User Interface Software and Technology (UIST '10), New York, N.Y., USA, October 2010.
  5. Reilly, D., Tang, A., Wu, A., Echenique, A., Chamoli, S., Massey, J. and Edwards, W.K. (2010). Or de l’Acadie: a TwinSpace demo. (demo) UIST ’10.
  6. Wu, A., Reilly, D., Hwang, J.Y., Echenique, A., Santos, E., and Edwards, W.K. (2010). A cross-reality shopping system. (contest demo and video) 3DUI '10.