In early 2020, Art Processors was brought on by the National Museum of Australia to create five digital interactive exhibits for its new Life in Australia gallery, opening in mid-2021. These interactives include a playful and educational window into the world of one of Australia's most celebrated creatures: the platypus.
As nocturnal and typically shy animals who spend most of their time underwater, platypuses aren't the kind of creature you can easily get to know. That's why we're working on a 3D experience that enables Museum visitors to not only view the platypus up close, but to actually interact with it. Using state-of-the-art motion tracking technology, the experience unfolds and adapts based on the movements of visitors standing in front of the aquarium-like digital display.
When designing a motion-controlled experience, the physical space is one of the most important factors to consider. To properly understand a space, we'll often build full-scale mockups or simulate it using projectors or screens. But when Melbourne went into lockdown at the onset of the COVID-19 pandemic, we had to turn our usual approach on its head. Suddenly we had a new problem to solve: how do we prototype a site-specific, motion-controlled experience while working from home?
Starting with what we know
A common first step when developing a motion-controlled experience is to create a simple 3D representation of the space. We do this for two reasons:
- It allows us to virtually place users to better understand how many people might use the experience at the one time.
- It allows us to virtually place the motion control cameras so we can understand how well they work within the space.
This is an example of a mock-up, where you can see two people interacting with four large screens and a still image of what the experience might look like. In the bottom right corner are views of the two motion control cameras used to control the experience.
Screens within screens
Once we've created a virtual mockup of the space, we would then usually create a physical mockup. In the example above, we had originally planned to create a curved wall in our Melbourne studio using basic materials and a large projector to fake the four screens. But with new restrictions in place, this approach was no longer going to work. We needed a way to achieve the same results from home.
Looking at the virtual mockup we'd made to test the space requirements for the experience, we asked ourselves: what if the screens in the mockup could show the actual experience instead of a just still image? This would enable us to see how the experience might look in the physical space, plus we could see how our content might fit with the curved wall.
In this video you can see this concept in action:
Let's quickly break down what we're looking at:
- This is the virtual mockup of the space with the live screens taking in video from an earlier prototype made to test water interaction
- The output from the water prototype
- A webcam view of the user's actions
- The motion-tracking software.
Bringing back depth
Overall, rendering the interactive prototype to the virtual screens worked well and allowed us to get a sense of how content might work across the four displays. But in shifting from a physical mockup to a virtual one, we also went from 3D to 2D: this lack of dimensionality made it harder for us to test which gestures might conflict with the physicality of the space.
To address this issue, we expanded the previous prototype to work with virtual reality (VR). In VR, the user can interact with a version of the mockup at the correct size and scale, helping us immensely with our spatial understanding of the experience.
In the next video you can see the VR version of this mockup. Notice that the layout is the same as the 2D version, but the background now shows what the user sees in VR.
Adapting to remote collaboration
Looking back, our next hurdle was figuring out the best way to collaborate remotely on more detailed design and development decisions around the platypus experience as we moved much deeper into production. At Art Processors, we like a challenge. Despite the realities of the 'new normal' we found ourselves contending with, this approach to prototyping proved to be a hugely effective way to collaborate remotely.
We’ll leave you with a photo of our former XR Developer Mitch McCaffrey conducting early testing on site at the National Museum of Australia: