As someone who designs for extended reality (XR), which includes augmented, virtual and mixed reality, what’s the benefit of coming from an architecture background? The ability to think and design spatially—in 3-D versus 2-D—as well as other skills, such as 3-D and visual communication. In practice, architects are required to be both detail oriented and strategic, and are also trained to be able to take large, complex problems and develop an understanding of how all the pieces fit together.
How did you learn the necessary skills to become a creative technologist? Through a lot of time, persistence and very kind people supporting me, like my partner. I also couldn’t imagine doing anything else, and was privileged enough to have the time to learn new skills. Digital fabrication and 3-D modeling are staples of architectural education, but I also started learning to code in grad school through Grasshopper, a visual programming language for Rhino, the main 3-D modeling software, and a common tool for generative design.
About four years ago, I tried on an Oculus Rift DK1 headset and realized that VR was the medium I really wanted to explore. Fast forward to 2017, when I started learning about the game engine Unity at a small boot camp at Noisebridge, a hackerspace in San Francisco that is still free and running. I’m still learning and hopefully always will be. Over time, I’ve been able to better identify what I need to know to accomplish something and target my learning accordingly—right now, I’m exploring the newish visual effects tools in Unity.
Through designing and prototyping for XR, what have you learned about how human emotion can affect our interactions with technology? Our emotional state can be significantly impacted through our interactions with technology, particularly if we are misclassified or mistargeted. I’ve been interested in affective computing, which is computing that is connected to human emotions, and the research behind codifying and representing human emotion though machine learning. In my AR app and installation artwork series Face2Face, I explore the capabilities and limitations of commercial facial analysis tools, like Amazon Rekognition. Many of these tools claim to deterministically classify human emotion, yet they can exhibit issues that arise from things like algorithmic bias and inconsistencies in environmental conditions like lighting. For example, in one exhibit, participants found that women of color were skewing towards being classified as frustrated or unhappy because of the heavy shadowing on their faces, an example of technology triggering a negative emotional response in turn.
What is the most important question you’re hoping to answer through PLACE, the interdisciplinary research group you cofounded? I’m not sure it’s just one question—it’s more of a general area of research. My cofounder Marisha Farnsworth and I believe that XR has the incredible potential to connect people, both to each other and to the environment, cities and spaces we inhabit. We’re also really interested in how XR can be used as a mechanism for knowledge transfer. Recently, at San Francisco Design Week, we exhibited a mixed reality piece called Phase Change. Users interacted with a one-to-one visualization of an iceberg melting at the average rate that polar ice is melting. This piece enabled users to engage with a real-world problem by presenting information in an immersive way.
What are some of the limitations that those who design for XR are trying to overcome now? AR and VR technologies are being adopted in a more mainstream way, which is exciting for the industry as a whole. Although, there are still some issues with form factor, price and resolution, these will be resolved over time. In my opinion, the Oculus Quest is the first headset on the market that starts to address these issues. And outside of user adoption, the other challenge is creating a thriving content ecosystem; in other words, the technology needs to offer users something. This is why we need more designers and creators on board to share their visions in the XR medium.
How do you see brands missing the mark when it comes to immersive experiences? Sometimes it feels more like a gimmick as opposed to an integral component of the narrative or experience they’re trying to create. I think it’s good practice to always ask: Why? How is this medium supporting the story you’re trying to tell?
What are the best immersive experiences you’ve enjoyed recently? Discrete Figures was an incredible show by Japanese arts and technology company Rhizomatiks and Tokyo-based dance troupe Elevenplay, held at the Gray Area Foundation for the Arts in San Francisco earlier this year. It combined dance and motion with real-time projection mapped AR. I also recently experienced an AR app called Museum of the Hidden City, which shares a historical narrative about gentrification and the redevelopment of the Fillmore District in San Francisco. Similar to the work being done by the immersive installation and theater group Nonchalance, within the experience, there are moments of situational awareness, which engage users more deeply with the city. I also loved virtual artist Chelley Sherman’s immersive VR piece Dispersion, which places the users in an incredible audiovisual landscape to interact with a fluid, dreamlike entity—it’s poetry.
How do your art projects keep you motivated? For me, each artwork or series is usually the product of a deep investigative process, and my way of coming to terms with the problems or incongruities in the world. They are almost always impact driven and subversive in some way, so I see them, to some degree, as a form of activism—a nice counterpart to my work as a designer in tech.
What advice do you have for designers who are interested in designing for XR? Do it! It’s an amazing field, with so much space to grow. I’m really excited to see how it develops over the next 5, 10, 20 years. Definitely reach out to people who are doing work that interests you. There are fantastic meet-up groups and communities in the XR space on Facebook. I would also leverage the incredible resources available online, like Unity tutorials and Devon Ko’s 3D for Designers educational resource.