These days, patrons ordering cocktails at Artechouse, a digital art exhibition center-slash-bar in Washington DC, choose from a menu that floats in space. Lists of ingredients scroll beneath beverage offerings, visible only to guests peering into their smartphones while using Artechouse’s free app. Although the drinks are real, the menu is virtual—an augmented reality carte du jour that hovers between visitors’ mobile devices and the world around them.
Artechouse’s menu was created by Kiel Mutschelknaus, a 33-year-old graphic designer and faculty member at Maryland Institute College of Art. As a work of text-based augmented reality, it highlights the seductive opportunities and the unique challenges faced by designers placing typography in three-dimensional realms.
The concept harks back to the 1990s, when renowned designer Muriel Cooper and her colleagues at MIT Media Lab explored what they called Information Landscapes, interactive screen-based planes of digital typography that swooped and rotated in three dimensions and suggested innovative ways of presenting information. What’s different today are the technologies: faster computer processors, higher-resolution displays, new developer tools like Apple’s ARKit and Google’s ARCore, and, perhaps most notably, a mobile device in nearly everyone’s hands. One result is the accelerated evolution of augmented reality (AR), virtual reality (VR) and mixed reality (MR).
“It’s the Lumières’ train moment again,” says Ben Rubin, director of the Center for Data Arts at the New School in Manhattan. He’s referring to the moment in 1896 when Auguste and Louis Lumière projected footage of a train arriving at La Ciotat station, both terrifying and delighting audiences and launching the medium of film. “It took decades for [film] to be realized technically, with sound and eventually color,” Rubin, 54, says. “But it also took decades to figure out what it is. How does it work? How do we perceive it? How do we use it?”
For designers, developers and artists delving into the still-emergent fields of AR, VR and MR, the same questions are vital today.
WHAT ABOUT TYPOGRAPHY?
When designing the Artechouse menu, Mutschelknaus selected IBM Plex Sans and IBM Plex Mono, which he describes as “clean and stripped down.” He opted for white because, he says, “I knew the bar would be dark, and even if the bartender wore a bright shirt or a light got turned on, the legibility would still be there.”
Legibility is inherently thorny in AR, VR and MR, where backgrounds, lighting conditions, and user perspectives and interactions are unpredictable. In three dimensions, typeface styles, sizes, weights and placement behave differently from their equivalents in print- and web-based design. What happens when users move closer to or farther from the text? What if they walk behind it and it’s backwards? What if the background interferes with the type in front of it? What about parallax, depth and layered focus? What about text that moves?
“People are just beginning to dig into these problems,” says Dan Rhatigan, senior manager of Adobe Type. “We are in the very early days of exploring what’s possible.”
Rhatigan’s team is working with new dynamic typeface technologies, like variable fonts and OpenType-SVG, that offer potential advantages in virtual environments. Meanwhile, he says, existing sans serif typefaces with clear shapes and open interior spaces, such as Myriad, Helvetica or Frutiger, are solid choices in three-dimensional settings.
The foremost objective remains legibility. “Because if whatever you’ve set in the typeface degrades before someone can read it,” Rhatigan says, “you’ve lost the battle.”
That battle inspired Zach Lieberman and his partner, Molmol Kuo, to create the AR app Weird Type. “The question that intrigued us,” Lieberman, 41, says, “was: What does it mean to write messages and put them in the air?”
Lieberman, a Brooklyn-based designer who is a cofounder of the School for Poetic Computation, describes himself as “an artist who writes code to create artwork.” With Weird Type, users can input words that hang in the real world, then choreograph them by selecting from seven modes that respond to touch and motion. In Ribbon, for example, they can “paint” a swirling strip of words; in Stamp, they can duplicate phrases, one atop the next.
Lieberman says his app is a way to investigate AR’s grammar, the yet-to-be-established conventions of what works and what doesn’t in the medium. “What if you create so much virtual content that you mask the real world?” he says. “And what about legibility? The type is flat, but it’s moving in three dimensions, and there are angles where it becomes illegible.”
Sharleen Chen, a designer in Los Angeles, tackled type-based grammar in MR while a student at California Institute of the Arts. Her 2017 MFA thesis, All Mixed Up, reconsiders the precepts of poster design by bringing five design principles into three dimensions. In “Perspective,” for instance, the word perspective morphs into the word matters as the viewer shifts position. “Interactivity” examines how a user can interact with and alter a poster’s design. “Mimicry” shows how nature, specifically wind and water, can be mimicked in MR.
“Mixed reality is most interesting when the digital object blends seamlessly into the real world,” Chen, 31, says. “This means our digital design can be susceptible to real-life elements, such as lighting and weather. Can and should type be readable on a dark, rainy day? Does the same type look the same on a sunny day?”
New York City–based typeface designer CJ Dunn addressed these questions and others when he designed his newest typeface, Louvette, and created what might be the first AR type specimen app, Louvette AR. “I wanted to see how my fonts would look in AR and what kinds of issues would present themselves,” he says. “We as type designers should be thinking about where the type will be used and how people are going to be reading, especially in emerging technologies.”
Louvette is a high-contrast display typeface that has sharp serifs and a range of hairline sizes, “so you can use it small with optimal hairlines and big with optimal hairlines,” Dunn, 37, says. Although he did not design Louvette specifically for AR, the typeface, he says, “addresses a typographic problem related to ‘perceived size,’ which we experience in AR differently from in print or on the web.”
His app demonstrates how this would be handled in AR. “As you walk toward the letters in space, the hairlines get thinner,” Dunn says. “When you get very close, you see the thinnest hairlines for the largest sizes—or, I should say, the largest perceived sizes. As you walk away from the letters, the hairlines get thicker so they don’t disappear.”
In the app, pairs of English words—one roman, one italic—form a circle suspended in space, and, slowly, each word changes, a letter at a time. The type is in shades of orange—Dunn’s solution to a variable background. “Having a strong hue is one way to stand out when you don’t know what the background will be,” he says.
Another way to maintain legibility against a fluctuating background is to treat the typography as a graphic component separate from the virtual space. That’s what the New York Times has done in most of its augmented and virtual reality stories.
The Times entered the VR arena in 2015, when it released its NYT VR app and delivered more than a million Google Cardboards to print subscribers so they could watch its VR film “The Displaced.” In 2018, the paper introduced its first AR experience on the general New York Times app, featuring four athletes competing in the Pyeongchang Olympics. Dozens more AR and VR projects have been completed since.
“We’re still reporting the facts,” Lian Chang, the Times’ product design director, multimedia, says, “but now, instead of just a photograph, you’re getting scale, texture and depth.”
As for text, words are used primarily as informational aids. In “The Displaced,” subtitles are duplicated on opposite sides of the scene to increase the likelihood that viewers looking around in 360 degrees will find them. In the Olympics article, pop-up explanatory captions are triggered by the viewer’s position relative to the athlete. To ensure contrast, the type, usually white, may have a drop shadow or appear on a dark field.
Chang, 41, distinguishes type treated as information from what he calls “ergodic type,” where, he says, “the user experience is inextricable from the storytelling.” With ergodic type, the type itself is part of the message being communicated. Examples can be seen in visual poetry, like that of E. E. Cummings, and in augmented reality artwork by Tamiko Thiel, 61, a media artist in Munich who has worked in VR since 1994 and AR since 2010. Some of her works use letterforms as part of the composition. For two AR projects, Transformation: Lehel and El Barrio Is Home!, Thiel collected handwritten reflections from local residents about their neighborhoods, then converted the notes into gold lettering that was visible to viewers using a free app at each site. In both, the intimacy of individual handwriting—legible or not—invoked the power of community.
Works like Thiel’s speak to the expanding expressiveness of these mediums and continue the vein of typographic explorations that Cooper and her colleagues were conducting in the ’90s. “We were experimenters,” says David Small, who, as a doctoral student at MIT, worked closely with Cooper at the Media Lab. Today, he is a designer based in Martha’s Vineyard and the founder of Small Design Firm.
Small, 53, speaks of the “incunabula,” which he defines as the 50-year period between the printing of the Gutenberg Bible and the turn of the 16th century: “the first 50 years of movable type,” he says.
“Initially, people tried to make printed books look like illuminated manuscripts,” he says, “with elaborate initial capitals in different colors. Then, within 50 years, they realized, hey, the printing press can do all sorts of stuff. Let’s forget about the past and figure out what works.”
People haven’t yet forgotten the past when it comes to augmented, virtual and mixed realities. They may be reluctant to download a necessary app or permit access to their cameras. They may be discouraged by low resolution and technical glitches, or by the cost of expensive headsets. Furthermore, they have to adapt to a new way of grasping information and seeing the world.
Small believes we are in a present-day incunabula, where reading has already shifted from paper to screens. “Why,” he asks, “should we continue to ape printing conventions when we can move type around in three dimensions?” ca