“A playful, wide and deep experience enabling museum visitors to connect with art in simple, physical ways and interact amongst themselves.” —juror Eva Mautino

“Gestural navigation put to good use to enable museum visitors to interact with the pieces in ways that they can’t do with the object itself—I do not particularly like the amusement park activities in the museum context, but I do like the segments that enable users to better understand the museum collection.” —juror Fanny Krivoy

Overview: While the Cleveland Museum of Art enjoyed a watershed moment in 2012 when it launched Gallery One, enabling visitors to use touchscreens to better understand the artworks, it didn’t fulfill the museum’s larger goals of increasing viewership and encouraging visitors to explore the other galleries. So the museum called upon Potion Design to reinvent the entire experience, changing the hardware, software and even the architectural layout of the space. The New York–based interactive design firm “embedded” sixteen custom games into the artwork and integrated six projection walls with software that tracks visitors’ body movements, gazes and facial expressions. The result is a fully immersive mixed-reality gallery that invites visitors to do everything from mirroring the pose of a character in an artwork to altering the emotion of a portrait by making a face.

• The system currently employs 1,300 artworks in the museum’s collection, and it is connected to 3,500 artworks on display.
• Potion designed custom computer vision and machine learning for facial tracking and analysis of artworks.
• On average, visitors are playing with ArtLens for more than nine and a half minutes; some interact for more than 20 minutes.

Comments by Matthew McNerney:
What are the project’s core features? “Sixteen custom games that employ three distinct visitor inputs—body gesture, gaze tracking and facial recognition; the ability for multiple players to use the system simultaneously; a fully updateable content managment system with custom-designed web tools to enable the museum to customize the experience at will; and seamless Bluetooth integration with the museum’s app so all artworks that a visitor learns about during game play will be saved to her or his app.”

Did you meet with any out-of-the-ordinary obstacles during development? “Digital projectors require the darkest environments to look their best, but our sensors and cameras needed the visitors to be well lit for the best experience. This required careful coordination to ensure that light was controlled perfectly between the art, the projectors and the visitors. We also needed to thoroughly vet the art curation in relationship to our games to ensure that there were no cultural insensitivities or misinformation given. And from a software perspective, training our software to accurately recognize emotions in real time, rather than from a photo, and developing an intuitive interface for visitors to control the games with their body gestures all proved challenging.”

How did this project compare with others you’ve worked on in the past? “The sheer scope of this project was daunting—sixteen distinct experiences across three different interfaces, all tied into a fully updateable system. It engaged more than half of our employees and required more prototyping than ever before. Most institutions have the aspiration to leverage their entire digital database, but very few are willing to put in the work to make that a reality. The Cleveland Museum of Art, having taken on this challenge once before, knew what it was signing up for and rose to the occasion.”