With Augmented Reality, You Can Now Superimpose Publicly Exhibited Artworks in Your Home | Innovation | Smithsonian Magazine

For a week in October, LED trucks featuring animated, illuminated quotes from conceptual artist Jenny Holzer’s latest public art project, You Be My Ally, drove around downtown Chicago and the city’s South Side. Those walking on the University of Chicago’s campus can currently use their phones to project all 29 quotes from the project, selected from texts from the University’s Core Curriculum, onto seven university buildings using a free web-based augmented reality app.

But for the first time, using the same app, art enthusiasts can also superimpose a public exhibit of Holzer’s work—each of the quotes—in their own houses, or wherever they might be. Instead of seeing the words Suddenly incoherence feels violent, a quote from “Citizen,” Claudia Rankine’s book-length poem, rolling up the sleek Ludwig Mies van der Rohe-designed façade of the university’s School of Social Service Administration, users can experience this zooming towards them from a distant spot in their kitchen or bathroom.

Museums and galleries have been exhibiting site-specific augmented reality works by artists for a few years, but the University of Chicago is one of a handful of global institutions taking AR one step further during the pandemic—to bring publicly exhibited artworks directly into people’s homes.

Holzer has created text-based public art installations all over the world, which have included light projections onto buildings, advertising billboards and LED signs. The initial plan for You Be My Ally had included one of Holzer’s signature light projections on campus, the LED trucks, and either a site-specific augmented reality component, or one that could be accessed anywhere.

“Jenny’s studio had first done an AR app as part of her Blenheim Palace project in the UK [in 2017], and she wanted to come back to that,” says Christine Mehring, an art history professor and adjunct curator at the University of Chicago’s Smart Museum of Art and one of the project’s coordinators.

Then Covid-19 hit, and the light projection, which would have encouraged large groups to gather to see it, was off the table. Instead, both site-specific AR works and an AR experience that people could enjoy anywhere would go ahead. “When it emerged that so many students and faculty would not be on campus, we really pushed the idea of having both in parallel,” says Mehring.

Four thousand miles away, London’s Serpentine Galleries found themselves grappling with a similar problem. They had just started exhibiting The Eternal Wave, a virtual reality experience by multimedia artist Cao Fei. The work, produced with Acute Art, a VR and AR production company that works exclusively with artists, begins in the gallery in a physical rendition of the kitchen in the Hongxia Theatre, originally a cinema for Beijing’s factory workers, which closed in 2008 and later became Cao Fei’s studio. From there, the visitor travels through various portals, moving through time and space to explore the computers of China’s early electronics industry and the area in and around the Hongxia Theatre.

“I think it was probably the most advanced artistic piece of VR ever produced,” says Daniel Birnbaum, curator of Acute Art, “but a few days after the opening, the gallery had to close because of Covid.” When the galleries did reopen on August 4, The Eternal Wave, which required the use of shared headsets, did not.

Working with the Serpentine Galleries, Acute Art came up with Plan B, which was to turn The Eternal Wave into an augmented reality experience. That included a site-specific AR version of The Eternal Wave—beginning in the kitchen space installed in the gallery and taking users on a multi-sensory journey—and a small sample of that work that users could access anywhere using the Acute Art app on their phones. The sample, a nine-minute video, takes users on an animated exploration of the theatre and beyond.

“We wanted a site-specific piece that people could access on their own devices,” says Kay Watson, the Serpentine’s digital curator. “But because exhibition capacity is now vastly reduced, it was also really important to offer another AR component that people could use at home.”

AR fine art that people can superimpose anywhere may be very new, but it has many interesting applications beyond the pandemic. Artists and art institutions can take artworks to a new and potentially global audience of art lovers, who may be unable to visit major galleries or museums.

Artists producing site-specific AR or VR works can add different but complementary AR experiences for users anywhere, without detracting from the original work. And, unlike site-specific AR art, users can become co-curators of the art by superimposing it against any backdrop. “It becomes this interactive game, almost like a participatory art form,” says Birnbaum.

That was crucial for You Be My Ally, which was conceived to bring elements of the Core Curriculum, selected from quotes suggested by University of Chicago students, to the biggest audience possible, and to invite users to find new meanings in its texts. “By projecting Reality is not is guaranteed [from The Human Condition by Hannah Arendt] on the Logan Center for the Arts on campus, maybe you think about illusionism and art as an alternate reality,” says Mehring. “When you see it in a public context, or even a political context, right now, in the context of fake news, the quote begins to mean something totally different.”

Enabling users to embed art in their own homes also forces them to think differently about it. “There’s something beautiful, evocative, but also unsettling about seeing something in your phone’s camera, in your own home, your most familiar environment, and have it not be what is actually in front of you,” Mehring says.

Acute Art produced some of its very first immersive artworks in 2017 and 2018 using virtual reality, a completely virtual experience where users wear headsets. At that time, Acute Art collaborated with Anish Kapoor, Marina Abramović, Olafur Eliasson and others. Art using augmented reality, where virtual elements are embedded in the real world that users see through their phone cameras, has come a long way since then. “We did AR experiments then, but I always thought these were limited in their reach, because these are things you see on your phone,” says Birnbaum.

When Acute Art produced density in 2019, a site-specific AR work by artist Koo Jeong A that took the form of a large, translucent, levitating ice cube and toured several different art institutions, it was praised for its low environmental impact. “We could show this quite beautiful AR object hovering in the air all over the world without shipping anything,” says Birnbaum. “It gave us a glimpse of future exhibitions and curatorial models that we have not fully explored yet.”

Birnbaum concluded that AR art might become an important tool to tackle climate change one day, but it was actually Covid-19 that created an immediate need for it. “Things that I thought might become more urgent in five or 10 years were urgent now,” he says. “No one could travel and everyone was at home. VR was hard with art museums and galleries closed, so AR has been our sole focus for the last half year. It’s evolved so quickly.”

Since March, Acute Art has produced augmented reality works for nine artists, including KAWS, Olafur Eliasson and Koo Jeong A, that, can be placed on any surface wherever the user is, either individually or together, using the Acute Art app. Those who missed the site-specific version of density can now conjure it up at home, hovering over the couch or floating over their beds.

While both site-specific AR art and AR art that can be accessed anywhere allow artists to bypass museums and galleries altogether, Birnbaum believes that arts institutions will still play a role, by incorporating more AR elements to reach audiences beyond their doors, particularly while physical travel is limited and exhibitions that rely on huge foot traffic are unrealistic.

The Serpentine was doing just that even before the pandemic hit. In February 2019, it launched a global open call for artist submissions to the Serpentine Augmented Architecture Program, a new collaboration with Google Arts and Culture and Sir David Adjaye. By July 2019, the gallery unveiled the first commission, The Deep Listener by Danish artist Jacob Kudsk Steensen.

Users who launch The Deep Listener app on site are taken on a guided tour of five species living in London’s Kensington Gardens and Hyde Park, the area surrounding the Serpentine Galleries. At each location, users can overlay an augmented reality experience and reactive audio about each species, which changes speed as you walk around. Users who launch the app anywhere else in the world access a slightly different AR experience.

The Serpentine has produced several other AR exhibits in 2019 and 2020, including The London Masaba (AR) Hyde Park launched in this past July, an exact virtual replica of Christo and Jeanne-Claude’s stacked barrel sculpture that floated on Serpentine Lake for three months in 2018.

Another collaboration with Acute Art, users of Acute Art’s app can see the site-specific sculpture floating on the lake outside the Serpentine Galleries, or place another version into their own environment. The site-specific work will become a permanent exhibit, reviving what was a temporary sculpture and providing a lasting tribute to Christo, who died this past May.

“AR was already enabling us to engage with audiences in new ways, but that’s particularly important now,” says Watson. “We’re really thinking about who our audiences are—online, offline, onsite, offsite—as we develop programs for the next few years, because things are going to be different.”

Watson rejects the idea that providing programming for offsite visitors might dissuade them from returning to institutions in person once the pandemic is over. “Visiting art institutions will always be important,” she says. “I’m longing for it, personally. I certainly don’t think of augmented reality as a replacement to that. It’s about expanding and extending participation.”

For the first time this year, the Serpentine Galleries produced a strategic briefing to share what they’ve learned about AR and other technologies with other cultural institutions. But Watson thinks it’s already easier for artists and institutions to experiment with AR today because the technology is more accessible. She points out that all of the Serpentine’s AR projects, including The Deep Listener, which they produced in house, were made using free and widely available game engines, the software used to render video game environments.

Free web-based tools allow artists to make their own AR filters for social media channels, like Instagram. In response to the pandemic, all the 70-plus artworks exhibited across New York City this October during Digital Art Month were AR works and all of them were rendered with Instagram filters. When people came across an artist’s name, bio and QR code in the street, they could scan the code to automatically activate the artist’s AR filters on Instagram, which could be superimposed on their current location, or used at home later.

“With this type of AR, the barriers to entry are lower. Artists who have never worked with Insta filters before have created their first AR works specificially for New York’s Digital Art Month,” says Elena Zavelev, founder and CEO of the Contemporary and Digital Art Fair, the art fair which co-sponsored the month-long festival. Digital Art Month: Miami is now in the works for December.

Zavelev thinks that AR will be an increasingly popular medium for artists, particularly compared to VR, which is expensive, difficult to produce and requires headsets. “AR is an elegant way to bring this fantasy layer into the real world just using your phone. You can just use an Instagram filter to place a fantastic creature in your house,” she says.

The Smithsonian Institution, which has gradually been reopening museums since late July, is collaborating with Facebook to create AR experiences for Instagram. This December, the Smithsonian will launch 10 Instagram filters that users can add to their stories that put objects from the museum collections in their homes. Transport a triceratops into your backyard, a pair of boots worn by the Wizard in the Broadway musical “The Wiz” into your closet, or the National Museum of Asian Art’s Cosmic Buddha into your living room. The project leverages the Smithsonian’s Open Access 3-D objects and adds interactive and educational features to the filters, from annotated facts to animations to sound effects.

Complex institutional AR projects usually require that artists collaborate with a technology partner, an obstacle and a leap of faith for many artists who have never worked in this medium. To tackle the remote AR component of You Be My Ally, for example, Jenny Holzer’s studio brought in the London digital agency Holition to join Mark Hellar, the technology consultant already working on the project.

The technology can be a barrier for users too, particularly for older generations. “For some people, the hurdle of even opening up the app is pretty high,” says Mehring. “And if it doesn’t work, or they can’t figure out how to make it work right away, you can lose so many people.”

However, You Be My Ally has convinced Mehring that as an artistic medium, AR, with its layers of artifice and reality, is here to stay. “One way of telling the history of art is as a history of illusionism, and AR ties right in there,” she says, adding that from the first generations of print makers to video artists, artists have always embraced new ways to bring art into people’s homes.

AR that can be accessed anywhere could be a new powerful tool for institutions to enhance their global impact well beyond the pandemic. “When institutions get artworks out to people where they live, they’re testing out their relevance,” says Mehring. “Art needs to live up to that challenge.”