- Date: 12 April 2010
- Artform: Visual arts
- Region: National
Cookies improve the way our website works, by using this website you are agreeing to our use of cookies.
For more information see our cookie policy.
Cookies improve the way our website works, by using this website you are agreeing to our use of cookies.
For more information see our cookie policy.
Select the image above to launch the Sancho Plan's digital commission, Augmented Harmony.The beats emanating from Bristol's Millennium Square announced that something was happening. The first sight was of onlookers, then a small group could be seen in front of them holding up signs which they moved to the music.
Walking in a little further it became clear what held everyone's attention. The BBC Big Screen mounted up high was showing a live image of the group at the front, but with animated characters added amongst them, rapping out the beats.
This interactive performance was created by The Sancho Plan, a group of writers, musicians, animators, designers and computer programmers, in collaboration with Bristol's Pervasive Media Studio, which hosted the project, and the BBC. As an experiment in inclusion, The Sancho Plan's aim was to give people a chance to make music together. They needed a technology to enable those without musical training to take part, and chose augmented reality (AR): the superposition of virtual content onto live camera feeds.
A video camera captured the dancers (even if they started by simply holding up the signs, the participants found themselves dancing eventually). Its live image was fed directly to the screen, but also through a computer that recognised the computer-generated symbols on the signs. The computer translated each of those symbols into a small monkey-like character who sat squarely on top of it, and played a track of beats. When you move the symbol, the monkey follows. When you hide the symbol, the monkey and its beats disappear. Monkeys harmonise their beats when others are present, and an animated link appears between them.
There is something timelessly fascinating about seeing transformations of ourselves looking back at us – whether in a rippling pool, a hall of mirrors, or a video screen. And now, when that 'mirror' is interactive, and responds to us with synthesised sound and vision, the possibilities are, as we know from fairy tales, considerable. Interactive technologies for blending the virtual and physical worlds are maturing.
The Sancho Plan's piece used AR to engaging and inclusive effect. There was no shortage of smiling participants, who had heard about the event or just happened to be near the square. A small sample of onlookers said it was 'amazing' or 'cool', and had appreciated the relationship between the symbols, the monkeys and the beats they rapped, even if most couldn't say what kind of technology they were seeing in action.
This was a relatively simple use of AR, which was aided by the symbols but could in principle have used the participants' bodies and gestures as triggers. The Sancho Plan recognises the need to explore more types of musical interaction. But the simplicity didn't seem to matter on this occasion. The participants experimented, achieving a visual effect even when they couldn't influence the sounds by doing so. They rotated the signs and swung them about. The groups, even those consisting of strangers, choreographed themselves spontaneously, moving around one another, stacking the signs, running, and dancing and waving to the beats. The monkeys sometimes couldn't quite keep up with the humans. At one point a little girl without a sign watched herself dancing – with the monkeys or the people behind her, it was hard to say.
AR is one of many ways for artists to embed virtual content and interactivity into our physical spaces: others include 'physical hyperlinks' such as barcodes and electronic tags linked to content, and media overlays triggered by GPS and the other sensors found on smart phones. But artistic exploration of how to weave interactivity with physicality is in its early days. The Sancho Plan brought a musical mirror to Millennium Square. We can look forward to more ways of seeing ourselves – and the physical world around us – through the eyes of the web.
- Tim Kindberg, 2010
The Sancho Plan are an award-winning artistic collaboration of writers, musicians, animators, designers and computer programmers who focus on creating live integrated experiences. Their unique audiovisual performances and installations have been presented in galleries, cinemas, theatres, clubs and festivals around the world.
Tim Kindberg is a computer scientist whose company, Matter 2 Media, develops technologies for urban interactivity. He can be reached at tim@matter2media.com.
Bringing together practitioners, researchers and users from diverse backgrounds, the Pervasive Media Studio seeks to explore these questions, establishing a globally-recognised centre of excellence in the heart of Bristol, UK.
The Sancho Plan: Ed Cookson, Neil Mendoza, Robin Butler
Pervasive Media Studio: Tarim, Clare Reddington, Geoffrey Taylor, Tim Kindberg, Phil Stenton
Do Tank Studios: Joe O'Brien, Adam Hoyle, JP Altier
BBC Big Screens: Erik Burnett-Godfree
Thanks to: Olly Venning, Edd Dawson-Taylor, Chris O'Shea, Darren Turrell
