How we invented telepathy
Hearing people’s inner thoughts
In Where We Meet, audiences can hear the inner thoughts of the characters present in the space as they walk close to them. But how does it work?
Here’s how we’ve invented telepathy!
Note: This is a bit techy, but still worth a read!
Technology that can be forgotten
The tech behind where we meet is actually quite complex. It’s a large networked architecture with an ecosystem of devices talking to each other. But from an audience perspective, the tech is almost invisible. When they enter the space, participants are given headphones to put on their heads and a pouch with a lanyard to put around their necks. Then they just walk around the space and interact with the piece with their physical body. The ‘tech’ is not in the way anymore.
If you’ve seen Punchdrunk’s Viola’s Room, it’s a very similar kit that is given to the audience. We just did it first… with no budget…
Audience tracking
The first technical challenge we needed to solve was to where each audience member was in the space. Or more precisely, how close they are to a character. So they could hear the right audio feed accordingly.
We had some additional challenges that existing solutions would not handle:
The piece is in the dark - any camera/computer vision solution would not work
The solution needs to identify individuals, so we know who’s who in the space
We tried and evaluated several technologies during a first R&D at Goldsmiths University in 2022. Here’s a high-level summary of our findings.
Bluetooth beacons
VERY imprecise and unstable values - would jump from 0.5 m to 3 m. Good for knowing where someone is at a building level, not in the performance space.
Shame, because it’s cheap!
Consumer-facing VR/AR technologies, such as Vive Trackers
Very easy to integrate, a lot of ready-made SDKs / tools
Had the most precise and stable readings
Trackers are a bit pricey and chunky
Limited tracking area (officially 10 x 10 m max, we pushed it to 12 x 12 m and 14 x 8 m)
Traditional motion capture, tested with Optitrack and passive markers
Can be very precise
But it is VERY expensive
Hard to tour with the setup and to install in a performance venue
Actually quite hard to scale up as it needs different patterns of markers to track people in the space and recognise them individually
Ultra Wide Band (UWB)
We unfortunately didn’t get a chance to experiment with this. We know it is used a lot in theatre for follow spots and has some great potential. We mainly found commercial-facing solutions that didn’t offer enough hacking options, with a steep entry price just to play and see if it’d work.
Custom solution with computer vision or sensors
Another option would have been to develop our own custom solution. We probably have the skills, but definitely not the budget/time. And it would have delayed any creative exploration too much.
We decided to go with Vive Trackers. It gave us the most embodied and natural feel in the space. We could build realistic ‘sound bubbles’ around each performer and get the feeling of entering their minds with a couple of steps. It also offered some very easy integration within the tools/frameworks we already knew how to use. The hardware is off the shelf. And it ended up being quite easy to attach to headphones and stay in place on the head (which is also ideal for tracking).
An ecosystem of distributed devices
The second challenge to solve was around the audio and how to stream individual feeds to each audience member so they hear what is relevant to their location in the space.
The first version of the show had been built in MaxMSP with separate audio channels for each audience member. This allowed to perfectly sync all the audio feeds between dancers and audience members and gave the illusion that everyone was listening to the same inner thought at the same time.
The challenge there was on the hardware. How to steam those audio channels to the audience? We started by using Silent Disco headphones. But realised that we could only go up to 6 radio channels (3 by default and up to 6 with a special license).
Then we switched to using In Ear Monitor systems that allowed a more granular control of the radio channels. It worked great. But we realised that the hiring cost would make it impossible to rehearse and tour in smaller venues. For reference, 5 days of hire would cost £1,000.
In 2024, we did a complete revamp of the project, to allow us to be completely autonomous with our tech and remove any hire from our performance budget - which was also a necessary strategy to ease the conversation with venues. As all the monologues are pre-recorded (spoiler alert!), we could work with devices that would hold all the audio files in advance. They didn’t need to be streamed from a central machine. And then we could send signals to the fleet to make sure audiences hear the right content at the right time.
We bought a series of second-hand Android devices and created an app that would run the entire show by being connected to our local network.
A big part of our eco-system was already running in a game engine (Unity) - which has a lot of pre-built tools for spatial audio - making it an even better technical opportunity.
The phones are then hidden in a pouch for the audience. They put it around their neck and forget about it.
Dancers app
Even if the monologues are pre-recorded, we wanted to give space to adapt to the audience in real-time. Each performance is unique and from one show to the other, the audience will behave differently. By giving more flexibility around the content, we can react to what is happening in the space and give a true feeling of liveness - and create some suspension of disbelief around the ability to hear the characters’ inner thoughts.
We decided to split the content into short clips - separated into different categories. The dancers are then also equipped with devices to decide on the fly which thoughts they want to share with the audience. And create a total illusion.
Interactive projections
Where We Meet contains interactive projections for each dancer, projecting onto the floor. The projections react to how close the audience is to the dancer. It is our main source of light in the space and acts as a bubble of protection around the dancer. But also towards the audience. As they approach closer, the boundary moves with them, leaving them in the dark and preventing them from being in the spotlight - and feeling exposed.
A spaghetti bowl of network messages
As you might have gathered so far, we are using a full ecosystem of devices and software that need to behave in sync. It reacts to the position of the audience and to the decisions the dancers make in real-time. And it needs to be controlled remotely so the show can start and finish at the same time for everyone.
To do so, we have built our own architecture of messages distributing messages between devices. They update all the devices in real-time so everyone is in sync with the ‘state’ of the show.
For the nerds out there
For the techy people amongst our readers, here are the tools/frameworks we use:
Tracking system, stage management and phone app built in Unity
Tracking with HTC Vive Trackers 3.0
Projections made with OpenFrameworks (we’re making a switch to TouchDesigner very soon)
Network messaging with OSC
Video output for projections via NDI
And to finish, here’s a moment of appreciation for the team. The tech is led by Clemence Debaig (me!), supported by Branden Faulls and some early input by Rob Hall. Please hire them for your projects!
—
Curious about Where We Meet?
Join us on the 12th of August 2024 for Camden Fringe at Theatre Technis in London!