The Fortune Teller
October, 2026
Every game and experience built so far has players interact with characters they cannot see. Either they are secret agents communicating over a terminal in The Heist, or variations of ghosts speaking from beyond in Doc Holystone's Box, The Specter of Grosvenor Place, Dragon's Layer, or the Triple Creek Murder. With each new version, I've added more interactivity with those characters, but there are only so many fictions that explain why you can't see the character you're playing with.
I've been working off and on for the last year to merge the emergent storytelling technology I developed at Rift Games with Unreal Engine virtual production technology to create an experience that is both emergent and visually immersive. And while I have a larger experience still in early development, I wanted to get something built now that proves I'm on the right track.
We host monthly supper clubs for our friends and I decided that the October get together would be the perfect venue to make this happen. My initial plan was to adapt The Triple Creek Murder and replace the disembodied voice of the main character with an interactive version of the Madame Leota crystal ball I originally built for the Haunted Mansion Halloween back in 2018. As often happens with these projects, my available time required me to scale back that ambition, and instead this project became an interactive fortune teller. Regardless of the scope reduction, I am quite pleased with the result.
I started by building a new Leota. The original projection-mapped a looping video onto a foam mannequin head. But because I wanted guests to get closer to her, I hoped to use rear projection to fully hide the projector. After much searching, I found two potential mannequin heads made with hard plastic. When they arrived I cut the backs off, scooped out the insides, and experimented with projecting from behind. Unfortunately both heads proved to be too opaque, leaving the image dim and blurry.
Resolved to use front projection, I next build a mockup table so I could experiment with different ways to hide the projector. I settled on mounting the projector inset to the table, covering it with a box. The box would doubled as the podium to receive the tarot cards.
I next moved to recreate Leota in Metahuman. I've been experimenting with Metahuman for some time now, learning how to use realtime performance, motion capture, and procedural animation to drive believable characters. I wanted to use Metahuman Creator to create a face that resembled the Haunted Mansion Leota, but lacking time (and probably the skill) to do it well, I turned to Fab and found a suitable Metahuman. I brought it into Unreal, removed the body and hair and began to work through projection mapping on to the mannequin head.
To get everything aligned, I built a virtual camera system that let me adjust the projection to account for the projector placement. Final alignment was done by projecting markers at the eyes, nose and mouth and aligning them with temporary markers on the mannequin.
I wanted the script to be fully dynamic based on the cards you play, so I turned to Claude. The game managed a state machine that tracked observed cards, duplicates, and timing delays and wrote suitable prompts to generate the required script as text.
Speech was rendered using ElevenLabs v4 Turbo, which generated impressive results with minimal latency. Claude additionally included audio tags to add emotion and range at key points of the dialog.
I did several experiments with lip-sync and settled on using a very easy to use package called Runtime Metahuman Lip Sync from Georgy Dev. I implemented my own audio streaming pipeline that streamed to and from Claude, through ElevenLabs and back to Unreal to control and minimize latency. Ultimately I was able to get latency from card detection to spoken audio to under two seconds. While not fast enough for an interactive conversation, it was more than sufficient for this performance. I did pre-render a library of utterances like “umm” and “ahh” that played during this time to cover any latency. Because those were also lip-synced, the final result blended perfectly together into a very convincing performance.
I then made a full deck of tarot cards following the same method as I did for Triple Creek. Three layers of card stock with a NFC sticker tag in-between the layers created a convincing card. Each card was then tagged with an identifier to detect the card on the reader. I used a smaller RFID reader this time attached to a ESP32 S3 Feather, which broadcast detected cards on the Director MQTT network.
I also wanted Leota to engage with the guest and not stare vacantly into space. I used a web camera to feed OpenCV running YuNet to perform face tracking. The camera process ran as another device on my Director MQTT network broadcasting guest position, allowing Leota to adjust her gaze. During readings, her attention moves between looking at the played card, the guest and the surrounding area as she thinks. I briefly experimented with my stereoscopic camera but realized I could achieve face detection much more cheaply with the webcam as I was getting close to being GPU and CPU bound with the rest of the performance.
When left alone Leota looked around and ran through various facial animation performances. If a guest was detected near by, she would enter “attract mode” calling out to the guest for them to join her. If a guest left part way through and stopped presenting cards, she urges them to continue, and then ultimately chastises them for not finishing.
I found a decorative table that matched the right dimensions for the final version. Using measurements from the mock up table, I cut a slot into the table top and made a custom projector mount that let me fine tune its position.
After testing in the workshop, I brought everything out to our backyard. The table on its own was too low to the ground so I built a pedestal to raise up the table, which also served as the perfect place to conceal the computer driving the show. The first full dry run happened on the Thursday before our party, and after making some last minute adjustments, I ran a soak test for several hours on Friday to make sure it could run and operate for many hours unattended.
When Saturday arrived as the sun began to set, I started the show and left it to be discovered by our guests. As they arrived and eventually found themselves on the upper patio, guests encountered Leota and delighted in having their fortunes told. People returned over the course of the evening, for repeated readings or to try it in groups. She made an excellent addition for our October get together and worked almost flawlessly all night. The only hiccup was when it got fully dark and the camera struggled with face detection. This meant sometimes Leota did not recognize a new approaching guest, confirming my earlier idea that I should use an infrared camera.
While the breadth of this project was not as large as some others I've attempted, I was extremely happy with the quality of the final experience. It fully validates that the larger project is not only doable, but that I should be able to finally reach the level of emergent immersive experiences I have been working towards.
Now, to find the time to get it done!