April, 2019

It's also good to share projects that didn't work out.

For his ninth birthday, our son asked for a Terraria themed birthday party. In addition to decorations I wanted to create something similar to the Pokewall I built for his birthday a few years before.

Human Controlled Terraria

In addition to Terraria, our son also loves building costumes from cardboard. The idea for this game was to let each kid create their own costume and load it into Terraria as their skin, and then build a camera controlled interface that lets them control the game through physical movements while wearing their costume. Although I did get the full system working, it was not stable enough to track a group of hyper, sugar infused nine year olds.

Camera Controlled Keyboard

We had already been modding Terraria together to create new weapons and armor. The plan was to use these same mods to load the kid designed skins, but the camera control system was built differently. Using OpenFrameworks along with its Kinect2 support, I first got skeletal tracking working so I could detect where players were.

Then I built a simple gesture recognizer that detected various poses and motions using the skeleton tracking as input. Next, I then extended the OpenFrameworks app to send keyboard events to the operating system. I was able to then remap the keybindings in Terraria to these key press events. At this stage I was able to fully control a character in Terraria by walking left and right, jumping, and swinging my arms to activate my weapon, and to reach across my waist to switch weapons from my inventory.

Unfortunately the gesture recognizer was never stable enough to really track a bunch of excited kids. We moved forward with the Terraria party and each kid still built their own cardboard costume and had fun LARPing Terraria through the house, so it was still a success. Perhaps someday I'll resurrect this code for another project.