Pokéwall
April, 2017
We keep escalating our kid's birthday party games and this year our son wanted a Pokémon themed birthday party. We planned to have games including dodgeball with rice filled "Pokéball" socks, a scavenger hunt for Pokémon figures in the yard, and originally, a corn hole game to toss Pokéballs into a target. Recently however, I had picked up an XBox One Kinect Sensor and a pico projector for another project, and decided that instead of tossing balls into plywood targets we really needed a game to try and catch real, moving Pokémon. I gave myself three evenings worth of time to pull it off, which would leave me time to still cut some holes in plywood as a backup plan.
This was another short project, so I used openFrameworks with the ofxKinectV2, ofxOpenCv addons, quickly mashing together some of my own code and provided sample code to get going as fast as possible.
Initially I intended to use the Kinect to track the ball's distance from the camera to determine when it reached the screen, but I found that simply using OpenCv blob detection I was able to track the ball well enough. My kids had a small red playground ball that was about the right size and I used a canvas painting drop cloth from Home Depot for the screen. Certainly not a professional quality projection surface, but I've found they're cheap, realatively bright and I don't worry about damaging them. I just tacked it up in front of our bookcase with nails in the moulding.
I encouraged everyone to throw underhanded, which increased angle of the arc the ball followed to reach the target. This meant the ball didn't enter the frame of the camera until it was very close to the screen, reducing the chance that the ball would be seen by the camera too early and register as a hit.
I made enough progress in the first evening to get blob tracking working at my desk. I knew matching the pico projector's image extents with the camera's extents would be messy (some day I need to solve this problem fundementally so I can just quickly pair any camera with any projector by projecting a callibration image), and I messed around a bit with OpenCv's region of interest settings, but didn't get satisfactory results.
The second evening I setup in the family room to test if this would actually work in practice and to better understand how the camera and projector callibration would work.
It took a bit to find the right lighting conditions that were dark enough for the projector, but light enough so the camera could see the ball. Once that was sorted, I went back to sorting out the calibration problem. I pretty quickly realized that since this was mapping to a fixed plane, I could just scale the blob coordinates by constants to match the projected image. I positioned the camera so the upper left corner of the camera matched the upper left corner of the projected image, then I wrote a callibration mode which allowed me to hold the ball in the lower right corner of the screen. By measuring the lower right corner position, I could then scale the camera to match.
I next quickly assembled a small stand from scrap lumber to keep the camera and pojector in place and asked our daughter to give it a try.
I set everything up again on the day of the party and marked off an area for the kids. I was very worried a bunch of excited seven year olds would knock the stand and mess up the callibration, so everything was anchored with tape. It looked pretty amateur, but it was completely functional.
Man we need to replace that carpet... once the kids get older and stop spilling stuff...
I originally planned to grab a few character images from the web to use, but I realized the Pokémon website has a full Pokédex and it was trivial to write a python script to scrape all 800 or so characters. I combined that with a few Pokémon sound effects found online and called it done.
Up to this point, our son did not know about this part of the party. We saved it for the end and after cake we led the kids downstairs. I explained they needed to throw underhanded and that sometimes the Pokémon might "block" the attack. In reality there were occasions when the ball moved too quickly to be detected and a hit would not register, but a little bit of in-game fiction can help explain away the occasional bug. They had a great time and everything worked without problems. The kids played for 45 minutes or so and when the part ended, everyone left having had a great time!
The code is available on GitHub without the assets, but there's a tool in the utils directory to help you.