Indiana Jones Adventure Halloween
I've never been fully happy with how these projects turn out because I've never been able to achieve the scale and level of detail I want. Some aspects often work out great, while others look more like a home made craft project than full, immersive experiences. I've been more successful when the physical scale of the project is smaller, like the Doc Holystone's Box, or when software and projected graphics dominate over physical construction, like the Forbidden Forest. So in August when I began thinking about what to do this year, my goal was to hit a quality bar I'd be happy with by using projected graphics to achieve the physical scale I wanted.
It didn't turn out that way. Ultimately this ended up being the largest physically constructed project and the most complex software project I've done so far. Despite that, by approaching the project differently I hit my overall quality bar for the first time and avoided any real last minute scramble to get it done.
A warning that this was a big project, and the write up is much larger than my others. You can jump straight to the end to see the final product if you want.
In the beginning phase of my projects I resist constraining my ideas by budget or complexity just to get a sense of what I would really want to do. Once I settle on an idea, I then start to bring it back to what seems doable. Sometimes those ideas start with a theme and then I build an experience around it, and other times I first start with the experience I want and then find a theme that fits. This year the experience came first. I had been thinking about how The Mandolorian uses realtime computer graphics to generate the majority of the sets by displaying them on a huge screen behind the actors. The Mandolorian backgrounds dynamically adjust their perspective based on where the camera is pointed, leaving the backgrounds feeling three dimensional. Knowing that I wanted to focus on graphics and software this time over physical construction, I started considering how I could similarly build an immersive space by rendering the sets around the viewer and projecting them on to large screens.
To reach the scale I wanted, I imagined a display across our entire front yard that would dynamically adjust its perspective for the people viewing it. However unlike the Mandolorian where there is a single camera viewing the scene, here there would be multiple viewers and no one perspective would be correct. As a solution I recalled the background painting used for the opening shot in the Disney film Pinocchio.
The perspective makes no sense when you look at the entire background because it's distorted, stretched, and pulled in odd directions. But when you look at it through the framing of the camera as it pans across the background, you only see parts of it at a time an the perspective appears correct. I hoped I could recreate this effect in real time for multiple viewers. If people stand close enough to the screen so they only see part at a time, I could dynamically correct the perspective for each viewer and blend between viewers that are close by.
This led me to sketching out what it would take to construct a large, long rear projection display being driven in realtime by the Unity game engine. Simultaneously I realized that I would need some sort of minimal structure to keep people physically close to the screen to fill each person's peripheral vision with a small part of the overall image. The design at this point ended up looking like a 24 foot long hallway with one physical wall and one rear projection wall. I settled on three, 8 wide by 4.5 foot high screens joined end to end to make the projection wall. I would use a set of depth sensing cameras to track viewers in the space and use three projectors to display the images.
I wanted to prototype the idea using equipment I already owned before committing to this direction and spending a bunch of money. I still had the rear projection screen material from the Forbidden Forest, a projector, and a Xbox One Kinect sensor. I figured this would be enough to roughly prototype the concept with a single display. First I constructed a single 8 x 4.5 foot frame, mounted the screen, and hung it in my garage. I then discovered that although there is Kinect support on a Mac using OpenFrameworks, there is no support in Unity, which I intended to use for this project. Rather than purchasing new camera equipment at this stage or porting a bunch of code from C++ to C#, I instead switched to using a pole mounted webcam and OpenCV with ArUco markers attached to a hat to track the viewer. Although this was perhaps the most ridiculous looking setup possible, it worked very well as a proof of concept prototype.
I was able to fairly quickly get a tracking setup that convincingly gave a 3D effect when looking at the screen while wearing the hat. I experimented with several different 3D scenes captured with photogrammetry that I found on Sketchfab to get a feel for what scale, depth and general blocking worked best. I was pretty convinced this would work but my experiments also showed I couldn't drive this from my Macbook Pro. I would instead require a gaming PC with a high end GPU, a set of matched projectors, and several depth sensing cameras to make this work for real.
I continued to consider various themes while I worked to build the first prototype. The long hallway suggested being indoors rather than out, and I played with the idea of revisiting Disneyland's Haunted Mansion, or one of the older Disneyland dark rides like Mr. Toad's Wild Ride. But as I sketched ideas, it became clear that you need a great deal of linear space for multiple scenes, and even a 24 foot long hallway was not going to be big enough for an entire experience. Instead I settled on an building an Egyptian tomb that you could look down into through the projected wall. It would be more of an immersive space than a directed experience. At the same time, although I was pretty convinced the effect would work, the price tag to build three panels was more than I really wanted to spend. I decided to build one complete panel as the next phase of prototyping and then decide if I should go further.
I was also rapidly realizing that the hallway required for this experience was going to be a major project on its own and that broke my rule of sticking to software as much as possible for this project. Extending across the sloping front lawn would require me to build a level floor to attach walls to and it probably needed a ceiling. It also needed to be sturdy enough for potentially hundreds of people to pass through. And it needed to look like a tomb inside and out, and not like a bunch of plywood. Although I know I have great neighbors, I could already imagine the awkward conversations as I took what would likely be several weeks to construct this thing in front of my house. But I had already been prototyping the projection piece in the garage. What if the entire tomb was in the garage? That would change the shape of the hallway from linear to a U shape, which would also mean I would have only one screen at the back, thereby reducing the cost of the project. Also, because viewers are walking towards the screen instead of across it, I only needed to manage a single viewer's perspective at a time, which would greatly simplify the software.
I was trying to think of a singular thing people could do in a tomb of that shape and I realized I should actually be building an Indiana Jones themed space. Walking down a corridor and then having the iconic giant boulder come rolling at you while you run out the other side seemed like a great choice. But ultimately whether it was going to be the boulder scene or some other portion from the Disneyland attraction, I knew Indiana Jones would be the right theme.
I built a tomb for one of my first projects and it was a great example of looking like a craft project. I rationalized at the time that it was in very low light, so you couldn't see the lack of convincing detail, but I knew this time I needed to be much better. I discovered a small industry that has popped up making faux rockwork panels for escape rooms and even more permanent panels for exterior architectural elements. I began pricing out what it would cost to get enough panels to cover the interior of my garage but at the same time I spent time reading articles about how people have made their own panels for craft and theatrical use. Extruded polystyrene foam (XPS) is used as insulation panels, often on basement foundation walls and other tight spaces. Unlike the foam board that is often used to pack electronics, XPS foam isn't made up of tiny beads that break apart. Its smooth surface can be carved with a very sharp knife, and also with heat.
There is still a shortage of building materials but I was able to purchase a few 2 x 2 foot panels for experimentation. Armed with an old Radio Shack soldering iron, I practiced carving stone blocks into the panels to learn the best heat, pressure and speed to get the look I wanted. The large collection of Indiana Jones Adventure photos from Daveland Web served as an excellent reference for color, shape, and texture. as I experimented with several types of additional distressing like using wire brushes and sandpaper to further rough up the panel surface.
These first attempts turned out looking more like wood than stone, but I was convinced I could create high quality faux stone panels based on what I learned.
The projectors I have require 12 feet of distance to fill the 8 x 4.5 foot panel, which is almost the entire depth of my garage, so I knew I would need short a throw projector for this to work. After scouring reviews, I settled on ordering an Optoma GT1080HDR, which would require only a little over three feet to fill the screen. This would allow the screen to move back leaving about 12 feet of depth in the garage for the hallway. When the projector arrived, I determined the final position of the screen and began blocking out the dimensions for the remainder of the temple.
At this point I fully committed to a physical build along with the software build. I had enough confidence with the software, plus good initial outcomes with the stonework that I decided I could pull off both at a quality level I'd be happy with. After a final supportive conversation with Beth, I began to transform the garage into a temple fit for Indiana Jones to explore. Our house is almost 100 years old and the garage floor is far from level. I also wanted to ensure the garage door could continue to function, which meant I needed to build a room within the room, and do it from the ceiling, which is level, down to the floor, which is not.
I started with blocks securely mounted to the garage walls that kept the temple walls within the garage door guide rails, and built the center wall header wide enough to contain the garage door opener mechanism. After testing several times to gain confidence that the garage door could travel unobstructed up above this frame, I extended the walls down to the floor, matching to the slope of the floor along the way. The temple needed to be durable, but not fully structural, so I used 1 x 2's for the framing.
By now I realized I had not moved my old commuter bike out from the garage. It was destined to remain backstage, behind the temple for the remainder of the project.
As I entered mid-September I knew getting the walls in and detailed was going to take time. On an early Saturday morning, Beth and I borrowed our good friend Tim's work truck and bought out all the available XPS foam from Home Depot in Oakland and Emeryville, along with more 1 x 2's, glue, and other supplies. The test fit for the initial panels showed that the framing needed to be more substantial to keep everything rigid. I was particularly concerned with people pounding on the walls with their fists and feet, so I added horizontal bracing at foot and hand heights. Once added, I started cutting and mounting the panels.
I framed an extra wall around the front of the screen to keep people a minimum distance away. This also allowed me to ensure the projected image would be fully cropped by the walls to leave a borderless image.
Through trial and error I learned that a jigsaw was the easiest way to cut XPS foam. For most of the long cuts I clamped a board on the panel to guide the saw, and shorter cuts I was able to freehand. If you do a similar project wear a mask when cutting. I didn't initially and inhaled small foam particles, which I'm sure is not good.
My workshop is connected to the garage through a single door, and I needed to make sure I could freely move work in and out. I left out the middle framing where the door was and left that panel detached so I could come and go. Also once most of the walls and ceiling were in place, I added work lights so I could see. At times I left the garage door open for natural light on weekends, but it remained closed when working in the evenings.
After all the panels were cut to fit I started gluing them into place. I ended up using almost four tubes of glue to attach all the panels. I clamped the panels into place for several hours so they would securely adhere to the framing. I didn't have enough clamps to do everything at once, so this took some time.
After spending hours looking at photos of the Disneyland Indiana Jones Adventure stonework and realizing that that attraction actually has multiple styles depending on where you are, I settled on the reference photos I thought worked best. I marked the proper stone spacing on a long board and used it to pencil in the horizontal divisions for the stones across each wall. I hand selected the spacing for the vertical divisions and penciled them in, too. Some used the existing seams in the foam panels and some did not to give a varied size to each stone block. After penciling in the stone, I traced over the lines with a hot soldering iron in several passes with different angles to give the stones their definition. I also added cracks and chips in various spots along the way. The fumes from this stage of the process were pretty bad so I kept a fan running with the garage door open to vent the area as I worked.
Using a wire brush to distress the foam panels left it with a wood-like grain, and so I experimented with other texturing processes. I settled with using water based spray on wall texture, that I then knocked down with a large putty knife after it was partially dried. Although the results were good, they don't actually match the pitted look of the Disneyland rockwork. If I were to this again, I want to try misting acetone or some other solvent onto the panel to see if that results in a better effect. In addition to different texturing processes, I experimented with several paint processes. I settled on using a base of interior flat acrylic paint and after it fully dried, I brushed on several layers of diluted tempera paint in various shades of brown and approaching black. The diluted tempera paint worked very well to get into the cracks and even run down the panel to look like weathered stone.
Although the panel fit was good, seams were still visible where panels met. Also, the panels come pre-scored to make it easy to snap into smaller sizes. I used some of those score marks for stone definitions, but many I did not want. Before painting could begin, I needed to fill all the cracks with joint compound. Ths stage was mildly terrifying as I wasn't fully sure how it would look in the end and it would be almost impossible to remove if it didn't work out.
I further distressed the corners and shaped the foam so the stone corners appeared stacked with missing mortar. Once the joint compound dried, I lightly sanded any putty knife marks, but left some roughness where I could.
The mortar between the stones at Disneyland is very dark, and so prior to painting the stone surface, I needed to paint in the mortar. I used interior flat acrylic paint that was close to black, but had a bit of brown in it. This was painted and worked into the cracks with a cheap, natural chip brush.
I overpainted the mortar lines so I could let the darker color come through the base to ultimately give the stone more color variation.
Next it was time to texture the walls. I did this entire process on the entrance corridor first, and actually changed my process for the exit corridor. For the entrance corridor I textured after I painted the mortar because I was worried about texture filling the carved mortar, so I spent a bunch of time masking off everything. This left noticeable gaps in the texturing where the tape was, so when I repeated the process for the exit corridor, I textured first and then painted.
After the texture was dry, I began to paint. I rolled on the base coat with a varying amount of paint to let the darker mortar come through in spots. After it dried, I began weathering each stone individually with multiple washes of tempera paint. Originally I mixed tempera and water and then applied with a chip brush, but over time I learned I could put undiluted tempera onto the brush, dab it on the stone, then return with a water-wet brush to dilute the paint directly on the stone. I repeated this process for several layers, alternating between a 2 and 4 inch brush to get the color variation I wanted. Occasionally the wall would become too wet, so I used a paint roller to absorb excess water, and then dry brushed the paint back into place. In the larger cracks and voids, especially in the corners, I applied undiluted tempera, and in some spots I touched up with black and brown Sharpie markers.
I spent far too much time trying to find the exact fixtures used at Disneyland and I ultimately decided they are probably custom made. The lights are designed to look like work lights that have been strung through the temple as needed so I ended up using exterior mountable junction boxes with bare light bulb attachments and then strung wiring between them to deliver power. I also selected to use Edison style bulbs to give the scene a warmer, older look, while the attraction uses more modern frosted bulbs. One of the key details at Disneyland is a large, old generator that sits in the camp outside the temple. The lights throughout the entire attraction flicker with the generator as it speeds up, slows down, and occasionally stutters. I wanted to recreate this effect, so I built programmable 12 volt dimmers controlled with a Raspberry Pi to drive the lights. I split the temple onto two, separate circuits so I could independently control the entrance and exit corridors.
Once the lights were installed I revisited some of the stone weathering to get the final color I wanted with the actual lighting.
This angle also shows that the center wall stonework is matched on each side. It's only fully noticeable at the end, but I wanted to complete the illusion that the wall is actually made of a large stack of stones.
Over breakfast one morning I was chatting with my daughter about how we should decorate the front yard and we were listening to some recordings of background music from the Indiana Jones attraction with the idea it could be playing outside. In the Disneyland ride, the line enters through Indiana Jones' abandoned camp and a radio is playing period 1930's music. As we listened to the recording, it was interrupted with a news flash that announced the Temple of the Forbidden Eye is drawing visitors, but that people are going missing once they enter. The announcer speculates that they looked into the eyes of the Idle. Immediately I realized the attraction backstory worked perfectly for this project and decided the projection piece would not be the rolling boulder, but instead should be the first scene of the Disneyland ride where you encounter the large statue of Mara.
I had started building the boulder scene using assets from the Unity Asset Store, but I would need custom art to pull off a convincing Mara scene. I reached out to my colleague Rich Larm to see if he knew of anyone I could commission to model Mara using reference photos from the ride, and after I further explained what I was trying to do, Rich volunteered himself! I handed off my reference photos and the work in progress Unity project to Rich and he then enlisted the help of Evan Davies, another colleague, to take a crack at the modeling while Rich focused on effects and lighting.
While all the physical temple construction was going on, I was working out what final hardware would be used to run the projection and tracking. For the depth camera, I came close to committing to use Intel's RealSense cameras but Intel announced they were discontinuing this product line. I instead selected the Stereolabs ZED 2. This camera requires NVIDIA CUDA to run the tracking models, so this finally pushed me over the edge to commit to a Windows PC for the graphics. Although all my early software work was on Windows, I haven't developed with Windows or owned a PC since 2008. Also the current GPU shortage meant delivery dates for PCs were into late November at the earliest. I did ultimately find an Alienware R12 in stock at the local Best Buy with an NVIDIA RTX 3070 GPU. I got that PC, plus a RAM and SSD upgrade and dove back into the world of Windows development. Soon after upgrading the RAM the PC stopped booting and I spent a few days with Alienware tech support to get unblocked. Although it was frustrating to have a brand new computer break, their support was excellent, and they sent a technician to my house with replacement parts to make the repairs.
The Stereolabs ZED 2 is a pretty amazing piece of hardware and the provided SDK is great. I did initially struggle to match the exact version of the CUDA SDK with the Stereolabs SDK, but once that was sorted it dropped directly into my existing Unity project and within a few hours I had full skeletal tracking working. I kept a Mac version of the project working so Rich and Evan could continue to work while I moved ahead refining how to use the skeletal tracking to correctly control the virtual camera. We started blocking out the final scene and struggled for a while to get the virtual perspective to match the real perspective.
As I worked further to integrate the projection to the physical space, I found I needed to be working directly in the temple to test. I set up a makeshift office to continue debugging and refining the work. Coding on such a large screen was great!
I wanted the temple scene to unfold as guests entered, so I built a state machine that sequenced the experience. I then defined areas of the physical temple that when the camera detected someone entering, the state machine would advance to the next sequence. Each spatial trigger was tunable, which allowed me to calibrate animations, sounds, and lighting to change when people reached specific locations.
Historically I've used a piece of software I built over the years called Director to coordinate all of the lighting, sound and animation. That system works best for linear sequences so for last year's Pirate Bay Halloween I started experimenting with a new design based around MQTT. I went further this year and began to lay the groundwork for Director2 using MQTT, which over time should evolve into my new coordination system. Instead of a separate coordination process, I used Unity directly as the controller for the show. I ported the Paho M2MQTT client to Unity and built up a system that allows Unity GameObjects to publish and subscribe to MQTT messages. I then used the Python Paho MQTT on the Raspberry Pi based controllers to synchronize the additional lighting and sound.
As state transitions occurred from the spacial triggers, messages were broadcast over the network to act as cues for the various digital actors like lights, sounds, and animation sequences. For development, especially when working on a Mac that did not have spatial triggers, a debug HUD could be enabled to trigger events directly for testing.
All the various pieces came together and the Temple of The Forbidden Eye took its final shape. Originally the entire scene was to be modeled in 3D but that proved to be a larger project expected. Rich instead did an amazing job creating a 3D space and filled it with many 2D elements, particle effects and lighting. I added animated lighting transitions and synchronized them with the physical space using the Director2 system. The camera tracking added a subtle, but immersive feeling effect to the rendered graphics that really connected the two spaces together.
I sourced the soundtrack from soundsofdisneyland.com and disneychris.com. Because people were walking through the temple and not on a ride vehicle that has repeatable timing, I edited the audio into clips that could overlap or be cut short. Each trigger would advance to the next audio clip, but I left enough audio at the end of each cue to account for people moving through at different paces.
We had friends over a few weeks before Halloween and after walking through the in-progress temple, I was asked if I was going to match the Disneyland attraction and have the three separate paths of treasure, eternal youth, or visions of the future. I only had a single audio clip, and thus only a single path. At that point I said that only a single path was planned, but after the evening was over I couldn't let the idea go and tracked down the remaining audio clips. I remixed them and added the ability to use random choices every time the sequence started.
Once I built that, I realized I needed to randomize the congratulatory audio at the end, too. In the attraction Indiana Jones has several possible lines he can say at the end as you exit. I also sourced that audio separated from the music track, and built support to play the six different lines at the end. It's a small detail that most people wouldn't notice, but I'm glad it made it in.
At Disneyland outside the Temple of The Forbidden Eye is the campsite of the Lost Delta Archeological Expedition. Since I had gone this far, it followed that guests should also enter through the camp into the temple. I didn't have an extremely detailed plan for the camp layout, but I knew it needed several key features to evoke the same feeling as the attraction. First it needed an old radio playing the 1930's music and news broadcasts. Second, it needed one or two tents set up in a campsite. Lastly it needed the bamboo and rope lined walkway to guide people through the space.
I spent a few evenings on Ebay and found an old US Army canvas pup tent, and a period, non-functioning radio. The tent arrived first, and that's when I discovered that the tents come in halves, and I had ordered only half a tent. I found a second tent, with both halves and ordered that, too. I sourced 80 feet of 2 inch diameter bamboo and a spool of 3/4 inch manila rope, which unfortunately only came in 600 foot lengths. The rope arrived on time, but the bamboo did not. After several days of waiting, I started a trace with FedEx to find the lost package, and magically it showed up the following day, which was only four days before Halloween.
I built a second set of lights for the camp and another dimmer so they would have the same generator driven flicker effect and had them ready to install when we built the camp.
Saturday morning, the day before Halloween, Beth and I started building the camp in full. We planned to have a preview night with friends that night and wanted everything ready to go. We set up the army tent and constructed a larger tent using canvas painting drop cloths that would serve as Indy's office. We scoured our house for additional props and artifacts and Beth filled in the camp with amazing details. I brought back my old Boy Scout training to lash bamboo poles together and then hung the lights. By early afternoon the camp was complete.
That night our friends arrived and toured the camp and temple. I found a few small bugs and needed adjustments, but everything worked very well. Unlike previous years where I was working non-stop to the very end, this year everything was ready to go with time to spare.
I fixed the last bugs and we made some small tweaks to how the camp was set up on Sunday morning. Our neighbor Christian stopped by the previous afternoon and offered to loan me a larger speaker to give the temple a bigger sound. On Sunday afternoon Christian came back with the speaker and we got the it set up and tested, which really improved the entire experience.
I did one final restart of all the software and let it run uninterrupted just to confirm nothing would crash as it ran for hours. It was at this point I smelled something burning in the yard. After hunting around I discovered that the dimmer I built had melted with the heat it generated. Although the dimmer was identical to the one I built for the temple, I theorized that because the cable runs were much longer between lights, the added resistance from the extra wire pushed the circuit beyond what it could handle. I quickly built a new dimmer with extra parts and tested it with half the lights. Over 45 minutes or so it too, became quite hot and was in the early stages of melting the breadboard. Although I was quite disappointed I would not have the signature lighting effect in the camp, I decided burning down my house was a worse option, so I removed the dimmer and instead hard wired the camp lights to power so they would remain on.
Smaller kids started to show up in the neighborhood around 4:30 to trick or treat. Several friends from work stopped by to see what I built and by 6:00 enough people were out that a line began to form.
Throughout the night people showed up in waves, but we rarely had nobody visiting. Often the line grew through most of the camp, and several times the wait time reached 15 minutes to get into the temple itself. Beth outfitted each of us with Indiana Jones hats and we dressed as guides. I stood at the camp entrance handing out candy and Beth stood at the temple entrance to meter people in and out over the night. She perfected her part, warning visitors "to not look in to the eyes of Mara" and got groups to verbally repeat the rules before entering. She also started asking for and packing in "single riders" Disneyland style, to keep the line moving.
Although we couldn't keep an exact count, based on the amount of candy we went through we think we had about 550 kids and about 250 adults come through. Almost everything worked without a hitch. Somewhere in the evening somebody pulled on one of the wires in the temple, which broke the lights in the exit corridor, and the radio in the camp struggled to play music at times because the Homepod speaker I hid inside kept falling off the wireless network, I think from all the cell phones with in the area generating interference.
So what did it look like? Here's a walkthrough from the end of the evening after the crowds died down
There are certainly things I'd do differently and things I wanted to do that got cut, but for the first time I felt I hit the quality bar and scale of the project I imagined at the beginning. It's always fun to combine all these different mediums into a singular project, and to be able to entertain so many people. I can't wait to do it again.
This project was not a singular effort. Thank you to all those that contributed, encouraged, and supported me: