American Express: Real-time Photo Manipulator

Processing (library), Java, OSC

For American Express's Unstaged concert experience, we teamed up with James Corden and Disclosure, a popular electronic band, to enable fans to become part of the concert experience. We created a real-time A/V show where fans could submit photos of themselves and have them displayed live during the concert.

Real-time Photo Manipulator

The program had a selection of scene layouts and manipulation effects baked into it. A VJ used a custom controller to live switch between the layouts, set the BPM for the effects to sync with, and control the types of effects and their amplitude. The program was continually loading new images from a backend, so the content was constantly changing. 

1. Responsibilities:

  1. Engineer in charge of the photo manipulation program. I wrote all of the pixel manipulation functions, wrote the memory management for loading and unloading images, created the layouts, and built the UI for controlling the program in real-time, in sync with the music
  2. Organize and collaborate with our back-end engineer to build an API for loading images from our servers at high velocity

2. Takeaways:

  1. I learned how to build out a program for a live event
    • We developed the program so that it could deliver a high-quality viewing experience even if certain parts of the program failed (such as our back-end disconnecting or running out of memory). This was incredibly important because the program was running live in front of thousands of fans. I've since utilized this lesson in almost every experiential project since.
    • I utilized the resources available to me online through forums and Touchdesigner's support team to create a program that efficiently pre-loads and manages thousands of frames of animation.
  2. I facilitated the development of the software, from initial concept through production tests
    • One of the more difficult parts of the build was figuring out the algorithms for all of the manipulations. While the warps and chops were straight forward, the rotations and waves were tricky. I eventually reached out for help from a colleague of mine and he taught me how to build basic GLSL shaders.
    • I set milestones and goals, collaborating with key stakeholders to create a satisfactory product.
    • Working with the back-end engineer we created tests to stress test the memory management on our production machines.

Massive Success

Although the program was only supposed to be used for a single night of the tour, the band liked the program / visuals so much that they took it on the road with them for the rest of their sold-out US tour.