XR Design and Immersive Audio Production
This page lists my research and development projects in Immersive Audio, AR, Motion Capture and 3D animation, and 360 Video. During my studies and experiences at record labels I was fortunate to be able to form relationships with people who gave me access to an array of audio recording technology and allowed me to experiment with it at live productions and with real audiences.
Multimodal Music Analysis
Role: Motion Capture Engineer
Type: Research and Engineering
This research project used motion capture and immersive audio recording to study performance from a quantitative multimodal perspective.
Abstract: This project introduces a workflow to record paired motion capture and sound of musical performances using an infrared light-based tracking system. This paper discusses example production cases ranging from short loops to full performances, addressing the production challenges and compromises, substantially different from a regular audio recording process. Motion capture is very versatile and can be applied in many disciplines including musical pedagogy, distributed performances, machine learning, and as creative assets. Paired audio-mocap data can be rendered into similar visual content for interactive virtual environments, live performances or fixed media. This extensible workflow can be adapted and extended for different purposes in musical performances to promote further research and creative content development.
This Project Involved:
- Motion Capture (motive/optitrack)
- Blender, Unity
- Reaper, Pro Tools
SONYC Immersive Audio
Sounds of NYC (SONYC) is an NSF Grant-funded urban noise pollution research project facilitated by NYU's Musical Audio Research Lab (MARL). During the summer of 2019 they partnered with the NYU Tandon School of Engineering's K12 STEM Education Program to bring music technology and noise pollution education to middle school teachers and students from the city. I was a part of a team of engineers and creative technologists who designed a curriculum and taught it to 25 middle school students and four middle school teachers from around the five city boroughs.
Final Project: The program ended with a series of final projects, where the students chose specific topics of noise pollution to research and implement the required music technologies to demonstrate and highlight their research. Below are two 360 video and immersive audio environments that the students of my group, which focused on immersive audio, produced with my help. The goal was to create immersive urban landscapes to demonstrate the different types of noise that humans of New York City live with and to bringing awareness to the health affects of noise pollution.
* Beware of your volume. The videos might come in loud *
Mobile AR Games
Role: AR Engineer
Type: For Fun
This development fed my curiosity about how computer vision is used to recognize image targets and anchors from a user's camera to augment the physical world in real time. I prototyped some mobile AR experiences using ARKit, vuforia and 8th Wall.
Video 1: AR ad activations against image targets (Old Winston Cigarette Ad against a Piero Fornasetti painting)
Video 2: Logo capture AR game - catch the logos to gain points!
360 Video Shoot at Boiler Room + Half Moon BK
My friend who founded Brooklyn radio station, Half Moon BK, collaborated with underground live-music company Boiler Room and allowed me to attend the show to shoot and produce 360 video content. I was responsible for every aspect of the recording and production of the content, including creative, engineering, editing and delivery.
Nervous about bringing a borrowed +$1,000 ambisonic microphone and rented 360 cameras to a small venue space in Brooklyn that was known to draw a large crowd, I headed over anyways and set up. The show was a lot of fun and I captured some cool 360 video shots to experiment with, along with ambisonic room recordings.
I captured over 1.5hours of 4K 360 video footage and 1st order ambisonic audio recording.
The show did draw a huge crowd that was packed into the small space and was eventually shut down by the police. Needless to say it was a crazy setup. I learned a lot about immersive new media production and all of the equipment remained unharmed :)
- Sennheiser AMBEO VR 3D Microphone
- Insta 360 One X
- GoPro Fusion
- Zoom H8 Portable Digital Recorder
- After Effects
Immersive Audio for Half Moon Radio
I went back to Half Moon BK to shoot a second live show in 360 video. Learning lessons from the first go-around, I scaled back my one-man production and the Boiler Room crew also scaled back crowd attendance.