Innovative Sound Reconstruction Using Dual-Shutter Optical Techniques
Written on
Chapter 1: Introduction to Dual-Shutter Optical Vibration Sensing
This year, I had the opportunity to attend CVPR in person, where I witnessed the impressive presentation of the Best Paper Honorable Mention titled "Dual-Shutter Optical Vibration Sensing," authored by Mark Sheinin, Dorian Chan, Matthew O’Toole, and Srinivasa Narasimhan. The core idea of their research is the ability to reconstruct sound using a laser beam and cameras on any vibrating surface. This innovative method allows for the isolation of musical instruments, focusing on specific speakers, and filtering out background noise, among other remarkable applications.
To get a better understanding of their methodology and hear the fascinating results, I encourage you to check out the video below or visit their website for more examples.
In one of the examples presented in the video, listeners can distinctly hear two separate guitars on each audio track.
Section 1.1: The Technology Behind the Method
This groundbreaking technique utilizes not only recorded sound but also employs a laser along with two cameras featuring rolling and global shutter capabilities. This visual approach simplifies the task of separating audio tracks compared to traditional methods. Interestingly, the technology allows for recordings through glass and from any vibrating object. In one demonstration, the researchers applied their technique directly on speakers to isolate the left and right audio channels, which a microphone would typically blend together.
Section 1.2: Overcoming Challenges in Visual Vibrometry
Traditionally, visual vibrometry, which is the basis of this technology, requires optimal lighting and high-speed cameras to capture vibrations at rates of up to 63 kHz. Remarkably, this new method achieves similar outcomes using sensors designed for only 60 and 130 Hz, and it can handle multiple vibrating objects simultaneously!
Chapter 2: The Process of Sound Reconstruction
To successfully reconstruct sound, the researchers must first comprehend the laser signals they receive and process them accurately. The procedure begins by directing a laser onto the surface of interest. The laser reflects off this surface into a focal plane, which serves as the source of information rather than the instruments themselves. This allows for an analysis of the subtle vibrations through the laser's response.
The resulting 2D laser response pattern, known as speckle, is captured by both cameras. The rolling-shutter camera records at 60 fps, capturing multiple images which are then utilized to represent a noisy and less accurate 63 kHz signal.
The global shutter camera is essential to overcome inconsistencies in speckle imaging caused by the object's surface roughness and motion. It captures a complete image of the speckle pattern, which serves as a reference frame for isolating significant vibrations from the rolling-shutter captures. This process is repeated throughout the video, enabling the extraction of sounds from recordings, including isolating individual instruments and filtering out background noise.
In conclusion, this overview highlights the incredible potential of dual-shutter optical vibration sensing. I strongly encourage readers to explore the full paper for additional insights. Congratulations to the authors on their honorable mention! I was thrilled to witness the live presentation and look forward to future research inspired by this paper.
Thank you for taking the time to read this article. I’d love to hear your thoughts on potential applications of this technology and any concerns it may raise. A special thank you to CVPR for hosting such an inspiring event in New Orleans. Stay tuned for next week’s paper review!
References
? Sheinin, Mark et al., 2022, Dual-Shutter Optical Vibration Sensing, Proc. IEEE CVPR.