May 2025
Sound of Body
Sound of Body is a wearable audiovisual system that turns biometric activity into a personalized audiovisual output. A sensor prototype captures heart rate and blood pressure in real time, and those signals are mapped to pitch, tempo, percussion, and projected visual motion so each participant generates a distinct melody and visual composition from their own body.
Research Context
Sound of Body began with a simple question. Music can change heart rate, breathing, and emotional state, but those internal changes are rarely treated as material for music itself. This project asks what happens when bodily signals are heard as input rather than only measured as data.
The research phase brought together auditory perception, music psychology, and semi-structured interviews. It looked at how sound is processed through the body, how arousal relates to heart activity, and how people describe the sounds they notice from their own bodies.
Design Process
From there, the project was organized as a sensing and mapping workflow rather than a fixed composition. Biometric signals, recorded body sounds, and generative visuals were designed as one linked system.
Heart rate controls pitch, diastolic pressure selects tempo and time signature, and systolic pressure shapes percussion timbre. Body-sound samples such as breath, borborygmi, skin friction, and joint sounds were edited in Adobe Audition and layered back into the output as additional sonic material.
Wearable Prototype
The prototype was built to test whether biometric change could drive a clear audiovisual response in real time. A wearable sensor captured heart rate and blood pressure while the live visual field updated from the same incoming stream, so the mapping could be tuned against actual bodily variation instead of prerecorded values.
System Design
The technical pipeline combined sensing, signal processing, audio editing, and projection. An STM32-based prototype transmitted live readings over UART serial at 115200 baud to Python, where the stream was parsed, stored, and mapped into musical parameters.
The Python pipeline used PyDub, MoviePy, and FFmpeg to assemble audio and video materials, while TouchDesigner handled the real-time sphere visualization. The same parameter stream drove both sound and image, so pitch shifted color, tempo changed rotational behavior, and percussion events shaped the pulse of the projected form.
Personalized Output
The overview video shows how wearable input becomes sound and projected visuals through one continuous flow. Each participant produces a different melody and visual response from their own biometric state.
The final output changes with the participant's biometric state at that moment. Because the music and visuals are generated from live readings rather than a fixed track, the same system can produce different audiovisual results across sessions.