May 2025
Sound of Body
Sound of Body is a wearable audiovisual system that turns biometric activity into a personalized audiovisual output. A sensor prototype captures heart rate and blood pressure in real time, and those signals are mapped to pitch, tempo, percussion, and projected visual motion so each participant generates a distinct melody and visual composition from their own body.
Research Context
Sound of Body began with a simple question. Music can change heart rate, breathing, and emotional state, but those internal changes are rarely treated as material for music itself. This project asks what happens when bodily signals are heard as input rather than only measured as data.
The research phase brought together auditory perception, music psychology, and semi-structured interviews. It looked at how sound is processed through the body, how arousal relates to heart activity, and how people describe the sounds they notice from their own bodies.
Signal Mapping
From there, the project was organized as a sensing and mapping workflow rather than a fixed composition. Biometric signals, recorded body sounds, and generative visuals were designed as one linked system.
Heart rate selects a harmonic tone between A2 and G4, diastolic pressure sets the beat interval from 15 down to 7.5 seconds, and systolic pressure picks the percussion timbre from gasp, sigh, and heartbeat samples. Body-sound recordings such as breath, borborygmi, skin friction, and joint sounds were edited in Adobe Audition and layered back into the output as additional sonic material.
Each sample carries its own feeling. Intestinal sound is among the first things anyone hears in the womb and lands as safety, while a knuckle crack lands closer to discomfort. Composing with these keeps the output recognizably from a body rather than a synth preset.
Wearable Prototype
The prototype was built to test whether biometric change could drive a clear audiovisual response in real time. A wearable sensor captured heart rate and blood pressure while the live visual field updated from the same incoming stream, so the mapping could be tuned against actual bodily variation instead of prerecorded values.
System Design
The technical pipeline combined sensing, signal processing, audio editing, and projection. A wrist-worn MKB0805 photoplethysmography module on an STM32 board measures heart rate and blood pressure over a 30-second window and streams readings over UART serial at 115200 baud into Python, where the stream is parsed, stored, and mapped into musical parameters.
The Python pipeline used PyDub, MoviePy, and FFmpeg to assemble audio and video materials, while TouchDesigner handled the real-time sphere visualization. The same parameter stream drove both sound and image, so pitch shifted color, tempo changed rotational behavior, and percussion events shaped the pulse of the projected form.
Personalized Output
The overview video shows how wearable input becomes sound and projected visuals through one continuous flow. Each participant produces a different melody and visual response from their own biometric state.
The piece runs as a small personal experience rather than a stage work. One participant wears the sensor band, the room stays dark, and the projection starts answering their pulse within seconds. Because every note and every color starts from live readings, no session repeats, and what a participant hears exists only while their body produces it.