Turning an image into sound

Signal Field grabs a random photo on the web and turns it into music. The sound then feeds back into the image to create visuals in your browser, cycling in an infinite loop. All built with code.

A while ago I started a project called LOOM, essentially it was a “generative ambient ecosystem” that would play different sounds and show different visuals based on the time of day. I completed the environment for the sounds to live in but the sound design was quite a massive task, and to be honest I have not been inspired to write music this year. Signal Field is an evolution of that idea, it combines a lot of what I learnt building Loom, RNDM and Terrain.

Every twenty-two seconds Signal Field gets a new random image from a select few sources on the web. Other than source, there is no curation or theme. Specific sources were needed as it requires access to permissive CORS headers to grab the image.

Once an image is chosen, it is analysed to get some data points. It does this by sizing the image down to 96x96 pixels and measures colour, brightness, contrast, temperature and identifies the 3 dominant colours in the image. It also keeps a brightness map of the whole frame so that it knows where the light and dark areas sit, not just the averages. One pass over 9,216 pixels produces every number the engine uses.

Those measurements get turned into musical decisions and applied to parameters across the signal chain and visual field so that every image creates something different. Image data influences the harmonic structure including the key, mode (minor, lydian etc), chord and tuning.

The image also builds the waveforms you hear – this was a real lightbulb moment for this project. I started with randomly choosing standard oscillators like a sine or square wave but the sound felt too dull and lacked a lot of harmonics. I rebuilt the voicing so that the image forms everything from the spectral tilt to the timbre and harmonic balance. The sound now feels much more alive.

A high-contrast image gives a brighter waveform with more harmonics; a soft, hazy image gives something rounder and darker. Underneath it all sits a deep bass drone on the root note.

The waveforms run through a filter, echo and reverb whose parameters are also decided by image data. Every image exists in it’s own space. When one image blends into another, the sounds also phase into their next form as a new set of parameters initiate. This can add a little dissonance but I quite like it, combined with the morphing dream-like quality of the visuals it starts to give the project a unique identity.

Initially that was how the system worked, after the parameters were set it would play the sound but there wasn’t enough variation in what i was hearing. I was thinking about the interaction I used with the playhead in Terrain and realised I could use a similar concept here, but take it further.

As the playhead moves across the image it is reading it like sheet music. The system lays out its chord notes as a grid: low notes at the bottom of the image, high notes at the top. Where the photo is bright, the note swells; Where it's dark, the note stays silent. So a bright cloud in the top corner of the photo becomes a high note that blooms as the line passes over it. The result is that the music's movement comes from the photograph's actual structure, its shafts of light, shadows and texture.

Underneath the waveforms is a layer of field recordings. I felt that the overall mix needed another element, and field recordings add a nice textural component. It also helps to bring a more organic feel alongside the digital synthesis. Image data decides if the sample will play, a simple yes or no, then the sample is fed into a delay which also has it’s parameters decided by the image data. Most of the field recordings are from my own collection of random recordings, sounds from around my town or on trips – I always try to collect sounds along the way.

While the sound plays the system is listening to its own output, those measurements feed straight back into the visuals. When the music swells, the image warps and smears more. Heavy bass makes the picture pulse, brighter notes split the colours apart at the edges. So the picture decides the sound, and the sound reshapes the picture, around and around. The image you see is being conducted by the music, and the music is being read off the image. Then twenty-two seconds later a new photo arrives, and the whole thing begins again.

I am still “tuning” the whole system as the sound can get a little unruly, particularly in the high end, but overall it’s in a good place so please check it out and post some feedback. Next I would like to introduce a time of day influence (like Loom) and explore the Web Audio API to see what other types of synthesis are available to vary the sound between images.

Try Signal Field →

Related Posts