Building an audio visual environment

In my last post Creating a generative ambient record that behaves like an ecosystem we started to flesh out some ideas for a new audio visual project, something that exists between an audio visual installation and concept album.

There are 3 key elements to this project - environment, audio and visual. Building the environment felt like the natural starting point as this will influence the sound and visuals and how they interact.

It took a lot of late nights to work out how this whole ecosystem could function. I’ve managed to get the fundamentals built and wanted to go over some of the core decisions that have shaped it so far. It is still very much a work in progress but it is moving in the right direction.

When you press play, the system creates a seed, a single number that acts like the DNA of that session.

The seed is used to initialise a deterministic PRNG stream, a repeatable flow of random numbers generated from a single starting value (the seed).

Once the seed is set, the stream produces the same sequence of values every time, so a listener can enter the seed to return to the same starting point. From there, the sound and visuals will always evolve in unpredictable directions. The seed is like a portal back to a particular unfolding.

Nothing is chosen by hand after that point. Instead, the seed guides every decision the system makes. It doesn’t dictate exact outcomes, but it sets the conditions in which things can happen.

From this one value, the system decides things like:

• Which sounds are allowed to appear

• How many layers can exist at once

• How long sounds live before fading away

• The overall mood and density of the piece

• The colour palette, motion, and pacing of the visuals

The same seed will always produce the same world. A different seed creates a completely new one. The seed doesn’t generate chaos, it creates structure. It defines ranges, probabilities, and tendencies, then lets the music and visuals unfold slowly within those boundaries.

Think of it less like random noise, and more like setting the weather at the start of a day. You don’t control each moment, but the conditions shape everything that follows.

To layer audio files into something cohesive and create rules for the system, we need to organise the sounds.

I am still working through the larger creative concept for the music but it will be ambient driven, including drones, field recordings and textural layers as a base for other sounds to build on. Grouping the sounds helps define some of the logic e.g. I want an exclusion list that ensures certain categories or tracks don’t play together.

In terms of architecture it looks like this:

Sounds live within a timeframe based on the listeners local system time:

• Dawn (05:00 - 08:59)

• Day (09:00 - 16:59)

• Dusk (17:00 - 20:59)

• Night (21:00 04:59)

Each sound has a category:

• Beds (long) - Fundamental tracks that anchor the system

• Movement (mid) - Layers shift perceptually without becoming busy

• Details (short) - Appear and leave before they become repetitive

Within each category, sounds are defined by types:

• Drone

• Field

• Bass

• Texture

• Motif

This creates some options for how tracks might be filtered and selected.

Within each type there are 3 tiers of rarity. Rarity affects selection weighting only.

• Common - Frequently played

• Uncommon - Played less often

• Rare - Rarely played
  

Check the users local time and grab X amount of tracks from the relevant timeframe to create the initial audio pool.

Check for tracks that are not allowed to play based on the following criteria:

• Already active - Cannot pick a currently playing file

• Still in cooldown - It ended recently so is temporarily ineligible

• Excluded - Cannot play when a currently active file is playing
  

Each eligible candidate gets a weight:

• Long unheard boost - Gradually increases the longer it’s been since last played.

• Rarity weight - Common, uncommon, rare

Then the PRNG (seeded) makes a weighted random choice. So rarity directly changes how often something appears relative to other eligible options.

After a track is chosen and loaded, the category determines how long it will play:

• Bed (drone, field, bass) - Long lifespan + long cooldown

• Movement (texture) - Average lifespan + average cooldown

• Detail (motif) - Shorter lifespan + shorter cooldown

Every few minutes the system checks to see if the maximum amount of layers are active, if there is room it begins the process again to select another track.

The seed is used to determine which image is loaded based on the users local time (dawn, day, dusk and night). It uses the Web Audio API’s Analyser node to measure the overall amplitude of the sound playing at any time.

The amplitude is then used to modify the speed at which the effects are applied to the image e.g. If I use pixel swapping – the more dense the sound, the faster the pixels will shuffle on screen, this creates the sense that the visuals are breathing with the audio. I will add some attack/release to make this process feel more organic.

So far I have some of the logic built and working and a basic UI which is enough for me to test with. I’d like to come up with something more interesting for the UI, so my next post will focus more on the visual part of the project.