Resonance Inference
In this mini-experiment, I further explore whether playable “instruments” can be extracted from a single audio recording by overfitting a simple model of resonating objects and a sparse control signal. Read more:
Zounds
Zounds is a blog about analyzing, synthesizing, and indexing audio.
Latest posts
Iterative Decomposition Model V3
There’s a new demo page for v3 of the iterative audio decomposition model I’ve been working on. The model is small and hasn’t had much (less than 24-hours) of training time so far, but doubles the duration of audio segments from the prior model . You can read more and listen to demos here
Playable State-Space Model From a Single Audio Sample
This work attempts to reproduce a short segment of “natural” (i.e., produced by acoustic instruments or physical objects in the world) audio by decomposing it into two distinct pieces: A state-space model simulating the resonances of the system a sparse control signal, representing energy injected into the system. The control signal can be thought of as roughly corresponding to a musical score,…
Gaussian/Gamma Splatting for Music
In this work, we apply a Gaussian Splatting-like approach to audio to produce a lossy, sparse, interpretable, and manipulatable representation of audio. We use a source-excitation model for each audio “atom”, implemented by convolving a burst of band-limited noise with a variable-length “resonance”, which is built using a number of exponentially decaying harmonics, meant to mimic the resonance of…
Sparse Interpretable Audio Model V2
Happy Leap Day! I’ve just published some high-level details and sound reconstructions from a newly-trained model that decomposes musical audio into an easy-to-manipulate format: event times and amplitudes time and amplitude-agnostic event vectors that describe a musical event A global context vector that determines room reverb Different “axes” of the sound can be manipulated independently, and…
Sparse Interpretable Audio Model
This post covers a model I’ve recently developed that encodes audio as a high-dimensional and sparse tensor, inspired by algorithms such as matching pursuit and dictionary learning. Its decoder borrows techniques such as waveguide synthesis and convolution-based reverb to pre-load inductive biases about the physics of sound into the model, hopefully allowing it to spend capacity elsewhere. The…
A Music Vocoder Using Conditional Generative Adversarial Networks
The last couple of posts have been all about audio analysis and search but in this one, I’ll return to some work that gets me a little closer to my ultimate goal, which is building synthesizers with high-level parameters, allowing the production of audio ranging from convincingly-real renderings of traditional acoustic instruments to novel synthetic textures and sounds. There’s clearly a lot of…
Cochlea: A RESTful API for Annotating Audio
Cochlea is an early-stage, RESTful API that allows users to annotate audio files on the internet. Segments or time intervals can be annotated with text tags or other arbitrary data. This may not sound very exciting on its own, but I believe that these simple primitives make possible incredibly diverse applications tailored to the needs of electronic musicians, sound designers and other folks…
Audio Query-By-Example via Unsupervised Embeddings
A couple of months ago, I gave a talk at the Austin Deep Learning Meetup about building Cochlea , a prototype audio similarity search engine. There was a lot to cover in an hour, some details were glossed over, and I’ve learned a few things since the talk, so I decided to write a blog post covering the process in a little more detail. Motivations and First Steps There are countless hours of audio…
Perceptual Audio Loss
Today, I perform a small experiment to investigate whether a carefully designed loss function can help a very low-capacity neural network “spend” that capacity only on perceptually relevant features. If we can design audio codecs like Ogg Vorbis that allocate bits according to perceptual relevance, then we should be able to design a loss function that penalizes perceptually relevant errors, and…