AI Music Generators - how they work and why it matters

In this article, co-authored with my colleague Raz Mesinai, the third in our series on AI music generation, we look at the fundamental architectures of platforms like Suno, Udio and Sonauto (now Treblo), their differences, and how those differences impact the artist’s ability to use them.

Let’s look first at how these systems take in music and sounds, like the tens of thousands of songs they used to train, with or without authorization. As the systems ingest the audio content, they tokenize it, just as an LLM tokenizes the text it is fed. The tokenization doesn’t involve “listening” to the music in any linear way we’re accustomed to but instead involved reducing it to symbols representing mere fractions of a beat. As the waveform arrives, its tens of thousands of amplitude samples per second are compressed into just 50–70 short frames per second, each frame a compact vector summarizing a sliver of sound perhaps 15–50 milliseconds long, far less than anything a human would recognize as a coherent sound.

The component that does this, the codec, is trained first on audio, with reconstruction and adversarial losses until it can compress → quantize → decode (back into the original audio) faithfully. This is the stage where it learns for itself which features are worth keeping.

The result is a codebook: a fixed dictionary of a thousand-odd sound-unit prototypes, against which any audio can be quantized into discrete tokens. Each prototype is a vector in a space of about a hundred dimensions (e.g. 128) with that space being how the system “understands” music: the vocabulary it reasons with. A visual representation, flattened to just two of those dimensions, might look like this:

What is fascinating, and simultaneously a bit spooky, is that the vocabulary that is used, and the dimensions along which that vocabulary exists, has no real bearing to the way we understand music. The genres no doubt land in different spaces, but the exact way in which everything is characterized, just as in LLMs, is unknown and in some sense incomprehensible to us as humans. In other words, the way AI music generators “understand’ music, likely has little correlation with the way we experience it.

So a song, in the context of an AI music generator, is the path the frames trace, stringing together the elements it "understands" to assemble the track. Individually, those elements rarely resemble anything we'd name, a booming kick, a subtle jazz hi-hat, a searing guitar, a faint flute; each is a far shorter, sub-instrument sliver of texture, and a recognizable sound emerges only across a whole run of them. Yet finite and lossy as this vocabulary is, it proves expressive enough to reconstruct almost any music, and to generate it anew.

Neither Suno nor Udio publishes how its models are built, but the available evidence, namely Suno’s open-source generative text-to-audio model, Bark, and its CEO’s stated preference for autoregression, points squarely at the token-based approach described above: a transformer that generates a song the way a language model writes a sentence, one sound-chunk at a time, left to right.

Generation begins not with sound but with words. The text prompt, and any lyrics, is first expanded by a language model into a rough plan including genre, tempo, key, a map of sections, all encoded into a conditioning vector. That vector doesn’t dictate the music but rather biases it. At every step the acoustic transformer predicts the next codec token conditioned on two things: the entire sequence of tokens it has already produced, and that prompt conditioning. The prompt, in effect, tilts the odds by nudging the path toward the hip-hop region of the vocabulary rather than the folk one, toward a build rather than a breakdown.

The song is then traced out one token at a time. The model predicts a token, appends it, and feeds the lengthened sequence back in to choose the next. In this way the path advances step by step, each new element selected with the whole prior path in view. This left-to-right commitment is the source of both their strength and their limit: because the model always sees everything that came before, it handles long-range structure well: the crescendo that resolves into the drop, the verse that yields to a chorus. But because it commits as it goes, it cannot easily reach back and revise the middle without redrawing everything after. A final codec decoder turns the finished token path back into a waveform.

It’s worth pausing here, because the difference between these systems and the diffusion models we turn to next comes down to a single design choice. Both classes are built on the same skeleton, namely an autoencoder that squeezes audio into a compact latent (a compressed representation that captures the essential structure of the sound in fewer bits than the raw waveform) and rebuilds it. They diverge only at the bottleneck, the narrow middle where that squeeze happens:

• Suno and Udio use a quantized bottleneck. The latent is snapped to a fixed codebook, turning sound into discrete token. Discrete tokens invite a language model, so generation is autoregressive: the next token, then the next, drawn left to right.
  

• Stable Audio and Treblo keep the bottleneck continuous. A variational-autoencoder latent, smooth and un-quantized. This is exactly what a diffusion model needs to wander through, adding and then removing noise across the whole track at once.

Same squeeze, two kinds of waist. Everything that follows, tokens versus latent, one sound-chunk at a time versus the whole track all at once, falls out of that one decision.

On the continuous side sit the diffusion systems. Stability AI’s Stable Audio is the well-documented example while Sonauto (Treblo) is the notable challenger, whose founders openly describe ripping out the tokenizer and replacing it with a variational-autoencoder bottleneck, then training a diffusion transformer on the continuous latent that remains.

The generation runs backwards, from noise. The audio is compressed, not quantized, into a continuous latent, and the model begins with that entire latent filled with pure noise. Over many steps it denoises the whole thing simultaneously, the track resolving coarse-to-fine like a photograph developing, with the prompt conditioning and, in Stable Audio’s case, an explicit sense of where the track starts and how long it runs steering every step. There is no path being drawn forward; the whole song sharpens into focus together.

That single difference, global refinement instead of forward commitment, is what lets these models edit and control so naturally. Because the entire track is held at once, a region can be masked and regenerated in place, the second verse rewritten while everything around it is preserved exactly, and continuous parameters like tempo or key nudged mid-generation. The trade sits at the other end: holding everything at once makes long-range discrete structure harder, so the tight EDM build-and-drop, or the precisely-aligned lyric that the autoregressive models nail, can be harder to pin down. A final decoder turns the finished latent back into a waveform.

There are many others of course, the table below provides a snapshot overview, but puts aside the issue that many of these systems work quite differently at the operational level, with some only providing WAV file output and other providing access to underlying stems for music production. A few, like ACE Studio, provide for integration with existing digital audio workstations (DAWs) like Ableton Live.

Suno and Udio share a model that leans heavily toward popular music structures in music production and arrangements, and understands a lot of advanced flows for getting great effects at “the drop” (like a crescendo and then a drum and bass rhythm following). Their models are more quantized, where even if you purposefully attempt to do something unconventional or “weird” it generally results in a degradation of the sound and a shift in tempo, a warping effect similar to Ableton. This is undoubtedly a result of the autoregressive step-by-step model as contrasted with the diffusion model and tells us something about the fundamental differences in the models.

The model used by Sonauto, called Melodia, leans a bit in the other direction, allowing for more creative and unconventional approaches to composition and generation, although it is clear that it is not specifically made for that. It can however, handle strange shifts in music styles and tempos. Because it relies heavily on latent audio diffusion rather than pure tokenized, left-to-right prediction, Melodia can map complex control layers onto the music. That being said it can be hard to get something solidly organized in that mainstream, EDM way.

Although not quite an apples-to-apples comparison, we prompted both Suno and Treblo to generate tracks based on, as well as describing, the history of, phonk, a subgenre of hip hop that emerged in the early 2010s in the United States. The tags and prompts, although not entirely uniform across the two platforms due to differences in the UI and guardrails, were based on phonk, doom metal, dark phonk and electronic.

The resulting Suno phonk history track is highly polished, biting, with the type of compression and mastering you would expect from an EDM producer, whereas the Treblo phonk history track is much more unpolished, weird, and free-format, not even picking up on the lyrics right away. It’s hard to separate differences in design from the actual architectural differences, but the the Treblo diffusion model seems to be much less constrained in its generation than the Suno autoregressive transformer model, which appears to spit out a regimented and polished track with little effort. Other tests we have performed suggest that the underlying architectures have a big impact on how the music is produced, with Suno being much less flexible from an artist’s viewpoint. Having said that, if you want to attempt to build a “banger factory” of bubble-gum pop or EDM tunes, Suno or Udio will do it for you. Of course good luck competing with the hundreds of thousands of tracks being uploaded to Spotify, Deezer and other streaming platforms daily as well as the roughly 7 million songs being generated on Suno every day, most of which (thankfully) never leave the platform.

Both of these models provide for music generation in an incredibly efficient and almost factory-like manner, with varying degrees of flexibility in terms of artist controllability. What’s also clear is that these are the most powerful samplers ever created, hence providing unlimited creative opportunities as well as being an unprecedented platform for misuse.

The copyright infringement suits are only one aspect of the opportunities for misuse AI music generators are creating. Adaptive music, which changes in real time in response to what is happening, such as in video games, is an active area of research, and a natural use for AI music generators, although it will take some time before they can do real-time generation. Adaptive music generation which incorporates monitoring of the user or crowd (e.g. emotional state) could lead to the possibility of music that can actively manipulate the listener. If you think social media addiction is scary, think about what AI based adaptive music manipulation could be used for.

Stay tuned (pun intended) for more on this in our next article.

In the meantime, check out some of these generators for yourself and share your tracks.