So your music helped train an AI model

The Atlantic piece on music datasets gets a bunch right - it’s fair to assume if your music meets the grade of a curated 12m dataset it has been included in big general models (all ours is in there).

It also misses a lot of important context. I’ve spent near a decade on this issue so apologize in advance for burdening you with the baroque complications. They are annoying, and important.

I wrote this extremely quickly, feel free to AMA.

For the sake of brevity, lets say there are 4 main reasons a musician might care if their work is used to train music models:

1. The music model may compete with your original music

2. You want to be paid for the use of your data

3. You are concerned about people using models to impersonate you

4. You would prefer AI models be banished from the universe

The piece alludes to this, although does so in an imprecise way.

The common argument goes like this: a company used your music to train a model that then may flood the market with songs that your music has to compete with.

Seems a clear case of infringement, your music was used to compete with your music! For better or worse, copyright law is much more complicated.

Copyright law does not protect you from general competition. It protects you from someone copying significant parts of your song and selling that copied material commercially. If, as is the case with general models like Suno, the songs being output from their model are defensibly new, you are no more protected from that competition as you would be from any human song on the market.

There is recent legal precedent establishing this. In Bartz v Anthropic, despite Anthropic admitting to buying and scanning second hand books (and pirating books they had to pay a hefty fine for), the judge found the AI outputs to be “spectacularly transformative” and as such, fair use. The same would apply to almost all AI music outputs. No copying / no competition. This will be contested, but they have a point.

The Atlantic piece shows examples of very popular songs being somewhat fuzzily reproduced by Suno. A novice reader might assume it is just as easy to produce fuzzy reproductions of their song in that same model. There is a great deal of research on this, and as things stand consensus appears to be that this phenomenon only occurs with songs that are drastically overrepresented in the dataset. So if an Ed Sheeran song appears in tens of thousands of youtube videos scraped for training, the model may overfit to reproduce passages of that song.

Its important to clarify two things here:

1) You aren’t Ed Sheeran

2) AI companies will argue this is a fringe failure of their model that they will likely have dedicated resources to address. It is in no-one’s interests for their model to reproduce popular songs.

We likely have a great deal of music in these models, and trust me I have tried all manner of tricks to reproduce our music in Suno for fun. I haven’t succeeded. You won’t either.

On principle I like the idea of data exchange being consensual, and worked on it for many years. For qualified reasons I won’t bore you with, I feel pretty confident that ship has sailed, at least when it comes to the pre-training of these large models.

Now, payment.

Suno openly admits to training on any recording of sufficient quality they can get their hands on. We don’t know exact figures, but it is safe to assume that number could be in the range of 30-50 million recordings.

For the purpose of argument, lets say 30m songs. We probably have 3 albums in these models, so lets say 30 songs.

How much should we be paid for that data?

If everyone got paid the same (which is never going to happen. I repeat, you are not Ed Sheeran), if we got $1 per song, we would receive $30 for our music, and Suno would need to spend $30m to license the whole dataset.

I doubt anyone is going to be happy with $1 per song. So lets start negotiating up!

Suno to date has raised $775m, lets drain them of all of it!

That equates to $25.83 per song. I’m still not happy!

The latest round of funding valued Suno at 5.4 billion dollars. Lets drain them of all of that! I don’t care if that’s not how company valuations work!

Even in that preposterous scenario, that would equate to a license of $180 per song, or for our 3 albums a licensing fee of $5400.

More acceptable, however you now have to factor that one company now has exclusive license to forever compete with you in the market. Your situation has not changed significantly, beyond making one average month worth of salary. To be clear, it is much more likely you will make $30.

I am very very open to having someone present me different math for this.

Some people will argue this is beside the point. Consent matters even if the payment is small. I understand that position.

I simply do not see governments putting barriers in front of what may be the most significant technological development of our time for policies that are unlikely to meaningfully improve the lives or incomes of most artists. Whether that is fair or not is a separate question.

It is equally reasonable to ask whether it is fair to impose broader economic costs on an entire country in pursuit of a principle that may ultimately return very little to almost all artists.

AI licensing is also a very different beast to other forms of licensing, because once a company has enough data to make a great model, they don’t need any more data ever again!

In the past couple of months, a team of researchers released ACE-step 1.5, an open weights model (you can download it to your hard drive) that many argue is competitive with what Suno sounded like around a year ago, particularly for pop music styles.

Crucially, ACE-step claims to be trained on a mix of licensed, royalty free and synthetic music.

Synthetic music is music generated by AI models for the purpose of training new AI models. As music generated by AI models is currently not covered under copyright, it is fair game to do so.

I played with it a little, and was initially underwhelmed as I don’t have much use for the generic sounding pop music is excels at.

Then I encountered this:

A user called mrcode had uploaded a LoRA, which is a means of customizing the outputs of an open model, presumably trained on the works of the Arctic Monkeys. I think it is fair to conclude this generation is a very convincing rendition of what it might sound like if that band covered an Elliot Smith song.

a model trained entirely on licensed datasets, and I am getting remarkable results from customizing it at home. You should try it, it’s remarkable.

2) unlikely to happen given how good the open model ecosystem, and synthetic training pipelines, are getting

I have already addressed how I find the examples of Ed Sheeran-ish generations from Suno are not very instructive.

However impersonation of the type I just demonstrated, where it is increasingly becoming trivial to mimic the style or voice of others, is a real concern.

Fortunately, there are legal frameworks around personality and publicity rights, even if enforcement will be uneven. Suno and other model companies will not permit anyone to type your name into their model and return songs that sound like you. If they did, they would be in big legal trouble, as personality rights give you the unique ability to profit from your name. Judging from how many people pay for Suno absent the ability to do that, it appears they don’t need to.

I am confident over the next 12-18 months we will begin to see labels and model companies explore options to allow for people to remix and spawn new works based on the named styles and voices of popular artists. They have little choice but to experiment with it, and it is a move I support.

We first proposed this category (we call it “Identity Play”) with our 2021 Holly+ project, and the logic is that while technically anyone will be able to mimic your style or voice alone in a basement somewhere, it will ultimately be more rewarding and beneficial to instead collaborate with your favorite artists and expose whatever you make to their distribution network of fans and labels. In the old world language, even though anyone can post an unofficial remix to youtube, it is much more likely for anyone to engage with that song if the artist themselves co-signs and publicizes it.

See Holly+ here

So while I feel many should take the prospect of impersonation seriously, I think it is just being real to conclude that the only way out of this is through, and maybe through experimentation we might find its quite interesting.

Having tried to patiently reason through all the complications of the AI culture war for a decade, when you finally boil down what many people’s objections to this area are, you find an earnest sense of unease with the field of AI more broadly. Conversations may start with an issue like training data, or deepfakes, but end in a genuine desire to see this all go away.

Who am I to tell you that is wrong? I am personally very optimistic and excited by everything changing, but I also appreciate this is a lot of change very quickly. It is a lot of new concepts that break old concepts. Most art and music people post has always been bad, but the volume of bad art and music enabled by infinity machines is a lot to stomach.

It doesn’t help that very few people are giving artists and musicians any clear advice as to how they might weather this transition. It also doesn’t help that almost everyone commenting on these issues suffers from an extreme lack of context. Most people in art and music know very little about AI, and most people in AI know very little about art and music. Some people are too eager to exploit that gap to panic and confuse people. The topic of AI is an emotional attention monster.

For the purpose of clarity, if anyone gives you the impression that AI will somehow go away, run very quickly away from them.

You do not have to change what you do, or adopt AI tools that disinterest you, and I am steadfast in my belief that music and art is a whole lot more complex and vital than the impressive ability to generate pleasing media.

However this is not going away. We are witnessing possibly the biggest infrastructural buildout in recorded history. AI policy will dominate elections in the coming years as countries compete for prime position in the next economy.

The math of data licensing is not going to get less complicated. The big record companies in court trying to broker deals with the big model companies have no interest or incentive to regulate AI out of existence.

No cavalry is coming. This is something we will have to deal with, and it may not be so bad so long as we can at least be on the same page about what our options are.

So rather than getting upset about the potentially lost cause of training data, maybe this is a time to think about what you want the world to look like, factoring we will live with increasingly powerful AI models at our fingertips, should we choose to use them.