Why are open-source advocates up in arms over California’s AI bill?

California bill SB-1047 met with a relatively quiet reception when it was announced in February, so why did a sudden firestorm of criticism erupt in April? 

The bill, introduced by state Sen. Scott Wiener, aims to rein in AI models that pose an "unreasonable risk" of causing more than $500 million in damages. Now open-source advocates claim the proposed legislation would stifle innovation and competition. 

The controversy erupted after the accelerationist organization Alliance for the Future alleged the bill was being “fast tracked,” and Fast.ai co-founder Jeremy Howard published a post on the bill’s harms to the AI industry. Howard’s assertion that SB-1047 would "stifle open-source AI and decrease safety" was widely quoted on social media. Alliance director Brian Chau summarized the bill as “Close source your models or else.” 

The opposition is unsurprising coming from Alliance for the Future, an effective accelerationist organization. A recent addition to the political landscape around AI, the effective accelerationist movement wants to accelerate technological progress through the unrestricted development of artificial intelligence -- especially open-source AI. The accelerationists have generally gotten support from Silicon Valley VCs like Andreessen Horowitz who want to see their investments in AI pay off. The AI Alliance, founded by Meta and IBM, has also opposed the bill.

What is surprising is the social media frenzy that converged around the claims that the bill was “fast tracked” and that it would at best stifle (and at worst criminalize) open-source models. 

Supporters of the bill have pushed back fiercely. Senator Scott Wiener, who introduced the bill, accused  Chau of “outright fear mongering” and insisted that it is “absolutely false” to say the bill criminalizes open-source. Meanwhile, Dan Hendrycks, director of one of the advocacy groups sponsoring the bill, said the goal was to prevent open-source models where the safety training could be removed “extremely easily and foreseeably.” 

SB 1047 is supported by two of the “Godfathers of AI,” Geoffrey Hinton and Yoshua Bengio. Professor Hinton said, “Powerful AI systems bring incredible promise, but the risks are also very real and should be taken extremely seriously. SB 1047 takes a very sensible approach to balance those concerns.” 

A former California legislative staffer dismissed claims the bill is on a "fast track," noting it must clear multiple committee reviews, one of which prepared this 38 page analysis. 

The bill's text requires developers to refrain from open-sourcing models that pose an “unreasonable risk” of causing critical infrastructure attacks or autonomous actions causing over $500 million in harm. 

But advocates for tighter AI regulation argue it actually creates a loophole by limiting shutdown requirements only to models "within custody, control, or possession" of the developer. So an open-source developer would be required to shut down the copies of the model they were running, but not the copies that were being run by other people. 

The bill is still being revised based on feedback, and it is likely that the open-source provisions will be workshopped further as the bill advances. Senator Wiener has already announced that the bill will limit coverage to models costing at least $100 million to train. 

As of May 21, SB 1047 has passed the California Senate 32-1 with bipartisan support and now advances to the Assembly. There is will go through the committee process all over again. 

As the legislation advances through the state capitol, both sides are digging in for what could be a defining battle. The future of open-source AI — and public safety — may hinge on it.