I expected this to connect the first half of the essay (investing in contributors) to the second half (banning LLMs) more explicitly:
Feedback given to people submitting purely LLM-authored PRs is ~useless once that PR is done: you might improve that PR but it isn't going to make any difference for later contributions. The LLM isn't learning from the feedback and the person submitting the PR usually lacks the context (and frequently the desire) to internalize it for future work. To use the metaphor of the article, effort spent on feedback for new contributors is worth it when contributing is an iterated game, but because LLMs don't learn from feedback it turns it into a one-shot game, which changes the payoff drastically.
Related, I joined a new company this year that makes heavy use of LLMs. My team skews early career, and I was mildly anxious about how coaching/mentor-level PR feedback would be received on their agentic PRs. I like reviewing code, and I'm not sure how I'd adapt to it being moot at work.
Turns out they really appreciate it. Small sample set, anecdotal, etc, but they've expressed concern about honing skills in the new era, so feedback with rationale + examples speaks to them. And for the most part, the feedback is incorporated by default in future PRs.
Obviously open source contributors have very different incentives, but it was still refreshing.
This was really good - the best articulation I've seen yet of a rationale for a blanket ban on AI in contributions given a specific project's goals and organization design.
I appreciate the betting on human contributors. It seems like many places are shutting out people in favor of letting LLMs do the work instead. It's refreshing to see the opposite take place here. I'm confident this approach will provide more experienced maintainers and end up an excellent investment in the long-term.
So while one could in theory be a valid contributor that makes use of LLMs, from the perspective of contributor poker it’s simply irrational for us to bet on LLM users while there’s a huge pool of other contributors that don’t present this risk factor.
This reminds me of when companies think that they can get better candidates by making candidates jump through more hoops. "We have 1,000 applicants, so let's ask them to explain their high school grades to us, even if they graduated 20 years ago. The lazy candidates will give up, leaving only the best ones."
The developers least interested in jumping through hoops are the ones that make the most valuable contributions. If you tried to get Jeff Dean to collaborate with you and told him that he's not allowed to ever use LLMs, he'd happily go off and work on the 10,000 other projects that would gladly take a Jeff Dean, LLM-assisted or not.
It comes down to conflicting assumptions about where the top performers are. I believe the top performers are overwhelmingly using LLMs, and it sounds like the Zig team believes that's not true.
Unfortunately the reality of LLM-based contributions has been mostly negative for us, from an increase in background noise due to worthless drive-by PRs full of hallucinations (that wouldn’t even compile, let alone pass CI), to insane 10 thousand line long first time PRs.
This is a bad argument for banning LLMs. These PRs will continue to flow in regardless of what rules you put in place because the contributors generating these PRs don't read the rules. It's like making it illegal to own a ski mask because you notice that many criminals use ski masks to hide their faces.
It comes down to conflicting assumptions about where the top performers are. I believe the top performers are overwhelmingly using LLMs, and it sounds like the Zig team believes that's not true.
That's a legitimate opinion to hold, but our experience with triaging PRs pre-LLM ban does not match your assumption at all.
These PRs will continue to flow in regardless of what rules you put in place because the contributors generating these PRs don't read the rules.
That's a shortsighted argument. We don't expect the policy to automagically make LLM contributions disappear, but instead:
users are warned upfront about the consequences of using LLMs, which is in general good practice whenever banning is involved (i.e. users who read the rules should not be surprised when their behavior leads to a ban).
users who have good intentions and who like to use LLMs know to not waste time (and money!) on generating code that we don't intend to merge anyway.
There is literally nothing to gain by not informing users of how we operate wrt LLM-assisted PRs.
That's a legitimate opinion to hold, but our experience with triaging PRs pre-LLM ban does not match your assumption at all.
I don't understand how you'd be able to tell.
You'd be able to identify low-value contributors who use LLMs irresponsibly, but you can't identify someone who's both competent and uses LLMs effectively.
For example, if Mitchell Hashimoto didn't disclose his LLM usage, would you have assumed he's a high-performing non-LLM contributor?
These PRs will continue to flow in regardless of what rules you put in place because the contributors generating these PRs don't read the rules.
That's a shortsighted argument. We don't expect the policy to automagically make LLM contributions disappear, but instead:
users are warned upfront about the consequences of using LLMs, which is in general good practice whenever banning is involved (i.e. users who read the rules should not be surprised when their behavior leads to a ban).
users who have good intentions and who like to use LLMs know to not waste time (and money!) on generating code that we don't intend to merge anyway.
There is literally nothing to gain by not informing users of how we operate wrt LLM-assisted PRs.
That's not the argument I'm making.
I'm saying that the LLM ban filters out potential high-value contributors who use LLMs and does not effectively filter the behaviors you described (hallucinated APIs, hopelessly broken code). By banning LLMs, you lose high-value contributors who like LLMs, but you'll continue to receive negative value contributions from people who ignore contribution guidelines.
I don't understand your mental model of a user who contributes a 10 KLOC PR that doesn't compile who would have not done it had you had a rule against LLMs. You already have rules against the behaviors you described and the PRs come anyway, so why is an extra rule going to stop them?
I'm saying that the LLM ban filters out potential high-value contributors who use LLMs and does not effectively filter the behaviors you described (hallucinated APIs, hopelessly broken code).
In your Jeff Dean high school analogy, you're assuming a filter whose behavior does not correlate to the value of a hiring candidate.
What the author is saying in the article is that they have consistent evidence that there is a positive correlation between LLM-using authors and low iterated-value-over-time contributors.
Discarding hiring candidates because they don't want to talk about their high school record is one thing. Discarding hiring candidates because, say, they can't demonstrate an ability to write and reason about code is very different.
What the author is saying in the article is that they have consistent evidence that there is a positive correlation between LLM-using authors and low iterated-value-over-time contributors.
I think that's a valid position, but my objection is that based on what they describe in the blog post, the justification for the LLM ban sounds weak.
The evidence the blog post offers is:
They're getting a lot of obviously useless PRs that are AI-generated
They're getting a smaller number of legitimate-seeming PRs that later turn out to have issues because the developer used AI and didn't understand the code
Those facts alone aren't enough to justify a blanket AI ban because, theoretically, 100% of the good external contributions could be AI-assisted as well.
It's possible that the Zig team has more data on this than they've shared here, but my objection is that the blog post does not make a convincing argument for the AI ban.
because, theoretically, 100% of the good external contributions could be AI-assisted as well.
I mean, that's possible. And, sure, we could discuss all sorts of hypothetical scenarios that a hypothetical open source project could hypothetically be in. If one were discussing a meta-policy around whether all possible open source projects should have AI bans, that would be an important discussion to have.
But the Zig team only cares about their one real project and in that project, it seems like they see self-stated AI use being correlated with lower-quality long-term contributor value. Is it possible that actually many of the high-value contributors are actually using AI and being sneaky about it? I suppose it's possible. But Occam's Razor suggests that's pretty unlikely.
because, theoretically, 100% of the good external contributions could be AI-assisted as well.
I mean, that's possible. And, sure, we could discuss all sorts of hypothetical scenarios that a hypothetical open source project could hypothetically be in.
Hi Bob, I'm a big fan of your work, so I find it disappointing that I tried to engage you in good faith and it feels like you're ignoring everything I said to focus on an uncharitable read of a minor detail.
I don't mean that it's important to consider the possibility that 100% of good contributions are LLM-assissted. I'm saying that we don't know the actual number and the blog post doesn't address it at all.
I don't think it's a wild hypothetical that 30-50% of good Zig contributions involved LLM assistance. And if the Zig team is happy to sacrifice those 30-50% or they think it's far lower, that's fine, but my point is just that the blog post would make a stronger argument if they focused not only on how LLMs produce noise but addressed the tradeoff of losing some percentage of good LLM-assisted contributions.
Is it possible that actually many of the high-value contributors are actually using AI and being sneaky about it? I suppose it's possible.
What's sneaky about using AI to contribute to Zig? Prior to the ban, there was no requirement to disclose AI usage, so I find it likely that there are good Zig contributors who used AI assistance.
I don't think it's a wild hypothetical that 30-50% of good Zig contributions involved LLM assistance.
It is. Like what are you basing it on? Because it flies against all data points we got from open source commiters and maintainers over the past 2 years. If this is your base model, you are going to have to do a lot of work to prove it before it can be used for this kind of analysis.
The point isn't proving an exact number (which I don't think is possible with the information available) but just finding out what the disagreement is.
If we agree that 30-50% is a reasonable estimate of how many good contributions used Zig, then what we disagree about is whether driving away 30-50% of potential contributors is a worthwhile trade in banning AI.
If we disagree about 30-50%, then it doesn't make sense to talk about whether the tradeoff is worth it because we have upstream disagreement that makes discussion of the tradeoff pointless.
My 30-50% is based on 90% of working developers I know personally now use LLMs, and most of the developers whose work I follow through blogs now use LLMs.
90% of working developers I know personally now use LLMs
Yes, that's going to happen when token usage is tracked, and factored into promotions. Ask if people think they're doing their best work, and you get a more interesting story.
AI is a pretty hated technology outside of some bay area and founder bubbles.
90% of working developers I know personally now use LLMs
Yes, that's going to happen when token usage is tracked, and factored into promotions. Ask if people think they're doing their best work, and you get a more interesting story.
Everyone in the 90% I know finds LLMs net positive to various degrees.
I only know of two developers whose employers pressure developers to use LLMs, but they're both on the high end of positive attitude towards LLMs, so they'd be using it anyway.
As additional anecdata, of the 7 developers I'm good friends with socially + me (all but 1 east-coast, outside of the bay area), 2 are required by their job to use AI, 2 are encouraged to do so. Of the ones required to use it, one is all-in, the other uses it sparingly as it is only sometimes useful for them. Of two that are encouraged to use it, one is positive and one negative. Of the remaining 4 who are not influenced by their job, they generally don't use it, either because they're skeptical or because they don't find it very useful.
Notably the one both required to use it and all-in on its use is the only one working at a Bay area startup.
I sincerely encourage you to find more communities to interact with. You seem to have fallen down the rabbit hole of programming "influencers" which is most definitely not representative of any widespread trends.
As a pro-craft activist who fights against the dangerous proliferation of LLMs, the vast majority of programmers I know are somewhere on the spectrum of fairly-critical to total abstention (like myself). They work on all sorts of commercial and open-source projects, some of which are quite well known in their respective fields. Am I claiming my anecdata is representative? No. I'm simply saying your anecdata is no better than mine.
Honestly, my best guess right now is that pro-AI sentiment across all developers is at best around two-thirds, and that percentage is actively falling. I expect it to dip below 50% as the bubble continues to burst and inference costs skyrocket.
I'm surprised you don't know more than 2 people at Google or Microsoft, to pick two well known, large companies heavily pressuring their employees into LLM use.
I deleted a longer draft of this comment because this discussion has been long, but I do just want to mention that Zig's no-AI policy went into effect in July 2025. (https://github.com/ziglang/zig/wiki/Contributing/61aff1ce9b3ebd8c38c0f5a18db00c06f0fa4022) Claude 4 had just come out and 3.7 was not good at writing Zig. With that context I don't think "30-50%" of good contributions were AI assisted.
Presumably, if they existed they could speak for themselves.
There is no point for anyone who would use LLM existence to speak for themselves because they are not permitted to contribute to the project for a while now. The policy is quite black and white.
It seems very strange to me to imagine that there is a significant group of developers who fulfil all of these criteria
I would take the position that the number of people is greater than zero but I'm not going to argue either way. All I can tell you that some quite prominent agentic coding figures are very happy Zig users and experienced developers. That's why this topic is somewhat interesting because Zig has a hard anti AI policy but the user base has some prominent agentic engineering users.
I know that those people exist and my personal feeling is this: If they now feel unable to contribute to Zig despite wanting to because they are unwilling to forego LLM usage, even only while contributing, then I worry about them and their long term health.
That seems entirely unnecessarily judgmental but since this is your personal feeling, you can have those.
I just responded to your initial comment because you said they could speak up for themselves and I think in the current configuration (AI being banned for more than a year), you won't find anyone who can speak up because they a) either are breaking the rules and won't say or b) do not contribute any more / won't start.
I don’t mean it adversarially, of course. I am sincerely concerned about a lot of people (I also read your earlier post where you expressed similar feelings).
edit: My concern is that LLM usage seems addictive in ways very similar to gambling.
I don't think I'm ignoring what you said, but I think you are giving them an uncharitable read.
They know their community way better than you or I ever will. If they feel they have strong signal that LLM use correlates with poor long term contributor value, I trust they know what they are doing.
I'm saying that the LLM ban filters out potential high-value contributors who use LLMs and does not effectively filter the behaviors you described (hallucinated APIs, hopelessly broken code). By banning LLMs, you lose high-value contributors who like LLMs, but you'll continue to receive negative value contributions from people who ignore contribution guidelines.
A policy by itself can never filter bad actors, duh. Enforcement is always a necessary complementary aspect. Same as with laws, they still need enforcement. But you still probably wouldn't want to live in a lawless society, especially when some people have powers that you don't have.
For example, if Mitchell Hashimoto didn't disclose his LLM usage, would you have assumed he's a high-performing non-LLM contributor?
If by construction somebody would be able to hide their usage of LLMs, then yes the only possible conclusion is that we wouldn't be able to notice (again, by construction) and would happily merge that code. If this happened in the Zig repo, then yes, those PRs might have been merged, and if those contributors decided to stop contributing after the AI ban, then those would be lost hands at contributor poker.
A policy by itself can never filter bad actors, duh. Enforcement is always a necessary complementary aspect. Same as with laws, they still need enforcement. But you still probably wouldn't want to live in a lawless society, especially when some people have powers that you don't have.
You're not addressing the point.
Zig already had policies that told people not to make 10 KLOC PRs that broke CI, but they did anyway. If policy is meaningless without enforcement, then why not enforce the rules that they were already violating?
If by construction somebody would be able to hide their usage of LLMs, then yes the only possible conclusion is that we wouldn't be able to notice (again, by construction) and would happily merge that code. If this happened in the Zig repo, then yes, those PRs might have been merged, and if those contributors decided to stop contributing after the AI ban, then those would be lost hands at contributor poker.
Okay, I'm glad we agree on that.
My critique is that the blog post would have made a stronger argument had it addressed the potential loss instead of focusing only on bad usage of LLMs.
By banning LLMs, you also lose out on potential good contributions. I can understand if you did that calculation and decided the tradeoff was worth it, but when the post says only that LLMs are banned because there was bad usage, it gives the impression that the decision was driven more by negative feelings toward LLMs than a rational evaluation of the tradeoffs of allowing or disallowing a popular tool.
Aside: I find "duh" as a response disrespectful, and if you continue being disrespectful when I'm giving you respectful responses, I will stop engaging with you.
Prior to the LLM ban, theres's nothing duplicitous about using LLMs and not disclosing it. I don't disclose my whole hardware and software stack as part of every open source contribution I make .
And after it, there is. If you use an LLM to contribute, your contributions aren't welcome. If you use an LLM to contribute and lie about it, you're an asshole. While I'm not involved in the zig community, so I can't speak for them, but if you were discovered, I imagine you would be unwelcome as well.
For whatever reason, when people say "I don't want you to spit on my sandwich", nobody goes "but I could do it so you'd never find out!". When they say "I don't want you to spit in my code", everyone who isn't contributing seems to come out of the woodwork to volunteer to secretly spit in the code.
And after it, there is. If you use an LLM to contribute, your contributions aren't welcome. If you use an LLM to contribute and lie about it, you're an asshole.
That's not relevant to the current discussion.
kristoff claimed "our experience with triaging PRs pre-LLM ban does not match your assumption at all."
I questioned whether he can make generalizations based on all LLM-assisted contributions when it seems like he's only sampling the contributions that were discovered as LLM-assisted because they were bad.
he's only sampling the contributions that were discovered as LLM-assisted
No, he is also sampling contributions where the contributor volunteered LLM usage.
The only undiscovered cases are contributions where the contributor used an LLM and created a high-quality contribution but did not disclose usage. Presumably that contributor then did not become a long-term contributor or alternatively stopped using LLMs, since they would then be asked to disclose further usage later.
Either way, this group of contributors are probably both small and not particularly valuable to the project long-term.
edit: Also, nothing is preventing such a contributor from contributing further, as long as they don't use LLMs. Someone who wants to contribute but is completely unable to do so without LLM assistance doesn't sound "most valuable" to me.
The only undiscovered cases are contributions where the contributor used an LLM and created a high-quality contribution but did not disclose usage. Presumably that contributor then did not become a long-term contributor or alternatively stopped using LLMs, since they would then be asked to disclose further usage later.
This is possibly true, but I'm disagreeing with the argument in the blog post, which doesn't reflect the claim you're making here.
My claim is in response to your claim. But sure, returning to the blog post I think it's pointless to speculate about potentially turning away good quality LLM-assisted contributions when the only problem at hand is an abundance of low quality LLM-assisted contributions.
There's a problem with this line of thinking, which is that pushing away good contributions is 99% not "at hand". Instead of contacting you and saying "I was gonna get into your project, but the anti-LLM policy put me off," the person just goes somewhere else. As such, it's a problem of unknowable size.
It's not unknowable, though! Measure the rate of high quality contributions and new contributors before and after the LLM ban. If the rate is significantly lower after the ban that would indicate that this theory has some weight.
There are two facts that seem to be universally true:
Every model released before the latest ones was absolute garbage. If you have tried to use LLMs but did not use the latest models you have no idea what's possible. (Illustrative example: The post I am replying to)
The latest model is not as good as the previous model, the previous model was better. (Illustrative example: Every single post on reddit.com/r/claude)
No one is asking you to disclose your whole stack, but in this case the Zig project is asking you to disclose your LLM usage. That is what makes non-disclosure duplicitous.
You and orib keep digging into this line of reasoning, and I'm confused if I'm missing something.
kristoff claimed that they banned LLMs based on behavior they observed before there was an LLM ban (i.e., before anyone was expected to disclose LLM usage). I'm arguing that before the ban, they don't have accurate data about who was using an LLM and who wasn't. The data is biased toward bad contributions + LLM usage because they're obviously detectable, whereas you can't tell whether good contributions involved LLM assistance.
The two of you keep saying, "If you use LLMs now, then you're dishonest," but how does that relate to whether the ban was justified in the first place?
It's a common misconception that we can't tell who is using LLM and who is not. I'm sure we didn't catch 100% of LLM-assisted PRs over the past few months, but the kind of mistakes humans make are fundamentally different than LLM hallucinations, making them easy to spot. Furthermore, people who come from the world of agentic coding have a certain digital smell that is not obvious to them but is obvious to those who abstain. It's like when a smoker walks into the room, everybody who doesn't smoke instantly knows it.
I'm not telling you not to smoke, but I am telling you not to smoke in my house.
the kind of mistakes humans make are fundamentally different than LLM hallucinations
This is saying "if theres an error, we can tell if its LLM or human". But it doesnt include the case when theres no errors, which contradicts your original claim:
It's a common misconception that we can't tell who is using LLM and who is not.
Hi Andrew, big fan of Zig and your work, and I've been a (small) Zig financial contributor for two years.
That said, I'm having a hard time believing this:
It's a common misconception that we can't tell who is using LLM and who is not.
You don't have ground truth, so how can you say this with confidence?
I'm sure we didn't catch 100% of LLM-assisted PRs over the past few months, but the kind of mistakes humans make are fundamentally different than LLM hallucinations, making them easy to spot.
I get that you can spot when developers submit AI-generated code and don't thoroughly review it themselves, but what if the developer uses an LLM for assistance but still refines it by hand so that the result is the same as hand-coded?
The way I use LLMs to contribute to other people's open source projects is that the result should look the same as hand coding. I don't think I'm that unusual in that respect. But maybe I'm one of the people you're describing that has a blind spot to their own anti-patterns.
Interviewer: What is your area of expertise?
Homer: I can tell the difference between butter and I Can't Believe It's Not Butter.
Interviewer: No, you can't, Mr. Simpson. No one can!
I think people are underestimating just how many hours of my life I've spent reviewing PRs. I'm telling you there are obvious, noticeable patterns - even when there are no "mistakes". It's usually quite obvious.
But like I said it's not foolproof. Ultimately I want an invitation tree like lobste.rs in order to enjoy a higher degree of trust with first-time contributors. As is, it has become toxic. I hate being the cops.
Yeah, I couldn't agree more. A lot of people who crank out code don't understand the massive shift that reviewing code brings. It's an entirely different skill and it's just as rooted in interpersonal skills as it is technical skills. The article y'all shared tried to explain it and it still went over people's heads. Reviewers can tell for the same reason people can tell when they're talking to a chatbot instead of a real person, the tells are technical and personal.
I also like the idea of an invitation tree, possibly balanced out with mentorships for enthusiastic newcomers.
andrewrk already commented a link to this post by mlugg, which says that “AI is entirely besides the point” and that mlugg disagrees with the architectural redesign that Bun did to get that speedup. You predicted that Zig would never see that speed increase, but that post says “we are going to make these enhancements, but […] we’re approaching the problem with the care it deserves, so that when we ultimately ship it, we don’t cause regressions.”
Therefore, the Zig project not copying those Bun changes is not evidence that the Zig project will never accept AI-generated code.
Look, I love software and the craft of it, so I get the concern. I just think we need to base opinions on evidence and not beliefs or conspiracy theories.
I would say that your argument is technically correct, which the best kind of correct. Yes, it is possible that before the LLM usage ban, there were a lot of good developers who wanted to contribute or maybe even did contribute without disclosing any use of LLMs. After the ban, those developers then all decided to stop trying to contribute, and also not to tell anyone about that. There are also a possible category of very skilled, very capable developers who came to Zig after the ban, want to contribute, are turned away by the policy and do not let Loris know about this.
This is of course possible but at least to me seems highly implausible.
For example, if Mitchell Hashimoto didn't disclose his LLM usage, would you have assumed he's a high-performing non-LLM contributor?
I don't think Mitchell Hashimoto would be dumb enough to contribute LLM-generated code to a programming language where bad code and the failure to find it in the PR could have far-reaching consequences for both the compiler and every user of it.
This idea that the best "performers" (whatever you actually mean by this) are using LLMs and that they couldn't possibly make contributions without them doesn't make a whole lot of sense.
If they wanted to get code into Zig and they were capable, they'd simply not use LLMs to contribute and that would be it. If they can't, or don't actually want to contribute to Zig, they won't. There needs to be no speculation as to who can and wants to contribute to Zig.
Basing your policies on wild hypotheticals that directly contradict your actual evidence and explicitly (very obviously) contradict your actual goals would be an incredibly bad move.
For example, if Mitchell Hashimoto didn't disclose his LLM usage, would you have assumed he's a high-performing non-LLM contributor?
I don't think Mitchell Hashimoto would be dumb enough to contribute LLM-generated code to a programming language where bad code and the failure to find it in the PR could have far-reaching consequences for both the compiler and every user of it.
This is like saying, "Mitchell Hashimoto wouldn't use Stack Overflow because he's too smart to just copy/paste code from StackOverflow without understanding it."
Using an LLM doesn't mean just blindly accepting its output without review. Competent developers use LLMs to assist in development and revise the LLM's output just like they can consult StackOverflow and write code based on what they learned.
This idea that the best "performers" (whatever you actually mean by this) are using LLMs and that they couldn't possibly make contributions without them doesn't make a whole lot of sense.
It's not that they couldn't but they don't want to because they perform better with them. Mitchell said in a recent blog post:
At this point I was firmly in the "no way I can go back" territory. I felt more efficient, but even if I wasn't, the thing I liked the most was that I could now focus my coding and thinking on tasks I really loved while still adequately completing the tasks I didn't.
I can write code in notepad.exe, but if a project required everyone to do that, I wouldn't contribute because I'd rather work on code where I can use the toolchain I like.
[...] the thing I liked the most was that I could now focus my coding and thinking on tasks I really loved while still adequately completing the tasks I didn't.
If the code that Mitchell in this hypothetical tried to get into Zig wasn't something he actually loved, then maybe it just doesn't belong in Zig.
Again, it's not very complicated: If you can do it, do it yourself. If you can't, just don't. We don't even really need to consider the "Well, person X can, but they really prefer not to" because it's irrelevant. The hypothetical of this hybrid case is pointless because anyone who actually could do these things with the policy in place and wanted to contribute would just do that: Contribute.
(In many of these cases we'd really only have to wait a year or two and many of these people will have had their abilities atrophy to the point where they're no longer able to do it as well, so I wouldn't really worry too much about the hybrid case even if it were relevant.)
If the code that Mitchell in this hypothetical tried to get into Zig wasn't something he actually loved, then maybe it just doesn't belong in Zig.
For me, development work often includes a mix of things I love and things I don't. I love creating a cleaner abstraction, but I don't love the tedious work of migrating all previous callsites. That's a task I now delegate to AI agents.
Again, it's not very complicated: If you can do it, do it yourself. If you can't, just don't. We don't even really need to consider the "Well, person X can, but they really prefer not to" because it's irrelevant. The hypothetical of this hybrid case is pointless because anyone who actually could do these things with the policy in place and wanted to contribute would just do that: Contribute.
If I'm understanding, you're saying that Zig banning LLM usage will not reduce good contributions because the good contributors are happy to eschew LLMs even if they like using them. Is that correct?
Or are you saying that developers who like LLMs are inherently inferior, so Zig banning LLM usage gets rid of inferior developers and their inferior contributions?
I think that both are untrue. If a project restricts me from using my preferred toolchain (even when the end result is the same to outside observers), I'd prefer to contribute to one of the many other projects that have no such restriction.
I've contributed a small amount to Zig but I won't in the future, not out of protest but because the LLM ban makes it more tedious to contribute than other projects available to me that give me the freedom to choose my own tools.
If I'm understanding, you're saying that Zig banning LLM usage will not reduce good contributions because the good contributors are happy to eschew LLMs even if they like using them. Is that correct?
I don't know whether "happy to" is the case, but any reasonable person who actually wanted to get code into Zig would just write it themselves. It's not a very radical idea.
It's no different than contributing to a project that technically uses C++ but you're not really supposed to use most of C++ in it. If that's the policy, I shouldn't expect to use a grab bag of modern C++-isms because I feel entitled to do so (or because I feel dependent on them). It has the same effect on quality as well; most people who can't accomplish something without using a ton of C++-isms in general (modern or not) are also worse programmers, so filtering them out might affect the top band (unlikely) but it definitely also filters out a lot at the lower band.
But again, the goal for the Zig project isn't even to filter people out, it's to invest maximally in the human beings making contributions in order to build their community.
Or are you saying that developers who like LLMs are inherently inferior, so Zig banning LLM usage gets rid of inferior developers and their inferior contributions?
Most of their contributions certainly will be, yes, and that's obviously exactly what the observation suggests so far.
I think that both are untrue. If a project restricts me from using my preferred toolchain (even when the end result is the same to outside observers), I'd prefer to contribute to one of the many other projects that have no such restriction.
I've contributed a small amount to Zig but I won't in the future, not out of protest but because the LLM ban makes it more tedious to contribute than other projects available to me that give me the freedom to choose my own tools.
It's your right to try to contribute to anything you want, for any reason you want. Since you're so passionate about this I suggest you start a fork of Zig (called Vig?) for people who want to use LLMs in their PRs.
I don't know whether "happy to" is the case, but any reasonable person who actually wanted to get code into Zig would just write it themselves. It's not a very radical idea.
I think the much more likely thing will be that people either break the rule or just don’t contribute.
Imagine you had a project that said you must not use an IDE to contribute or you may not work on a windows computer to do it. Those are all rules that will be impossible to police and people won’t follow them or they are too off putting.
I think the much more likely thing will be that people either break the rule or just don’t contribute.
We don't know what the shadow number of contributions that have been made using LLMs to a meaningful degree is, of course, but for now the policy is clearly stopping the most egregious cases and most likely a good chunk of the ones who pretended they made their contributions themselves.
Imagine you had a project that said you must not use an IDE to contribute or you may not work on a windows computer to do it. Those are all rules that will be impossible to police and people won’t follow them or they are too off putting.
I don't think this is a very apt comparison because an IDE doesn't have a meaningful impact on the quality of the output, whereas LLMs still cannot produce good code beyond single procedures or ridiculously simple boilerplate.
For the case that actually mirrors that of an IDE, no one can actually tell that you used an LLM to ask questions about a code base, and what does it really matter anyway? The worst that would happen when your PR is made is that you'd then be mistaken and the project maintainers would set you straight; then you've gained actual, real knowledge and can work (once again, "manually") on solving the real problem the real way. In either of those situations you're still fine and not at all a risk of being turned away because of the policy. You also then get the benefit of the stated goal of the Zig project in that they are investing in someone learning real things about the language, the compiler and the domain.
The amount of SpecLang you'd have to put into your process to produce something good out of LLMs (the very frontier, even) for anything larger than a function or two rivals that of just writing the code yourself, which means that the policy's real effect is simply keeping LLM code out of the project, which is not a bad policy at all.
The AI rule is or will be the same.
It might be. It might also seem to users of LLMs that the LLMs produce good enough code in the future because their ability to spot bad code gets dulled by their use of them, just as using them as a crutch for thinking is known to cause a dulling of the mind itself. I think this is a whole lot likelier than LLMs actually becoming good enough to produce code that passes the human test, and the truth is that a project always has the right to refuse your contribution just because regardless, so any kind of borderline cases can be handled just about any way the project chooses, and without a massive amount of entitlement there is very little one can really demand in that situation.
I'm not saying the AI-generated JS engine is better than a human-written one, but I can't understand how you can believe the upper limit on LLMs is a single function or boilerplate in the face of projects like this.
This is not true at all. It sounds like you haven't used LLMs in at least a year or you only use LLMs for niche languages.
I've used GPT-5.4 on every thinking level, Composer-2, Opus4.6. Without fail they can't seem to produce anything at all without bloating the solution by at least 2x, the highest I've observed is 6x. That's not a great basis to build things with at all.
I've run a ton of experiments using more or less specs up-front, varying the size of the thing they should spit out, etc.; I've done more than I could ever be asked to evaluate these things. Some of the bloat comes from over-guarding against things that can't actually happen and any programmer who knows the domain or even just language constructs would know that to be the case.
Over-guarding might be a good thing, except I also find that the output often has obvious holes in it where it's not doing what it should, once again obvious to anyone who's been in the domain for even a short time.
You can talk about languages all you want, but I've found that they can't even do basic memory management which they for sure have tons of examples of, and it's an exercise in and of itself to make it understand what it should allocate temporarily in a main loop and what that means at something like a frame boundary, for example.
Honestly, if you think they spit out acceptable results that's more a reflection of your standards than anything else, at least from what my own observations tell me. I'm not impressed with LLMs and by extension I'm even less impressed with the programmers who use them and claim them as sufficient, even with detailed specs and babysitting.
You and @gonz might be using different definitions of “good code”. That LLM-made JavaScript engine shows that LLMs can write large amounts of functional, correct code (which might be your definition), but it does not demonstrate that the LLM-generated code is also efficient and easily understandable by a human (which might be part of gonz’s definition).
A discussion on the quality or merit of LLMs does not seem to be relevant here because the stance of the Zig project has little to do with quality but it's a philosophical take. Zig does not want any LLM code or issues, regardless of quality. I think the IDE comparison is quite appropriate as a result. We don't need to debate the benefits or downsides of IDEs or LLMs, because that's not what is being judged here.
This policy will have effects on who participates and it will create a filter but I don't think it will change behavior. Personally AI has become a tool I use, and when there is a project that disallows me to use it, I'm more likely to not participate (because I chose to be a honest person and follow the rules of someone else's turf).
A value judgment on LLMs is necessarily part of the argument: we have a bunch of closed boxes (PRs) and when spending time opening them, we discovered that the ones that have "LLM-assisted" printed on them yield on average subpar outcomes alongside boxes that do not present the stamp, but for which we can tell that it's missing because of clerical errors.
If LLM usage did not lead to subpar outcomes in our experience, we wouldn't be having this discussion at all. We might be having a different discussion about the ethics of using LLMs maybe, but we wouldn't be talking about literal game theory.
We source core team members from the pool of trusted contributors, we get a ton of what is essentially free labor from trusted contributors, and we also built a community whose culture is heavily influenced by these people. And this funnel starts at looking at PRs, a process that the existence of LLMs has actively made worse. As mentioned in bold in the original blog post, it's a key part of our business model. I don't know how you could get any more practical than this.
A discussion on the quality or merit of LLMs does not seem to be relevant here because the stance of the Zig project has little to do with quality but it's a philosophical take.
With all due respect Armin, I think that more often than not you interpret our communications based on your own preconceptions rather than on what's actually printed on paper. Maybe, despite your beliefs, there are some concrete reasons for this policy. Also maybe you shouldn't have cringed as hard when we left GitHub, or at least should have reevaluated your initial impression after new evidence of GitHub being broken came in, even if you don't consider us people of real weight. Just my two cents.
With all due respect Armin, I think that more often than not you interpret our communications based on your own preconceptions rather than on what's actually printed on paper. Maybe, despite your beliefs, there are some concrete reasons for this policy.
To be clear I’m not criticizing the policy. You have your reasons for it, but your policy is set because your expertise with LLM assisted or generated code is bad, and as a result you banned the use outright. That is absolutely fine! But as a result it’s a policy that does not set a quality bar for contributions but it sets a more philosophical gate. LLM generated code does not fit into the philosophy of Zig.
To be clear: that is fine by me and I won’t criticize it. Your project, your rules.
Also maybe you shouldn't have cringed as hard when we left GitHub
Maybe. I should probably do a lot of things in life, but at the end of the day I’m a person with emotions and feelings and the responses one has are those. The way the zig project expresses itself in the blog just doesn’t really vibe with me but that is also okay — at least for me! I use and like the language for the most part, but that won’t make me automatically agree to all decisions and it won’t change me as a person either.
There are just things people will never be in agreement on for more deeply rooted reasons.
even if you don't consider us people of real weight
Zig has plenty of weight, but yes, I think Mitchell’s move in my book will pull more people along than the Zig project which is much more niche in comparison.
This policy will have effects on who participates and it will create a filter but I don't think it will change behavior. Personally AI has become a tool I use, and when there is a project that disallows me to use it, I'm more likely to not participate (because I chose to be a honest person and follow the rules of someone else's turf).
Well, all the policy actually does is bring us back to the time before LLMs:
If you can write the code and you want to, you're welcome to contribute
If you can't write the code but you want to, you're not welcome to contribute it until you have the skill to write it or at the very least be guided towards a version that is up to par
If you can write the code but don't want to, then you didn't need to show up at all
This de facto is literally just how things were before LLMs, but people are making it seem radical out of nowhere.
The only thing that actually changed in the greater ecosystem is that people can now pretend that they can write the code, so I think that really is the operative difference in why some people find this policy so offensive.
People keep using this word without making it clear whether this is a qualitative evaluation or a quantitative one. If it is quantitative, then it can be measured, and if it can be measured, then it can be controlled. If it is qualitative and it never has a tangible impact, then it literally doesn't matter (unfortunately). If it is qualitative because it may eventually have a tangible impact, then you might have an argument- but if NOTHING is measurable about it TODAY, then that would frankly be... a really strange phenomenon, because it means that there is an effect that spontaneously emerges in the future that shows absolutely zero evidence of existing even in a "seed form" today.
I'm getting the impression that you think 'qualitative' means 'unobservable' or 'not objective'... that is a misunderstanding of the word, I think. It just means observations that are more clearly described with words than with numbers.
For example, think of a colleague whose explanations you really like, and find easy to follow. Can you quantify, with a number, how clear their explanations are, or what makes them so clear? Probably you can't put a number on it. But would you say that their clarity of speech/writing has "no tangible impact" on your relation and work?
Despite the widespread myth that only quantitative measurements and descriptions are of use to science, many non-quantifiable things, i.e. things that cannot be usefully measured on numerical scales, can be given precise objective descriptions that can play a valid role in scientific theories and explanations. Thus linguists describe language, chemists describe matter, [and anthropologists describe behaviour and culture —Sietsebb]
—The Fontana Dictonary of Modern Thought (1977), lemma 'quantification'
Bloated output by at least 2x the amount that a (competent) human would write, guarding against things that can't happen and not handling things that can. I've tested a tons of different permutations of spec-driven development and so on, which people claim works, but it all spits out low quality code that has obvious bugs in it.
If there are bugs, then why didn't you test for them, or ask it to? (I know it lies sometimes. Adversarial prompting, where you basically get another AI or instance to poke holes in things, helps.)
2x code output? Well, yes. That does happen. Have you seen human DNA or done any research on just how terrible it is? It still leads to (mostly) working humans. Basically, the user of the product of whatever that code is contributing to won't give a shit, and they will get that functionality 5x faster, so it doesn't matter that it took 2x the code, or that the code is ugly, or that it doesn't use comptime as much as it could, or that it has a bloated test suite.
If there are bugs, then why didn't you test for them, or ask it to?
Do you expect to cover literally every edge case in your spec? At some point just writing the thing myself is a whole lot faster than sitting there meticulously writing the spec that's supposed to cover everything, and it reveals more about the design as well.
2x code output? Well, yes. That does happen. Have you seen human DNA or done any research on just how terrible it is? It still leads to (mostly) working humans. Basically, the user of the product of whatever that code is contributing to won't give a shit, and they will get that functionality 5x faster, so it doesn't matter that it took 2x the code, or that the code is ugly, or that it doesn't use comptime as much as it could, or that it has a bloated test suite.
Right. Ship garbage, because no one cares what your code looks like? I care what it looks like, I have to work with it, modify it, add things to it and understand it for the purposes of understanding the exact data transformations in it. Some of us have higher standards than you people.
I literally shipped this demo in a week. https://incitez-web.pages.dev/?demolinks It's a complete rewrite of the eyecite citation search engine. The original was in Python, I rewrote the whole thing in Zig, using the original test suite as a spec. It is 50 times faster, 15% or so more accurate, and since it is Zig, it compiled to WASM, so this runs completely in the browser- no attorney/client privilege is violated.
Go ahead and play with it looking for bugs.
It was actually done in an hour. The week is what it took to refine it, because (surprise) I care about code too (and yes, at first there were a bunch of little things to fix). But this would have taken any human literally months to do manually to ANY level of quality. This one passes the entire eyecite test suite AND found bugs in it (corrected and sent pull requests to eyecite's repo) AND goes over and above it in terms of citation-finding. Did I mention it is FIFTY times faster?
Here's the repo. Go ahead and code-review the code. (The web-demo repo is private, but most of what it uses is from here, plus my pmarreck/docscan lib (public) to parse pdfs, doc's and docx's.)
I haven't written a single line of Zig in my life, but I've been a software developer for 30 years now and I know pretty damn well what failure modes to look for. https://github.com/pmarreck/incitez
Your bespoke production is like hand-building a car or hand-sewing an article of clothing. It may be better, but very few people are going to pay for that.
LLMs are better at converting already existing code, it's true.
You didn't "rewrite" anything, the LLM did.
If the original was written in Python and your vibecoded version is only 50 times faster that's painfully bad.
You're telling on yourself by pushing something you know basically nothing about when you're basically letting an LLM do everything for you in a language you don't even know or understand.
Pay me and I'll review it and tell you whether it sucks as much as I think it probably does.
You've never written Zig, but you spam your GitHub with Zig projects. I'm sure they're happy to have such a productive member of the community around.
False. 1) I'm responsible for it if something goes wrong in a prod env with it. That is all that matters. 2) This was not just an ask, it was a cooperation similar to pair programming.
and your vibecoded version is only 50 times faster that's painfully bad
So now you're just deliberately being an asshole based on literally zero information? Nice. You also forgot the speed of development. 1 week to go 50x faster AND run in the browser AND pass all the same tests. That is literally impossible without LLM assistance, because the overhead of doing that conventionally with a TEAM would have already cost that much time just to communicate/coordinate- and much of their code would have sucked, to boot. To call that anything but impressive is just fucking oxbrained, frankly.
You're telling on yourself by pushing something you know basically nothing about when you're basically letting an LLM do everything for you in a language you don't even know or understand.
Again, you're completely missing the point. That codebase you stopped updating 2 years ago that you wrote in the dead of night to meet a deadline, you also know nothing about, regardless of whether you wrote it or not. That doesn't mean it isn't valuable. You've simply shifted the ignorance. I have learned when I've needed to, on the hot paths, which is where it counts. "Letting an LLM do everything for you" is pure ignorance, dude- I'm there every step of the way, making difficult tech decisions in some cases, and vetting the diffs, calling anything suspicious into question.
Pay me and I'll review it and tell you whether it sucks as much as I think it probably does.
Dude, with the attitude you have, you're headed for a firing in a matter of time, and I wouldn't bat a fucking eye. Good Lord, you are obstinate. Anyway, there are multiple types of suckitude and only some matter- the measurable ones. Guess what's good at creating measurements.
You've never written Zig, but you spam your GitHub with Zig projects. I'm sure they're happy to have such a productive member of the community around.
And all of those projects work well. I don't know what the Zig community would think of it, and I wouldn't care. Just like I don't care what you think. Just leave your blinders on while the tidal wave comes, my dude. I'm sure it'll all work out great.
Jesus Christ, the whole lot of you obstinate skeptics are just lost. To make it abundantly clear to your level: NO CUSTOMER IS GOING TO CARE ABOUT YOUR BESPOKE, SLOW-ASS-CODED POLICE FORCE MANAGEMENT SOFTWARE (by the way, there is nothing at all sophisticated about that type of software and the LLM will eat your lunch) IN 12-24 MONTHS IF YOU ARE NOT DELIVERING AS QUICKLY AS COMPETITORS, EVEN IF YOU PROVIDE SLIGHTLY BETTER QUALITY (and it would not just have to be slightly better to justify that massively increased delay, it would have to be 100% better! You willing to take that bet? I'm not.)
But hey, it looks like you're doing government contract work, so perhaps you are slightly safer than those in the normal private sector. For now.
Are these LLM contributions coming from known-good developers, or randoms?
Because if you're grouping them together, that's a pretty serious conflation problem. It would be like grouping together people who know how to use a tool with people who don't, and then blaming the tool
This reminds me of when companies think that they can get better candidates by making candidates jump through more hoops. "We have 1,000 applicants, so let's ask them to explain their high school grades to us, even if they graduated 20 years ago. The lazy candidates will give up, leaving only the best ones."
On the /r/ProgrammingLanguages subreddit we've been dealing with a lot of AI slop in recent months. To combat that we tried a bunch of different things, the most recent one being that posts referring to GitHub are filtered (= basically hidden) automatically and the author is notified about needing to copy-paste a standard phrase into a comment. Essentially a dumbed down Reddit Turing test.
On the surface that may seem pointless: surely the LLM users will just do that and lie about it? And indeed I've seen this sentiment (essentially "you can't stop LLM users") in other places.
In reality this approach has thus far proven remarkably effective in spite of how stupid it is. Why? Because many LLM users are extremely lazy, so much in fact that just copy-pasting a phrase is apparently too much to ask.
What I'm ultimately saying is is that while it may seem LLM users can't be stopped unless somebody comes up with a magical oracle, in reality you can filter out let's say 80% of the noise by applying a few rules that seem dumb at first but end up being surprisingly effective.
What I'm ultimately saying is is that while it may seem LLM users can't be stopped unless somebody comes up with a magical oracle, in reality you can filter out let's say 80% of the noise by applying a few rules that seem dumb at first but end up being surprisingly effective.
I think automated rules to trip up lazy LLM contributions make sense, but isn't that the opposite of what we're discussing here?
As far as I can tell, the Zig team isn't adding any automation. They're just adding a new rule that contributors are expected to read that says "No LLMs." So, the lazy vibe coders will just skip reading that rule and continue the low-effort contributions, but it filters out the developers who want to use LLMs but still do their normal due diligence in verifying their code.
Do you have an example of a developer who wants to contribute to the Zig project but is prevented from doing so by the LLM policy? Because it seems to me like someone who is both good and willing should be able to find a way to contribute that doesn't violate the policy.
Interesting, did you contribute to Zig before? Really, the question I would like to ask is this: Do you worry about having made yourself dependent on a tool that is changing so rapidly and unpredictably and has such big open questions around copyright and legality, to the degree that you want to spend significant time working on an open source project but feel unable to do so without using the tool?
I only contributed to issue discussions and a little to the ecosystem by implementing zig-cross (based on an article written by kristoff). I have been using LLVM for quite a few years and contributed to a bunch of different projects in the ecosystem. There are a few issues in the zig c++ driver that I have on my TODO list, but never enough time ;(
Do you worry about having made yourself dependent on a tool [...]
My open source contributions started in 2013 or so (when I was ~17). I see the ecosystem as a campsite: leave it better than you found it. In this way I made a lot of minor/'drive-by' contributions to projects in my ecosystem (security/reverse engineering). When I got my first job in 2018 my open source contributions suffered majorly. LLMs allow me to get back some of this time and I personally do not think my OSS contribution quality suffered (if anything I suspect the quality has gone up since I can actually spend time on tests and design iterations). My GH is public so I guess you be the judge...
Do you have an example of a developer who wants to contribute to the Zig project but is prevented from doing so by the LLM policy?
How would I have this information?
The effect isn't measureable because the effect creates the absence of an artifact. Developers who use LLMs and want to contribute will see the rule and decide not to contribute. Maybe some will post somewhere publicly, but the vast majority will just silently move on.
I'm not able to name a specific person because how would I? But I think it's obvious that this ban shuts out some developers who might otherwise make positive long-term contributions to Zig.
Which of these points do we disagree on?
At least 20% of skilled developers regularly use LLMs in their work
Among developers who might want to contribute to Zig, the proportion who use LLMs is the same +/- 10% (so at least 10-30% of skilled developers who might want to contribute to Zig also use LLMs)
Some skilled developers who both use LLMs and might be interested in contributing to Zig would likely choose not to if they're unable to use their preferred toolchain.
Well, #1 seems like one of those stats that are in the 90% of stats that are made up on the spot.
Even if it were true, the population of LLM users in #2 would not be influenced by #1. Using LLMs and wanting to contribute to Zig are independent variables.
But 3 still works, sure. You can make that argument, but from my experience I'd rather not have a contributor that gets upset at being asked to follow a simple rule that they don't necessarily agree with.
I actually think it's more important to take care of the existing contributors. Establish a clear culture that helps them be happy and creates an environment where they look forward to working with other people on the project.
Think about projects like sqlite, where their insular team and uniquely worded code of conduct seems shocking at first, but actually makes them extraordinary effective as a team.
Honestly the backlash to these "no LLM" policies reminds me of the loud handful that despised Codes of Conduct when those were gaining traction. Like, if you're the time of person that goes into rule lawyering when you're asked not to be jerk then I probably don't want to work with you in the first place.
But 3 still works, sure. You can make that argument, but from my experience I'd rather not have a contributor that gets upset at being asked to follow a simple rule that they don't necessarily agree with.
This is in effect just saying "I don't want you because you disagree with the rule." That's the exact thing the rule was being critiqued for!
You're basically saying "I'd be fine with a LLM-using contributor but not if they get upset at not being allowed to use LLMs."
It's not quite that. It's more "I don't want you because you are unwilling to agree to the rules that we as a project have agreed to." Which is not unreasonable. The critique is that this is shutting a lot of people out that would otherwise have been valuable contributors. That's a fine opinion to have, but there is very little evidence to suggest that this outweighs the tangible benefit of shutting out a lot of low effort slop contributions.
Note that both of these are measurable. The reviewers of the Zig project can measure the rate of good and bad contributions before and after the ban and see if there is any detectable decrease in good contributions, and if the ban had little or no effect on bad contributions.
Another metric that the reviewers can use is their own happiness: Did the ban make their lives better? Reviewing slop is very demoralizing. Anything that would reduce the risk of wasting time on slop would be a huge win.
Well, #1 seems like one of those stats that are in the 90% of stats that are made up on the spot.
I'm not claiming it's a real stat. I don't think we have reliable stats on this. I'm asking if 20% is a reasonable lower bound because if we have vastly different assumptions for this, it affects the rest of the model.
What do you think the lower bound is?
Even if it were true, the population of LLM users in #2 would not be influenced by #1. Using LLMs and wanting to contribute to Zig are independent variables.
They're not statistically independent. If #1 were 100%, then #2 definitionally must also be 100%.
Or, more relevant to this discussion, if the rate of LLM usage among all developers goes from 10% to 40%, I'd expect the rate of LLM usage among potential high-value Zig developers to also increase a similar amount.
If you think that LLM usage among potential Zig contributors would remain constant (or change in unrelated ways) while global LLM usage quadrupled, can you share why you think that?
I actually think it would be higher among potential Zig contributors because using Zig indicates openness and curiosity about newer tools.
But 3 still works, sure. You can make that argument, but from my experience I'd rather not have a contributor that gets upset at being asked to follow a simple rule that they don't necessarily agree with.
It seems like this is more about your view of the rule rather than contributors being able to follow "a simple rule."
For example, if the rule was that you can't use Apple products or VS Code for any Zig contributions, would you say, "Oh, that's fine because it's just a simple rule." Or would you think, "Hey, what does it matter which tools I use as long as I do due diligence and take responsibility for the final result?"
Not a zig dev, but: So far, the number of skilled developers committing LLM-generated code I've seen contributing on GitHub has been far below 1%. At best 3 people vs >1000 contributors.
And even those 3 weren't LLM regulars, but 3 skilled devs contributing who also happened to have a side project with some LLM assistance.
In my own experience even Opus 4.6 and Gemini 3.1 Pro are unable to generate even intern-quality code. And that's not because I'm just not able to prompt them correctly, even if I ask people who frequently use AI to show me their code (which they claim is fine), it's always an absolute mess that I'd never use in production myself.
Not a zig dev, but: So far, the number of skilled developers committing LLM-generated code I've seen contributing on GitHub has been far below 1%. At best 3 people vs >1000 contributors.
How are you counting who's using LLMs and who's not?
In my own experience even Opus 4.6 and Gemini 3.1 Pro are unable to generate even intern-quality code. And that's not because I'm just not able to prompt them correctly, even if I ask people who frequently use AI to show me their code (which they claim is fine), it's always an absolute mess that I'd never use in production myself.
Do you have examples of prompts and bad outputs from Opus 4.6?
In my experience, Opus 4.6 generates code better than most fresh CS grads.
It's a personal project, so it's not as polished as what I'd produce if I were writing code for paying users and working with teammates, but I still consider it good quality.
There's a ton of people in the comments who need to remind themselves that "absence of evidence is not evidence of absence."
They're engaged in the digital equivalent of thinking they can identify plastic surgery because they can spot bad plastic surgery. And even if they were right for now, I wouldn't bet on that remaining true as models advance.
First, did the ban work? The reviewers can self-evaluate: Do they feel like the burden of reviewing slop contributions lessened? If so, the ban worked.
Second, did the ban result in a lower rate of quality contributions? This can be measured statistically.
There's a ton of people in the comments who need to remind themselves that "absence of evidence is not evidence of absence."
Except it is.
Absence of proof is not proof of absence, but when you expect evidence for X, and you don’t see any, your probability for X has to go down. We can even quantify it.
Let P(X) your prior probability that X is true.
Let P(E|X) your probability of observing some specific evidence if X is true.
What should be P(X|~E), your posterior probability that X is true, after you failed to observe such evidence?
So we got "E is evidence for X means P(E|X) > P(E). Kind of obvious, but I need it to analyse my Bayes formula from above: P(X|~E) = P(X) × (1 - P(E|X)) ÷ (1-P(E)). Here we go again (recall that probabilities are all in ]0, 1[):
You're making a couple mistakes here and regurgitating all of Bayes' Law is the opposite of convincing.
First, Bayes' Law is about what you believe is likely, not necessarily what is true, so invoking it doesn't actually refute what I said. Your conclusion "Absence of evidence is evidence of absence" is simply wrong under a Bayesian lens. But that's an old frequentist criticism.
More relevantly, even under Bayesian logic, saying "you don’t see any [evidence]" is combining two different scenarios:
you see no evidence because there was none
you see no evidence because you missed it
Given 2, any posterior updating should also include P(I can't reliably identify LLM output|I didn't categorize anything as LLM-created).
Ah, yes, frequentism. The thing we should have ditched like 80 years ago. But no, scientific papers still use it today, I’m sick of it. Also, Probability Theory: the Logic of Science (chapters 1 and 2 at least), should be required reading for any aspiring scientist. I mean how can you read that and still be a frequentist? The axioms of probability theory are so few, so obvious, which one could you possibly reject? I understand the use of frequentist recipes as shortcuts, imperfect approximations of correct reasoning… but actually believing this stuff? That’s alien to me.
First, Bayes' Law is about what you believe is likely, not necessarily what is true, so invoking it doesn't actually refute what I said.
Do remember than in Bayesian thinking, probability is in the mind: how "likely" something "is" is how sure I am that something is true. Or how sure you are. Depends which priors we’re talking about. I often say my probability, or your probability to reflect this. Strictly speaking, "the" probability does not make sense.
Don’t mistake this for epistemic relativism though. While each of us have different prior information, it doesn’t mean we can believe whatever we like. The correct probability is derived from the information we have access to, not from how we feel about it.
To be absolutely rigorous, we should talk about P(A|I), meaning how likely it is that A is true, given background information I. And the full Bayes formula should be written:
P(A|BI) = P(A|I) × P(B|AI) ÷ P(B|I)
Your conclusion "Absence of evidence is evidence of absence" is simply wrong under a Bayesian lens.
Did you mean "under a frequentist lens"? Because if you really meant "Bayesian", you have to know this is probability theory 101 you’re disputing here.
you see no evidence because there was none
you see no evidence because you missed it
I considered that, and swept it under the rug for simplicity. But if we must, it’s just an additional stage: not seeing evidence where I expected some increases my probability that there is no evidence, which in turn increases my probability that there is nothing.
Absence of observation is evidence of absence of evidence.
Evidence of absence of evidence is evidence of absence.
Therefore, Absence of observation is evidence of absence.
The fundamental mistake in your comments is writing about Bayes in a non-Bayesian manner.
If you want to argue that the Bayesian interpretation is the only legitimate one, you should frame it in purely Bayesian language, because you come across as deeply confused when you talk about it in terms of universal probabilities.
E.g.:
Do remember than in Bayesian thinking, probability is in the mind: how "likely" something "is" is how sure I am that something is true. ... Strictly speaking, "the" probability does not make sense.
This is what I mean. You can think that "the" probability doesn't make sense, but that's a minority viewpoint, especially in science, and pretending otherwise does you no favors. Don't respond to people saying "the" probability as if they're actually saying "my" probability, because many of them are explicitly not doing that.
Most people talking about "the" probability are very much talking about the true probability, however it's defined outside of humans, brains, and evidence. (Call it E[some event] of an infinite random process if you like, there's plenty of definitions.)
You lamented that science has not adopted Bayes enough, but I can tell you from my own time in academic neuroscience, it's not solely due to slow adoption of new methods. (Many new statistical techniques get adopted within a decade or two.)
It's mostly due to Bayes's fundamental change in interpretation, and not that they just haven't heard a good enough argument for Bayes yet.
It's great for an experiment to tell you how to update your priors, but most scientists are interested in discovering the true likelihood of something independent of experiments. When I originally said "absence", I am talking about the true prevalence of a thing, not just my personal estimation.
One personal estimation I do have is that the likelihood you've been an academic scientist is low, otherwise you'd have had a more complete picture of why Bayesian stats haven't taken over. It's not just inertia, laziness, or that frequentism doesn't have serious flaws.
I considered that, and swept it under the rug for simplicity. But if we must, it’s just an additional stage: not seeing evidence where I expected some increases my probability that there is no evidence, which in turn increases my probability that there is nothing.
Seriously? But it also increases your prior that you're missing things. To be intellectually honest, you have to update all relevant priors, not just the ones you want.
This is like someone with dementia claiming they don't have dementia because they don't notice they're forgetting things.
[…] if we must, it’s just an additional stage: not seeing evidence where I expected some increases my probability that there is no evidence, which in turn increases my probability that there is nothing.
Seriously?
Of course. Here’s how it works:
Let’s assume E is evidence for X. Which means, finding E makes X more plausible, and proving that there is no E makes X less plausible.
At first we’re not sure about E. Haven’t seen it, haven’t even tried. But we do know that if we actively look for it, there’s a chance we’ll find it, if there’s any.
We look for E for a while. Still nothing. This makes E less plausible.
X is now less plausible.
Here’s a concrete example with the same structure:
Let’s say the presence of some suspect’s fingerprints on the crime scene is evidence for him having done it. Which means, finding such fingerprints makes his guilt more plausible, and proving that here aren’t any fingerprints makes his innocence more plausible.
Forensics have yet to arrive at the crime scene. We’re not sure about those fingerprints. But we do know that they have a chance of finding them, if there are any.
Forensics came, they looked, they found nothing. They could have missed something of course, they’re not perfect. Still, the presence of the suspect’s fingerprints is now less plausible.
The suspect’s guilt is now less plausible than before.
I believe you can accept (1) and (2). You may be tempted to dispute that (3) follows from (2), or that (4) follows from (1) and (3). But try that in law enforcement and they’re going to look at you funny.
But it also increases your prior that you're missing things.
If you mean something like "can’t miss the shots you don’t take", sure. But that’s irrelevant to my main point: looking for evidence, and failing to find it, makes the thing less plausible.
You can think that "the" probability doesn't make sense, but that's a minority viewpoint, especially in science
Depends on the field. When you can have controlled studies it matters less (or at least, less visibly so). And when the alternative to publishing p-values is perishing, it’s hard to let go of the old ways.
[…] the true probability, however it's defined outside of humans, brains, and evidence. ([…] there's plenty of definitions.)
Plenty of definitions, but can you find one that makes sense in a deterministic world?
Because last time I checked the laws of physics as we know them are deterministic. Same initial conditions, same results, every time. Even in quantum mechanics, if you just look at the equations and don’t interpret them. (Just because the photon bounced off the beam splitter doesn’t mean it didn’t also get through. That’s how we get interferences.)
For instance, when you throw a fair die, the reason you ascribe a uniform probability as to which face will end up, is because we humans aren’t consistent at throwing die, the process is chaotic (diverges quickly), we can’t observe the initial conditions precisely enough, and the die is well balanced an symmetric. Not because its mystical propensity to land on one of its faces is equally distributed.
One personal estimation I do have is that the likelihood you've been an academic scientist is low, otherwise you'd have had a more complete picture of why Bayesian stats haven't taken over. It's not just inertia, laziness, or that frequentism doesn't have serious flaws.
Then what is it? Care to outline that "more complete picture", or at least estimate how many pages it would require?
It's mostly due to Bayes's fundamental change in interpretation
A fundamental change in basic thinking, yes, but this goes way beyond interpretation. Interpretation would be like squabbling over Copenhagen vs Everett while the colleagues are building lasers, interferometers, and quantum computers: a philosophical perspective, of no practical consequence.
Not so with statistics. Here we’re talking about how to conduct and prioritise research, what policies should be derived from the results, and ultimately avoid countless unnecessary deaths.
I take you’ve heard of Ronald Fisher? Champion of frequentism, and staunch critic of a 1950 paper that said tobacco caused lung cancer. "Correlation does not imply causation", but it damn sure makes it more plausible. Ignoring that basic fact likely participated in delaying tobacco prevention policies, indirectly causing who knows how many preventable deaths.
Stupid fallacies like "Absence of Evidence is Not Evidence of Absence" can have far reaching consequences. That’s why I couldn’t let it slide. It sounds very reasonable on the surface, but it’s false, dangerous… and seeing trained scientists still making such a basic mistake today scares me.
I don’t want to work with high-performing slop merchants, so I’m happy their talents are desired elsewhere. At least half the happiness I derive from open source comes from my interactions with the people I collaborate with, if not more.
The developers least interested in jumping through hoops are the ones that make the most valuable contributions.
This is probably true, but relies on a flawed analogy. The Zig team is not requiring anyone to jump through hoops: they're selecting from among all possible contributors the subset which they believe might be the most valuable to the health of the project in the long term. For you, perhaps because you place individual reputation above other qualities, those contributors would be the LLM-equipped Jeff Deans of the world; for Zig, clearly, they're the people who signal their willingness to put in their own time and effort for the good of the whole.
It comes down to conflicting assumptions about where the top performers are. I believe the top performers are overwhelmingly using LLMs, and it sounds like the Zig team believes that's not true.
You can believe that, of course, based on your own experiences and personal bubble. I, for the same reasons, believe the exact opposite. Your argument also implicitly relies on an assumption we have little evidence for but mounting evidence against: that "top performers", however you define them, remain top performers after starting to use LLMs. This is far from clear and, so far, the limited data we have (together with classic cognitive psychology findings; see "Ironies of Automation") suggests otherwise.
The Zig team is not requiring anyone to jump through hoops
Yes, and you are not looking for a paid job here, so you are not required to jump through hoops in order to get hired. If you don't agree to the no-AI policy, you are free to contribute to OpenClaw, start your own open source project, make a sloppy AI-generated fork of Zig, or do whatever you want with your free time.
I think this argument depends on your time horizon. Is the set of developers-who-make-the-most-valuable-contributions static? How did they become such developers? Surely Jeff Dean didn't get to where he is by having AI do all the work for him.
We're in an interesting moment of time where we have a large pool of developers who definitely learned how to develop without AI, but that pool is going to shrink relative to the total over time. For the moment, maybe you're right (or maybe not) that the best developers are all using LLMs. But assuming our industry lasts beyond this moment, having at least some kind of policy that clearly invests in training seems smart.
I believe the top performers are overwhelmingly using LLMs, and it sounds like the Zig team believes that's not true.
This depends on what you mean by "top performer". I believe this is true in many projects, but only among contributors who already have deep familiarity with the codebase. I think this is almost always false for people who are not already deeply familiar with a codebase: a "top performing" new contributor is one who is making small, targeted improvements, and these are the kinds of changes for which having LLMs author code is the least helpful. The kinds of changes where LLMs have the largest positive impacts (such as big refactorings or changes which require lots of boilerplate) are precisely those which are least suited for new contributors.
Personally, I've found LLMs massively useful for small targeted changes. I certainly can't read 100KB of code in a few seconds! You do have to do very careful work reviewing for style and design though.
The LLM can identify the relevant area for you but once it has done so it doesn't do much to speed up the actual writing of the code (since there is definitionally not much to write). Certainly it can often do that part too, but for small changes it's not much slower to do it by hand.
... Though I guess it is unclear to me whether Zig would consider "I used an LLM to explain the codebase then wrote this change myself" to be a violation of their AI policy.
Agreed. As a maintainer, I can produce a style-compliant, well-tested patch using an LLM much faster than I can review an AI-generated PR from a new contributor.
Dealing with an LLM through a proxy is more work for a worse result; without the right context, the code just doesn't hit the mark.
The developers least interested in jumping through hoops are the ones that make the most valuable contributions.
Evidence suggests that this statement is false. And since you assume it as an axiom, so are the rest of your arguments.
Simply moving off of Github (ie. adding a hoop to jump through) seems to improve the quality of contributions to a project. Many projects have noted this. Putting a single hoop of "need to make a new login" scrapes off so, so, so many garbage contributors that you seem to be only left with real contributors.
In the limit, people will get super annoyed if you put too many hoops in the way. However, psychology also shows that if you overcome a hoop you become invested and that tends to increase your probability of continuing your activity.
There is also a quiet implication in your statement that the "rockstar programmers" are the ones to make the biggest contributions. Evidence suggest that for most projects "rockstar programmers" are a detriment. What most projects need are people who can argue but still cooperate with each other rather than "rockstar" technical people.
You keep repeating "evidence suggests" without providing any evidence.
The developers least interested in jumping through hoops are the ones that make the most valuable contributions.
Simply moving off of Github (ie. adding a hoop to jump through) seems to improve the quality of contributions to a project. Many projects have noted this.
Yes, I agree that some filters work well. Moving to Codeberg is probably a good filter as good developers don't much mind the one-time cost of joining a growing forge, whereas low-effort developers won't bother.
My argument isn't that hoops exclusively filter out the top, but I was disagreeing with this point from the post:
So while one could in theory be a valid contributor that makes use of LLMs, from the perspective of contributor poker it’s simply irrational for us to bet on LLM users while there’s a huge pool of other contributors that don’t present this risk factor.
I think that this is flawed thinking because it assumes that the filter only cuts out the bottom or cuts out a random sample, but I think that it disproportionately affects the top, not because all the best programmers use LLMs (although incidentally, I think most of them do) but because top programmers have other options for where to apply their skills.
There is also a quiet implication in your statement that the "rockstar programmers" are the ones to make the biggest contributions. Evidence suggest that for most projects "rockstar programmers" are a detriment.
I think you're using "rockstar programmer" to mean someone who's talented but a jerk. I'm not talking about attracting them.
Most open-source projects see a more exaggerated version of the Pareto principle where 5% of contributors do 95% of the work (including unglamorous work like debugging, code review, documentation).
I think that adding tedious work to the contribution process disproportionately affects people most likely to end up in that top 5% because they have many options available to them of other projects that offer a more pleasant experience and would welcome talented contributors.
Very well written. Some months ago I argued against a suggestion to ban LLM contributions for one project, not out of love for LLMs, but on the basis that it would create more work and arguments and be impossible to detect and enforce. I'm pleased to say that this post has changed my mind.
I guess this is in response to the post by Bun about their fork of Zig to speed up debug builds: https://xcancel.com/bunjavascript/status/2048427636414923250; they've said this was LLM-written and they can't upstream it per policy. My understanding from some of the people who've reviewed this code is that it is pretty bad and wouldn't make the cut to be upstreamed anyway 🤷♂️
That response is everything that was going through my head when I read the bun announcement. I was like, "would they really rather vibecode their own poorly thought-out fork without input from the zig lang devs instead of just reading the release notes?"
Disclosure policies like Ghostty's map cleanly onto the contributor poker framing: if someone discloses LLM use, you calibrate the bet accordingly rather than refusing the hand. More information, not less.
That said, Zig's problem seems to be as much about review capacity as trust. Knowing a PR is AI-assisted doesn't make it cheaper to read. A disclosure policy probably works well for most projects; a ban starts making sense specifically when the noise-to-signal ratio has already made the queue unmanageable, which is where Zig found itself.
A recent poll suggests that the 'youth' is a population often resentful of AI, given its marketing as their direct competitors, even though 69% of Gen Z trust work more if done without LLMs. I think creating spaces where LLMs are not allowed is good to counteract the productivity FOMO arms-race. For example, it is difficult for students and young academics to abstain from LLMs under the promise of future expertise, when they are currently being compared to / graded on a curve against their (LLM using) peers. LLM usage is high, and simultaneously 80% of Gen Z respondents felt that LLM usage was likely to make future learning more difficult.
I don't think you have to subscribe to inevitablism; it is not a forgone conclusion that prompts are the new abstraction level. And, if they are, there is definitely still space for LLM-free zones, especially amongst systems programming and language design, where the novelty of problems may be less compatible with the pattern matching inherent to LLMs. For the opinion of a high quality contributor and LLM-user, here's José Valim (creator of Elixir) from a recent episode of the Thinking Elixir podcast:
Al has been pretty bad for development work on the type system. And in general, things related to the Elixir compiler, because a lot of the time it's not an implementation issue.
...
So if I ask it to do it, it's going to miss a bunch of important corner cases, then you have to be playing wacamole later and pick those out.
...
Some of the problems that we are doing, the way to solve the problem and think about the scenarios is actually going for it and implementing it and considering the cases. If you automate that, sometimes you're automating the thinking away.
My anecdata/experience as a student says that (even before LLMs) there are ~5 students (being charitable) in a class of ~150 who have/obtain the ability become systems programmers who can contribute to a project like Zig. Most of them do not need to be in academia either, they are usually self-motivated individuals with agency who like to tinker. Cheating on (group) assignments is (and has been) pervasive in academia long before LLMs and I simply don't believe Gen Z students are doing it because of societal pressures.
it is not a forgone conclusion that prompts are the new abstraction level
Yeah I do believe this either, but LLMs have their use cases so if you have ~1000 people in the world who can contribute to your project in the first place, asking them to not use LLMs for anything feels like a pool that is likely to shrink over time, not grow.
Cheating on (group) assignments is (and has been) pervasive in academia long before LLMs and I simply don't believe Gen Z students are doing it because of societal pressures.
That's fair enough, but don't LLMs broaden the cheating possibilities, and make it much easier (and more alluring) to do so? For instance, I couldn't cheat my way around creating an interpreter at university, except through some code reuse, which would have been trivial to detect. I don't know if a good solution has been found yet in terms of problem setting either.
[...] a pool that is likely to shrink over time, not grow.
I suppose this will be borne out in the number of Zig contributors in the future, but it seems like something that can only be evaluated in a decade, by which point the development landscape will have shifted again. I appreciate your personal experience of having LLMs free up time to contribute to open source, and I guess development will return primarily to those who could allocate the time pre-LLM.
If the question is not about free time, but instead "Will future developers be able/willing to work without the use of LLMs?" - I'm not sure what to say other than I hope we don't become that dependent on centralised resources.
Some small handful of people will be both able to think like a computer (the main requirement) and also to memetically resist the ELIZA effect in small doses. If enough of them choose to not use LLMs for everything then we will be fine. A bigger concern is how much of society seems to simply accept the output of LLMs, which is obviously bad writing, as if it were good writing.
I'm quite sure that many of their valued contributors are LLM users. And despite their "AI Ban", the proportion of their contributors which use LLMs will grow. And at the same time quality will increase and they will become more successful.
The irritating thing about this policy is the out-group disdain for "LLM users" as if it's not a transformative technology which allows 10x more people reach their full potential in programming. Obviously it produces spam, but yeah. Not everybody is a rich English-speaking university educated person and everybody else is best to use LLMs to learn coding in their own spoken language etc.
The irritating thing about this policy is the out-group disdain for "LLM users"
This policy does not exclude LLM users in any way. We don't quiz you on whether you have ever used an LLM, you're just asked to not use one when contributing to Zig.
Obviously it produces spam, but yeah.
Glad to see that you gave deep thought to the reasoning expressed in the original article.
Why do they not use AI to help them vet new contributors ("please investigate this contributor's profile age and activity to vet their quality" etc.)? Why do they not use AI to help them assess PR's? They can do all that while maintaining their anti-AI-code ban.
Because if their stance is "nothing AI related ever," that's a religious stance, not a rational one
I expected this to connect the first half of the essay (investing in contributors) to the second half (banning LLMs) more explicitly:
Feedback given to people submitting purely LLM-authored PRs is ~useless once that PR is done: you might improve that PR but it isn't going to make any difference for later contributions. The LLM isn't learning from the feedback and the person submitting the PR usually lacks the context (and frequently the desire) to internalize it for future work. To use the metaphor of the article, effort spent on feedback for new contributors is worth it when contributing is an iterated game, but because LLMs don't learn from feedback it turns it into a one-shot game, which changes the payoff drastically.
Yep. It's sound logic, theirs and yours.
Related, I joined a new company this year that makes heavy use of LLMs. My team skews early career, and I was mildly anxious about how coaching/mentor-level PR feedback would be received on their agentic PRs. I like reviewing code, and I'm not sure how I'd adapt to it being moot at work.
Turns out they really appreciate it. Small sample set, anecdotal, etc, but they've expressed concern about honing skills in the new era, so feedback with rationale + examples speaks to them. And for the most part, the feedback is incorporated by default in future PRs.
Obviously open source contributors have very different incentives, but it was still refreshing.
This was really good - the best articulation I've seen yet of a rationale for a blanket ban on AI in contributions given a specific project's goals and organization design.
I appreciate the betting on human contributors. It seems like many places are shutting out people in favor of letting LLMs do the work instead. It's refreshing to see the opposite take place here. I'm confident this approach will provide more experienced maintainers and end up an excellent investment in the long-term.
This reminds me of when companies think that they can get better candidates by making candidates jump through more hoops. "We have 1,000 applicants, so let's ask them to explain their high school grades to us, even if they graduated 20 years ago. The lazy candidates will give up, leaving only the best ones."
The developers least interested in jumping through hoops are the ones that make the most valuable contributions. If you tried to get Jeff Dean to collaborate with you and told him that he's not allowed to ever use LLMs, he'd happily go off and work on the 10,000 other projects that would gladly take a Jeff Dean, LLM-assisted or not.
It comes down to conflicting assumptions about where the top performers are. I believe the top performers are overwhelmingly using LLMs, and it sounds like the Zig team believes that's not true.
This is a bad argument for banning LLMs. These PRs will continue to flow in regardless of what rules you put in place because the contributors generating these PRs don't read the rules. It's like making it illegal to own a ski mask because you notice that many criminals use ski masks to hide their faces.
That's a legitimate opinion to hold, but our experience with triaging PRs pre-LLM ban does not match your assumption at all.
That's a shortsighted argument. We don't expect the policy to automagically make LLM contributions disappear, but instead:
users are warned upfront about the consequences of using LLMs, which is in general good practice whenever banning is involved (i.e. users who read the rules should not be surprised when their behavior leads to a ban).
users who have good intentions and who like to use LLMs know to not waste time (and money!) on generating code that we don't intend to merge anyway.
There is literally nothing to gain by not informing users of how we operate wrt LLM-assisted PRs.
I don't understand how you'd be able to tell.
You'd be able to identify low-value contributors who use LLMs irresponsibly, but you can't identify someone who's both competent and uses LLMs effectively.
For example, if Mitchell Hashimoto didn't disclose his LLM usage, would you have assumed he's a high-performing non-LLM contributor?
That's not the argument I'm making.
I'm saying that the LLM ban filters out potential high-value contributors who use LLMs and does not effectively filter the behaviors you described (hallucinated APIs, hopelessly broken code). By banning LLMs, you lose high-value contributors who like LLMs, but you'll continue to receive negative value contributions from people who ignore contribution guidelines.
I don't understand your mental model of a user who contributes a 10 KLOC PR that doesn't compile who would have not done it had you had a rule against LLMs. You already have rules against the behaviors you described and the PRs come anyway, so why is an extra rule going to stop them?
In your Jeff Dean high school analogy, you're assuming a filter whose behavior does not correlate to the value of a hiring candidate.
What the author is saying in the article is that they have consistent evidence that there is a positive correlation between LLM-using authors and low iterated-value-over-time contributors.
Discarding hiring candidates because they don't want to talk about their high school record is one thing. Discarding hiring candidates because, say, they can't demonstrate an ability to write and reason about code is very different.
I think that's a valid position, but my objection is that based on what they describe in the blog post, the justification for the LLM ban sounds weak.
The evidence the blog post offers is:
Those facts alone aren't enough to justify a blanket AI ban because, theoretically, 100% of the good external contributions could be AI-assisted as well.
It's possible that the Zig team has more data on this than they've shared here, but my objection is that the blog post does not make a convincing argument for the AI ban.
I mean, that's possible. And, sure, we could discuss all sorts of hypothetical scenarios that a hypothetical open source project could hypothetically be in. If one were discussing a meta-policy around whether all possible open source projects should have AI bans, that would be an important discussion to have.
But the Zig team only cares about their one real project and in that project, it seems like they see self-stated AI use being correlated with lower-quality long-term contributor value. Is it possible that actually many of the high-value contributors are actually using AI and being sneaky about it? I suppose it's possible. But Occam's Razor suggests that's pretty unlikely.
Hi Bob, I'm a big fan of your work, so I find it disappointing that I tried to engage you in good faith and it feels like you're ignoring everything I said to focus on an uncharitable read of a minor detail.
I don't mean that it's important to consider the possibility that 100% of good contributions are LLM-assissted. I'm saying that we don't know the actual number and the blog post doesn't address it at all.
I don't think it's a wild hypothetical that 30-50% of good Zig contributions involved LLM assistance. And if the Zig team is happy to sacrifice those 30-50% or they think it's far lower, that's fine, but my point is just that the blog post would make a stronger argument if they focused not only on how LLMs produce noise but addressed the tradeoff of losing some percentage of good LLM-assisted contributions.
What's sneaky about using AI to contribute to Zig? Prior to the ban, there was no requirement to disclose AI usage, so I find it likely that there are good Zig contributors who used AI assistance.
It is. Like what are you basing it on? Because it flies against all data points we got from open source commiters and maintainers over the past 2 years. If this is your base model, you are going to have to do a lot of work to prove it before it can be used for this kind of analysis.
The point isn't proving an exact number (which I don't think is possible with the information available) but just finding out what the disagreement is.
If we agree that 30-50% is a reasonable estimate of how many good contributions used Zig, then what we disagree about is whether driving away 30-50% of potential contributors is a worthwhile trade in banning AI.
If we disagree about 30-50%, then it doesn't make sense to talk about whether the tradeoff is worth it because we have upstream disagreement that makes discussion of the tradeoff pointless.
My 30-50% is based on 90% of working developers I know personally now use LLMs, and most of the developers whose work I follow through blogs now use LLMs.
Yes, that's going to happen when token usage is tracked, and factored into promotions. Ask if people think they're doing their best work, and you get a more interesting story.
AI is a pretty hated technology outside of some bay area and founder bubbles.
Everyone in the 90% I know finds LLMs net positive to various degrees.
I only know of two developers whose employers pressure developers to use LLMs, but they're both on the high end of positive attitude towards LLMs, so they'd be using it anyway.
As additional anecdata, of the 7 developers I'm good friends with socially + me (all but 1 east-coast, outside of the bay area), 2 are required by their job to use AI, 2 are encouraged to do so. Of the ones required to use it, one is all-in, the other uses it sparingly as it is only sometimes useful for them. Of two that are encouraged to use it, one is positive and one negative. Of the remaining 4 who are not influenced by their job, they generally don't use it, either because they're skeptical or because they don't find it very useful.
Notably the one both required to use it and all-in on its use is the only one working at a Bay area startup.
I sincerely encourage you to find more communities to interact with. You seem to have fallen down the rabbit hole of programming "influencers" which is most definitely not representative of any widespread trends.
As a pro-craft activist who fights against the dangerous proliferation of LLMs, the vast majority of programmers I know are somewhere on the spectrum of fairly-critical to total abstention (like myself). They work on all sorts of commercial and open-source projects, some of which are quite well known in their respective fields. Am I claiming my anecdata is representative? No. I'm simply saying your anecdata is no better than mine.
Honestly, my best guess right now is that pro-AI sentiment across all developers is at best around two-thirds, and that percentage is actively falling. I expect it to dip below 50% as the bubble continues to burst and inference costs skyrocket.
Can you clarify? What's the mistaken belief I have? And where did you get the impression that my opinions about AI come from influencers?
I'm surprised you don't know more than 2 people at Google or Microsoft, to pick two well known, large companies heavily pressuring their employees into LLM use.
I deleted a longer draft of this comment because this discussion has been long, but I do just want to mention that Zig's no-AI policy went into effect in July 2025. (https://github.com/ziglang/zig/wiki/Contributing/61aff1ce9b3ebd8c38c0f5a18db00c06f0fa4022) Claude 4 had just come out and 3.7 was not good at writing Zig. With that context I don't think "30-50%" of good contributions were AI assisted.
I've been using Claude Opus the past couple months to write Zig and IMHO it is now good at writing it.
Presumably, if they existed they could speak for themselves. Their relative absence indicates to me that this group of people is insignificant.
There is no point for anyone who would use LLM existence to speak for themselves because they are not permitted to contribute to the project for a while now. The policy is quite black and white.
It seems very strange to me to imagine that there is a significant group of developers who fulfil all of these criteria:
...yet, so many (SO MANY) LLM advocates seem very happy to speak up in forums and threads like these.
I would take the position that the number of people is greater than zero but I'm not going to argue either way. All I can tell you that some quite prominent agentic coding figures are very happy Zig users and experienced developers. That's why this topic is somewhat interesting because Zig has a hard anti AI policy but the user base has some prominent agentic engineering users.
I know that those people exist and my personal feeling is this: If they now feel unable to contribute to Zig despite wanting to because they are unwilling to forego LLM usage, even only while contributing, then I worry about them and their long term health.
That seems entirely unnecessarily judgmental but since this is your personal feeling, you can have those.
I just responded to your initial comment because you said they could speak up for themselves and I think in the current configuration (AI being banned for more than a year), you won't find anyone who can speak up because they a) either are breaking the rules and won't say or b) do not contribute any more / won't start.
I don’t mean it adversarially, of course. I am sincerely concerned about a lot of people (I also read your earlier post where you expressed similar feelings).
edit: My concern is that LLM usage seems addictive in ways very similar to gambling.
I don't think I'm ignoring what you said, but I think you are giving them an uncharitable read.
They know their community way better than you or I ever will. If they feel they have strong signal that LLM use correlates with poor long term contributor value, I trust they know what they are doing.
A policy by itself can never filter bad actors, duh. Enforcement is always a necessary complementary aspect. Same as with laws, they still need enforcement. But you still probably wouldn't want to live in a lawless society, especially when some people have powers that you don't have.
If by construction somebody would be able to hide their usage of LLMs, then yes the only possible conclusion is that we wouldn't be able to notice (again, by construction) and would happily merge that code. If this happened in the Zig repo, then yes, those PRs might have been merged, and if those contributors decided to stop contributing after the AI ban, then those would be lost hands at contributor poker.
You're not addressing the point.
Zig already had policies that told people not to make 10 KLOC PRs that broke CI, but they did anyway. If policy is meaningless without enforcement, then why not enforce the rules that they were already violating?
Okay, I'm glad we agree on that.
My critique is that the blog post would have made a stronger argument had it addressed the potential loss instead of focusing only on bad usage of LLMs.
By banning LLMs, you also lose out on potential good contributions. I can understand if you did that calculation and decided the tradeoff was worth it, but when the post says only that LLMs are banned because there was bad usage, it gives the impression that the decision was driven more by negative feelings toward LLMs than a rational evaluation of the tradeoffs of allowing or disallowing a popular tool.
Aside: I find "duh" as a response disrespectful, and if you continue being disrespectful when I'm giving you respectful responses, I will stop engaging with you.
Wouldn't duplicitousness (ie. a "high-performer" concealing their LLM usage) be a disqualifying character trait all on its own?
Prior to the LLM ban, theres's nothing duplicitous about using LLMs and not disclosing it. I don't disclose my whole hardware and software stack as part of every open source contribution I make .
And after it, there is. If you use an LLM to contribute, your contributions aren't welcome. If you use an LLM to contribute and lie about it, you're an asshole. While I'm not involved in the zig community, so I can't speak for them, but if you were discovered, I imagine you would be unwelcome as well.
For whatever reason, when people say "I don't want you to spit on my sandwich", nobody goes "but I could do it so you'd never find out!". When they say "I don't want you to spit in my code", everyone who isn't contributing seems to come out of the woodwork to volunteer to secretly spit in the code.
That's not relevant to the current discussion.
No, he is also sampling contributions where the contributor volunteered LLM usage.
The only undiscovered cases are contributions where the contributor used an LLM and created a high-quality contribution but did not disclose usage. Presumably that contributor then did not become a long-term contributor or alternatively stopped using LLMs, since they would then be asked to disclose further usage later.
Either way, this group of contributors are probably both small and not particularly valuable to the project long-term.
edit: Also, nothing is preventing such a contributor from contributing further, as long as they don't use LLMs. Someone who wants to contribute but is completely unable to do so without LLM assistance doesn't sound "most valuable" to me.
This is possibly true, but I'm disagreeing with the argument in the blog post, which doesn't reflect the claim you're making here.
My claim is in response to your claim. But sure, returning to the blog post I think it's pointless to speculate about potentially turning away good quality LLM-assisted contributions when the only problem at hand is an abundance of low quality LLM-assisted contributions.
There's a problem with this line of thinking, which is that pushing away good contributions is 99% not "at hand". Instead of contacting you and saying "I was gonna get into your project, but the anti-LLM policy put me off," the person just goes somewhere else. As such, it's a problem of unknowable size.
It's not unknowable, though! Measure the rate of high quality contributions and new contributors before and after the LLM ban. If the rate is significantly lower after the ban that would indicate that this theory has some weight.
However, in this case the ban was introduced when LLMs were mid. So with this method you can at best measure LLM contributors at a point in time.
There are two facts that seem to be universally true:
Every model released before the latest ones was absolute garbage. If you have tried to use LLMs but did not use the latest models you have no idea what's possible. (Illustrative example: The post I am replying to)
The latest model is not as good as the previous model, the previous model was better. (Illustrative example: Every single post on reddit.com/r/claude)
I don't think that was true for 4.5. Every few iterations there's a model that somehow suddenly everyone agrees is a genuine step forward.
And yet, at that time, the LLM-huffers were still saying that they were making massive productivity improvements. Were they lying?
No one is asking you to disclose your whole stack, but in this case the Zig project is asking you to disclose your LLM usage. That is what makes non-disclosure duplicitous.
You and orib keep digging into this line of reasoning, and I'm confused if I'm missing something.
kristoff claimed that they banned LLMs based on behavior they observed before there was an LLM ban (i.e., before anyone was expected to disclose LLM usage). I'm arguing that before the ban, they don't have accurate data about who was using an LLM and who wasn't. The data is biased toward bad contributions + LLM usage because they're obviously detectable, whereas you can't tell whether good contributions involved LLM assistance.
The two of you keep saying, "If you use LLMs now, then you're dishonest," but how does that relate to whether the ban was justified in the first place?
It's a common misconception that we can't tell who is using LLM and who is not. I'm sure we didn't catch 100% of LLM-assisted PRs over the past few months, but the kind of mistakes humans make are fundamentally different than LLM hallucinations, making them easy to spot. Furthermore, people who come from the world of agentic coding have a certain digital smell that is not obvious to them but is obvious to those who abstain. It's like when a smoker walks into the room, everybody who doesn't smoke instantly knows it.
I'm not telling you not to smoke, but I am telling you not to smoke in my house.
This is saying "if theres an error, we can tell if its LLM or human". But it doesnt include the case when theres no errors, which contradicts your original claim:
and is what OP is arguing against
Hi Andrew, big fan of Zig and your work, and I've been a (small) Zig financial contributor for two years.
That said, I'm having a hard time believing this:
You don't have ground truth, so how can you say this with confidence?
I get that you can spot when developers submit AI-generated code and don't thoroughly review it themselves, but what if the developer uses an LLM for assistance but still refines it by hand so that the result is the same as hand-coded?
The way I use LLMs to contribute to other people's open source projects is that the result should look the same as hand coding. I don't think I'm that unusual in that respect. But maybe I'm one of the people you're describing that has a blind spot to their own anti-patterns.
Thank you for the kind words.
I think people are underestimating just how many hours of my life I've spent reviewing PRs. I'm telling you there are obvious, noticeable patterns - even when there are no "mistakes". It's usually quite obvious.
But like I said it's not foolproof. Ultimately I want an invitation tree like lobste.rs in order to enjoy a higher degree of trust with first-time contributors. As is, it has become toxic. I hate being the cops.
Yeah, I couldn't agree more. A lot of people who crank out code don't understand the massive shift that reviewing code brings. It's an entirely different skill and it's just as rooted in interpersonal skills as it is technical skills. The article y'all shared tried to explain it and it still went over people's heads. Reviewers can tell for the same reason people can tell when they're talking to a chatbot instead of a real person, the tells are technical and personal.
I also like the idea of an invitation tree, possibly balanced out with mentorships for enthusiastic newcomers.
Bun just increased debug compilation speed in their Zig fork by 4x, but you'll never see it because it was helped along with Opus.
Is there a point at which your anti-AI rule would relax? Because if not, it's not falsifiable.
andrewrk already commented a link to this post by mlugg, which says that “AI is entirely besides the point” and that mlugg disagrees with the architectural redesign that Bun did to get that speedup. You predicted that Zig would never see that speed increase, but that post says “we are going to make these enhancements, but […] we’re approaching the problem with the care it deserves, so that when we ultimately ship it, we don’t cause regressions.”
Therefore, the Zig project not copying those Bun changes is not evidence that the Zig project will never accept AI-generated code.
OK, thanks for the clarification.
Look, I love software and the craft of it, so I get the concern. I just think we need to base opinions on evidence and not beliefs or conspiracy theories.
I would say that your argument is technically correct, which the best kind of correct. Yes, it is possible that before the LLM usage ban, there were a lot of good developers who wanted to contribute or maybe even did contribute without disclosing any use of LLMs. After the ban, those developers then all decided to stop trying to contribute, and also not to tell anyone about that. There are also a possible category of very skilled, very capable developers who came to Zig after the ban, want to contribute, are turned away by the policy and do not let Loris know about this.
This is of course possible but at least to me seems highly implausible.
I don't think Mitchell Hashimoto would be dumb enough to contribute LLM-generated code to a programming language where bad code and the failure to find it in the PR could have far-reaching consequences for both the compiler and every user of it.
This idea that the best "performers" (whatever you actually mean by this) are using LLMs and that they couldn't possibly make contributions without them doesn't make a whole lot of sense.
If they wanted to get code into Zig and they were capable, they'd simply not use LLMs to contribute and that would be it. If they can't, or don't actually want to contribute to Zig, they won't. There needs to be no speculation as to who can and wants to contribute to Zig.
Basing your policies on wild hypotheticals that directly contradict your actual evidence and explicitly (very obviously) contradict your actual goals would be an incredibly bad move.
This is like saying, "Mitchell Hashimoto wouldn't use Stack Overflow because he's too smart to just copy/paste code from StackOverflow without understanding it."
Using an LLM doesn't mean just blindly accepting its output without review. Competent developers use LLMs to assist in development and revise the LLM's output just like they can consult StackOverflow and write code based on what they learned.
It's not that they couldn't but they don't want to because they perform better with them. Mitchell said in a recent blog post:
I can write code in notepad.exe, but if a project required everyone to do that, I wouldn't contribute because I'd rather work on code where I can use the toolchain I like.
If the code that Mitchell in this hypothetical tried to get into Zig wasn't something he actually loved, then maybe it just doesn't belong in Zig.
Again, it's not very complicated: If you can do it, do it yourself. If you can't, just don't. We don't even really need to consider the "Well, person X can, but they really prefer not to" because it's irrelevant. The hypothetical of this hybrid case is pointless because anyone who actually could do these things with the policy in place and wanted to contribute would just do that: Contribute.
(In many of these cases we'd really only have to wait a year or two and many of these people will have had their abilities atrophy to the point where they're no longer able to do it as well, so I wouldn't really worry too much about the hybrid case even if it were relevant.)
For me, development work often includes a mix of things I love and things I don't. I love creating a cleaner abstraction, but I don't love the tedious work of migrating all previous callsites. That's a task I now delegate to AI agents.
If I'm understanding, you're saying that Zig banning LLM usage will not reduce good contributions because the good contributors are happy to eschew LLMs even if they like using them. Is that correct?
Or are you saying that developers who like LLMs are inherently inferior, so Zig banning LLM usage gets rid of inferior developers and their inferior contributions?
I think that both are untrue. If a project restricts me from using my preferred toolchain (even when the end result is the same to outside observers), I'd prefer to contribute to one of the many other projects that have no such restriction.
I've contributed a small amount to Zig but I won't in the future, not out of protest but because the LLM ban makes it more tedious to contribute than other projects available to me that give me the freedom to choose my own tools.
I don't know whether "happy to" is the case, but any reasonable person who actually wanted to get code into Zig would just write it themselves. It's not a very radical idea.
It's no different than contributing to a project that technically uses C++ but you're not really supposed to use most of C++ in it. If that's the policy, I shouldn't expect to use a grab bag of modern C++-isms because I feel entitled to do so (or because I feel dependent on them). It has the same effect on quality as well; most people who can't accomplish something without using a ton of C++-isms in general (modern or not) are also worse programmers, so filtering them out might affect the top band (unlikely) but it definitely also filters out a lot at the lower band.
But again, the goal for the Zig project isn't even to filter people out, it's to invest maximally in the human beings making contributions in order to build their community.
Most of their contributions certainly will be, yes, and that's obviously exactly what the observation suggests so far.
It's your right to try to contribute to anything you want, for any reason you want. Since you're so passionate about this I suggest you start a fork of Zig (called Vig?) for people who want to use LLMs in their PRs.
I think the much more likely thing will be that people either break the rule or just don’t contribute.
Imagine you had a project that said you must not use an IDE to contribute or you may not work on a windows computer to do it. Those are all rules that will be impossible to police and people won’t follow them or they are too off putting.
The AI rule is or will be the same.
We don't know what the shadow number of contributions that have been made using LLMs to a meaningful degree is, of course, but for now the policy is clearly stopping the most egregious cases and most likely a good chunk of the ones who pretended they made their contributions themselves.
I don't think this is a very apt comparison because an IDE doesn't have a meaningful impact on the quality of the output, whereas LLMs still cannot produce good code beyond single procedures or ridiculously simple boilerplate.
For the case that actually mirrors that of an IDE, no one can actually tell that you used an LLM to ask questions about a code base, and what does it really matter anyway? The worst that would happen when your PR is made is that you'd then be mistaken and the project maintainers would set you straight; then you've gained actual, real knowledge and can work (once again, "manually") on solving the real problem the real way. In either of those situations you're still fine and not at all a risk of being turned away because of the policy. You also then get the benefit of the stated goal of the Zig project in that they are investing in someone learning real things about the language, the compiler and the domain.
The amount of SpecLang you'd have to put into your process to produce something good out of LLMs (the very frontier, even) for anything larger than a function or two rivals that of just writing the code yourself, which means that the policy's real effect is simply keeping LLM code out of the project, which is not a bad policy at all.
It might be. It might also seem to users of LLMs that the LLMs produce good enough code in the future because their ability to spot bad code gets dulled by their use of them, just as using them as a crutch for thinking is known to cause a dulling of the mind itself. I think this is a whole lot likelier than LLMs actually becoming good enough to produce code that passes the human test, and the truth is that a project always has the right to refuse your contribution just because regardless, so any kind of borderline cases can be handled just about any way the project chooses, and without a massive amount of entitlement there is very little one can really demand in that situation.
This is not true at all. It sounds like you haven't used LLMs in at least a year or you only use LLMs for niche languages.
If LLMs can't go beyond single procedures, how are people creating entire JavaScript engines from scratch with LLMs?
I'm not saying the AI-generated JS engine is better than a human-written one, but I can't understand how you can believe the upper limit on LLMs is a single function or boilerplate in the face of projects like this.
I've used GPT-5.4 on every thinking level, Composer-2, Opus4.6. Without fail they can't seem to produce anything at all without bloating the solution by at least 2x, the highest I've observed is 6x. That's not a great basis to build things with at all.
I've run a ton of experiments using more or less specs up-front, varying the size of the thing they should spit out, etc.; I've done more than I could ever be asked to evaluate these things. Some of the bloat comes from over-guarding against things that can't actually happen and any programmer who knows the domain or even just language constructs would know that to be the case.
Over-guarding might be a good thing, except I also find that the output often has obvious holes in it where it's not doing what it should, once again obvious to anyone who's been in the domain for even a short time.
You can talk about languages all you want, but I've found that they can't even do basic memory management which they for sure have tons of examples of, and it's an exercise in and of itself to make it understand what it should allocate temporarily in a main loop and what that means at something like a frame boundary, for example.
Honestly, if you think they spit out acceptable results that's more a reflection of your standards than anything else, at least from what my own observations tell me. I'm not impressed with LLMs and by extension I'm even less impressed with the programmers who use them and claim them as sufficient, even with detailed specs and babysitting.
You and @gonz might be using different definitions of “good code”. That LLM-made JavaScript engine shows that LLMs can write large amounts of functional, correct code (which might be your definition), but it does not demonstrate that the LLM-generated code is also efficient and easily understandable by a human (which might be part of gonz’s definition).
A discussion on the quality or merit of LLMs does not seem to be relevant here because the stance of the Zig project has little to do with quality but it's a philosophical take. Zig does not want any LLM code or issues, regardless of quality. I think the IDE comparison is quite appropriate as a result. We don't need to debate the benefits or downsides of IDEs or LLMs, because that's not what is being judged here.
This policy will have effects on who participates and it will create a filter but I don't think it will change behavior. Personally AI has become a tool I use, and when there is a project that disallows me to use it, I'm more likely to not participate (because I chose to be a honest person and follow the rules of someone else's turf).
A value judgment on LLMs is necessarily part of the argument: we have a bunch of closed boxes (PRs) and when spending time opening them, we discovered that the ones that have "LLM-assisted" printed on them yield on average subpar outcomes alongside boxes that do not present the stamp, but for which we can tell that it's missing because of clerical errors.
If LLM usage did not lead to subpar outcomes in our experience, we wouldn't be having this discussion at all. We might be having a different discussion about the ethics of using LLMs maybe, but we wouldn't be talking about literal game theory.
We source core team members from the pool of trusted contributors, we get a ton of what is essentially free labor from trusted contributors, and we also built a community whose culture is heavily influenced by these people. And this funnel starts at looking at PRs, a process that the existence of LLMs has actively made worse. As mentioned in bold in the original blog post, it's a key part of our business model. I don't know how you could get any more practical than this.
With all due respect Armin, I think that more often than not you interpret our communications based on your own preconceptions rather than on what's actually printed on paper. Maybe, despite your beliefs, there are some concrete reasons for this policy. Also maybe you shouldn't have cringed as hard when we left GitHub, or at least should have reevaluated your initial impression after new evidence of GitHub being broken came in, even if you don't consider us people of real weight. Just my two cents.
To be clear I’m not criticizing the policy. You have your reasons for it, but your policy is set because your expertise with LLM assisted or generated code is bad, and as a result you banned the use outright. That is absolutely fine! But as a result it’s a policy that does not set a quality bar for contributions but it sets a more philosophical gate. LLM generated code does not fit into the philosophy of Zig.
To be clear: that is fine by me and I won’t criticize it. Your project, your rules.
Maybe. I should probably do a lot of things in life, but at the end of the day I’m a person with emotions and feelings and the responses one has are those. The way the zig project expresses itself in the blog just doesn’t really vibe with me but that is also okay — at least for me! I use and like the language for the most part, but that won’t make me automatically agree to all decisions and it won’t change me as a person either.
There are just things people will never be in agreement on for more deeply rooted reasons.
Zig has plenty of weight, but yes, I think Mitchell’s move in my book will pull more people along than the Zig project which is much more niche in comparison.
That's a new usage of the word "niece" for me. What do you mean by it here?
That's because I can't type. It was supposed to be "niche".
Ah! Sorry for not spotting that. It didn't trip the fuzzy match in my brain, and I found myself wondering if "niece" had grown a new meaning.
Well, all the policy actually does is bring us back to the time before LLMs:
This de facto is literally just how things were before LLMs, but people are making it seem radical out of nowhere.
The only thing that actually changed in the greater ecosystem is that people can now pretend that they can write the code, so I think that really is the operative difference in why some people find this policy so offensive.
People keep using this word without making it clear whether this is a qualitative evaluation or a quantitative one. If it is quantitative, then it can be measured, and if it can be measured, then it can be controlled. If it is qualitative and it never has a tangible impact, then it literally doesn't matter (unfortunately). If it is qualitative because it may eventually have a tangible impact, then you might have an argument- but if NOTHING is measurable about it TODAY, then that would frankly be... a really strange phenomenon, because it means that there is an effect that spontaneously emerges in the future that shows absolutely zero evidence of existing even in a "seed form" today.
I'm getting the impression that you think 'qualitative' means 'unobservable' or 'not objective'... that is a misunderstanding of the word, I think. It just means observations that are more clearly described with words than with numbers.
For example, think of a colleague whose explanations you really like, and find easy to follow. Can you quantify, with a number, how clear their explanations are, or what makes them so clear? Probably you can't put a number on it. But would you say that their clarity of speech/writing has "no tangible impact" on your relation and work?
Despite the widespread myth that only quantitative measurements and descriptions are of use to science, many non-quantifiable things, i.e. things that cannot be usefully measured on numerical scales, can be given precise objective descriptions that can play a valid role in scientific theories and explanations. Thus linguists describe language, chemists describe matter, [and anthropologists describe behaviour and culture —Sietsebb]
—The Fontana Dictonary of Modern Thought (1977), lemma 'quantification'
Bloated output by at least 2x the amount that a (competent) human would write, guarding against things that can't happen and not handling things that can. I've tested a tons of different permutations of spec-driven development and so on, which people claim works, but it all spits out low quality code that has obvious bugs in it.
If there are bugs, then why didn't you test for them, or ask it to? (I know it lies sometimes. Adversarial prompting, where you basically get another AI or instance to poke holes in things, helps.)
2x code output? Well, yes. That does happen. Have you seen human DNA or done any research on just how terrible it is? It still leads to (mostly) working humans. Basically, the user of the product of whatever that code is contributing to won't give a shit, and they will get that functionality 5x faster, so it doesn't matter that it took 2x the code, or that the code is ugly, or that it doesn't use comptime as much as it could, or that it has a bloated test suite.
Future AI will clean SOME of that up.
Do you expect to cover literally every edge case in your spec? At some point just writing the thing myself is a whole lot faster than sitting there meticulously writing the spec that's supposed to cover everything, and it reveals more about the design as well.
Right. Ship garbage, because no one cares what your code looks like? I care what it looks like, I have to work with it, modify it, add things to it and understand it for the purposes of understanding the exact data transformations in it. Some of us have higher standards than you people.
Our industry has become a clown college.
Now give me the arguments that only count towards LLM code and not human output in general
[Comment removed by author]
I literally shipped this demo in a week. https://incitez-web.pages.dev/?demolinks It's a complete rewrite of the eyecite citation search engine. The original was in Python, I rewrote the whole thing in Zig, using the original test suite as a spec. It is 50 times faster, 15% or so more accurate, and since it is Zig, it compiled to WASM, so this runs completely in the browser- no attorney/client privilege is violated.
Go ahead and play with it looking for bugs.
It was actually done in an hour. The week is what it took to refine it, because (surprise) I care about code too (and yes, at first there were a bunch of little things to fix). But this would have taken any human literally months to do manually to ANY level of quality. This one passes the entire eyecite test suite AND found bugs in it (corrected and sent pull requests to eyecite's repo) AND goes over and above it in terms of citation-finding. Did I mention it is FIFTY times faster?
Here's the repo. Go ahead and code-review the code. (The web-demo repo is private, but most of what it uses is from here, plus my pmarreck/docscan lib (public) to parse pdfs, doc's and docx's.) I haven't written a single line of Zig in my life, but I've been a software developer for 30 years now and I know pretty damn well what failure modes to look for. https://github.com/pmarreck/incitez
Your bespoke production is like hand-building a car or hand-sewing an article of clothing. It may be better, but very few people are going to pay for that.
You're completely missing the point.
False. 1) I'm responsible for it if something goes wrong in a prod env with it. That is all that matters. 2) This was not just an ask, it was a cooperation similar to pair programming.
So now you're just deliberately being an asshole based on literally zero information? Nice. You also forgot the speed of development. 1 week to go 50x faster AND run in the browser AND pass all the same tests. That is literally impossible without LLM assistance, because the overhead of doing that conventionally with a TEAM would have already cost that much time just to communicate/coordinate- and much of their code would have sucked, to boot. To call that anything but impressive is just fucking oxbrained, frankly.
Again, you're completely missing the point. That codebase you stopped updating 2 years ago that you wrote in the dead of night to meet a deadline, you also know nothing about, regardless of whether you wrote it or not. That doesn't mean it isn't valuable. You've simply shifted the ignorance. I have learned when I've needed to, on the hot paths, which is where it counts. "Letting an LLM do everything for you" is pure ignorance, dude- I'm there every step of the way, making difficult tech decisions in some cases, and vetting the diffs, calling anything suspicious into question.
Dude, with the attitude you have, you're headed for a firing in a matter of time, and I wouldn't bat a fucking eye. Good Lord, you are obstinate. Anyway, there are multiple types of suckitude and only some matter- the measurable ones. Guess what's good at creating measurements.
And all of those projects work well. I don't know what the Zig community would think of it, and I wouldn't care. Just like I don't care what you think. Just leave your blinders on while the tidal wave comes, my dude. I'm sure it'll all work out great.
Jesus Christ, the whole lot of you obstinate skeptics are just lost. To make it abundantly clear to your level: NO CUSTOMER IS GOING TO CARE ABOUT YOUR BESPOKE, SLOW-ASS-CODED POLICE FORCE MANAGEMENT SOFTWARE (by the way, there is nothing at all sophisticated about that type of software and the LLM will eat your lunch) IN 12-24 MONTHS IF YOU ARE NOT DELIVERING AS QUICKLY AS COMPETITORS, EVEN IF YOU PROVIDE SLIGHTLY BETTER QUALITY (and it would not just have to be slightly better to justify that massively increased delay, it would have to be 100% better! You willing to take that bet? I'm not.)
But hey, it looks like you're doing government contract work, so perhaps you are slightly safer than those in the normal private sector. For now.
Are these LLM contributions coming from known-good developers, or randoms?
Because if you're grouping them together, that's a pretty serious conflation problem. It would be like grouping together people who know how to use a tool with people who don't, and then blaming the tool
On the /r/ProgrammingLanguages subreddit we've been dealing with a lot of AI slop in recent months. To combat that we tried a bunch of different things, the most recent one being that posts referring to GitHub are filtered (= basically hidden) automatically and the author is notified about needing to copy-paste a standard phrase into a comment. Essentially a dumbed down Reddit Turing test.
On the surface that may seem pointless: surely the LLM users will just do that and lie about it? And indeed I've seen this sentiment (essentially "you can't stop LLM users") in other places.
In reality this approach has thus far proven remarkably effective in spite of how stupid it is. Why? Because many LLM users are extremely lazy, so much in fact that just copy-pasting a phrase is apparently too much to ask.
What I'm ultimately saying is is that while it may seem LLM users can't be stopped unless somebody comes up with a magical oracle, in reality you can filter out let's say 80% of the noise by applying a few rules that seem dumb at first but end up being surprisingly effective.
I think automated rules to trip up lazy LLM contributions make sense, but isn't that the opposite of what we're discussing here?
As far as I can tell, the Zig team isn't adding any automation. They're just adding a new rule that contributors are expected to read that says "No LLMs." So, the lazy vibe coders will just skip reading that rule and continue the low-effort contributions, but it filters out the developers who want to use LLMs but still do their normal due diligence in verifying their code.
Do you have an example of a developer who wants to contribute to the Zig project but is prevented from doing so by the LLM policy? Because it seems to me like someone who is both good and willing should be able to find a way to contribute that doesn't violate the policy.
That would be me :)
Interesting, did you contribute to Zig before? Really, the question I would like to ask is this: Do you worry about having made yourself dependent on a tool that is changing so rapidly and unpredictably and has such big open questions around copyright and legality, to the degree that you want to spend significant time working on an open source project but feel unable to do so without using the tool?
I only contributed to issue discussions and a little to the ecosystem by implementing zig-cross (based on an article written by kristoff). I have been using LLVM for quite a few years and contributed to a bunch of different projects in the ecosystem. There are a few issues in the
zig c++driver that I have on my TODO list, but never enough time ;(My open source contributions started in 2013 or so (when I was ~17). I see the ecosystem as a campsite: leave it better than you found it. In this way I made a lot of minor/'drive-by' contributions to projects in my ecosystem (security/reverse engineering). When I got my first job in 2018 my open source contributions suffered majorly. LLMs allow me to get back some of this time and I personally do not think my OSS contribution quality suffered (if anything I suspect the quality has gone up since I can actually spend time on tests and design iterations). My GH is public so I guess you be the judge...
How would I have this information?
The effect isn't measureable because the effect creates the absence of an artifact. Developers who use LLMs and want to contribute will see the rule and decide not to contribute. Maybe some will post somewhere publicly, but the vast majority will just silently move on.
I just don’t understand why this theoretical scenario seems worth considering if there isn’t even one example you can point to?
I'm not able to name a specific person because how would I? But I think it's obvious that this ban shuts out some developers who might otherwise make positive long-term contributions to Zig.
Which of these points do we disagree on?
Well, #1 seems like one of those stats that are in the 90% of stats that are made up on the spot.
Even if it were true, the population of LLM users in #2 would not be influenced by #1. Using LLMs and wanting to contribute to Zig are independent variables.
But 3 still works, sure. You can make that argument, but from my experience I'd rather not have a contributor that gets upset at being asked to follow a simple rule that they don't necessarily agree with.
I actually think it's more important to take care of the existing contributors. Establish a clear culture that helps them be happy and creates an environment where they look forward to working with other people on the project.
Think about projects like sqlite, where their insular team and uniquely worded code of conduct seems shocking at first, but actually makes them extraordinary effective as a team.
Honestly the backlash to these "no LLM" policies reminds me of the loud handful that despised Codes of Conduct when those were gaining traction. Like, if you're the time of person that goes into rule lawyering when you're asked not to be jerk then I probably don't want to work with you in the first place.
This is in effect just saying "I don't want you because you disagree with the rule." That's the exact thing the rule was being critiqued for!
You're basically saying "I'd be fine with a LLM-using contributor but not if they get upset at not being allowed to use LLMs."
It's not quite that. It's more "I don't want you because you are unwilling to agree to the rules that we as a project have agreed to." Which is not unreasonable. The critique is that this is shutting a lot of people out that would otherwise have been valuable contributors. That's a fine opinion to have, but there is very little evidence to suggest that this outweighs the tangible benefit of shutting out a lot of low effort slop contributions.
Note that both of these are measurable. The reviewers of the Zig project can measure the rate of good and bad contributions before and after the ban and see if there is any detectable decrease in good contributions, and if the ban had little or no effect on bad contributions.
Another metric that the reviewers can use is their own happiness: Did the ban make their lives better? Reviewing slop is very demoralizing. Anything that would reduce the risk of wasting time on slop would be a huge win.
I'm not claiming it's a real stat. I don't think we have reliable stats on this. I'm asking if 20% is a reasonable lower bound because if we have vastly different assumptions for this, it affects the rest of the model.
What do you think the lower bound is?
They're not statistically independent. If #1 were 100%, then #2 definitionally must also be 100%.
Or, more relevant to this discussion, if the rate of LLM usage among all developers goes from 10% to 40%, I'd expect the rate of LLM usage among potential high-value Zig developers to also increase a similar amount.
If you think that LLM usage among potential Zig contributors would remain constant (or change in unrelated ways) while global LLM usage quadrupled, can you share why you think that?
I actually think it would be higher among potential Zig contributors because using Zig indicates openness and curiosity about newer tools.
It seems like this is more about your view of the rule rather than contributors being able to follow "a simple rule."
For example, if the rule was that you can't use Apple products or VS Code for any Zig contributions, would you say, "Oh, that's fine because it's just a simple rule." Or would you think, "Hey, what does it matter which tools I use as long as I do due diligence and take responsibility for the final result?"
Not a zig dev, but: So far, the number of skilled developers committing LLM-generated code I've seen contributing on GitHub has been far below 1%. At best 3 people vs >1000 contributors.
And even those 3 weren't LLM regulars, but 3 skilled devs contributing who also happened to have a side project with some LLM assistance.
In my own experience even Opus 4.6 and Gemini 3.1 Pro are unable to generate even intern-quality code. And that's not because I'm just not able to prompt them correctly, even if I ask people who frequently use AI to show me their code (which they claim is fine), it's always an absolute mess that I'd never use in production myself.
How are you counting who's using LLMs and who's not?
Do you have examples of prompts and bad outputs from Opus 4.6?
In my experience, Opus 4.6 generates code better than most fresh CS grads.
For example, this project is mostly AI-generated from a template, but I guide the AI and ask for revisions.
It's a personal project, so it's not as polished as what I'd produce if I were writing code for paying users and working with teammates, but I still consider it good quality.
The main point we disagree on is this:
LLM usage in itself is a negative contribution.
Right, this is a fundamental disagreement.
If you think it's impossible to make a positive contribution while using LLM assistance, then discussing anything else about an AI ban is pointless.
[Comment removed by author]
There's a ton of people in the comments who need to remind themselves that "absence of evidence is not evidence of absence."
They're engaged in the digital equivalent of thinking they can identify plastic surgery because they can spot bad plastic surgery. And even if they were right for now, I wouldn't bet on that remaining true as models advance.
There is evidence.
First, did the ban work? The reviewers can self-evaluate: Do they feel like the burden of reviewing slop contributions lessened? If so, the ban worked.
Second, did the ban result in a lower rate of quality contributions? This can be measured statistically.
Except it is.
Absence of proof is not proof of absence, but when you expect evidence for X, and you don’t see any, your probability for X has to go down. We can even quantify it.
Let
P(X)your prior probability thatXis true.Let
P(E|X)your probability of observing some specific evidence ifXis true.What should be
P(X|~E), your posterior probability thatXis true, after you failed to observe such evidence?Recall Bayes’ formula:
P(A|B) = P(A) × P(B|A) ÷ P(B)Therefore:
Note that "
Eis evidence forX" means that it is more probable whenXis true, than whenXis false:Another way to look at it, is to look at
P(E)itself:Assuming
P(E|X) > P(E|~X)(Eis evidence forX), then:So we got "
Eis evidence forXmeansP(E|X) > P(E). Kind of obvious, but I need it to analyse my Bayes formula from above:P(X|~E) = P(X) × (1 - P(E|X)) ÷ (1-P(E)). Here we go again (recall that probabilities are all in]0, 1[):Phew. Now I’ve got proof: when
Eis evidence forXand you get~E, then your probability forXgoes down. Absence of evidence is evidence of absence.You're making a couple mistakes here and regurgitating all of Bayes' Law is the opposite of convincing.
First, Bayes' Law is about what you believe is likely, not necessarily what is true, so invoking it doesn't actually refute what I said. Your conclusion "Absence of evidence is evidence of absence" is simply wrong under a Bayesian lens. But that's an old frequentist criticism.
More relevantly, even under Bayesian logic, saying "you don’t see any [evidence]" is combining two different scenarios:
Given 2, any posterior updating should also include P(I can't reliably identify LLM output|I didn't categorize anything as LLM-created).
Ah, yes, frequentism. The thing we should have ditched like 80 years ago. But no, scientific papers still use it today, I’m sick of it. Also, Probability Theory: the Logic of Science (chapters 1 and 2 at least), should be required reading for any aspiring scientist. I mean how can you read that and still be a frequentist? The axioms of probability theory are so few, so obvious, which one could you possibly reject? I understand the use of frequentist recipes as shortcuts, imperfect approximations of correct reasoning… but actually believing this stuff? That’s alien to me.
Do remember than in Bayesian thinking, probability is in the mind: how "likely" something "is" is how sure I am that something is true. Or how sure you are. Depends which priors we’re talking about. I often say my probability, or your probability to reflect this. Strictly speaking, "the" probability does not make sense.
Don’t mistake this for epistemic relativism though. While each of us have different prior information, it doesn’t mean we can believe whatever we like. The correct probability is derived from the information we have access to, not from how we feel about it.
To be absolutely rigorous, we should talk about
P(A|I), meaning how likely it is thatAis true, given background informationI. And the full Bayes formula should be written:Did you mean "under a frequentist lens"? Because if you really meant "Bayesian", you have to know this is probability theory 101 you’re disputing here.
I considered that, and swept it under the rug for simplicity. But if we must, it’s just an additional stage: not seeing evidence where I expected some increases my probability that there is no evidence, which in turn increases my probability that there is nothing.
Absence of observation is evidence of absence of evidence.
Evidence of absence of evidence is evidence of absence.
Therefore, Absence of observation is evidence of absence.
The fundamental mistake in your comments is writing about Bayes in a non-Bayesian manner.
If you want to argue that the Bayesian interpretation is the only legitimate one, you should frame it in purely Bayesian language, because you come across as deeply confused when you talk about it in terms of universal probabilities.
E.g.:
This is what I mean. You can think that "the" probability doesn't make sense, but that's a minority viewpoint, especially in science, and pretending otherwise does you no favors. Don't respond to people saying "the" probability as if they're actually saying "my" probability, because many of them are explicitly not doing that.
Most people talking about "the" probability are very much talking about the true probability, however it's defined outside of humans, brains, and evidence. (Call it
E[some event]of an infinite random process if you like, there's plenty of definitions.)You lamented that science has not adopted Bayes enough, but I can tell you from my own time in academic neuroscience, it's not solely due to slow adoption of new methods. (Many new statistical techniques get adopted within a decade or two.)
It's mostly due to Bayes's fundamental change in interpretation, and not that they just haven't heard a good enough argument for Bayes yet.
It's great for an experiment to tell you how to update your priors, but most scientists are interested in discovering the true likelihood of something independent of experiments. When I originally said "absence", I am talking about the true prevalence of a thing, not just my personal estimation.
One personal estimation I do have is that the likelihood you've been an academic scientist is low, otherwise you'd have had a more complete picture of why Bayesian stats haven't taken over. It's not just inertia, laziness, or that frequentism doesn't have serious flaws.
Seriously? But it also increases your prior that you're missing things. To be intellectually honest, you have to update all relevant priors, not just the ones you want.
This is like someone with dementia claiming they don't have dementia because they don't notice they're forgetting things.
Of course. Here’s how it works:
Eis evidence forX. Which means, findingEmakesXmore plausible, and proving that there is noEmakesXless plausible.E. Haven’t seen it, haven’t even tried. But we do know that if we actively look for it, there’s a chance we’ll find it, if there’s any.Efor a while. Still nothing. This makesEless plausible.Xis now less plausible.Here’s a concrete example with the same structure:
I believe you can accept (1) and (2). You may be tempted to dispute that (3) follows from (2), or that (4) follows from (1) and (3). But try that in law enforcement and they’re going to look at you funny.
If you mean something like "can’t miss the shots you don’t take", sure. But that’s irrelevant to my main point: looking for evidence, and failing to find it, makes the thing less plausible.
Depends on the field. When you can have controlled studies it matters less (or at least, less visibly so). And when the alternative to publishing p-values is perishing, it’s hard to let go of the old ways.
Plenty of definitions, but can you find one that makes sense in a deterministic world?
Because last time I checked the laws of physics as we know them are deterministic. Same initial conditions, same results, every time. Even in quantum mechanics, if you just look at the equations and don’t interpret them. (Just because the photon bounced off the beam splitter doesn’t mean it didn’t also get through. That’s how we get interferences.)
For instance, when you throw a fair die, the reason you ascribe a uniform probability as to which face will end up, is because we humans aren’t consistent at throwing die, the process is chaotic (diverges quickly), we can’t observe the initial conditions precisely enough, and the die is well balanced an symmetric. Not because its mystical propensity to land on one of its faces is equally distributed.
Then what is it? Care to outline that "more complete picture", or at least estimate how many pages it would require?
A fundamental change in basic thinking, yes, but this goes way beyond interpretation. Interpretation would be like squabbling over Copenhagen vs Everett while the colleagues are building lasers, interferometers, and quantum computers: a philosophical perspective, of no practical consequence.
Not so with statistics. Here we’re talking about how to conduct and prioritise research, what policies should be derived from the results, and ultimately avoid countless unnecessary deaths.
I take you’ve heard of Ronald Fisher? Champion of frequentism, and staunch critic of a 1950 paper that said tobacco caused lung cancer. "Correlation does not imply causation", but it damn sure makes it more plausible. Ignoring that basic fact likely participated in delaying tobacco prevention policies, indirectly causing who knows how many preventable deaths.
Stupid fallacies like "Absence of Evidence is Not Evidence of Absence" can have far reaching consequences. That’s why I couldn’t let it slide. It sounds very reasonable on the surface, but it’s false, dangerous… and seeing trained scientists still making such a basic mistake today scares me.
With "due diligence" anyone can write safe C code. It all comes down to discipline.
I don’t want to work with high-performing slop merchants, so I’m happy their talents are desired elsewhere. At least half the happiness I derive from open source comes from my interactions with the people I collaborate with, if not more.
This is probably true, but relies on a flawed analogy. The Zig team is not requiring anyone to jump through hoops: they're selecting from among all possible contributors the subset which they believe might be the most valuable to the health of the project in the long term. For you, perhaps because you place individual reputation above other qualities, those contributors would be the LLM-equipped Jeff Deans of the world; for Zig, clearly, they're the people who signal their willingness to put in their own time and effort for the good of the whole.
You can believe that, of course, based on your own experiences and personal bubble. I, for the same reasons, believe the exact opposite. Your argument also implicitly relies on an assumption we have little evidence for but mounting evidence against: that "top performers", however you define them, remain top performers after starting to use LLMs. This is far from clear and, so far, the limited data we have (together with classic cognitive psychology findings; see "Ironies of Automation") suggests otherwise.
Yes, and you are not looking for a paid job here, so you are not required to jump through hoops in order to get hired. If you don't agree to the no-AI policy, you are free to contribute to OpenClaw, start your own open source project, make a sloppy AI-generated fork of Zig, or do whatever you want with your free time.
I think this argument depends on your time horizon. Is the set of developers-who-make-the-most-valuable-contributions static? How did they become such developers? Surely Jeff Dean didn't get to where he is by having AI do all the work for him.
We're in an interesting moment of time where we have a large pool of developers who definitely learned how to develop without AI, but that pool is going to shrink relative to the total over time. For the moment, maybe you're right (or maybe not) that the best developers are all using LLMs. But assuming our industry lasts beyond this moment, having at least some kind of policy that clearly invests in training seems smart.
This depends on what you mean by "top performer". I believe this is true in many projects, but only among contributors who already have deep familiarity with the codebase. I think this is almost always false for people who are not already deeply familiar with a codebase: a "top performing" new contributor is one who is making small, targeted improvements, and these are the kinds of changes for which having LLMs author code is the least helpful. The kinds of changes where LLMs have the largest positive impacts (such as big refactorings or changes which require lots of boilerplate) are precisely those which are least suited for new contributors.
Personally, I've found LLMs massively useful for small targeted changes. I certainly can't read 100KB of code in a few seconds! You do have to do very careful work reviewing for style and design though.
The LLM can identify the relevant area for you but once it has done so it doesn't do much to speed up the actual writing of the code (since there is definitionally not much to write). Certainly it can often do that part too, but for small changes it's not much slower to do it by hand.
... Though I guess it is unclear to me whether Zig would consider "I used an LLM to explain the codebase then wrote this change myself" to be a violation of their AI policy.
Agreed. As a maintainer, I can produce a style-compliant, well-tested patch using an LLM much faster than I can review an AI-generated PR from a new contributor. Dealing with an LLM through a proxy is more work for a worse result; without the right context, the code just doesn't hit the mark.
I believe we need to set a privilege level for who can use LLM, but defining a cutoff is difficult. llvm-project story https://discourse.llvm.org/t/concerns-about-influx-of-ai-generated-bug-fixes/90381/56?u=maskray
Evidence suggests that this statement is false. And since you assume it as an axiom, so are the rest of your arguments.
Simply moving off of Github (ie. adding a hoop to jump through) seems to improve the quality of contributions to a project. Many projects have noted this. Putting a single hoop of "need to make a new login" scrapes off so, so, so many garbage contributors that you seem to be only left with real contributors.
In the limit, people will get super annoyed if you put too many hoops in the way. However, psychology also shows that if you overcome a hoop you become invested and that tends to increase your probability of continuing your activity.
There is also a quiet implication in your statement that the "rockstar programmers" are the ones to make the biggest contributions. Evidence suggest that for most projects "rockstar programmers" are a detriment. What most projects need are people who can argue but still cooperate with each other rather than "rockstar" technical people.
You keep repeating "evidence suggests" without providing any evidence.
Yes, I agree that some filters work well. Moving to Codeberg is probably a good filter as good developers don't much mind the one-time cost of joining a growing forge, whereas low-effort developers won't bother.
My argument isn't that hoops exclusively filter out the top, but I was disagreeing with this point from the post:
I think that this is flawed thinking because it assumes that the filter only cuts out the bottom or cuts out a random sample, but I think that it disproportionately affects the top, not because all the best programmers use LLMs (although incidentally, I think most of them do) but because top programmers have other options for where to apply their skills.
I think you're using "rockstar programmer" to mean someone who's talented but a jerk. I'm not talking about attracting them.
Most open-source projects see a more exaggerated version of the Pareto principle where 5% of contributors do 95% of the work (including unglamorous work like debugging, code review, documentation).
I think that adding tedious work to the contribution process disproportionately affects people most likely to end up in that top 5% because they have many options available to them of other projects that offer a more pleasant experience and would welcome talented contributors.
Very well written. Some months ago I argued against a suggestion to ban LLM contributions for one project, not out of love for LLMs, but on the basis that it would create more work and arguments and be impossible to detect and enforce. I'm pleased to say that this post has changed my mind.
I'm glad you published this post showing the reasoning for the ban. It will save people from getting the wrong idea.
I guess this is in response to the post by Bun about their fork of Zig to speed up debug builds: https://xcancel.com/bunjavascript/status/2048427636414923250; they've said this was LLM-written and they can't upstream it per policy. My understanding from some of the people who've reviewed this code is that it is pretty bad and wouldn't make the cut to be upstreamed anyway 🤷♂️
It's not, but this is.
That response is everything that was going through my head when I read the bun announcement. I was like, "would they really rather vibecode their own poorly thought-out fork without input from the zig lang devs instead of just reading the release notes?"
Oh wow, that reminds me of the caveats about parallel checking in TypeScript 7.0
Disclosure policies like Ghostty's map cleanly onto the contributor poker framing: if someone discloses LLM use, you calibrate the bet accordingly rather than refusing the hand. More information, not less.
That said, Zig's problem seems to be as much about review capacity as trust. Knowing a PR is AI-assisted doesn't make it cheaper to read. A disclosure policy probably works well for most projects; a ban starts making sense specifically when the noise-to-signal ratio has already made the queue unmanageable, which is where Zig found itself.
How do you see this pool evolve over the years? From my perspective the “youth” is using LLMs for everything, especially in academia etc…
A recent poll suggests that the 'youth' is a population often resentful of AI, given its marketing as their direct competitors, even though 69% of Gen Z trust work more if done without LLMs. I think creating spaces where LLMs are not allowed is good to counteract the productivity FOMO arms-race. For example, it is difficult for students and young academics to abstain from LLMs under the promise of future expertise, when they are currently being compared to / graded on a curve against their (LLM using) peers. LLM usage is high, and simultaneously 80% of Gen Z respondents felt that LLM usage was likely to make future learning more difficult.
I don't think you have to subscribe to inevitablism; it is not a forgone conclusion that prompts are the new abstraction level. And, if they are, there is definitely still space for LLM-free zones, especially amongst systems programming and language design, where the novelty of problems may be less compatible with the pattern matching inherent to LLMs. For the opinion of a high quality contributor and LLM-user, here's José Valim (creator of Elixir) from a recent episode of the Thinking Elixir podcast:
My anecdata/experience as a student says that (even before LLMs) there are ~5 students (being charitable) in a class of ~150 who have/obtain the ability become systems programmers who can contribute to a project like Zig. Most of them do not need to be in academia either, they are usually self-motivated individuals with agency who like to tinker. Cheating on (group) assignments is (and has been) pervasive in academia long before LLMs and I simply don't believe Gen Z students are doing it because of societal pressures.
Yeah I do believe this either, but LLMs have their use cases so if you have ~1000 people in the world who can contribute to your project in the first place, asking them to not use LLMs for anything feels like a pool that is likely to shrink over time, not grow.
That's fair enough, but don't LLMs broaden the cheating possibilities, and make it much easier (and more alluring) to do so? For instance, I couldn't cheat my way around creating an interpreter at university, except through some code reuse, which would have been trivial to detect. I don't know if a good solution has been found yet in terms of problem setting either.
I suppose this will be borne out in the number of Zig contributors in the future, but it seems like something that can only be evaluated in a decade, by which point the development landscape will have shifted again. I appreciate your personal experience of having LLMs free up time to contribute to open source, and I guess development will return primarily to those who could allocate the time pre-LLM. If the question is not about free time, but instead "Will future developers be able/willing to work without the use of LLMs?" - I'm not sure what to say other than I hope we don't become that dependent on centralised resources.
Some small handful of people will be both able to think like a computer (the main requirement) and also to memetically resist the ELIZA effect in small doses. If enough of them choose to not use LLMs for everything then we will be fine. A bigger concern is how much of society seems to simply accept the output of LLMs, which is obviously bad writing, as if it were good writing.
This Software Unscripted interview does talk about this problem: https://www.youtube.com/watch?v=VedijY4W5bU
Well if that's how things are are going to be, we're hopefully going to hoard in the Zig community all the people who like systems thinking :^)
I'm quite sure that many of their valued contributors are LLM users. And despite their "AI Ban", the proportion of their contributors which use LLMs will grow. And at the same time quality will increase and they will become more successful.
The irritating thing about this policy is the out-group disdain for "LLM users" as if it's not a transformative technology which allows 10x more people reach their full potential in programming. Obviously it produces spam, but yeah. Not everybody is a rich English-speaking university educated person and everybody else is best to use LLMs to learn coding in their own spoken language etc.
This policy does not exclude LLM users in any way. We don't quiz you on whether you have ever used an LLM, you're just asked to not use one when contributing to Zig.
Glad to see that you gave deep thought to the reasoning expressed in the original article.
Why do they not use AI to help them vet new contributors ("please investigate this contributor's profile age and activity to vet their quality" etc.)? Why do they not use AI to help them assess PR's? They can do all that while maintaining their anti-AI-code ban.
Because if their stance is "nothing AI related ever," that's a religious stance, not a rational one