I Gave Claude A Shot At Out-Writing Me. It Failed in the Best Possible Way.

Can AI do my job better than I can? Variations of this question have been saturating the Internet since the mass adoption of LLM’s, and the debate has ben supercharged in the last month with the release of Substack’s new AI-detection feature. It just so happened that at the same time I was working on a new video about the existential threat posed by the rise of AI-powered drone warfare. I have been a little late getting onto the Claude bandwagon, but several friends of mine have told me that the output on the latest models is so good that it would be irresponsible for me to not at least give it a try. So, a few weeks before Hank Green admitted that he had a LLM problem, I gave the most advanced Claude model a shot at helping me out with a script that I had thoroughly outlined. Things did not go as I had hoped.

I am not an AI absolutest in any sense of the term. I see it as a tool that has some useful applications—and have heavily relied on AI transcripts for interviews for at least a year now. When ChatGPT first came out I asked it to write some sample scripts to me, but the output was so bad that I walked away and never looked back. However, I keep reading that models are getting better and better—and I have a slightly better idea of how to use them now—that I decided to give it my best shot.

I already had a detailed seven page outline, including specific quotes from tape that I had pulled, links to reference material and an idea for how the script should flow from one topic to another, and an idea for how to conclude the topic. Then I wrote a thorough prompt explaining how I wanted Claude to interpret the material and approach the script. I even included an AI-generated style guide where I fed previous scripts into Claude to train on. This is the exact prompt I put together (as well as the script that Claude actually wrote).

I’m telling you all of this because I want to show you that I gave it a good faith effort—the same sort of effort that I would put into training an actual human to do my job. The hope was that Claude would produce something truly brilliant. The LLM would extract the essence of my vision for the script and return with a close-to-finished product. I had no illusions that I wouldn’t have to make some tweaks to the final version, but going through all this process I had a reasonable expectation that it would return something useable.

Instead I got this.

You are welcome to read the whole thing at your leisure, but I would recommend not wasting your time because it goes off the rails in pretty much the first paragraph. If you looked at my original outline above you will immediately notice that I had helpfully written an intro that I know is right for my channel. Claude chose to ignore it entirely and skip to a later part of the outline. After that it more or less ran in circles—regurgitating the same basic ideas and only tangentially following the outline I’d given it. It was also filled with all of the normal AI cliches. At a stylistic level, it also had no idea where to put B-roll.

After seeing the output, I decided that the old ways of writing scripts were still the best. It took me another three days to get the draft to a place that I thought was pretty good, and another three or four to film and edit it. You can see the final human-made output here.

Then again, since this is a YouTube video, after all: why not just watch the final product?

I think it came out pretty well. It turns out that, at least for now, my job remains safe.