Pink elephants blowing their noses

Yesterday, an old acquaintance posted a picture on LinkedIn that (probably unintentionally) demonstrates a big problem problem with AI. He works at a company called “Pink Elephant” in Heidelberg and posted that “trunk disease” is rampant there, which is why no one is available to talk to at the moment. He illustrated this with the image shown in this post.

This shows how AI can deliver results that look absolutely perfect but are still useless. The AI couldn’t sensibly transfer the act of blowing one’s nose to the elephant because it had no training data for this. Anything outside the convex hull of the training data is just free hallucination! The AI completely misplaced the tissue.

However, we expect exactly this kind of performance from AI, namely that it creates code that represents the analogue of an elephant blowing its trunk. On a nice picture, we can easily see that it failed on our expectation (or that our expectations where set too high). But with a mountain of AI-generated code, it’s not so easy to see that the task has been failed. On the contrary: the code looks perfect, it even “works,” but no one notices that the snot ends up on the carpet.

On the same day, I was asked for my opinion on Amazon Q Developer. I’m not familiar with this development tool, and I’m sure it has great features. But I’d bet $100 that this software essentially just makes it easier to make elephants blow their snot onto the carpet. It doesn’t magically turn developers into mathematicians who have an idea of what the convex hull of a set is and thereby have an idea of where the limits of their tool might be.

You might argue that this is a very theoretical scenario. Unfortunately, that is not the case. Just a month ago, we had to put a feature into production that was developed with significant AI support. One aspect of this feature involves sending several thousand reminder emails. In a realistic test, it quickly took up all of the available hard drive space. It took us a few days to identify an innocent-looking line of code as the culprit, which was supposed to do nothing more than set a specific email header. It passed the code review because this line was apparently correct. For me, it was the equivalent of pink elephant snot on the carpet.