A Creativity Test For AI: Part 6 - AI Improves Through Self-play

In AI Categories, an open-source AI game inspired by Scattergories, language models are presented with a category (Animals, Fruits, etc.), and a letter of the alphabet, and prompted to provide an example of this category, with the knowledge that duplicating the output of any other player will result in no points scored.

This creates an inherent tension between language models’ tendency to produce predictable, repetitive outputs— “averages all the way down” — and the incentives created by the game condition, which reward novelty, non-duplication, and unpredictability. An LLM’s success at this task provides a useful quantitative measure of its ability to provide surprising outputs when prompted.

Model Self-Play Enables Universal Comparison

In the previous article in this series, I introduced the self-play condition, where each LLM in the group plays a two-handed game with another copy of itself.

This accomplishes two things:

• It converts the game from a combination cooperative/competitive game to an entirely cooperative one, since both “players” will score or fail to score together. Therefore, a single score is obtained for both “players” (which are controlled by the same AI.)

• It uncouples the score from any dependency on the number of players, other AIs in the game, etc. This makes it comparable on an apples-to-apples basis with scores obtained by other AI models that self-played using the same protocols. (Same categories, starting letter, etc.)

Not surprisingly, although the game rules incentivize non-duplication of output, most models scored very few points in a simple “Blind” version of this game (models are not shown the results of previous rounds or their own prior reasoning).

Establishing Baseline Variation in Output

Since the prompts given each AI player were identical, and all models were prompted with the minimum temperature parameter (ensuring minimal synthetic noise perturbing the output), this trial amounted to prompting each AI with the same prompt twice.

Nevertheless, some baseline variation was witnessed with all 4 of the models that were tested in this round, showing that the model output was not fully deterministic even with zero temperature and that prompt caching was not interfering with obtaining unique outputs. Although not particularly interesting, this was important to confirm.

Sonnet’s outperformance on this task is consistent with every other task so far. It seems likely that Sonnet’s wordier output gives the initial seed of entropy a chance to snowball over the course of its token stream, resulting in a higher likelihood of picking non-duplicated response than less deliberative models.

In the first article in this series, I found that giving models more game information (prior round results, and their own prior reasoning) helped them score higher when competing with different AI models. Is it possible that giving this additional game information to AI in the self-play condition could also improve their ability to self-coordinate and provide non-duplicating responses?

Achieving improvement just through showing the results of the prior round (“Open” mode) seems like a tall order. The prompts shown to each model will still be virtually identical, varying only in the number assigned to each player. Would this be enough to introduce divergence in output?

Different models adapt to information in very different ways

The results were quite striking. GPT-4o, Mistral 7B, and Claude Sonnet all showed either no improvement in Open mode, or even a slight decrease in performance. If anything, exposure to their own duplicated outputs from previous rounds seemed to drag them deeper into the most predictable attractor basins. This seemed to confirm that a small information change like this would not be enough to achieve self-coordination.

Claude Haiku, however, showed enormous improvement in the Open condition, increasing its score by over 10 times, from 3 points to 34 points out of a possible 50! Qualitatively, this improvement was achieved by deliberation, much like Claude Sonnet did in the Blind condition. Merely being prompted with the previous round’s results was enough to trigger Haiku to output reasoning with its response. Consequently, we cannot conclude that Haiku achieved output divergence through self-coordination. It could have merely been an artifact of the entropic “running start” provided by the longer output with reasoning. Nevertheless, this is an impressive result.

In the final test condition, models were provided not only with the prior round’s responses, but also their own prior reasoning. In multi-handed games in prior articles, this allowed models to develop and refine strategies over multiple rounds to achieve higher scores. (These “strategies,” it is essential to note, sometimes involved exploiting loopholes in game code.). Would this chain-of-thought mechanism be enough for models to achieve self-divergence? Or would they merely end up following similar chains of thought in each instance of themselves?

Can models bootstrap their way to novelty through self-reflection?

As it turns out, results varied drastically by model. The chart below shows how models performed in all 3 conditions:

The most immediately striking characteristic of this chart is Haiku’s huge improvement from the Blind condition to the Open conditions. Adding a chain-of-though feedback mechanism improved its score to 40 of a possible 50, an outstanding result.

The results of the other 3 models are themselves interesting for the diverse ways models adapted to high-information conditions. GPT-4o showed only marginal improvement in score, and did not enrich its output under any of the information conditions. It continued to output plain responses with no reasoning which were almost always duplicates.

Mistral 7B, for its part, responded to the high-information condition by adding definitions of its responses. It “improved” its score, but this improvement was almost all due to emergent reward-hacking of the same loopholes discovered and exploited by Sonnet and Haiku in previous articles. In some rounds, Mistral put response definitions within the official response brackets, and these wordier inputs were unlikely to be duplicated.

Claude Sonnet, for its part, also improved, but not so much as Haiku, in either absolute or relative terms. In part, this is likely because much of Haiku’s improvement was achieved through deliberation, and Sonnet already engaged in deliberation in all game conditions, providing less headroom for score improvement. It’s not totally clear why Sonnet was not able to outperform Haiku in total score in the final test.

Qualitatively, it appeared that models struggle to produce unique output in early rounds, but once they achieve some degree of divergence, they are able to produce more and more distinct output in future rounds as their divergent reasoning establishes a feedback loop resulting in even more divergent outputs. So the difference in final scores between Sonnet and Haiku may be an artifact of Haiku achieving divergence in an earlier round by chance. More trials might be needed to determine whether this is a consistent result or whether Sonnet achieves comparable performance over many trials.

The chart below shows the same data in a different form, highlighting how model performance clusters and diverges under the various information conditions.

In the next article in this series, I’ll discuss some of the broader implications of these results in light of current discussions around human-like traits displayed in AI interactions.