Preamble
This is Part 5 of The Advent of Code.... Assistants. If you have not read Part 1 to 4 yet, maybe you want to head over there.
The repository associated with this project is available on CodeBerg.
I have also been taking a break from writing these articles for a week because... life!
Day 7
Today, we had to split a tachyon beam. And I am going to start splitting hairs!
In its review Claude mentions that I am using
Heavy abstraction with Beam objects, Action types, Grid helper
It's not the first time Claude makes this kind of observation... and it's starting to bug me!
Difference between abstraction and modelization
Disclaimer: I know this is a highly philosophical topic and this section represents my own views even if I try to stay objective
Abstraction
By its name, an abstraction is a technique that makes things more abstract. Abstractions are both useful but sometimes damaging/dangerous.
The aim of an abstraction is to make something complex look easy or completely hide it from the user of the abstraction. An ORM, for example, is an abstraction of SQL and the RDBMS being used.
Oftentimes abstractions are kind of "obscure", and for good reasons!
Modelization
Modelization is the act of trying to more-or-less faithfully represent a real-life scenario in code. Or in other words: being idiomatic.
An example of that would be the following problem where:
Train A leaves Chicago, heading towards Toledo, Ohio traveling at 70 miles per hour. At the same time, Train B leaves Toledo, heading to Chicago, and travels at 60 miles per hour. The distance between the two cities is 260 miles. When do the trains meet?
And creating classes like the following:
@dataclass
class City:
name: str
location: Tuple[float]
def dist_between(self, other_city):
...
return distance
@dataclass
class Train:
departed_from: City
departed_at: time
speed: int
def distance_travelled_after(self, time: deltatime):
...
return distance
This is modeling. It allows referring to "relatable" things like City and Train instead of abstract variables like da, t or db, making things more concrete instead of abstract.
To be fair, there is a level of abstraction added in both of the above classes. The methods inside the classes are a bit of abstraction. But once again, one of the things those methods achieved is making the whole thing more "relatable"... more "expressive"...
I'll let you be the judge, which one is most idiomatic?
sa * 3 / 60
or
train_a.distance_travelled_after(deltatime(minutes=3))
Yet in Claude's reviews, it feels like any class definition is just an abstraction and sounds like it's a bad thing.
I strongly disagree and will address that by the end of this article.
Complexity Vs Coding practices
Here again I am going to knit-pick one of Claude's comments:
Complexity: Human solution significantly more complex with state machines and method dispatch
Before going further, let me explain what method dispatch is for those who might not be familiar with the term.
The following is a method dispatch (it is a way of calling different methods depending of the value of a variable):
class Example:
def some_method(self, some_value: str, other_value):
getattr(
self,
f'handle_val_{some_input}',
self.unhandled
)(other_value)
def handle_val_a(self, other_value):
# Do something
...
def handle_val_b(self, other_value):
# Do something else
...
...
def unhandled(self, other_value):
# Skip or raise an exception
...
As opposed to:
def some_function(some_value, other_value):
if some_value == 'a':
# Do something
...
elif some_value == 'b':
# Do something else
...
...
else:
# Skip or raise an exception
Method (or function) dispatch is often used to keep methods small and make code more readable (to humans) compared to long (often nested) if/elif chains.
It is also generally accepted as being more maintainable since handling a new value for the some_value variable becomes "adding a method" instead going back to the if/elif chain and and having to wedge a new elif in there.
I use getattr and an f-string in the example above to fetch the appropriate method to use, but there are a few variations to this technique, like using a dictionary mapping values to methods or method names.
Fun side-note: method dispatch is the technique used by Django's CBVs to know whether to call get or post based on the request method.
Of course, as with any programming technique, there are some trade-offs. Method dispatch is probably overkill when the actual handling is just a single line of code for example. Yet, even in those cases, it might be considered "defensive" programming to just go ahead, if you have a feeling that, over time, a single line of code might grow into 2... then 5... then 256.
This is also Dynamic Programming and is easy to do in Python, a dynamic language, and harder to do in some other, non-dynamic, languages. Implementing something similar from scratch in C with a linked list of function pointers is... Not Fun!
Over-engineering Vs Future-proofing
I will admit that some of the code I wrote in part A was useless. But this is the nature of AoC. A lot of the time, going the little extra mile is what makes solving part B (almost) trivial instead of being a nightmare.
This was not really the case for today... Although there are ways to solve part B in the same way used to solve part A, they are either very slow, too memory-intensive, or both.
At the end of the day, I tried to go back and re-do part A by using the same technique as part B and although I was able to use MILP (with a binary variable to represent the state of indicator lights), it proved to be much slower than the original solution.
But when Claude refers to over-engineered solution, I think it mostly refers to the modelling of the problem. And I think this is telling of the state of our profession, Claude's preferrences are just reflecting a bias found in its training data after all.
And I agree that taking the time to model things correctly (even though I still take some shortcuts because this is AoC) is not necessary for the single purpose of a challenge such as AoC. And each solution can be a copy-pasted one-off throw-away solution...
But even with all that in mind taking the time to correctly model your problems has multiple advantages
Readability when things don't work
Having a clearly modelled problem is easier to reason about.
As an example with today's part A, pressing a button has an effect (of toggling some lights). If my final result is wrong, once I have validated (since it's in its own method, I can test that in isolation) that my press method does what it's supposed to do, it is out of the way when looking at my code. I can now focus on where the problem most likely is: which button I press and how many times.
Reusability
Even if AoC is a set of "unrelated" problems, there are things that are constant:
- you get a text file as an input
- you get a small test input that can be used to validate your code
- each day has 2 parts
And some that are recurrent:
- part 2 almost always re-uses some items and concepts from part 1
- some concepts are used in several challenges, like grids, coordinates, string parsing, etc
Acknowledgements
Some of Claude's criticism of my solution are well-founded and it is the slowest of all. I may re-visit it after day 12. But for now, even if slower, I find 40 and 400 ms acceptable for AoC.
But?.. How did the AIs do
- Claude did fine
- GPT had the fastest solver and best knowledge of algorithmics
- Kimi-K2, again today, started going in circles and required human guidance to solve the challenge
Claude's full review
Read the full review on CodeBerg
Day 8
Creating a fire-hazard by daisy-chaining Christmas lights
Obscure algorithms and their names
During its review, Claude informed me that, contrary to the human, all AIs used Kruskal's MST with Union-find.
I suppose this is the right time for me to make a confession: During my CS degree I literally slept through 99% of the theoretical algorithmics classes. But even from studying the course (yes, I did pass the exam), none of those names are familiar to me. Maybe it has to do with the fact my degree was more focused toward CS applied to business or it might also be my memory.
In any case, what I am sure of is that people who are self-taught or are not CS-graduates (like project managers, product owners, actual clients, etc) will have no idea what those names are either.
Side-note: This statement is not meant in any way, shape or form, to disparage people who are self-taught. What I know today comes from writing code, not going to school. We are all self-taught. Some of us are self-taught and also happen to have a CS degree.
The funny part is that I re-implemented Kruskal's algorithm in my solution, because it felt like the obvious way to do it (compute all the distances, sort, pick the shortest). It is close to brute-force. It is not some smartly optimized algorithm that requires a namesake (no offense to Joseph Kruskal), it is a sensible (at least to me) way to approach the problem without any special trickery or optimization.
For the Union-find part of the challenge, my very unoptimized solution (looping over all circuits and using box in circuit), could probably be made faster by using a btree or similar. I am not sure it is worth it though since my solution runs on par with Claude's and Kimi-K2's (slightly slower than Claude's, faster than Kimi's) who allegedly used the most performant algorithm and didn't deal with the overhead of OOP.
The one thing that would grant the biggest performance boost would be to not calculate the distance but go with the squared distance instead, which is what GPT-OSS did. Square root calculations are quite expensive and the distance here serves no actual purpose other than sorting. Sorting on the squared distance does not introduce any bug at this point. In production-ready code, it would be an issue to advertise that value as "distance" though.
GPT-OSS always (during this experiment at least) optimizes... and that trick made its solution significantly faster.
Confusing language
Today's challenge had some slightly confusing language, indicating that sometimes connecting 2 junction boxes together wouldn't have any effect because they were already in the same circuit. And it was also asking you to make 1000 connections.
When I first read the challenge, I had doubts whether those 1000 connections were supposed to be an absolute number or were we meant to only count the successful ones.
Instinctively I (maybe you too) and all the AIs (although we should not talk about instinct here) went first for the meaningful connections.
That was the wrong choice!
Since I had noted the ambiguous language in the first place, I quickly realized my mistake when running the code and changed my counting method.
All the AIs needed to be nudged in the right direction, either going in circles as to why their code was not working or even confidently deciding to use the 9th connection instead of the 10th (on the test input)
Getting frustrated with Claude's review
When asking Claude to review today's code, it was very eloquent in the fact that the AIs all were able to identify the name of the algorithms and how readable their code was (I honestly did not find it very readable myself) and how, for AoC, one should just use throw-away code and not bother with re-use.
After questioning its reasoning, Claude conceded a few points...
Initial draft issues:
- Treated code duplication as "architectural difference" rather than critical anti-pattern
- Rewarded GPT for hardcoded 1000 value (actually should be penalized - breaks test workflow)
- Understated the severity of 70-88% code duplication
- Called GPT the winner despite massive duplication
We were moving in the right direction!
We still had different standards for readability. I asked Claude to amend its review once more with the following request:
Please also analyze how the human uses 'business terms' (ie: the same words used in the problem) while the AIs use mostly technical terms and algorithm names in docs, making this less relatable to the business side but more relatable to people who know the names of every algorithms
You can read the dump of that review session in the repo
After that 3rd version, I accepted its review for the day.
Full review
Read Claude's final full review on CodeBerg
After eight days of comparing human and AI approaches to the same problems, some patterns have emerged that go beyond just solving puzzles.
Conclusion and take-aways
I do not care if humans participating in AoC use throw-away code and/or copy-paste code from one solution to the next. This is something I have done multiple times for AoC.
I have also regretted multiple times not having spent a bit of time on some sort of architecture and now finding myself copy-pasting opening a file, reading it and parsing a grid.... 🙃
And after doing that for a couple of years, I realized that having some infrastructure in place was helpful. So did any other person who published AoC infrastructure packages on PyPI. This very project uses adventofcode-initializer to help with downloading inputs, transforming challenges to markdown and the tedium of removing and re-adding part 2 for every AI run (so they can't peek ahead).
When it comes to these agents though, this one-off / copy-paste approach bothers me! They are not designed for AoC, they are designed to be helpful in a professional context.
Even if they recognize this is AoC and they know about AoC! Many people participate in AoC for many different reasons. There are those who do it as a speed-coding contest (you definitely don't want to write code like them in a professional context) but there are those who do it to learn a new language or technique or simply learning to code. Arguably, there are better ways to learn to code for the first time but some learn best when they are stuck in the mud and that's also valid. AoC challenges are also used for interview questions (don't get me started on that).
In all those cases, promoting anti-patterns is definitely not the way to go
And, no matter how you look at it, writing in a review "as a senior software engineer" (this is part of the review command) stating that 80% code duplication is "an architectural difference" or a "stylistic choice" is not a good thing.
All of the gripes I listed above should be parametrized by default in any model used by coding agents.
This can be somewhat mitigated by marking those things as important in your AGENTS.md (or similar file). But it should be built directly into the system prompt. Something along the lines of:
Unless the user specifically requests a one-off script
follow coding best practices,
including but not limited to ...
The only model in this experiment on which I can actually change the system prompt is GPT-OSS. Depending on the agent used, I can make attempts at modifying the system prompt of the other models using various techniques that may involve altering their source code... but without any warranty since I have no control over the software running the models (which might/should? override my attempts).
Anyway those should probably be the defaults. This is a failing from the companies that create these agents.
Once again, we also have to remember that the models are inherently biased by their training data. Does this mean that, in our profession as a whole, we do not value these practices enough? Does it mean the training data does not feature enough professional-grade code?
Probably a bit of both.
Despite that, there is also enough data in their training set for the models to "know" that things like code duplication are an anti-pattern. It only took a bit of nudging for Claude to recognize it.
In another project I have been using it on, it also says "method dispatch" is a better design practice than if/elif chain and writes several lines about why this is the case. So why is it calling the pattern "too complex" in this one?
Maybe the enshitification of models has already started? And producing poor code "by design" is a way to ensure future business?
Or maybe I am just being too cynical...
See the other articles in the series:
Comments
(via Mastodon or BlueSky )