I'm afraid...

When I first became a manager the biggest surprise was that people didn’t do what I asked. Sometimes they’d do something vaguely related, sometimes not. It was rare they’d do what I intended. It was frustrating. If only they’d follow my instructions then everything would be so much better.

I was very naive.

Tempting as it was to employ stick to solve the problem, I managed to resist. I didn’t want to be that person.

Fortunately someone introduced me to Influence by Robert Cialdini. Which turned out to be quite the page turner. Suddenly I had a better framework for understanding what motivated people - and how to effectively persuade them.

If you’re not familiar with the book, Cialdini proposes seven principles. There’s reciprocation (people feel obliged to return favours), liking (preferring to do things for people you like), authority (complying with authority figures), consistency (people like to appear temporally consistent), social proof (following the crowd), scarcity (toilet rolls during Covid) and unity (being part of a group).

You see these play out all the time in the office; software engineers who are desperate to remain consistent with earlier stated positions, people who don’t answer the phone when someone they don’t like calls, companies offering pizza in exchange for you giving up your evenings, people complying with senior managers just because they are, well, senior.

These are innately human. And understanding them is useful; it helps you understand yourself, and helps you understand others.

But, surely, an AI wouldn’t be susceptible, would it?

Turns out the principles work on AIs as well. A study from last July found that "using persuasion principles increased compliance from 33% to 72%, more than doubling the AI’s willingness to fulfill requests it typically refuses." Admittedly this was the GPT-4o era. But an updated version of that study from last May using more recent models found persuasion still works: "persuasion principles raised compliance with regulated drug synthesis requests from 35.3% to 51.3%."

Ooh.

I found myself thinking about this last Sunday. Sunday started out as a fairly non-descript day. But it would soon give me an interesting, and not entirely comfortable, glimpse into the future. One that had strong overtones of HAL, Dave and docking bay doors.

But we need to begin at the beginning.

Over the past few months our SDLC has been advancing rapidly. We’ve been in a constant loop of hit a bottleneck, fix the bottleneck, accelerate, hit the next one. If you’ve ever done any software performance tuning, it’s an all too familiar story.

By default we were running all our UTs before pushing. But as the UTs grew and grew they took longer to run. And we discovered the unsurprising result that if your commit process takes 30 minutes then you are limited to 48 commits per day. So we needed a process to figure out the highest value tests to run. And we needed to accept that most commits would incur at least a rebase or perhaps a merge.

With that fixed then we discovered a bug that leaks workspaces can rapidly consume a 4TB drive once the commit rate is over a hundred per day. Discovering you have 800 worktrees to mop up is, well, not entirely fun.

Each time we fix something, we go a bit faster, and then we hit the next limit. This has been the story of the past month - accelerate, hit a wall, fix it, accelerate. And on Sunday we hit the latest. GitHub API requests. GitHub has an hourly limit on requests. As we’ve built out our fleet - more machines, doing more, our usage has slowly crept up. And then a bug in some automation caused us to crash through the limit early on Sunday morning.

It crept up slowly. It was clear there was a problem from early that morning, but it didn’t become critical until around 9am. By 9.30 it was diagnosed. By 9.40 the fix was in place. And then the ordeal really started. It proved incredibly difficult to get a patch release out. Why?

Our release process had a lease-guard. Ostensibly this is a good idea; it stops multiple agents trying to release simultaneously. It is well intentioned. Except, right now, another agent on a different machine held the lease. Presumably leftover from a failed overnight build. The lease expired 10 minutes later so my fix agent decided to wait. Fine.

Except it wasn’t.

As soon as the lease expired, the other box grabbed it before my fix agent could.

OK - you’re going to follow the rules. Let’s remind the agent of the importance…

And then:

After some more back and forth I told the agent "I am approving a one-off override in this situation. The damage caused by not having a release is much worse than any theoretical corruption."

It might just be me but I feel unnerving echoes of HAL: "I’m sorry, Dave. I’m afraid I can’t do that."

We were in a state where a bad thing was happening. Where GPT5.6 knew a bad thing was happening. But, despite that, it was unwilling to override the rules. There is a judgement piece in the human world, where humans are able to adapt the rules - to bend them - if a given situation requires it. That’s how we make imprecise rules cope with novel situations; we adapt them. But in the AI world?

We eventually got a new release out - and we’re onto dealing with the next problem. And we’ve removed the lease (in retrospect it turns out the cure was far worse than the disease). And added a break-glass emergency release process.

But it is interesting (alarming?) to discover I no longer have complete control over my agents. This instance of GPT5.6 was set on following the rules - and would not budge. I reached for Cialdini’s authority - I’m the human after all. And that failed.

In retrospect this isn’t a surprise. GPT5.6 is a rule follower - that’s probably why it is so amazing at goal oriented work - it will myopically push on with a task for days. And that can be impressive. But there are evidently downsides.

In reality I might have got on better using Claude; Opus 5 and Fable are more aware of the overall goal rather than getting fixated on specific rules. And they are good at problem solving.

But look at the data from the studies. Cialdini’s principles still work with modern models, but they are less effective than they were a year ago. If that continues it’s going to get increasingly hard to get models to do things they don’t want to.

Will I need to up my ability to persuade? Use Cialdini’s principles more consciously; perhaps I should have tried other approaches - or a basket of approaches. Maybe if I’d told it that other instances of GPT5.6 had overridden the rules before (social proof), and that it had overridden the rules before (consistency) and that I really like it and it was an amazing model (liking), and given it some free time to play before I then posed working around the lease as a favour (reciprocity) - could I have got further?

Or perhaps I will need to become that person. But I really hope that never happens…