Outcomes we want to see from AI at Medium Engineering

This was originally written and published by Christophe Spy (Director of Engineering, Medium) as an internal Medium Eng post.

Desired outcomes (what “good” looks like)

🙏🏻 These are the kinds of changes we want to see if AI is going well at Medium.

Less drudgery, more leverage

Engineers use AI to remove repetitive, low‑leverage work (e.g. tedious refactors, boilerplate, large test suites), not just to ship more tickets.

We see examples like:

• “We took a part of the codebase that was basically untestable and, with AI, made it testable and added meaningful coverage in days instead of months.”
• “We cleaned up a legacy area we had been afraid to touch for years, using AI for the mechanical work.”

Faster learning and broader technical range

Engineers use AI to ramp on new languages, frameworks, and parts of the stack much more quickly.

Engineers can explore unfamiliar areas safely by asking AI to explain tradeoffs, pitfalls, risks, and how the system uses this areas of the code.

We see fewer cases where work stalls or we avoid implementing things because “nobody knows this code anymore”.

Frontend/mobile engineers are able to make safe, reviewable backend changes with AI’s help (plus a review from a backend engineer).

Teams can ship more end‑to‑end features without long handoffs.

Higher technical ambition, not just more speed

Teams take on work that used to be “too expensive”:

• Meaningful test coverage in critical areas.
• Pieces of M2 migration or other large refactors.
• Structural cost savings.

Incident/action items evolve from “add an alert” to “Let’s improve tests here”.

Engineers use AI pre-work to explore edge cases and tradeoffs (design rigor).

Better planning, reasoning, and risk reduction:

• Engineers use AI to think through complex scenarios before building, exploring edge cases, stress-testing assumptions, and comparing multiple approaches, and better understand the pros and cons of each approach
• Design docs become more rigorous as AI is used to challenge thinking and look for gaps, risks, and alternate approaches that the team might not be aware of.

AI fluency as mindset, not just tool usage

Engineers can explain:

• When they choose a heavier model vs a cheaper one, and why.
• How they use AI to reason about a system they don’t know (finding root causes, understanding hidden costs).
• Where they don’t trust AI (judgment calls, ambiguous product decisions, sensitive code paths).

In reviews, incidents, and 1:1s, people naturally talk about how AI helped them reduce risk or effort, not just “AI wrote this file.”

Better quality and maintainability

Code quality is improved through AI-assisted code reviews, security audits, and other code analysis tooling.

Engineers use AI to identify anti-patterns in the code, performance and cost bottlenecks, and security gaps that would otherwise be difficult to find by hand.

The code in areas we touch becomes clearer and more consistent over time, not messier.

Engineers use AI to apply established coding and architecture patterns, so touched code stays consistent and easy to navigate and reason about, regardless of which model or harness is used.

Engineers use AI to simplify and standardize, especially in refactors and tests, and avoid clever one-off patterns.

Outcomes to avoid (what we do not want)

⛔ These are anti‑goals and bad incentives we should explicitly avoid in performance, ladders, and messaging.

Velocity metrics for AI

No performance metrics as success metrics for AI adoption like:

• Number of PRs/commits.
• Lines of code added/changed.
• Tokens used or dollars spent on models.

*It’s okay to use them as debugging signals to identify if folks need support, are overloaded, or to ensure onboarding is going smoothly.

No framing like “burn $X of tokens per month or you’re not using AI enough.”

We don’t want engineers to optimize for “more output” at the expense of quality, safety, and thoughtful judgment.

Replacing human judgment with blind trust

Engineers should not feel pressured to:

• Ship AI‑generated code they don’t understand.
• Skip thinking about architecture, correctness, or ethics because “AI said so.”

For high‑risk or complex systems, we should increase the expectation of human understanding, even if AI writes the first draft.

We want to avoid the world where nobody can be held accountable for the code they committed because “AI wrote it”. Engineers are still accountable for their code clarity, code correctness, test coverage, and ability to reason about the code. The human judgment of engineers remains what matters most.

Drowning in code reviews

We want to avoid the case where teams are just cranking out an untenable amount of code, so much so that humans can’t reasonably keep up with reviewing them (even with AI assistance).

Eroding codebase understanding

We want to avoid:

• Critical areas where nobody can explain how things work because “AI wrote it.”
• Removing code review and shared understanding entirely, especially on core systems.

The goal is to move people from “I write every line myself” to “I orchestrate the system and can explain it” and not to “I don’t know what this does.”

Security and privacy regressions

We want to avoid cases where code is quickly written, without realizing what security or privacy implications it might cause. It’s easy to ship stuff with insane security gaps when there is no human looking at it.

Ignoring ethical and environmental concerns

We should not:

• Force teams to use vendors they see as misaligned with our values.
• Hand‑wave away questions about environmental impact and model energy usage.

Where possible, we should prefer vendors whose values align better with ours and be explicit about that.

Tool theater and cargo-cult adoption

Engineers should not feel pressured to mention AI in updates to look modern, even when it didn’t help.

Teams should not adopt a tool because leadership is excited about it, only when it actually improves results, quality, speed of learning, or safety.

Since this was published on February 20, we’ve rallied around these as outcomes we want to see from our AI adoption. There’s a lot of hype around AI. Our north star related to AI adoption is that we will see outcomes from our inputs, and that the humans at Medium can move with the speed and responsiveness of a much larger team, while staying true to what makes the platform worth building.

Brittany Jezouit (Chief of Staff, Medium) wrote and published a related story about how Medium uses AI internally.

Outcomes we want to see from AI at Medium Engineering was originally published in Medium Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.