What a Leaked API Key Taught Our Team

A colleague asked me last week why our team cares so much about least privilege. I gave a rambling answer, so here is the longer, more organized version.

When we finally measured, the numbers were worse than anyone guessed. Around 42% of the time spent on a typical change went into working around least privilege rather than building anything.

The problem

The setup was ordinary: OpenSSL at the edges, Vault doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

The first symptom was subtle. Nothing failed outright, but least privilege kept drifting in the wrong direction, and every fix seemed to make the next one harder.

What we changed

Rather than a big-bang rewrite, we migrated one module at a time over about 11 weeks, keeping both paths alive behind a flag.

Stripped of the project-specific bits, it looks like this:

cosign sign --yes ghcr.io/acme/api@sha256:3f1c...
cosign verify \
  --certificate-identity-regexp 'https://github.com/acme/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/acme/api@sha256:3f1c...

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

• Make the default the safe choice, not just the documented one.
• Write the migration guide before writing the migration.
• Measure before and after, even if it's just a rough number in a spreadsheet.
• Delete dead code aggressively; it's the cheapest refactor there is.

What it cost us

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Vault for everything. That's a fair question with a boring answer: we tried, and it made least privilege harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Closing thoughts

If you take one thing away from this, let it be this: least privilege is a design decision, not a cleanup task. Make it early and make it on purpose.