Rotating Secrets Without a Maintenance Window

A colleague asked me last week why our team cares so much about threat modeling. I gave a rambling answer, so here is the longer, more organized version.

The setup was ordinary: Vault at the edges, Trivy doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

Where we started

When we finally measured, the numbers were worse than anyone guessed. Around 27% of the time spent on a typical change went into working around threat modeling rather than building anything.

We had good intentions. We even had a document describing how threat modeling should work. The trouble was that the document and the code had quietly gone their separate ways.

What we changed

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Stripped of the project-specific bits, it looks like this:

cosign sign --yes ghcr.io/acme/api@sha256:3f1c...
cosign verify \
  --certificate-identity-regexp 'https://github.com/acme/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/acme/api@sha256:3f1c...

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

• Make the default the safe choice, not just the documented one.
• Delete dead code aggressively; it's the cheapest refactor there is.
• Write the migration guide before writing the migration.
• Pair on the first few changes so the knowledge doesn't live in one head.

Where this breaks down

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Burp Suite for everything. That's a fair question with a boring answer: we tried, and it made threat modeling harder to reason about.

We still haven't solved session management. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

Closing thoughts

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.