The system that secures itself

Yesterday we had a team meeting that was supposed to be about our new products in quantitative finance model risk. We ended up spending the whole hour on a different kind of model risk.

If you've been following along this month, you know why. They are calling it "Black October": a backdoored LiteLLM release on PyPI, RCEs in Next.js and Node, a certificate authority whose intermediate keys turned out to be recoverable (one of them was literally "4"), and a new one seemingly every few hours. The scale and the severity are both off the charts. Anyone responsible for pretty much any information system right now can relate to this feeling of overwhelm.

So we made a list. How do we make the company more robust to attack, both financially (nobody runs through our API keys) and in terms of our customers' information? How do we shrink our attack surface? Most of that list is boring and that's fine.

But it was also an important reminder that one of the very first examples we shipped on run.prose.md was a security audit.

The security audit we already had

The original version was really simple. You point it at a repo, and the agent decides what to look for and writes a report. That's a nice demo. It's not enough when the vulnerabilities are coming in faster than anyone can read about them, and when none of them could possibly be in the pre-training data of the models.

The first thing that came across my feed was pwn.watch by @thekitze, an aggregator of live vulnerabilities and incidents. So I asked an agent to "start with the smallest possible change to the .prose.md": point our security audit at those lists.

It was about two lines. And just from that change we found two real vulnerabilities in two different repositories, one public and one private. One of them was a login CSRF in our own web app. It's fixed and deployed now.

(We're also proudly sponsoring pwn.watch now.)

OK, now do it for every repo

That was still me pressing a button. What we really need is for this to run automatically: daily for every repository, and maybe every few hours for the critical ones.

So I gave our agent access to every repo in the org and asked: "can you try something where we scan every repo every day … maybe we need to offset the times so as not to spike the system?"

Here's the first full pass:

Repos in the org	67
Scanned	55
Flagged for action	9
Worth watching	24
Clean	10
Nothing to scan	11
Total cost	~$16

It found more things the one-off audits had missed. It also showed us something we couldn't see before: 12 repos (18%) were never scanned at all. They were refused as "too large" because a size check was counting git history that the scan never even checks out. You only find that kind of gap when you actually try to cover everything. Since fixed: our biggest repo now scans for about a dollar.

Finding isn't fixing

This is where I realized the way it was written was helpful for finding vulnerabilities, but it wasn't actually doing anything about them. A long list of problems is just more work for someone who's already overwhelmed. What we really want is an agent that can fix the issue right after it finds it.

One humbling thing we learned along the way: when we compared the agent's findings to a plain old deterministic scanner (osv-scanner), the agent reading code on its own had missed most of the vulnerable packages. I think the lesson is that LLMs are pretty bad vulnerability scanners, but they can be really good vulnerability fixers. So we split the job up. The platform does the exact lookup (locked versions against known advisories), and the agent does the part it's actually good at: figuring out what really ships, picking the most important thing, making the smallest fix, and running your tests to prove it works.

And if it can't fix something, the absolute fallback is always the report: which file, which package, which version, who needs to do what.

The features we had to add

Getting the agent to actually fix things meant adding a bunch of features to the product. Some of them are really simple:

• Fetch a specific URL. So the agent can read a feed or confirm an advisory.
• Memory between runs. The output of one run becomes the input to the next. If nothing is new, the agent doesn't waste a bunch of tokens re-discovering that there's nothing more to do.
• Linux sandboxes. This was a good excuse to finally put Linux containers into our production app. The agent gets a real shell to run your package manager and your tests, with network access only to the package registries. (And the same 72-hour cooldown I use on my own machine: "Nothing from any package manager goes on my machine that's like less than 3 days old." It gives the rest of the world time to find the bad releases, and that rule alone would have blocked the LiteLLM backdoor.)
• A package scanner and a secret scanner. The secret scanner never shows the secret to the model, and it doesn't try to "fix" a leaked key. Rotating credentials is human work, so it gives you a checklist instead.

My favorite part is how we decided what to build. This is the fun part of dogfooding agentic products with agents - using the tool to improve the tool. Every scan reports its own "capability gaps": basically "I found this, but I couldn't fix it because I didn't have that tool." The first full pass became the baseline. Once the Linux sandbox and registry access were in, an agent fixed a known CVE end to end for two cents. The tests failed, it bumped the dependency, the tests passed, and the only change was the lockfile. To be clear, that was on a test repo with an old, well-known CVE (minimist, CVE-2021-44906), not one of our production repos.

The system that secures itself

So that's what we built: a system that secures itself. Using our own product to secure our own company is kind of a core tenet for us, and it runs as an ordinary workload, with the same jobs, the same wallet and the same sandbox that anyone else gets.

A full scan of a typical repo takes about 7 minutes and costs less than 50 cents with GPT 6.1 Sol. There's nothing hard about it for the agent. The thing that matters is that you're doing it regularly and catching things quickly. A perfect job for a robot.

Try it on your repos

So if you're curious, try it out right now:

1. Go to run.prose.md/rawwerks/comprehensive-security-audit
2. Sign in with GitHub.
3. Connect it to your repository, and set it to go off daily, hourly, whatever.

And let us know what it finds!

A few really important things:

• Make it your own! You don't have to use our workflow as-is. Even on the hosted version, you can change the program, or ask our in-app agents to change it for your situation: the sources you want to scan, or the way you want to address the problem, using different kinds of multi-agent patterns.
• You don't have to run it in our cloud. You can take the program and run it yourself. The OpenProse language is open source and will "compile" in any modern coding agent harness (Claude Code, Codex, pi with subagents, OMP, Hermes, OpenCode, etc.)

Stay safe out there…and set a cooldown!

---

P.S. One more riff. It's absolutely ridiculous that in late 2026, in the age of agentic coding, my agents were still committing as me on GitHub. I'm not reviewing the majority of code by hand anymore. I'm defining the invariants and the outcomes. So why is it still using my username?

Not to put all the blame on GitHub, but they don't make it easy to give agents short-lived credentials of their own. So now our agents act as their own GitHub App, with one-hour tokens scoped to the repos they're working on, from a tiny broker that only our team can log into. Every token is logged against the agent session that asked for it. We run on it ourselves. We've published it as a template at github.com/openprose/agent-identity: you make your own App and run your own broker. This is deliberately not a product. The whole point is to have fewer trusted parties involved, not more.