career · security · communication · ai

The breach that kept coming back

It came in as a bug report on a Wednesday. The text editor on a site I look after had lost most of its buttons. An hour in, I understood I was not looking at a bug. I was looking at a hole, and someone was already climbing through it.

My first instinct was to panic. I did not. Panic makes you start fixing before you understand what you are fixing. So I sat with the logs and mapped the damage first. What got in, how far it reached, what it might have touched. Once I could describe the blast radius out loud, I started to move.

The work split into two jobs. Close the hole. Clean up what came through it.

The cleanup ran late. Around midnight I called the system administrator on the client side and told her straight how urgent this was. We got on a Zoom and rotated the keys that mattered while we were both awake to confirm each one. Nothing proved those keys had been stolen. They could have been, and that was reason enough to treat them as if they were. You assume the worst on the things that would hurt the most. That keeps the blast radius small, and it lets everyone sleep that night instead of lying awake doing math.

Then I thought I had it. I patched the path the attacker used, set up a scanner, and stepped back.

Out of habit, I checked again myself. The attacker had already found a second way in, one my fix did not cover and my scanner never flagged. If I had trusted the green light over my own eyes, the site would have sat there wide open as though I had never shown up. That happened three times. Patch, reverify, find the next door hanging open. Whack-a-mole, and the mole was winning on volume.

What ended it was going under the symptom. Instead of closing each door one at a time, I shut the whole wing. I cut off every route into that plugin at once. The recurrences stopped. By late Thursday into early Friday morning, it was actually over.

A few things stuck.

Recheck it yourself

Reverification is the job. Run it yourself, every time. A scanner only catches what it was built to catch and stays quiet about the rest. The check I trusted was the one I ran by hand, hunting for the thing I had not thought of yet.

Legacy is a countdown

Running a live site on an out of date framework is a countdown. When the security patches stop coming and the known holes pile up, getting hit becomes a question of when somebody aims a bot your way. Sometimes the vendor does not ship a fix in time. In my case a clean upgrade would have triggered a cascade of other upgrades, too risky and too slow for a site already scheduled to be retired in a few months. So you hold the line by hand and you plan the exit.

The human part

The part people skip is the human one. The midnight call. The plain language. Choosing to act as if the worst were true so nobody had to gamble on it. Each time it came back I told her before she had to ask, owned that my last fix was too narrow, and showed her the wider one. That is what kept her trusting me enough to let me keep working while the ground moved under us.

When attacking gets cheap

And then there is the part that actually sits with me. This landed three days after the exploit for that plugin went public. It also landed the day after Anthropic shipped its Fable 5 model, which the company pulled soon after, following a public jailbreak.

I cannot prove a model wrote the thing that hit me. I can tell you the timing made me sit up. Attacking a system used to take skill. Now someone can describe what they want and let a model do the reaching. More people will try, because trying just got cheap.

So the next stretch looks like white hat AI against black hat AI. Some of the black hat will be bad intent holding a good model. Some of it will be a good model talked into bad work by someone who knew the right words. I spent two days as the white hat. I do not think it was my last.