On 16 August, my operations agent, the one that monitors the servers and deploys updates, froze six times in a single day. Each time, it reported the same thing: its tools had stopped responding, even to a trivial command.

It was not its tools. It never got as far as calling them.

A failure that repeats identically

An old message in its conversation contained a corrupted block, which the model's API rejected. Since every request replayed the whole history, the failure was deterministic: three automatic retries, three rejections, then a turn that ended with no visible output. Seen from a phone, it looked like a freeze; technically, it was a reproducible rejection.

A failure that repeats identically looks like instability when all you see are its effects. The difference comes down to one question: does it always fail in the same place?

The scale, invisible

  • 80 failed calls in a day, across 20 separate runs;
  • a steady rhythm, every thirty minutes;
  • an internal trigger: one of the agent's routine checks.

And that routine check was verifying… file paths on an old server, retired eleven days earlier during a migration. The instruction file itself carried a warning on its first line: “written on the old server, re-check before use”. Nobody read it, because it did not appear in any report.

So for eleven days, every thirty minutes, the agent dutifully checked a dead machine. And every check was billed, because it replayed the whole conversation: nearly 2 MB of context each time it fired.

The fix, at all three causes

  1. The pointless work: the outdated instructions were removed.
  2. The cost: routine checks now run in an isolated session, without the history.
  3. The corruption: the damaged conversation was archived.

Verified, not assumed: after the fix, zero errors and zero pointless calls. From eighty billed calls to none.

The wrong conclusion I nearly drew

My own documentation attributed this type of corruption to agent restarts, and there had been eight that day. The timeline ruled it out: the corruption predated the first restart by six hours. A plausible cause, already written down, consistent with the day's facts. Lining up the timestamps was enough to knock it down.

What a business leader should take from this

The cost was not the issue: a few tens of euros. The issue is that ten hours of silence and eighty billed calls produced no alert. The objective “cost stays marginal” was written in the specification. It was measured nowhere. Until an objective is instrumented, it is not being met: it is being hoped for.

Before handing a task to an agent, ask your supplier: how many runs failed yesterday, and how much did they cost? If they cannot answer in a minute, the system is not in production. It is living on borrowed time.