On 27 May, an internal service ended up exposed to the internet, listening on every address instead of the local machine only. An API key was used up to the tune of about 250 dollars before anyone noticed. The incident was fixed and recorded in the operations agent's memory: date, cause, impact, fix. A flawless note.
On 3 August, two other services were exposed in exactly the same way: the password vault and a file server that was being actively scanned.
The memory had kept the case, not the rule
A note gets reread when someone thinks to look for it. A check runs even when nobody thinks of it. That is the whole difference between an organisation that documents its mistakes and one that stops repeating them.
The same day, another discovery
While auditing that memory, a second problem came to light. A scheduled script, which nobody had documented, was adding an “all is well” statement to it every night, without having measured anything. 51 identical blocks took up more than half of the agent's long-term memory, the part it rereads every time it starts.
A task that writes “all is well” without having checked anything is worse than no task at all: it manufactures false assurance.
The rule adopted
Every incident experienced leads to a daily check:
- exposed service → list of all open ports, with an alert on any new one;
- service down for seven weeks with no alert → check that every expected service is running;
- disk filled up by caches → free-space threshold and cache volume;
- memory bloating silently → memory size compared with a ceiling;
- security test run from the machine being tested → always validate from a third-party network.
In your company
The question to ask after every incident, human or machine, is not “did we document it properly?” but “what will now detect it without us?”. If the answer is “someone who rereads the note”, the incident will happen again.