Why your automation fails silently, and the checks that catch it
Every real outage in my own job-search automation exited 0 and wrote a clean log. These are the checks that catch that kind of failure now.
By Firas Mohammed · · 4 min read
The short answer
Automation fails silently because a scheduler judges a job by its exit code, and a broken job usually exits 0. In my own job-search automation every real outage had that shape: the task reported success while a lane produced nothing, or junk. One search lane returned the same 7 results for every keyword for days.
The check that catches this asks whether the output is plausible. Does the count move when the query moves? Do the results match what was asked? Has the lane been quiet longer than its own history allows?
What silent looked like
The system runs hourly on a Windows laptop, reading job boards and my mail. Each of these ran for days under a green status.
- A job-board search lane read only the 7 result cards painted on screen. "Head of Operations" came back with Senior Document Controller and Warehouse Assistant.
- A full sweep needed about 68 minutes against a 40-minute timeout. Results were saved only after the last lane, so every sweep was thrown away at minute 40 while the log printed "Found 40 roles".
- An API account hit its spend cap. The mail step exited with the wrong code, skipped its fallback and stopped reconciling.
- A missing request timeout let one hung connection stall the sweep. Five lanes went quiet for 11 to 12 days.
Rows from that search lane before and after the fix. All 42 were junk.
Five checks that catch it
- Watch the count. Real search results vary. A lane that returns the same number for different queries is reading a fixed piece of the page. A count equal to the page size is just pagination, so that one only gets a note. A check that cries wolf gets ignored.
- Run a canary with a control query. Once a day each lane runs real keywords plus one off-topic control: "Nurse" for an operations search. If the two result sets overlap, the keyword isn't being applied. A stale canary verdict is a finding too. In October one lane's canary count went 25, 7, 0 over three days while the login check said OK. The site had moved its search page.
- Set quiet limits from each lane's own history. My first alert used a fixed 4 days and flagged a one-off sweep, 94 rows on a single day, after 63 days of silence. The limit now is the median gap between active days times 3, with a 4-day floor. Lanes active on fewer than 3 days get none. An hourly lane alerts after 4 days, and a lane I run by hand can sit quiet for 6.
- Alert on transitions only. A message goes out when a problem appears and when it clears. An hourly "still broken" ping gets muted, and a muted alert looks like coverage. That same first alert carried a fixing command that would have hidden the finding.
- Treat zero as an error. One step printed a green "all present" tick over an empty list. Zero scanned is a hard failure now.
Repairs need proof
My status report already printed the fixing command for every finding. A person still had to type it. So I added an hourly sentinel: detect, repair, verify, escalate.
It verifies by evidence. After a repair it collects the status again, and a remedy that exits 0 and changes nothing counts as failed. Each fault gets 4 repairs per 24 hours, then it escalates as flapping, because something fixed four times a day is a bug. Remedies come from a fixed allowlist, and it never edits code. On its first run it found the dashboard down and brought it back.
In September my phone said a repair was exhausted, 4 of 4. Nothing had been repaired once. The repair refused to restart a dead browser because records on disk said a filled form was open in it, and attempts that never reported back counted as failures. An attempt with no recorded result tells you nothing.
Where I stay in the loop
Anything that leaves the machine under my name starts as a draft. Outgoing messages sit in a queue with a veto window, and I can stop any of them before they go. Replies in a live conversation are always drafted and flagged for me.
Forms are approved from my phone with a single-use token that expires in 15 minutes. Before acting, the system re-reads every field on the live page and stops if anything changed.
Questions people ask
How do I know a scheduled job is actually working?
Check what it produced. Compare today's count with the job's own history, run a query with a known answer once a day, and treat an empty result as an error. The exit code tells you the process ended.
Should automation repair itself?
For a short allowlist of safe restarts, yes, with a repair budget and a check after each repair. Anything that edits code, changes records or sends under your name goes to a person.
The system behind this
The full system, my own job-search automation on an open-source engine, is in the case study. The same approval rule applied to AI agents is in the two-key gate note.
If your automation reports green while doing nothing, this is work I take on.