Bottom Line Studio
Seven checks. Only six watched.
A new rollout check made it into the pipeline and missed the alert job’s hand-typed list. We caught the gap before the first scheduled run. The fix changed how monitoring found its checks.
A quiet alert can hide a missing check.
Sound familiar? You added a data check last sprint. No alert has fired. Does that mean the check passed, or that nobody is watching it?
The check shipped. The alert coverage didn’t.
On a client go-live this summer, the one where a data contract caught 14 bad Stage 1 dates, a set of automated checks guarded the new pipeline. An alert job watched those checks and posted to the team when one failed.
The alert job knew which checks to watch from a list. Someone had typed that list by hand.
Then we added a seventh check, for the rollout itself. It went into the pipeline. It didn’t go onto the list. We caught it the day before the alert’s first scheduled run. Had we missed it, that check could have failed every morning and nobody would have heard.
It was the second time the list had fallen behind. The first time, we fixed the list. The second time, we fixed the design.
Hand-typed lists aren’t a sign of a careless team. They’re the fastest way to get monitoring running on day one, and on day one they’re correct. The trouble starts on day thirty, when the system has grown and the list hasn’t. Nothing fails. The list just quietly stops describing reality, and an alert that never fires looks exactly like a pipeline that never breaks.
Read the schema, then check the set
The analytics engineer taking over the platform reviewed the alerting and suggested the fix: stop telling the alert job what exists. Let it look.
- Iterate the schema. The job asks the database for every check that exists and watches all of them. A new check is covered the moment it ships.
- Assert the expected set. The job compares the identities it discovers with an expected set. A missing or renamed check must produce a loud mismatch. Review intentional changes to that set alongside the code. Equal counts alone are not enough.
You need both halves
Either half alone brings the old problem back in a different form. Iterating alone covers new checks but lets a deleted one disappear without a sound. The expected set alone is just a typed list with extra steps. Together, adding a check is automatic and removing one becomes a decision someone has to make on purpose.
| A typed list | Iterate the schema only | Iterate + expected set | |
|---|---|---|---|
| A new check is added | Silent until someone updates the list | Covered automatically | Covered; reconcile the expected set |
| A check is renamed or dropped | The list points at nothing | Vanishes without a sound | Fails loudly |
| Who has to remember | Everyone, every time | Nobody, until something goes missing | An owner reviews intentional changes to the expected set. |
Check yours in ten minutes
- Find every hand-typed list of things to watch: alert configs, freshness monitors, “dashboards to refresh” lists.
- Compare each list to what actually exists. Compare identities as well as counts. Seven expected and seven found can still be the wrong seven.
- Replace the list with a query against the place the checks really live.
- Keep an expected set and fail when what’s found doesn’t match it.
- Break something on purpose and confirm the alert fires. The Pipeline Guardrail Audit walks through that test and eleven others.
When did you last break a check on purpose to prove the alert worked?