Propose Freely. Merge Never.

A guardrail that has never fired is not proven. It is unproven. A quiet alert channel means either that nothing has broken, or that the thing meant to notice is broken — and from the outside those look identical.

This audit exists because we trusted one that had been silently incapable of working for months. It read job state from a source where suspended jobs do not appear, so the single condition it existed to detect was, structurally, invisible to it. It reported success every hour. A production pipeline sat broken for three days.

Work through the four parts below in order. Each check tells you what to look at, why it matters, what a passing answer looks like, and how to prove it rather than assume it. Nothing here is vendor-specific; the query shapes are pseudocode you translate to your own warehouse.

The rule that makes the whole audit work

Never mark a check as passing because the code looks correct. Mark it passing only when you have broken something on purpose and watched the system react. Reading is not testing.

Part 1 — Can your monitoring observe the failure state?

The most expensive failures are not wrong logic. They are correct logic pointed at a place where the answer never appears.

The state source actually contains the state you're watching for

CheckFor every guardrail, find the table, view or system function it reads job state from. Then confirm that a job in the failed state and a job in the suspended state both appear there.

WhyMany schedulers expose "currently running" and "recently run" views that omit suspended objects entirely — a suspended job has no scheduled run, so it appears nowhere. Monitoring built on those views can detect a job that fails but never one that stopped.

GoodYou can point at a specific column, in a specific source, that reports the suspended state — and you have seen it do so.

Prove it

-- suspend a non-critical job, then run the guardrail's
-- own state query verbatim. does the job appear?
SHOW <jobs> IN <scope>;              -- includes suspended
SELECT * FROM <running_jobs_view>;   -- probably does not

The guardrail runs in every environment it claims to cover

CheckGrep every guardrail for hardcoded database, schema, account or environment names. Confirm a deployed, running copy exists per environment.

WhyGuardrails are usually written and tested in development, then "deployed" to production by copying the file. A hardcoded environment name survives the copy and the production instance monitors development forever.

GoodEither the environment is derived from execution context, or there is one self-contained copy per environment and you have listed them.

Prove itList the deployed guardrail objects in production and read the body of each. Do not trust the repository — read what is actually deployed.

Every object the guardrail names still exists under that name

CheckExtract every literal object name from your guardrail code and resolve each one against the live catalogue.

WhyA loop that iterates over a name pattern matching nothing does not error. It completes successfully, in zero iterations, and reports success — indefinitely, and at whatever frequency you scheduled it.

GoodEvery name resolves, or the guardrail raises when it matches zero objects.

Prove it

-- the highest-value assertion in this whole document
IF (matched_count = 0) THEN
    RAISE 'guardrail matched no objects — check names';
END IF;

Part 2 — Do failures actually reach a human?

A detection that stops at a log file is not an alert. It is a record you will read after the incident, during the post-mortem, at the point where it is worth nothing.

You have traced one real failure end to end

CheckBreak something deliberately in a non-critical path. Follow it from the failing check, through detection, to the message arriving on a device someone looks at.

WhyEvery link in that chain is individually plausible and collectively untested. The break is almost never in the check — it is in the hop between detection and delivery.

GoodYou have the actual notification, with a timestamp, and you know how long the whole path took.

Prove itScreenshot it. This is the single highest-value artefact from the audit, and it is the one people skip.

The recipient is able to receive the message

CheckConfirm each recipient is verified with whatever integration sends the alert, that the channel still exists, and that the person is still at the company and still owns that system.

WhyMost notification integrations only deliver to verified addresses, and adding a name to a configuration file does not verify it. Alerts route to departed colleagues and archived channels routinely.

GoodEvery recipient has confirmed receipt of a real message within the last quarter.

Prove itAsk them. "Did you get this?" is a complete test.

More than one person receives it

CheckCount the distinct humans on the critical alert path. If the answer is one, that is your bus factor for the entire monitoring system.

WhySingle-recipient alerting works until that person is on holiday, and holidays are exactly when a multi-day outage becomes possible.

GoodA shared channel or distribution list, not an individual — and someone else has confirmed they see it.

Prove itSend a test while the primary recipient is away and confirm the secondary acts on it.

Part 3 — Would you notice if coverage quietly decayed?

Detecting a test that fails is easy. Detecting a test that stopped running is the one almost nobody builds, and it is where confidence outlives coverage.

Test results are durable, not just logged

CheckConfirm your important tests write their failing rows somewhere queryable, rather than emitting a warning into a build log.

WhyIf tests run as warnings so they cannot block the refresh — usually the right call — then nothing persists their results by default. A warning nobody reads is indistinguishable from a pass.

GoodA dedicated schema where one table per test holds that test's failing rows. Empty is the healthy state.

Prove it

SELECT table_name, row_count
FROM <catalogue>
WHERE table_schema = 'TEST_FAILURES';

You assert the expected set, not just scan what's present

CheckConfirm your monitoring compares the set of tests it finds against the set it expects, as an equality — not merely iterating whatever happens to be there.

WhyIterating what exists cannot see a deletion. A removed or renamed test simply stops having a result, which reads as silence, which reads as healthy. Set equality catches both directions: something missing means a test disappeared, something unexpected means a rename left a stale artefact whose old rows will look like a live failure forever.

GoodBoth a missing and an unexpected comparison, and an alert on either.

Prove itDelete a test in a branch, run the monitor, confirm it complains. Then rename one and confirm it complains differently.

The expected set is derived, not hand-maintained

CheckFind where the list of monitored tests lives. If it is a literal list in a procedure, it has already drifted or it will.

WhyHardcoded inventories go stale silently — someone adds a test and forgets the list. That is the same class of gap the tests exist to catch, reproduced inside the monitoring itself. We have watched exactly this drift twice on one system.

GoodThe monitor discovers what to watch (iterate the schema) and separately holds a small, reviewed expectation set that fails loudly when reality diverges.

Prove itAdd a new test, run a build, and confirm the monitor picks it up with no code change.

Part 4 — Can the system recover without someone awake?

If recovery requires a human noticing, your recovery time is however long it takes someone to look. Over a long weekend, that is days.

Recovery runs in dependency order

CheckConfirm automated resume walks the job graph from leaf to root, not in arbitrary or alphabetical order.

WhyResuming a root before its children means the next scheduled run fires into a partially-resumed graph, which fails in a new and more confusing way than the original problem.

GoodExplicit child-to-parent ordering, with the scheduled root resumed last.

Prove itSuspend the whole chain in a test environment and watch the order it comes back.

Recovery triggers a run instead of waiting for the next tick

CheckAfter resuming a suspended root, confirm the guardrail fires an immediate execution.

WhyResuming a daily job at 09:00 without triggering it means the data stays stale until 02:00 tomorrow. You recovered the scheduler and left the business without numbers for a full day.

GoodResume, then execute immediately — but only when the root was the object actually suspended.

Prove itSuspend the root, wait for the guardrail, and confirm a run starts within its interval rather than at the next scheduled time.

You rehearse it on a schedule

CheckPut a recurring calendar entry — quarterly is enough — to break one thing deliberately and confirm the whole chain reacts.

WhyEverything above decays. Names change, people leave, integrations get reconfigured, someone comments out a check during an incident and never restores it. An audit is a snapshot; the rehearsal is what keeps it true.

GoodA dated record of the last rehearsal and what it caught.

Prove itThe calendar invite exists and the last one actually happened.

Scorecard

Mark a check proven only if you broke something and watched the system react. Everything else is assumed, and assumed is what the three-day outage was.

#	Check	Status
01	State source contains the state you watch for	proven
02	Runs in every environment it claims to cover	proven
03	Every named object still resolves	proven
04	One real failure traced end to end	proven
05	Recipients can actually receive	proven
06	More than one human on the path	proven
07	Test results are durable, not just logged	proven
08	Expected set asserted as an equality	proven
09	Expected set is derived, not hand-kept	proven
10	Recovery runs in dependency order	proven
11	Recovery triggers rather than waits	proven
12	Rehearsal is scheduled and has happened	proven

How to read your score

10–12 proven: your guardrails are real. Schedule the rehearsal and move on.
6–9: normal, and the gaps are worth an afternoon.
Under 6: you do not currently know whether your monitoring works. That is not a criticism — almost nobody does until they check. Start with 01, 03 and 04, which find the most for the least effort.

If you'd rather not run this alone

We help GTM and RevOps teams audit what they already have, fix what the audit finds, and build the guardrails that make it safe to put AI anywhere near production data. The first conversation is free and specific to your stack — bring your scorecard.

calendly.com/ian-bottomlinestudio/30min