Bottom Line Studio AI
Two version pins drifted, and the nightly build went dark
A production data pipeline stopped compiling code that had worked for months. Nobody had touched it. Two settings that had to move together had quietly stopped agreeing.
A green local build is evidence about your laptop. Production runs somewhere else.
Sound familiar? A pipeline nobody touched stops running. Everything passes on your machine. And the scheduler has already given up — no error in anyone’s inbox, just a job that isn’t there anymore.
What happened
On a client’s data platform, the nightly production build stopped. Not slowed down — stopped. After repeated failures the scheduler auto-suspended the job, which is exactly what you want it to do and deeply confusing to walk into cold.
The cause was a coupling that lived nowhere in one place. A newer syntax for declaring test arguments needed two things at once: a runtime at or above a certain version, and a config flag turning on the stricter behavior. Both were in place. Then the runtime pin in one environment moved below the line. The flag stayed on.
The result wasn’t one broken test. Every test in the project failed to compile, because the syntax the project used was no longer syntax the production runtime understood. Repeated failures tripped the auto-suspend. That’s the right call for a job that keeps failing. It’s also silent: a suspended job stops sending errors, because as far as the scheduler is concerned it isn’t running anymore.
Local checks didn’t catch it, and that’s the expensive part. The local toolchain was a different implementation of the same tool, and it was stricter: locally, the new syntax was mandatory. In production, below the version line, it was unreadable. A clean local run proved nothing about production. Every instinct says “it compiles on my machine, we’re fine.” That instinct was the problem.
The fixes, in order of value
Two habits cover most of it: keep coupled settings next to each other, and make CI the judge instead of your laptop. Here’s what that looks like in practice.
- Make CI run the production version. The single biggest one. This class of break gets caught at pull-request time by a machine, not overnight by a scheduler.
- Pin coupled settings in the same change. If a flag needs a minimum version, the pin and the flag go in one pull request, with a comment at each site naming the other.
- Test risky changes against the real runtime. Anything touching syntax, versions or project-wide config runs once on the production runtime before merge. Zero errors, and no deprecation warnings you haven’t read.
- Read every line an auto-fixer changes. Migration tools are useful. They’re not a reviewer, and a diff that parses locally can still break production.
- Alert on suspension as its own event. A suspended job doesn’t fail; it just stops existing. List the jobs that should be running, compare to what is, and page on the gap.
The eventual fix was a one-line pin. The cost was a day spent working out why a pipeline nobody had touched had stopped compiling. Nobody forgets a coupling like this on the day they set it up. They forget it four months later, when a routine bump moves one pin and not the other — which is why the note at each site matters more than the memory of whoever wrote it.