The Alert That Never Fired

A five-area code review of oo-workers before the v1.29.0 release found three criticals. The worst was quiet: browser-check runs that completed, recorded results, and never alerted.

The alert that never fired#

A browser-check run is N Playwright tests, each writing its own execution row, so "did the site go down?" is a per-run aggregate. The old detector grouped rows by a ±30 second startedAt bucket. That sort of worked on the master path. On the region-dispatch path results come back one at a time, nothing aggregated them, and the alerting function was never called.

The fix replaces timing with structure:

• A qa_runs row is created at run start with an expected_tests count; every execution row carries its run_id.
• Completion is count-based: the writer checks whether all expected rows have a completedAt.
• Alerting is an idempotent claim: UPDATE qa_runs SET outcome, alerted_at WHERE alerted_at IS NULL. When the last two results race, exactly one alert fires.
• Comparison is scoped to (project, region), so a healthy region can no longer mask another region's outage.

Credentials that went nowhere#

The Playwright runner accepted a credentials parameter and never read it. Authenticated checks silently ran logged out, and a loosely written "dashboard loads" test passes fine against a login redirect. Credentials are now injected into the spawned test's environment, so scripts read process.env.LOGIN_EMAIL and actually sign in.

The import that lied#

oo-workers can be seeded from an ObserveOne export: obs export produces a JSON bundle of your monitors, alert channels, and status pages, and the self-hosted instance imports the whole bundle in one shot. That import ran as a single Postgres transaction while counting per-row insert errors as "skipped" and moving on. Postgres aborts a transaction at the first error, so every insert after a bad row failed and the final COMMIT rolled everything back, while the importer still reported partial success. Each row now runs in its own savepoint, so a bad row rolls back alone and "imported 214, skipped 3" means what it says.

Redacting what the URL carries#

A check URL can embed basic-auth credentials (https://user:pass@host). Error messages interpolated the raw URL and flowed to logs, Slack, webhooks, and email. URLs are now credential-stripped before they reach any error text.