Heartbeats

Monitor cron jobs, backups and queue workers by having them ping ObserveOne when they finish. A missing ping past the period plus grace raises the alert.

A heartbeat watches a job ObserveOne cannot reach: a nightly backup, a cron task on a private box, a queue worker inside a VPC. Nothing gets probed. The job tells ObserveOne it finished, and silence is what raises the alarm.

How the timing works#

A heartbeat has two numbers. Period is how often you expect the ping: a slider of presets in the dashboard, from every 30 seconds up to daily, or --period in seconds from the CLI. Grace is the slack you allow on top, in seconds on both, and a backup that usually takes 40 minutes but occasionally runs long wants a grace of 3600 (1 hour).

After each ping, the next one is due at last ping + period + grace. Miss that deadline and the heartbeat goes down and alerts through whatever channels you attached.

Set grace deliberately. Too small and a job that runs a few minutes long pages you every week until you stop believing the alert. Too large and a job that died at midnight goes unnoticed until morning.

Create one#

In the dashboard, open Heartbeats, choose New, name it, and drag the period slider. Grace lives under Advanced Settings.

The Create Heartbeat Monitor form with a name field and the expected-period slider
Period is a slider of presets here; the CLI takes the same value in seconds.

From the terminal:

obs heartbeat create --name "Daily Backup" --period 86400 --grace 3600

Either way you land on the heartbeat, where the ping URL is waiting.

A newly created heartbeat showing status Pending, its ping URL, and an empty ping history
A fresh heartbeat sits at Pending until the first ping arrives.

That key in the URL is the whole credential: anyone holding it can report your job as healthy.

Ping it from your job#

Call the URL when the work has succeeded, not when the script starts.

#!/bin/bash
set -eo pipefail
pg_dump mydb | gzip > /backups/mydb.sql.gz
curl -fsS --retry 3 https://<host>/ping/<key>

pipefail is doing real work there. Without it a pipeline reports the exit status of its last command, so a pg_dump that dies still leaves gzip exiting 0 on a valid empty archive, the script pings, and you have built a monitor that reports success forever.

Reading the states#

  • pending: created, no ping yet.
  • up: the last ping arrived on time.
  • late: the deadline has passed and the check has not caught up yet. It does not alert.
  • down: the missed ping is confirmed. This is the state that alerts.
  • paused: switched off with obs heartbeat toggle <id>, and not alerting.

obs heartbeat runs <id> lists recent pings with their arrival times.

After an outage#

obs heartbeat reset <id> wipes the ping history, zeroes the missed-ping count and puts uptime back to 100%. It also clears the deadline, so the heartbeat cannot go down until the next real ping. Reset a job you have fixed, not one you are still debugging.

To silence one without switching it off, obs heartbeat toggle-muted <id>. Pings and states keep recording, alerts stop.

Next steps#

Was this page helpful?