Monitoring¶

A failed pass says so. The night that never ran — the scheduler stopped, the worker was never deployed, the machine was off — says nothing at all. The morning check is the answer to that silence.

The morning check¶

python manage.py backup_check
# backup_check: ok, newest whole backup 2026-03-10 03:00 UTC

It asks the store for the newest whole backup under:

the config/ folder, when CONFIG is set, and
every database's folder — db/, db-<alias>/ for each alias the pass would dump today,
or, with neither configured, every media folder.

Objects marked .without-media. do not count: a deploy at noon cannot stand in for a night that never ran. Media does not count either while there are databases or a configuration: tonight's media does not stand in for a database that was not written.

finding	message	exit
every folder has a whole backup within `STALE_HOURS` (26)	none	0
a database with no backup at all, beside databases that are backed up	none — logged at INFO, `Databases not backed up yet` (below)	0
some folder's newest is older, or there is none	MISSING, naming the oldest	non-zero (`BackupMissingError`)
the store could not be read	UNCHECKED — as loud as missing	non-zero (`BackupUncheckedError`)
backups are off on this machine	none	0, "nothing to do"

The error names every stale folder:

CommandError: backup_check: No whole backup within 26 h under acme/prod/: db-analytics/ (none), config/ (2026-03-08 03:00 UTC)

STALE_HOURS is 26 so that a pass at 03:00 one night and a slow one the next still pass a check at 09:00. Raise it if your nightly pass runs less often than daily.

A database added today¶

With ALIASES a callable, the list of databases is computed at every check — and a tenant created after the nightly pass has no backup at all at the morning check. That is not a missed night, so:

a database folder with no backup at all, beside databases that are backed up, is "not backed up yet": logged at INFO (Databases not backed up yet, the kinds in ob_kinds), not MISSING. Its first nightly pass backs it up;
a database whose backups exist but are stale is MISSING, as before;
the only database, with no backup at all, is still MISSING — there is nothing beside it to show that the passes run;
a database whose dump fails every night is not silent either: each night's FAILED message names it.

Schedule it as in Scheduling; from Python it is offsite_backup.check_last_night(), which returns the oldest of the newest whole backups (or None where backups are off) and raises the same errors.

The last-success file¶

'LAST_SUCCESS_FILE': '/var/run/acme/backup-last-success',

After every successful pass the file is rewritten with the pass's time in ISO format:

2026-03-10T03:00:12.402113+00:00

Something already watching the machine can read its age — a node-exporter textfile collector, a Docker healthcheck, a monitoring agent's file-age check. It is written by every successful pass, deploy passes and passes by hand included, so it answers "did anything succeed lately", not "did last night run"; the morning check — or backup_status, per kind — answers that. A file that cannot be written is logged (Last-success marker not written) and does not fail the pass. Mount the path where your monitor can see it — the worker writes it at night, the deploy's one-off container before a migration.

The age of every kind¶

The morning check says MISSING once a day, in the chat. A monitor wants the same answer whenever it asks, silently, and as a number:

python manage.py backup_status                # or: offsite-backup status
# acme/prod/ (stale after 26 h)
#   config/: 2026-03-10 03:00 UTC, 6.2 h ago
#   db/: 2026-03-10 03:00 UTC, 6.2 h ago
#   media/: 2026-03-10 03:00 UTC, 6.2 h ago

For every kind — the config, every database the pass would dump today, and every media source — it reads the newest whole backup from the store, as the morning check does (objects of a pass without media do not count), and its age. Nothing is notified.

finding	exit
every kind has a whole backup within `STALE_HOURS` (or `--stale-hours H`)	0
a kind's newest is older, or there is none (`STALE` on its line)	1
the store could not be read	2
backups are off on this machine	0, "nothing to do"

That makes it a healthcheck as it is — a cron job's exit status, a Docker HEALTHCHECK, a monitoring agent's command check, a push to a dead man's switch on success. --json is for a script; from Python, offsite_backup.freshness() returns the same figures (kinds, each with kind, newest, seconds, stale) and raises BackupUncheckedError when the store cannot be read.

Prometheus¶

python manage.py backup_status --prometheus /var/lib/node_exporter/textfile/offsite_backup.prom

writes the text exposition format for node-exporter's textfile collector (or any scraper that reads a file) — atomically, so a scrape never reads half a file — and still exits as above. Run it from cron every few minutes, or after the pass:

offsite_backup_up{prefix="acme/prod"} 1
offsite_backup_checked_timestamp_seconds{prefix="acme/prod"} 1773133200
offsite_backup_last_success_timestamp_seconds{prefix="acme/prod",kind="db"} 1773111600
offsite_backup_stale{prefix="acme/prod",kind="db"} 0

metric
`offsite_backup_up`	1 when the store was read; 0 — and nothing else is written — when it was not
`offsite_backup_checked_timestamp_seconds`	when the store was asked
`offsite_backup_last_success_timestamp_seconds{kind}`	when the newest whole backup of the kind was written; absent for a kind with none
`offsite_backup_stale{kind}`	1 when the kind has no whole backup within the threshold

An alert, for example:

- alert: OffsiteBackupStale
  expr: offsite_backup_stale == 1 or offsite_backup_up == 0
  for: 30m
- alert: OffsiteBackupNotChecked
  expr: time() - offsite_backup_checked_timestamp_seconds > 3 * 3600

The second catches the textfile itself going stale — the job that writes it stopped.

The log¶

Everything is logged to the offsite_backup logger, with fields in the record's extra (prefixed ob_, so a JSON formatter can keep them apart from your own):

level	message	fields
INFO	`Backup written`	`ob_prefix`, `ob_objects`, `ob_bytes`
INFO	`Database backed up` (a one-off backup)	`ob_alias`, `ob_key`
INFO	`Nightly backup skipped: ...` (off, done, or being done)	—
INFO	`Databases not backed up yet` (the morning check)	`ob_kinds`
ERROR (with traceback)	`Database not backed up`	`ob_alias`
ERROR (with traceback)	`Store not written; left out of this pass` — one of several stores refused an object	`ob_store`
WARNING	`Backup still locked; left for a later night`	`ob_key`, `ob_until`
WARNING	`Backup depth not read`	`ob_folder`
WARNING	`Last-success marker not written`	`ob_path`, `ob_error`
WARNING	`Media file not listed`, `Media file not readable; left out`	`ob_path`, `ob_error`
WARNING	`Configuration path not found; left out`, `Media directory not found` — a mount missing where the pass ran (`W005` under `check --deploy`)	`ob_path`
INFO	`Restore drill passed`	`ob_prefix`, `ob_stamp`
ERROR (with traceback)	`Database not restored by the drill`, `Archive not opened by the drill`	`ob_key`, `ob_step`
ERROR (with traceback)	`Drill database not emptied`	`ob_alias`
WARNING	`Live database not counted` (the drill)	`ob_alias`, `ob_error` (the class only)
WARNING	`Media volume not found` — a `VolumeSource` Docker volume that does not exist (`W005` under `check --deploy`)	`ob_volume`
WARNING	`Media file is a hard link; left out` — in a Docker volume	`ob_path`
ERROR	`Notifier not built`, `Notifier raised`	`ob_notifier`, `ob_error`
ERROR	`Backup message not formatted`	`ob_error`
ERROR	`Telegram refused the backup message`	`ob_status`, `ob_description`
ERROR	`Telegram unreachable`, `Backup message not sent through the bot`	`ob_error` (the class only)
ERROR	`Webhook refused or unreachable`	`ob_status` or `ob_error` (the class only), `ob_attempts`
ERROR	`Webhook body not built`	`ob_error`
INFO / ERROR	`Backup written` / `Backup failed` ... from `LogNotifier` (INFO for a written backup and a passed drill)	`ob_prefix`, `ob_text`

No message, token, password or private key is ever logged.

Heartbeats of scheduled tasks¶

A project that tracks when each of its scheduled tasks last ran — a heartbeat table, a "last run" dashboard — sees the package's tasks as never run until each has run once: the nightly pass after its first 03:00, the check after its first 09:00, the drill after its first month. Register them under their task names, the same for Celery and taskiq:

task	default schedule (UTC)
`offsite_backup.nightly`	03:00
`offsite_backup.check`	09:00
`offsite_backup.drill`	only when scheduled — e.g. the 1st of the month, 05:00 (Scheduling)

What the store holds¶

python manage.py backup_list prints, per kind, the nights kept, the oldest night, the objects and bytes, the newest stamp, how long the newest object is locked and how many objects came from passes without media. --json for scripts. See Restore.