On-prem backup and restore

Backup, business continuity, and disaster recovery for a self-hosted AgentPlaybooks instance. This runbook supports NIS2 Article 21(2)(c) (backup management and disaster recovery) for operators who run the product on their own infrastructure.

This is an operational guide, not legal advice. Confirm retention, classification, and reporting duties with your IBF (information-security officer) and counsel before production use.

The restore drill required by the test plan — restore from backup onto an empty machine, confirm secrets are readable, confirm the audit trail, record RTO — is automated as scripts/self-host/restore-drill.sh and .github/workflows/self-host-restore-drill.yml.

Related: Self-Hosting (deployment), Cross-platform Agent Backups (apb push / apb pull of playbook files — not a vault or database backup). The same runbook lives at docs/self-host/backup-restore.md in the repository.

What to back up

A self-hosted instance has a stateless application container (docker-compose.yml at the repository root) and a stateful data plane (self-hosted Supabase, or scripts/self-host/compose.yml for the Postgres + object-storage half used by this runbook).

The application image is rebuilt from this repository. Do not treat container layers as the backup of record.

apb push portable snapshots store skills and MCP references, not vault values. They do not replace this runbook.

Key-loss warning

If SECRETS_ENCRYPTION_KEY is lost, every stored secret is gone.

The vault derives a per-user AES-256-GCM key with HKDF from that 64-character hex master key, the owner’s user id, and (for v2 ciphertext) the playbook id and secret name. The database holds only ciphertext. There is no re-encryption tooling and no key-escrow path in the product.

• Generate the key once: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
• Store it outside the backup archive, under a different access-control list than BACKUP_ENCRYPTION_PASSPHRASE
• Rotating it without a re-encrypt procedure makes existing rows undecryptable
• A successful database restore with the wrong key still looks healthy: playbooks and audit rows return, and secret.encrypted_value is present — decryption fails only when a caller reads a secret

Treat a missing master key as a reportable incident under your organisation’s process, not as a restore retry.

Scheduling

Cron example (host that can see the compose project):

# /etc/cron.d/agentplaybooks-backup
SHELL=/bin/bash
# 02:15 local, every day
15 2 * * *  root  set -a; . /etc/agentplaybooks/backup.env; set +a; /opt/agentplaybooks/scripts/self-host/backup.sh >>/var/log/apb-backup.log 2>&1

Load scripts/self-host/backup.env.example into /etc/agentplaybooks/backup.env (mode 0600).

Encryption of backups

Archives are encrypted at rest with AES-256-CBC, PBKDF2, 310 000 iterations, via OpenSSL, using BACKUP_ENCRYPTION_PASSPHRASE.

• This passphrase must not be SECRETS_ENCRYPTION_KEY
• Backup files are written mode 0600 into BACKUP_DIR (default ./backups, mode 0700)
• --no-encrypt exists only for isolated lab debugging; do not use it in production
• Optionally copy the .tar.gz.enc file to an offsite location you control (BACKUP_OFFSITE_RSYNC)

Ciphertext of vault secrets is encrypted again inside the archive. That is defence in depth, not a substitute for the master key.

Retention

backup.sh deletes archives older than BACKUP_RETENTION_DAYS (default 90) and then keeps only the newest BACKUP_KEEP_COUNT (default 14).

Pick numbers that match your classification under 7/2024 (VI. 24.) MK rendelet / organisational policy. A starting point many operators use:

• 14 daily copies on-site
• 4 weekly copies offsite
• 6 monthly copies offsite

The script implements a simple day/count cap so a full GFS calendar can live in your backup product (Veeam, Borg, restic) wrapping these archives. Do not lower retention in order to pass a disk-usage check.

Offsite copy

Set BACKUP_OFFSITE_RSYNC to a second machine or cluster (example: backup@dr.example.org:/var/backups/agentplaybooks/). backup.sh runs rsync -a after a successful encrypt.

Follow a 3-2-1 pattern: three copies, two media, one offsite. The compose volumes on the primary host are not an offsite copy.

RPO / RTO guidance

These are starting targets for an on-prem instance. Tighten them if your security class, IBF, or contract says so.

Record every restore drill’s RTO_SECONDS (CI logs and your BCP folder). If the measured RTO exceeds the target, the fix is operational (faster disks, warm standby, smaller database) — not a change that weakens audit or tests.

A reasonable first target for a significant-class on-prem instance: RPO ≤ 24 h, RTO ≤ 4 h. High class typically needs a tighter pair and a documented warm standby.

Restore procedure (empty machine)

• Install Docker and clone this repository (or copy the compose files and scripts).
• Restore SECRETS_ENCRYPTION_KEY from the separate secret store into the application environment. Do not skip this step.
• Restore BACKUP_ENCRYPTION_PASSPHRASE so the archive can be decrypted.
• Copy the newest .tar.gz.enc from offsite to the machine.
• Start an empty data plane:

docker compose -f scripts/self-host/compose.yml down -v
   docker compose -f scripts/self-host/compose.yml up -d

For official self-hosted Supabase, start that stack with fresh volumes instead, and point COMPOSE_FILE / DB_SERVICE / STORAGE_SERVICE at it.

• Restore:

export BACKUP_ENCRYPTION_PASSPHRASE=…
   export SECRETS_ENCRYPTION_KEY=…
   ./scripts/self-host/restore.sh --archive backups/agentplaybooks-backup-YYYYMMDDTHHMMSSZ.tar.gz.enc

• Start (or rebuild) the application from the root docker-compose.yml, with the same NEXT_PUBLIC_SUPABASE_* values the browser can reach and with SECRETS_ENCRYPTION_KEY set.
• Verify:

- A playbook that existed before the incident still loads - A vault secret decrypts (reveal or use_secret with a known test value) - audit_logs row count is at least the pre-incident count - If the audit hash-chain verify command exists on this revision, run it

• Write the printed RTO_SECONDS into the drill record.

restore.sh drops and recreates the target database. Do not point it at a live instance you still need.

Scripts

# Backup (from the repository root)
set -a; . /etc/agentplaybooks/backup.env; set +a
./scripts/self-host/backup.sh

# Restore onto empty volumes
./scripts/self-host/restore.sh --archive backups/agentplaybooks-backup-….tar.gz.enc

# Automated drill (also run by CI)
./scripts/self-host/restore-drill.sh

Point the same scripts at an official Supabase compose project:

export COMPOSE_FILE=/opt/supabase/docker/docker-compose.yml
export COMPOSE_PROJECT_NAME=supabase
export DB_SERVICE=db
export STORAGE_SERVICE=storage
export STORAGE_PATH=/var/lib/storage

Restore drill

The drill:

• Starts scripts/self-host/compose.yml
• Applies supabase/schema.sql
• Seeds one playbook, one AES-256-GCM secret, three audit_logs rows, and a storage object
• Runs backup.sh
• docker compose down -v (volumes must disappear)
• Starts a fresh stack and runs restore.sh
• Asserts: secret decrypts to the same plaintext, playbook name/GUID match, audit_logs count is unchanged, storage object matches
• If an audit hash-chain verify command exists (npm run audit:chain, scripts/verify-audit-chain.sh, or scripts/verify-audit-hash-chain.ts), runs it; otherwise prints a notice and continues
• Prints RTO_SECONDS

CI: .github/workflows/self-host-restore-drill.yml (pull request and main when these files change, plus a weekly schedule).