bmc_hub/docs/BACKUP_SYSTEM.md

BMC Hub Backup System - Environment Configuration Guide

Quick Start

Add these lines to your .env file to enable the backup system:

# ===== BACKUP SYSTEM =====

# Safety switches (start with safe defaults)
BACKUP_ENABLED=true                # Enable backup system (default: false)
BACKUP_DRY_RUN=false              # Disable dry-run to actually perform backups (default: true)
BACKUP_READ_ONLY=false            # Allow restore operations (default: true)

# Backup formats
DB_DAILY_FORMAT=dump              # Compressed format for daily backups (default: dump)
DB_MONTHLY_FORMAT=sql             # Plain SQL for monthly backups (default: sql)

# Backup scope
BACKUP_INCLUDE_UPLOADS=true       # Include uploads/ directory (default: true)
BACKUP_INCLUDE_LOGS=true          # Include logs/ directory (default: true)
BACKUP_INCLUDE_DATA=true          # Include data/ directory (default: true)

# Storage configuration
BACKUP_STORAGE_PATH=/opt/backups  # Production path (default: /opt/backups)
# BACKUP_STORAGE_PATH=./backups   # Use this for local development
BACKUP_MAX_SIZE_GB=50             # Maximum storage size (default: 50)
STORAGE_WARNING_THRESHOLD_PCT=80  # Warn at 80% usage (default: 80)

# Retention policy
RETENTION_DAYS=30                 # Keep daily backups for 30 days (default: 30)
MONTHLY_KEEP_MONTHS=12            # Keep monthly backups for 12 months (default: 12)

# Offsite uploads (SFTP/SSH)
OFFSITE_ENABLED=false             # Disable until configured (default: false)
OFFSITE_WEEKLY_DAY=sunday         # Day for weekly upload (default: sunday)
OFFSITE_RETRY_MAX_ATTEMPTS=3      # Max retry attempts (default: 3)
OFFSITE_RETRY_DELAY_HOURS=1       # Hours between retries (default: 1)

# SFTP/SSH connection (configure when enabling offsite)
SFTP_HOST=backup.example.com
SFTP_PORT=22
SFTP_USER=bmc_backup
SFTP_PASSWORD=                    # Leave empty if using SSH key
SSH_KEY_PATH=/path/to/id_rsa      # Path to SSH private key (preferred)
SFTP_REMOTE_PATH=/backups/bmc_hub

# Mattermost notifications
MATTERMOST_ENABLED=false          # Disable until webhook configured (default: false)
MATTERMOST_WEBHOOK_URL=https://mattermost.example.com/hooks/xxx
MATTERMOST_CHANNEL=backups
NOTIFY_ON_FAILURE=true            # Send alerts on failures (default: true)
NOTIFY_ON_SUCCESS_OFFSITE=true    # Send alerts on successful offsite uploads (default: true)

Configuration Steps

1. Basic Setup (Local Development)

BACKUP_ENABLED=true
BACKUP_DRY_RUN=false
BACKUP_READ_ONLY=false
BACKUP_STORAGE_PATH=./backups

Restart the application:

docker-compose restart api

2. Enable Offsite Uploads

Using SSH Key (Recommended)

OFFSITE_ENABLED=true
SFTP_HOST=your-backup-server.com
SFTP_USER=backup_user
SSH_KEY_PATH=/path/to/id_rsa
SFTP_REMOTE_PATH=/backups/bmc_hub

Using Password

OFFSITE_ENABLED=true
SFTP_HOST=your-backup-server.com
SFTP_USER=backup_user
SFTP_PASSWORD=your_secure_password
SFTP_REMOTE_PATH=/backups/bmc_hub

3. Enable Mattermost Notifications

1. Create an incoming webhook in Mattermost
2. Copy the webhook URL
3. Add to .env:

MATTERMOST_ENABLED=true
MATTERMOST_WEBHOOK_URL=https://your-mattermost.com/hooks/xxxxxxxxxxxxx
MATTERMOST_CHANNEL=backups

Scheduled Jobs

When BACKUP_ENABLED=true, the system automatically schedules:

• Daily Backup: 02:00 CET - Full backup (database + files) in compressed format
• Monthly Backup: 1st day at 02:00 CET - Full backup in plain SQL format
• Weekly Offsite: Sunday at 03:00 CET - Upload all pending backups to offsite
• Backup Rotation: Daily at 01:00 CET - Delete expired backups
• Storage Check: Daily at 01:30 CET - Check disk usage

Manual Operations

Via UI Dashboard

Visit: http://localhost:8000/backups

Features:

• Create manual backups (database, files, or full)
• View backup history with sizes and checksums
• Restore from backup (with confirmation)
• Manual offsite upload
• View notifications
• Monitor storage usage

Via API

# Create manual backup
curl -X POST http://localhost:8000/api/v1/backups \
  -H "Content-Type: application/json" \
  -d '{"job_type": "full", "is_monthly": false}'

# List backups
curl http://localhost:8000/api/v1/backups/jobs

# Get backup details
curl http://localhost:8000/api/v1/backups/jobs/1

# Restore from backup (⚠️ DANGER - enters maintenance mode)
curl -X POST http://localhost:8000/api/v1/backups/restore/1 \
  -H "Content-Type: application/json" \
  -d '{"confirmation": true}'

# Upload to offsite
curl -X POST http://localhost:8000/api/v1/backups/offsite/1

# Check storage
curl http://localhost:8000/api/v1/backups/storage

# Check maintenance mode
curl http://localhost:8000/api/v1/backups/maintenance

Database Migration

Migration has already been applied. If you need to re-run:

docker-compose exec -T postgres psql -U bmc_hub -d bmc_hub < migrations/024_backup_system.sql

Troubleshooting

Backup not running

• Check BACKUP_ENABLED=true in .env
• Check logs: docker-compose logs api | grep backup
• Verify scheduler status via API: curl http://localhost:8000/api/v1/backups/scheduler/status

Offsite upload failing

• Verify SFTP credentials
• Test SSH connection: ssh -i /path/to/key user@host
• Check retry count in backup history
• Review notifications in dashboard

Storage full

• Increase BACKUP_MAX_SIZE_GB
• Reduce RETENTION_DAYS
• Manually delete old backups via UI
• Enable offsite uploads to move backups off-server

Restore not working

• Set BACKUP_READ_ONLY=false in .env
• Restart API: docker-compose restart api
• Verify backup file exists on disk
• Check maintenance mode overlay appears during restore

Safety Features

The system includes multiple safety switches:

1. BACKUP_ENABLED - Master switch, disabled by default
2. BACKUP_DRY_RUN - Logs operations without executing
3. BACKUP_READ_ONLY - Blocks destructive restore operations
4. OFFSITE_ENABLED - Disables external uploads until configured
5. MATTERMOST_ENABLED - Prevents notification spam

Always test with BACKUP_DRY_RUN=true first!

Production Checklist

• Set strong SFTP credentials
• Configure SSH key authentication
• Test restore procedure (on test system first!)
• Configure Mattermost notifications
• Set appropriate retention periods
• Verify backup storage capacity
• Document restore procedures
• Schedule restore drills (quarterly)
• Monitor backup success rate
• Test offsite download/restore

Backup Storage Structure

/opt/backups/
├── database/
│   ├── db_20251213_020000_daily.dump
│   ├── db_20251201_020000_monthly.sql
│   └── ...
└── files/
    ├── files_20251213_020000.tar.gz
    └── ...

Support

For issues or questions, check:

• Logs: /logs/app.log
• API Docs: http://localhost:8000/api/docs
• Dashboard: http://localhost:8000/backups