bmc_hub/docs/BACKUP_SYSTEM.md

# BMC Hub Backup System - Environment Configuration Guide

## Quick Start

Add these lines to your `.env` file to enable the backup system:

```bash
# ===== BACKUP SYSTEM =====

# Safety switches (start with safe defaults)
BACKUP_ENABLED=true                # Enable backup system (default: false)
BACKUP_DRY_RUN=false              # Disable dry-run to actually perform backups (default: true)
BACKUP_READ_ONLY=false            # Allow restore operations (default: true)

# Backup formats
DB_DAILY_FORMAT=dump              # Compressed format for daily backups (default: dump)
DB_MONTHLY_FORMAT=sql             # Plain SQL for monthly backups (default: sql)

# Backup scope
BACKUP_INCLUDE_UPLOADS=true       # Include uploads/ directory (default: true)
BACKUP_INCLUDE_LOGS=true          # Include logs/ directory (default: true)
BACKUP_INCLUDE_DATA=true          # Include data/ directory (default: true)

# Storage configuration
BACKUP_STORAGE_PATH=/opt/backups  # Production path (default: /opt/backups)
# BACKUP_STORAGE_PATH=./backups   # Use this for local development
BACKUP_MAX_SIZE_GB=50             # Maximum storage size (default: 50)
STORAGE_WARNING_THRESHOLD_PCT=80  # Warn at 80% usage (default: 80)

# Retention policy
RETENTION_DAYS=30                 # Keep daily backups for 30 days (default: 30)
MONTHLY_KEEP_MONTHS=12            # Keep monthly backups for 12 months (default: 12)

# Offsite uploads (SFTP/SSH)
OFFSITE_ENABLED=false             # Disable until configured (default: false)
OFFSITE_WEEKLY_DAY=sunday         # Day for weekly upload (default: sunday)
OFFSITE_RETRY_MAX_ATTEMPTS=3      # Max retry attempts (default: 3)
OFFSITE_RETRY_DELAY_HOURS=1       # Hours between retries (default: 1)

# SFTP/SSH connection (configure when enabling offsite)
SFTP_HOST=backup.example.com
SFTP_PORT=22
SFTP_USER=bmc_backup
SFTP_PASSWORD=                    # Leave empty if using SSH key
SSH_KEY_PATH=/path/to/id_rsa      # Path to SSH private key (preferred)
SFTP_REMOTE_PATH=/backups/bmc_hub

# Mattermost notifications
MATTERMOST_ENABLED=false          # Disable until webhook configured (default: false)
MATTERMOST_WEBHOOK_URL=https://mattermost.example.com/hooks/xxx
MATTERMOST_CHANNEL=backups
NOTIFY_ON_FAILURE=true            # Send alerts on failures (default: true)
NOTIFY_ON_SUCCESS_OFFSITE=true    # Send alerts on successful offsite uploads (default: true)
```

## Configuration Steps

### 1. Basic Setup (Local Development)

```bash
BACKUP_ENABLED=true
BACKUP_DRY_RUN=false
BACKUP_READ_ONLY=false
BACKUP_STORAGE_PATH=./backups
```

Restart the application:
```bash
docker-compose restart api
```

### 2. Enable Offsite Uploads

#### Using SSH Key (Recommended)
```bash
OFFSITE_ENABLED=true
SFTP_HOST=your-backup-server.com
SFTP_USER=backup_user
SSH_KEY_PATH=/path/to/id_rsa
SFTP_REMOTE_PATH=/backups/bmc_hub
```

#### Using Password
```bash
OFFSITE_ENABLED=true
SFTP_HOST=your-backup-server.com
SFTP_USER=backup_user
SFTP_PASSWORD=your_secure_password
SFTP_REMOTE_PATH=/backups/bmc_hub
```

### 3. Enable Mattermost Notifications

1. Create an incoming webhook in Mattermost
2. Copy the webhook URL
3. Add to `.env`:

```bash
MATTERMOST_ENABLED=true
MATTERMOST_WEBHOOK_URL=https://your-mattermost.com/hooks/xxxxxxxxxxxxx
MATTERMOST_CHANNEL=backups
```

## Scheduled Jobs

When `BACKUP_ENABLED=true`, the system automatically schedules:

- **Daily Backup**: 02:00 CET - Full backup (database + files) in compressed format
- **Monthly Backup**: 1st day at 02:00 CET - Full backup in plain SQL format
- **Weekly Offsite**: Sunday at 03:00 CET - Upload all pending backups to offsite
- **Backup Rotation**: Daily at 01:00 CET - Delete expired backups
- **Storage Check**: Daily at 01:30 CET - Check disk usage

## Manual Operations

### Via UI Dashboard
Visit: `http://localhost:8000/backups`

Features:
- Create manual backups (database, files, or full)
- View backup history with sizes and checksums
- Restore from backup (with confirmation)
- Manual offsite upload
- View notifications
- Monitor storage usage

### Via API

```bash
# Create manual backup
curl -X POST http://localhost:8000/api/v1/backups \
  -H "Content-Type: application/json" \
  -d '{"job_type": "full", "is_monthly": false}'

# List backups
curl http://localhost:8000/api/v1/backups/jobs

# Get backup details
curl http://localhost:8000/api/v1/backups/jobs/1

# Restore from backup (⚠️ DANGER - enters maintenance mode)
curl -X POST http://localhost:8000/api/v1/backups/restore/1 \
  -H "Content-Type: application/json" \
  -d '{"confirmation": true}'

# Upload to offsite
curl -X POST http://localhost:8000/api/v1/backups/offsite/1

# Check storage
curl http://localhost:8000/api/v1/backups/storage

# Check maintenance mode
curl http://localhost:8000/api/v1/backups/maintenance
```

## Database Migration

Migration has already been applied. If you need to re-run:

```bash
docker-compose exec -T postgres psql -U bmc_hub -d bmc_hub < migrations/024_backup_system.sql
```

## Troubleshooting

### Backup not running
- Check `BACKUP_ENABLED=true` in `.env`
- Check logs: `docker-compose logs api | grep backup`
- Verify scheduler status via API: `curl http://localhost:8000/api/v1/backups/scheduler/status`

### Offsite upload failing
- Verify SFTP credentials
- Test SSH connection: `ssh -i /path/to/key user@host`
- Check retry count in backup history
- Review notifications in dashboard

### Storage full
- Increase `BACKUP_MAX_SIZE_GB`
- Reduce `RETENTION_DAYS`
- Manually delete old backups via UI
- Enable offsite uploads to move backups off-server

### Restore not working
- Set `BACKUP_READ_ONLY=false` in `.env`
- Restart API: `docker-compose restart api`
- Verify backup file exists on disk
- Check maintenance mode overlay appears during restore

## Safety Features

The system includes multiple safety switches:

1. **BACKUP_ENABLED** - Master switch, disabled by default
2. **BACKUP_DRY_RUN** - Logs operations without executing
3. **BACKUP_READ_ONLY** - Blocks destructive restore operations
4. **OFFSITE_ENABLED** - Disables external uploads until configured
5. **MATTERMOST_ENABLED** - Prevents notification spam

Always test with `BACKUP_DRY_RUN=true` first!

## Production Checklist

- [ ] Set strong SFTP credentials
- [ ] Configure SSH key authentication
- [ ] Test restore procedure (on test system first!)
- [ ] Configure Mattermost notifications
- [ ] Set appropriate retention periods
- [ ] Verify backup storage capacity
- [ ] Document restore procedures
- [ ] Schedule restore drills (quarterly)
- [ ] Monitor backup success rate
- [ ] Test offsite download/restore

## Backup Storage Structure

```
/opt/backups/
├── database/
│   ├── db_20251213_020000_daily.dump
│   ├── db_20251201_020000_monthly.sql
│   └── ...
└── files/
    ├── files_20251213_020000.tar.gz
    └── ...
```

## Support

For issues or questions, check:
- Logs: `/logs/app.log`
- API Docs: `http://localhost:8000/api/docs`
- Dashboard: `http://localhost:8000/backups`