Backup & Restore

How to back up and restore CRAFT data including PostgreSQL databases, Keycloak realms, OpenFGA state, and Infisical secrets.

Backup and Restore Platform Data

This guide covers backup and restore procedures for all stateful components of CRAFT. Each service owns its own database, and infrastructure services (Keycloak, OpenFGA, Infisical) maintain independent state that must be backed up separately.

Stateful Components

The platform has the following stateful components that require backup:

Component	Storage	Data
Governance DB	PostgreSQL	Organizations, projects, users, role assignments
Assets DB	PostgreSQL	Artifacts, data connections, files, models
Utils DB	PostgreSQL	Data catalog, scheduling, context packs, memories
Keycloak	PostgreSQL	Realms, users, IdP configurations, sessions
OpenFGA	PostgreSQL	Authorization model, relationship tuples
Infisical	Internal store	Application secrets, data connection credentials
Redis	In-memory + AOF	Cache, session state, event streams
Solution DBs	PostgreSQL	Data Insights sessions, Data Governance profiles

Each service has a separate PostgreSQL database. There are no cross-service foreign keys, so databases can be backed up independently without coordination.

PostgreSQL Backup

Automated Backups (Cloud-Managed)

For cloud-managed PostgreSQL (Cloud SQL, RDS, Azure Database):

GCP Cloud SQL

Cloud SQL provides automated daily backups with configurable retention.

# Enable automated backups
gcloud sql instances patch <instance-name> \
  --backup-start-time=02:00 \
  --retained-backups-count=30 \
  --retained-transaction-log-days=7

# Create an on-demand backup
gcloud sql backups create --instance=<instance-name>

# List available backups
gcloud sql backups list --instance=<instance-name>

AWS RDS

# Enable automated backups with 30-day retention
aws rds modify-db-instance \
  --db-instance-identifier <instance-id> \
  --backup-retention-period 30

# Create a manual snapshot
aws rds create-db-snapshot \
  --db-instance-identifier <instance-id> \
  --db-snapshot-identifier emergence-backup-$(date +%Y%m%d)

Azure

# Azure Database for PostgreSQL has automated backups enabled by default
# Configure retention (7-35 days)
az postgres flexible-server update \
  --resource-group <rg> \
  --name <server-name> \
  --backup-retention 30

Manual Backups (On-Premises)

For self-managed PostgreSQL deployments:

# Back up all platform databases
DATABASES="keycloak openfga infisical governance assets utils prefect datareadiness talk2data"
BACKUP_DIR="/backups/emergence/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

for db in $DATABASES; do
  pg_dump -h localhost -U postgres -Fc -f "$BACKUP_DIR/$db.dump" "$db"
done

# Verify backups
for dump in "$BACKUP_DIR"/*.dump; do
  pg_restore --list "$dump" > /dev/null && echo "OK: $dump" || echo "FAILED: $dump"
done

Use pg_dump -Fc (custom format) for backups. It supports parallel restore and selective table restoration, and compresses data automatically.

Point-in-Time Recovery

For production deployments, enable WAL archiving for point-in-time recovery:

# postgresql.conf
archive_mode = on
archive_command = 'cp %p /archive/wal/%f'
wal_level = replica

Keycloak Backup

Keycloak state is primarily stored in its PostgreSQL database, but realm configuration can also be exported as JSON for version control.

# Export realm configuration (includes users, IdPs, clients)
# Run from within the Keycloak container or pod
/opt/keycloak/bin/kc.sh export \
  --dir /tmp/keycloak-export \
  --realm <realm-name> \
  --users realm_file

# Copy the export out of the container
kubectl cp <keycloak-pod>:/tmp/keycloak-export ./keycloak-backup/

Realm exports include user accounts but not user credentials (passwords). Users will need to reset passwords after a realm import. SSO users are unaffected since their credentials are managed by the external IdP.

OpenFGA Backup

OpenFGA stores its authorization model and relationship tuples in PostgreSQL. The database backup covers all OpenFGA state. For additional safety, export the authorization model:

# Export the current authorization model
curl -s "http://<openfga-host>:8080/stores/<store-id>/authorization-models" \
  -H "Authorization: Bearer $TOKEN" | jq . > openfga-model-backup.json

Secrets Backend Backup

The platform supports two secrets backends. Back up whichever you are using.

Infisical

Infisical manages application secrets including data connection credentials. Back up the Infisical database (PostgreSQL infisical database) and preserve the encryption keys.

Infisical secrets are encrypted at rest. Backing up the database preserves the encrypted data, but you must also preserve the ENCRYPTION_KEY and AUTH_SECRET for restoration, without them, encrypted data cannot be decrypted.

pg_dump -h localhost -U postgres -Fc -f infisical-backup.dump infisical
# Also export the encryption key from the infisical-bootstrap-secret:
kubectl get secret infisical-bootstrap-secret -n em-runtime -o yaml > infisical-bootstrap-secret.yaml

ESO + GCP Secret Manager

Secrets stored in GCP Secret Manager have built-in version history. No separate backup action is required, GCP retains all secret versions.

To export a snapshot of all current secret values:

# List all em-runtime secrets
gcloud secrets list --project=your-project-id --filter="name:em-runtime"

# Verify each secret has at least one active version
gcloud secrets versions list em-runtime --project=your-project-id

For disaster recovery, ensure IAM bindings for the ESO service account are documented and reproducible via Terraform.

Redis Backup

Redis serves as a cache and event stream. While cache data is ephemeral, you may want to back up Redis for faster recovery:

# Trigger a manual RDB snapshot
redis-cli -a $REDIS_PASSWORD BGSAVE

# Copy the dump file
kubectl cp <redis-pod>:/data/dump.rdb ./redis-backup/dump.rdb

Restore Procedures

Restore PostgreSQL Databases

# Restore a single database from a custom-format dump
pg_restore -h localhost -U postgres -d governance -Fc --clean --if-exists governance.dump

# Restore all platform databases
for db in keycloak openfga infisical governance assets utils prefect datareadiness talk2data; do
  pg_restore -h localhost -U postgres -d "$db" -Fc --clean --if-exists "$db.dump"
done

Restore Keycloak Realms

# Import realm configuration
/opt/keycloak/bin/kc.sh import \
  --dir /tmp/keycloak-import \
  --override true

Restore from Cloud Managed Backups

GCP Cloud SQL

# Restore from an automated backup
gcloud sql backups restore <backup-id> \
  --restore-instance=<instance-name>

AWS RDS

# Restore from a snapshot (creates a new instance)
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier <new-instance-id> \
  --db-snapshot-identifier <snapshot-id>

Backup Schedule Recommendations

Component	Frequency	Retention	Method
PostgreSQL (all DBs)	Daily + continuous WAL	30 days	Cloud-managed or pg_dump
Keycloak realm exports	Weekly	90 days	JSON export
OpenFGA model	On change	Indefinite	JSON export in version control
Redis	Daily	7 days	RDB snapshot
Secrets backend (Infisical)	Daily	30 days	Database backup with encryption keys
Secrets backend (GCP SM)	N/A, managed	Indefinite	GCP retains all versions; document IAM bindings in Terraform

Disaster Recovery Checklist

1. Restore PostgreSQL databases

   Restore all platform databases from the latest backup. Verify row counts and data integrity.

2. Restore Keycloak

   Import realm configurations. SSO users will re-authenticate via their IdP. Local users may need password resets.

3. Verify OpenFGA schema

   Confirm the authorization model is loaded. The Governance service re-applies the schema on startup.

4. Restore secrets backend

   Infisical: Restore the Infisical database and encryption keys. Verify data connection credentials are accessible. ESO + GCP Secret Manager: Verify GCP Secret Manager secrets are intact and ESO ClusterSecretStore can authenticate via Workload Identity.

5. Restart platform services

   Follow the startup order: PostgreSQL -> Redis -> Keycloak -> OpenFGA -> Governance -> Assets/Utils -> Solutions.

6. Validate end-to-end

   Run health checks on all services. Test authentication, permission checks, and data connection queries.

Next Steps

Deployment Overview

Review the full deployment architecture and infrastructure requirements.

Helm Configuration

Configure Helm values for backup-related settings.

GDPR Compliance

Understand data retention requirements for GDPR compliance.

Network Security

Secure backup data in transit and at rest.