Upgrades

Version upgrade procedures, Helm chart promotion workflow, migration steps, and rollback strategies for the em-runtime platform.

Upgrades

This guide covers the em-runtime Helm chart upgrade process, version promotion through environments, migration procedures, and rollback strategies.

Upgrade Procedures

Standard Helm Upgrade

For direct Helm deployments (not using ArgoCD):

# Check current version
helm list -n $NAMESPACE

# Review what will change
helm diff upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

# Perform the upgrade
helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version $NEW_VERSION \
  --namespace $NAMESPACE \
  -f values.yaml

Pre-Upgrade Checklist

Always back up PostgreSQL, Redis, and the em-runtime-secrets Secret before upgrading.

Before upgrading:

1. Review the changelog for breaking changes and required migrations
2. Back up databases:
   # PostgreSQL
   pg_dumpall -h $PG_HOST -U postgres > backup-pre-upgrade.sql
   
   # Kubernetes Secrets
   kubectl get secret em-runtime-secrets -n $NAMESPACE -o yaml > em-runtime-secrets-backup.yaml
   kubectl get secret infisical-bootstrap-secret -n $NAMESPACE -o yaml > infisical-bootstrap-backup.yaml
   
3. Verify current health:
   kubectl get pods -n $NAMESPACE
   curl https://$HOSTNAME/api/governance/health
   
4. Test in non-production first — promote through dev and staging before production

Database Migrations

EM-Runtime services run Alembic database migrations automatically on startup. Each service manages its own database migrations independently.

Migration Behavior

Aspect	Behavior
Trigger	Automatic on pod startup
Direction	Forward only (upgrade)
Scope	Per-service, per-database
Safety	Idempotent — safe to run multiple times

Migration Order

Migrations are applied in service startup order:

PostgreSQL ready -> Keycloak (schema) -> OpenFGA (schema) -> Infisical (schema)
                 -> Governance (alembic) -> Assets (alembic) -> Utils (alembic)

If a migration fails, the pod will restart and retry. Check pod logs for migration errors before investigating further.

Rollback

Helm Rollback

# List revision history
helm history em-runtime -n $NAMESPACE

# Rollback to previous revision
helm rollback em-runtime $PREVIOUS_REVISION -n $NAMESPACE

Database Rollback Considerations

Alembic migrations are forward-only in production. If a database migration must be reversed, you must restore from backup.

• Schema-only changes (add column, add table): Generally safe to keep during rollback; old code ignores new columns
• Data migrations: Require database restore from pre-upgrade backup
• Destructive migrations (drop column, rename): Require database restore

Direct Helm

Always specify --version in production:

helm upgrade em-runtime oci://ghcr.io/emergenceai/charts/em-runtime \
  --version 4.8.6 \
  --namespace production \
  -f production-values.yaml

Upgrade Troubleshooting

Pods stuck in CrashLoopBackOff after upgrade

Check pod logs for migration errors:

kubectl logs -n $NAMESPACE deploy/em-runtime-governance --previous

Common causes:

• Database migration conflict (two migrations with same parent)
• Missing database or extension
• Incompatible secret format

Init containers timing out

Init containers wait for dependencies (PostgreSQL, Redis, Keycloak). Check that all infrastructure services are healthy:

kubectl get pods -n $NAMESPACE -l app.kubernetes.io/component=database
kubectl logs -n $NAMESPACE -l app=keycloak

HTTPS/CORS errors after upgrade

Verify that hostname and scheme values match your Gateway and DNS configuration:

kubectl get secret em-runtime-secrets -n $NAMESPACE \
  -o jsonpath='{.data.base-url}' | base64 -d

ArgoCD shows OutOfSync after promotion

The CMP plugin may need to rebuild the chart dependency. Force a sync:

kubectl get application em-runtime -n argocd -o jsonpath='{.status.sync.status}'
# If OutOfSync, trigger a hard refresh
argocd app get em-runtime --hard-refresh
argocd app sync em-runtime

Next Steps

Values Reference

Complete reference for all Helm chart values.