Secrets Infrastructure

Deploy and configure ESO + GCP Secret Manager or Infisical for secrets management in CRAFT.

Secrets Infrastructure

This guide covers deploying and configuring the secrets management backend for CRAFT. Two options are supported, choose based on your deployment environment.

For a comparison of the two backends and guidance on which to choose, see Secrets Management.

---

Option A: ESO + GCP Secret Manager

For GKE and GCP-based deployments. Infrastructure is provisioned by Terraform; Kubernetes resources are managed by ArgoCD.

Prerequisites

• GKE cluster with Workload Identity enabled
• Terraform-provisioned GCP infrastructure
• ArgoCD deployed and managing the cluster

1. Terraform Provisions GCP Secrets

The gke Terraform module automatically creates the required secrets in GCP Secret Manager when you run terraform apply. No manual secret creation is needed for infrastructure secrets, the module provisions them with placeholder values that you update after provisioning.

After running Terraform:

1. Navigate to GCP Secret Manager in your project
2. Update each provisioned secret with its actual value
3. ESO will sync the updated values into Kubernetes on the next sync cycle

2. Install External Secrets Operator

ESO can be installed as an ArgoCD-managed application or deployed manually:

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set installCRDs=true

3. ClusterSecretStore

ESO requires a ClusterSecretStore that configures how to authenticate to GCP Secret Manager. This is provisioned by Terraform/ArgoCD:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: "your-project-id"   # Replace with your GCP project
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: "your-cluster-name"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

4. ExternalSecret Resources

Each service has ExternalSecret resources that map GCP SM secrets to K8s Secrets. These are managed by ArgoCD in the kubernetes/em-runtime/ directory:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: em-runtime-credentials
  namespace: em-runtime
  annotations:
    reloader.stakater.com/auto: "true"   # Enable auto-restart on change
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-manager
    kind: ClusterSecretStore
  target:
    name: em-runtime-credentials
    creationPolicy: Owner
  data:
    - secretKey: postgres-password
      remoteRef:
        key: em-runtime   # GCP SM secret name
        property: postgres-password
    - secretKey: redis-password
      remoteRef:
        key: em-runtime
        property: redis-password

5. Install Stakater Reloader

Reloader watches K8s Secrets for changes and triggers rolling restarts of annotated pods:

helm install reloader stakater/reloader \
  --namespace em-runtime \
  --set reloader.watchGlobally=false

Pods are automatically enrolled when their Deployment or the Secret has the annotation:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

6. Verify Sync

# Check ExternalSecret status
kubectl get externalsecrets -n em-runtime

# Check the synced K8s Secret exists
kubectl get secret em-runtime-credentials -n em-runtime

# Check Reloader is watching
kubectl logs -n em-runtime -l app.kubernetes.io/name=reloader | head -20

Manual Sync (Immediate Rotation)

To force an immediate sync without waiting for the refresh interval:

kubectl annotate externalsecret em-runtime-credentials \
  --namespace em-runtime \
  force-sync="$(date +%s)" \
  --overwrite

---

Option B: Infisical

For on-premises and cloud-agnostic deployments. Infisical runs as a Helm subchart within em-runtime.

Prerequisites

• Kubernetes cluster (any CNCF-conformant distribution)
• Helm chart deployed with Infisical subchart enabled

1. Enable Infisical in Helm Values

Infisical is enabled by default. Verify in your values file:

infisical:
  enabled: true
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"

2. Bootstrap Infisical

On first deployment, Infisical initializes with the admin email configured in Helm:

bootstrap:
  adminEmail: "[e-mail]"

Access the Infisical UI and complete initial setup:

kubectl port-forward svc/em-runtime-infisical -n em-runtime 8080:80
# Open http://localhost:8080

3. Create Machine Identity

Create a Kubernetes machine identity in Infisical for each service that needs secret access:

1. In the Infisical UI, navigate to Project → Access Control → Machine Identities
2. Create a machine identity with the Kubernetes authentication method
3. Configure the service account name and namespace

4. Inject Credentials

Infisical credentials are passed to services via environment variables in Helm values:

governance:
  extraEnvVars:
    - name: INFISICAL_URL
      value: "http://em-runtime-infisical.em-runtime.svc.cluster.local"
    - name: INFISICAL_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: infisical-machine-identity
          key: clientId
    - name: INFISICAL_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: infisical-machine-identity
          key: clientSecret

5. Verify

# Check Infisical pod is running
kubectl get pods -n em-runtime -l app.kubernetes.io/name=infisical

# Check governance service logs for Infisical connection
kubectl logs -n em-runtime -l app=em-runtime-governance | grep -i infisical

---

Backup and Recovery

ESO + GCP Secret Manager

GCP Secret Manager has built-in version history. To recover a previous secret value:

# List secret versions
gcloud secrets versions list your-secret-name --project=your-project-id

# Access a specific version
gcloud secrets versions access VERSION_ID --secret=your-secret-name --project=your-project-id

Infisical

Back up the Infisical PostgreSQL database (infisical database) using standard PostgreSQL backup tools:

kubectl exec -n em-runtime deployment/em-runtime-infisical -- \
  pg_dump -U infisical infisical > infisical-backup-$(date +%Y%m%d).sql

Include this backup in your regular database backup rotation. See Backup and Restore for the full backup procedure.

Next Steps

Secrets Management Overview

Concepts, architecture, and backend comparison.

Backup and Restore

Full backup procedures including secrets backends.