Manage Secrets

Retrieve secrets at runtime — environment injection via em-service, file mounts at /mnt/secrets/, and adding new secrets to your chart.

Manage Secrets

This page is about how a solution gets secrets at runtime. The platform handles sourcing, rotation, and delivery — your job is to declare what you need and read it correctly.

For the platform-side architecture (Infisical vs ESO + GCP Secret Manager, rotation mechanics, Workload Identity), see Security › Secrets Management and Deployment › Infrastructure › Secrets.

Secret pipeline

Read the diagram left-to-right: the upstream backend is the source of truth; ESO or Infisical syncs values into a K8s Secret in your namespace; em-service injects them into your pod as env vars (via envFrom) and/or files (via volume mount). On rotation, Reloader detects the K8s Secret change and rolls your Deployment.

Two delivery shapes

The em-service chart delivers secrets to your pod in two ways. Pick whichever fits the consumer:

Shape	Format	Best for
Environment variable	Loaded via envFrom from a synthesized <service>-secrets Kubernetes Secret	Most use cases (DB URLs, API keys, OIDC client secrets)
File mount	Files at /mnt/secrets/<key>	Multi-line secrets (PEM keys, certs); SDKs that expect a file path (gcloud, kubeconfig)

Both are populated from the same upstream secret backend; the difference is purely how your code reads the value.

Steps

1. Declare the secret in your chart

   Reference the secret by secretKeyRef in your values.yaml. The Kubernetes Secret itself (<service>-secrets) is created by the platform’s secret pipeline — you do not check secret values into the chart.

   charts/<solution>/values.yaml

   api:
     env:
       # Plain values: use the simple form
       LOG_LEVEL: "INFO"
       KEYCLOAK_ISSUER_URL: "https://keycloak.example.com/realms/acme"
       KEYCLOAK_AUDIENCE: "<solution>-api"
     envVars:
       # Secret values: use envVars with valueFrom.secretKeyRef
       - name: DATABASE_URL
         valueFrom:
           secretKeyRef:
             name: <solution>-secrets
             key: database-url
       - name: LLM_GATEWAY_API_KEY
         valueFrom:
           secretKeyRef:
             name: <solution>-secrets
             key: llm-gateway-api-key
   

   For file-mount secrets, declare a volume and mount it (the platform’s em-service chart supports custom mounts via volumeMounts and volumes):

   api:
     volumeMounts:
       - name: tls
         mountPath: /mnt/secrets/tls
         readOnly: true
     volumes:
       - name: tls
         secret:
           secretName: <solution>-tls
   
2. Read at runtime

   Environment variables: standard os.environ. File mounts: read from disk lazily.

   import os
   from pathlib import Path
   
   # Env-injected secret
   db_url = os.environ["DATABASE_URL"]
   
   # File-mounted secret
   tls_cert = Path("/mnt/secrets/tls/tls.crt").read_text()
   tls_key  = Path("/mnt/secrets/tls/tls.key").read_text()
   

   Two rules:

   1. Never log a secret value. Wrap reads in repr-safe types or use pydantic.SecretStr.
   2. Don’t re-read on every request. Read once at startup; re-read only when a rotation signal fires (see Stakater Reloader below).
3. Add a new secret

   Adding a secret requires two changes:

   1. Add it to your chart (the envVars snippet above) referencing the key name.
   2. Add the value to the upstream secret backend in each environment.

   The upstream backend depends on where you’re deploying:

   Deployment	Where to add the secret
   Local dev (docker-compose / Kind)	.env (gitignored) or kubectl create secret generic <solution>-secrets --from-literal=key=value
   Self-hosted (Infisical)	Infisical UI for the project; ESO syncs to the cluster
   Cloud (GCP Secret Manager via ESO)	gcloud secrets create + grant Workload Identity SA secretmanager.secretAccessor

   See Security › Secrets Management and Deployment › Infrastructure › Secrets for backend setup.

4. Local development

   For local-dev secrets:

   • Use .env (in .gitignore) for env vars; set -a; source .env; set +a before uv run.
   • Mount fixture files for file-style secrets via docker-compose volumes.
   • Never commit secrets — even fixture/dummy ones if they look real. Use values like dev-only-not-a-real-key.
   • Generate fresh keys for tests (e.g., the dev-private.pem from Authenticate Users).

Where the values come from

Solutions don’t talk to the secret backend directly — the platform’s secret pipeline (ESO for cloud, Infisical for self-hosted) materializes upstream secrets into Kubernetes Secrets in your namespace. Your chart’s secretKeyRef references those K8s Secrets. The platform handles:

• Authentication to the upstream backend (Workload Identity in GCP, machine identities in Infisical)
• Sync intervals (ESO defaults to 1h; configurable per ExternalSecret)
• Resync on upstream change

You do not see the upstream backend from your service code. Just declare the K8s Secret and key.

Rotation

The platform pairs secrets with Stakater Reloader. When the underlying K8s Secret changes (because ESO synced an update from upstream), Reloader rolls your Deployment automatically — if you opt in by annotating the Deployment.

em-service adds the annotation by default for the secrets it injects via envFrom. For custom volume mounts (like <solution>-tls), add the annotation manually:

charts/<solution>/values.yaml

api:
  podAnnotations:
    secret.reloader.stakater.com/reload: "<solution>-tls"

Verify the rollout fires:

# Update the upstream secret, then watch
kubectl -n em-<solution> rollout status deployment/<solution>-api

Verification

# Confirm the K8s Secret exists with the expected keys
kubectl -n em-<solution> get secret <solution>-secrets -o jsonpath='{.data}' | jq 'keys'

# Confirm the env var reaches the pod
kubectl -n em-<solution> exec deployment/<solution>-api -- printenv DATABASE_URL | head -c 60; echo

# Confirm the file mount reaches the pod (if applicable)
kubectl -n em-<solution> exec deployment/<solution>-api -- ls -la /mnt/secrets/tls

If the secret is declared but the pod fails with secret <name> not found: ESO/Infisical hasn’t synced yet, or the upstream secret doesn’t exist. See Troubleshooting › secrets.

Next steps

em-service env model

Reference for env vs envVars precedence and dedup rules.

Use shared storage

Storage credentials follow the same pattern as DB creds.

Secrets management

Platform-side architecture (ESO vs Infisical).

Troubleshooting › secrets

Pod CrashLoopBackOff with “secret not found” and similar.