Automating secret rotation in Kubernetes

Minimizing mistakes by removing the human element

Márk Sági-Kazár

2023-02-04 @ FOSDEM ’23


Márk Sági-Kazár

Engineering Technical Lead @ Cisco

Help engineering teams run their business on Kubernetes


Once upon a time..

Why is secret rotation important?

  • Maintain security of sensitive information
  • Meet compliance requirements
  • Reduce the risk of a data breach

Challenges of secret rotation

  • Complexity
  • Time-consuming and error prone process
  • Disruption of service availability

Secret rotation should be…

  • possible
  • automated
  • periodic

Secret rotation flow

    actor Operator
    participant Provider as Secret provider
    participant Store as Secret store
    participant Deploy as ???
    participant Production

    Deploy->>Store: Watch for changes
    activate Deploy
    Operator->>Provider: Generate new secret
    Provider-->>Operator: Return new secret
    Operator->>Store: Rotate secret in store
    Store-->>Deploy: Notice secret change
    deactivate Deploy
    Deploy->>Production: Deploy new secret

Secret rotation in Kubernetes

Deploying secrets to Kubernetes

  • External Secrets:
  • Synchronize secrets from an external store to Kubernetes secrets
  • Mount secrets as usual (env var, file)


Turn on envelope encryption!!!

Triggering workload rollout

    participant Store as Secret store
    participant ExternalSecrets as External secrets
    participant Kubernetes
    participant Reloader

    ExternalSecrets->>Store: Watch for changes
    Reloader->>Kubernetes: Watch for changes
    Store-->>ExternalSecrets: Notice secret change
    ExternalSecrets->>Kubernetes: Deploy new secret
    Kubernetes-->>Reloader: Notice secret change
    Reloader->>Kubernetes: Trigger workload rollout


Thank you

Any questions?