Automating secret rotation in Kubernetes

Minimizing mistakes by removing the human element

Márk Sági-Kazár


2023-02-04 @ FOSDEM ’23

whoami

Márk Sági-Kazár

Engineering Technical Lead @ Cisco


Help engineering teams run their business on Kubernetes

@sagikazarmark

https://sagikazarmark.hu

Once upon a time..

Why is secret rotation important?

  • Maintain security of sensitive information
  • Meet compliance requirements
  • Reduce the risk of a data breach

Challenges of secret rotation

  • Complexity
  • Time-consuming and error prone process
  • Disruption of service availability

Secret rotation should be…

  • possible
  • automated
  • periodic

Secret rotation flow

sequenceDiagram
    actor Operator
    participant Provider as Secret provider
    participant Store as Secret store
    participant Deploy as ???
    participant Production

    Deploy->>Store: Watch for changes
    activate Deploy
    Operator->>Provider: Generate new secret
    Provider-->>Operator: Return new secret
    Operator->>Store: Rotate secret in store
    Store-->>Deploy: Notice secret change
    deactivate Deploy
    Deploy->>Production: Deploy new secret

Secret rotation in Kubernetes

Deploying secrets to Kubernetes

  • External Secrets: https://external-secrets.io
  • Synchronize secrets from an external store to Kubernetes secrets
  • Mount secrets as usual (env var, file)

Important

Turn on envelope encryption!!!

Triggering workload rollout

sequenceDiagram
    participant Store as Secret store
    participant ExternalSecrets as External secrets
    participant Kubernetes
    participant Reloader

    ExternalSecrets->>Store: Watch for changes
    Reloader->>Kubernetes: Watch for changes
    Store-->>ExternalSecrets: Notice secret change
    ExternalSecrets->>Kubernetes: Deploy new secret
    Kubernetes-->>Reloader: Notice secret change
    Reloader->>Kubernetes: Trigger workload rollout

Demo

https://github.com/sagikazarmark/demo-fosdem23-kube-secret-rotation

Thank you

Any questions?



@sagikazarmark

https://sagikazarmark.hu