Kubernetes secret management

Márk Sági-Kazár

2023-06-30 @ Let’s Code meetup


Márk Sági-Kazár

Open Source Tech Lead @ Cisco

CNCF Ambassador




What is secret management?

Sensitive information, necessary for an application to function, is made available to it.

Why is it important?

  • Maintain security of sensitive information
  • Reduce the risk of security (and other) incidents
  • Meet compliance requirements

Common requirements

  • Encryption: Secrets should be encrypted both at rest and in transit.
  • Rotation: The ability to rotate secrets regularly.
  • Access control: Only authorized users/applications can access the secrets.
  • Auditability: Track who is accessing which secrets and when.
  • Compliance: Adhere to regulatory requirements related to data security.


  • No one-size-fits-all solution
  • Secret management infra can easily become a SPOF
  • Solutions often become complex, increasing the chance of introducing a vulnerability

Kubernetes secrets

  • Standard Kubernetes resource
  • Values are base64 encoded

⚠️ Plug the holes! ⚠️

  • Turn on encryption at rest
  • Configure least-privilege access to Secrets

Official guide: Good practices for Kubernetes Secrets

Deploying secrets to Kubernetes

External Secrets Operator


Triggering workload rollout

    participant Store as Secret store
    participant ExternalSecrets as External secrets
    participant Kubernetes
    participant Reloader

    ExternalSecrets->>Store: Watch for changes
    Reloader->>Kubernetes: Watch for changes
    Store-->>ExternalSecrets: Notice secret change
    ExternalSecrets->>Kubernetes: Deploy new secret
    Kubernetes-->>Reloader: Notice secret change
    Reloader->>Kubernetes: Trigger workload rollout

What could possibly go wrong?

Who knows, so monitor everything


Potential high cardinality labels (drop metrics/labels you don’t need)

Changes take effect with a delay

  1. Change some configuration ✏️
  2. Wait until the next secret sync period 🤞
  3. Hope nothing breaks 🙏

Solution: create (and modify) test secrets at the same time.

To sum up ESO

  • Understand how (and when) changes will take effect
  • Monitor and alert for failures

Kubernetes without secrets 😱

Access secret store directly

  • Integrated into the application


  • “Inject” secrets into the application

Secret injection in Kubernetes

  • Inject a custom init into Pods using a mutating admission webhook
  • Get secrets from secret store in the custom init
  • Inject secrets as environment variables


  • Started at Banzai Cloud
  • Vault Swiss Army knife


Bank-Vaults secret injection

  • Secret references: vault:path/to/secret#KEY
  • Mutating webhook
    • Detect secret references
    • Mutate Pods
  • Custom init replaces secret references with actual values


Secret changes do not take effect (ie. trigger workload reload) at the moment.

Risks and mitigations

Risk: Secret store is a SPOF

Mitigation: Maintain a cluster-local instance

Risk: Webhook is a SPOF

Mitigation: Configure webhook according to best practices





Final thoughts

It seems wisest to assume the worst from the beginning…and let anything better come as a surprise.

Jules Verne

Thank you

Any questions?