Kubernetes Secrets Management: Protecting Sensitive Data

Kubernetes secrets management is one of those topics that looks solved on day one and quietly becomes a liability by day one hundred. You create a Secret, reference it from a Deployment, and the application starts. It works, so it feels secure. The uncomfortable truth is that a stock Kubernetes Secret protects almost nothing on its own, and the gap between "it works" and "it is protected" is where real incidents happen. This deep dive is part of our broader Kubernetes security guide. Here we focus on one thing and do it thoroughly: how to store, distribute, and protect sensitive data such as database passwords, API tokens, TLS keys, and cloud credentials across a cluster's lifetime. What Kubernetes Secrets are, and the big caveat A Kubernetes Secret is an API object designed to hold small amounts of sensitive data so that it does not have to be baked into a Pod spec or a container image. Compared with putting a password directly in a Deployment manifest, that is genuinely a step forward: the value lives in its own object, can be mounted or injected on demand, and can be governed by access controls. Here is the caveat that surprises many teams: by default, Secret data is only base64-encoded, not encrypted. Base64 is an encoding, not a cipher. It has no key and provides no confidentiality. Anyone who can read the Secret object can decode it in a single command: kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d Worse, Secret values are stored in etcd, the cluster's backing datastore. If etcd is unencrypted, then a backup file, a snapshot, a stolen disk, or direct etcd access exposes every secret in the cluster in a form that is trivial to read. So the mental model to adopt is simple: a Secret is a convenient container for sensitive data, not a guarantee that the data is secure. Everything below is about closing that gap. Encryption at rest: EncryptionConfiguration and a KMS provider The first control to add is etcd encryption at rest. Kubernetes supports encrypting Secret resources before they are written to etcd, configured through an EncryptionConfiguration file that the API server loads via --encryption-provider-config. You can encrypt with a locally managed key, but the strongest and most operationally sound option is a KMS provider. With KMS, the data-encryption keys are themselves wrapped by a key held in an external key management service, so the raw key never sits in a config file on the control plane, and you can rotate and audit it independently. A KMS-backed configuration looks roughly like this: apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources:   - resources:       - secrets     providers:       - kms:           apiVersion: v2           name: my-kms-provider           endpoint: unix:///var/run/kmsplugin/socket.sock       - identity: {} Two practical notes. First, order matters: whichever provider is listed first is used to encrypt new writes, while all listed providers can decrypt, which is exactly how you migrate from identity (plaintext) to encryption and later rotate keys. Second, enabling encryption only affects future writes. To encrypt secrets that already exist, force a rewrite after the change: kubectl get secrets --all-namespaces -o json | kubectl replace -f - Encryption at rest is the control that turns "anyone with an etcd snapshot owns your credentials" into "an attacker also needs the KMS key." That is a meaningful raise in cost for the attacker. Who can read secrets: RBAC and the danger of broad get/list Encryption protects data at rest, but inside a running cluster the more common exposure path is authorization. If a user, service account, or workload can call the API and read a Secret, encryption at rest does not stop them because the API server decrypts on the way out. This is why RBAC on secrets deserves special scrutiny. A role that grants get, list, or watch on the secrets resource is, in effect, a role that can read credentials. Granted broadly, at cluster scope or with a wildcard, it becomes a path to every secret in the cluster. Treat these verbs as high-privilege and scope them tightly: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata:   namespace: payments   name: read-payments-db-secret rules:   - apiGroups: [""]     resources: ["secrets"]     verbs: ["get"]     resourceNames: ["payments-db"] Note the resourceNames field, which narrows the grant to a single named Secret rather than all secrets in the namespace. Prefer namespaced Roles over ClusterRoles for secret access, avoid list and watch unless a workload truly needs to enumerate secrets, and remember that a Pod's service account inherits whatever that account can do. For a full treatment of least-privilege role design, see our guide on Kubernetes RBAC. Avoiding leakage: images, env vars, and logs Some of the worst secret exposures never touch etcd at all. They leak because the secret was copied somewhere careless. Do not bake secrets into container images. A credential added during a build stays in the image layers and travels to every registry and node that pulls it. Image layers are not a safe place for anything sensitive. Be cautious with environment variables. Injecting a Secret as an env var is convenient, but environment variables have a habit of ending up in crash dumps, error trackers, kubectl describe output for the Pod spec, and child processes. Where you can, prefer mounting secrets as files through a volume. Mounted files are read on demand, can be updated in place, and are less likely to be captured incidentally. Beware logs. Application logs, debug traces, and startup banners that echo configuration are a classic leak channel. A secret that is safely encrypted in etcd is worthless as a protection if the app prints it to stdout on boot. Mounting a secret as a file is straightforward: volumes:   - name: db-creds     secret:       secretName: payments-db containers:   - name: api     volumeMounts:       - name: db-creds         mountPath: /etc/secrets         readOnly: true External secret stores and the Secrets Store CSI Driver For many teams the right long-term answer is to stop treating the cluster as the source of truth for secrets and instead keep them in a dedicated external secrets manager, backed by a KMS, with its own access policies, versioning, and audit trail. The cluster then pulls secrets at runtime rather than storing them permanently. The common, vendor-neutral pattern for this is the Secrets Store CSI Driver. It mounts secrets from an external store directly into a Pod as a volume, so the sensitive value is delivered to the workload that needs it without necessarily persisting as a native Kubernetes Secret. A SecretProviderClass object describes which external secrets to fetch, and the Pod references it as a CSI volume: apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata:   name: payments-db spec:   provider:   parameters:     objects: |       - objectName: "payments-db-password"         objectType: "secret" volumes:   - name: secrets-store     csi:       driver: secrets-store.csi.x-k8s.io       readOnly: true       volumeAttributes:         secretProviderClass: payments-db The benefits are centralized policy, one place to rotate and revoke, and an audit trail that lives outside the cluster. Another related pattern uses an operator that syncs from an external store into native Secrets, which is easier to adopt but reintroduces the in-cluster copy, so pair it with encryption at rest and tight RBAC. Rotation and short-lived credentials Static secrets that never change are a standing risk: the longer a credential lives, the more places it leaks to and the larger the blast radius when it does. Good secrets management assumes rotation from the start. Two complementary habits help here. First, rotate regularly and make rotation a routine, automated operation rather than a manual scramble after an incident. External secret stores make this far easier because rotation happens in one place and workloads pick up the new value on their next mount or refresh. Second, prefer short-lived, dynamically issued credentials wherever the platform allows it, such as database credentials that are generated on demand and expire in hours, or cloud access granted through workload identity federation instead of long-lived static keys. A credential that expires on its own is one you do not have to remember to revoke. Sealed and encrypted secrets in Git for GitOps GitOps is a wonderful operating model, and it collides head-on with secrets: you want your entire desired state in version control, but you must never commit plaintext secrets to a repository. Git history is durable and widely replicated, so a secret committed once is effectively leaked forever, even after a later "fix" commit. The resolution is to commit only encrypted secret material. Two established, vendor-neutral approaches: Sealed secrets, where a controller in the cluster holds a private key and you commit a public-key-encrypted object that only that controller can decrypt. The encrypted object is safe to store in Git; the plaintext never leaves your machine or the cluster. File-level encryption of secret manifests using an encryption tool with keys held in a KMS, so the values in the committed YAML are ciphertext and are decrypted only in the delivery pipeline or by an in-cluster operator. Either way, the rule is the same: the repository contains ciphertext, and the decryption key lives somewhere the repository cannot reach. Never let a plaintext secret land in a commit, a pull request diff, or a CI log. Auditing secret access You cannot protect what you cannot see. The Kubernetes audit log records requests to the API server, including reads of Secret objects, and it is one of the most valuable and most underused security signals in a cluster. Configure an audit policy that captures access to the secrets resource, ship those logs to a system you actually review, and alert on the anomalous patterns: an unexpected service account reading secrets, a sudden list across all namespaces, or access from a workload that has no business touching credentials. Auditing serves two purposes. Day to day, it is detection. After an incident, it is the record that tells you exactly which secrets were touched and therefore which ones need rotating. Both are far better than guessing. Shift left: catch secret problems before they ship Everything above is a runtime control. The cheapest place to fix a secrets problem, though, is before it ever reaches a cluster, which is the heart of secure software development. Two practices pay for themselves quickly: Detect hardcoded secrets in CI. Scan source, manifests, and commit history for tokens, keys, and passwords so a credential is caught in the pull request instead of after it has been deployed and cached in a dozen places. Scan manifests and IaC. Automated infrastructure-as-code scanning flags the misconfigurations we have discussed: unencrypted etcd, over-broad RBAC on secrets, secrets injected as env vars, and Deployments that reference sensitive data unsafely, all before they are applied. Shift-left turns secrets management from a firefighting exercise into a guardrail that runs on every change. How Rainforest helps Rainforest is built to catch exactly these issues early. Its secret scanning looks across your code and configuration for hardcoded credentials and accidentally committed secrets, so a leaked token is flagged in the pull request rather than discovered in an incident review. Its IaC scanning inspects your Kubernetes manifests and infrastructure definitions for the misconfigurations that undermine secrets management: missing encryption at rest, overly permissive RBAC, and unsafe secret handling in Pod specs. Together they give you a shift-left safety net that complements the runtime controls in this guide, all from one application security testing platform. If you would like to see how that looks against your own manifests and repositories, book a demo and we will walk through it with you.