↓ Skip to main content
Blog | Haeger
  1. Posts/

Setting Up Sealed Secrets in a New Kubernetes Cluster

·6 mins·
DevOps Kubernetes Security
Table of Contents

Kubernetes Secrets are not encrypted, they are base64-encoded. Commit one to your repository and you have committed plaintext credentials that anyone with read access can decode. That rules out the obvious GitOps approach: everything in Git, except the secrets, which you then apply by hand and go hunting for the next time you rebuild.

Sealed Secrets fixes this. A controller in your cluster encrypts secrets so the ciphertext can safely live in your repository. This guide walks through setting it up in a fresh cluster, including the two things worth getting right from the start.

How it works
#

On first start, the controller generates an RSA key pair and stores it as a Secret in the cluster:

  • The public key encrypts. The kubeseal CLI on your machine uses it.
  • The private key decrypts. It never leaves the cluster.

You create a SealedSecret locally, commit it, and the controller turns it into a regular Secret inside the cluster. The ciphertext is only usable by that one cluster. That is the heart of the mechanism and also where most people trip up later.

Step 1: Install the controller
#

Using Helm:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

The namespace is a convention, but it matters: kubeseal looks for a controller named sealed-secrets-controller in kube-system by default. Install it elsewhere and you will be passing flags on every invocation.

Verify:

kubectl get pods -n kube-system -l name=sealed-secrets-controller

Step 2: Install the CLI
#

brew install kubeseal          # macOS

Linux binaries are on the releases page. Keep the CLI roughly in step with the controller, large version gaps between the two are an unnecessary source of trouble.

kubeseal --version

Step 3: Back up the sealing key
#

This is the most important step, and it’s almost always skipped. Do it right after installing.

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-key-backup.yaml

This file is the master key to every secret you will ever seal. It belongs in a password manager or an encrypted offline backup, never in the same repository as your sealed secrets. Otherwise you have defeated the very thing you adopted this for.

Why the urgency? See “Cluster binding” below.

Step 4: Seal a secret
#

The common case: registry credentials so your cluster can pull private images. Create the secret locally without applying it:

kubectl create secret docker-registry dockerhub-secret \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=<user> \
  --docker-password=<token> \
  --namespace=my-app \
  --dry-run=client -o yaml > secret.yaml

--dry-run=client is the critical part: the secret lands on disk, not in the cluster. Use a registry access token rather than your account password. Tokens can be revoked individually.

Then seal it:

kubeseal --format yaml < secret.yaml > sealed-secret.yaml

kubeseal fetches the public key from the controller itself. You can also work offline by exporting the key once:

kubeseal --fetch-cert > pub-cert.pem
kubeseal --format yaml --cert pub-cert.pem < secret.yaml > sealed-secret.yaml

That’s the usual approach for CI pipelines, which shouldn’t need cluster access.

Now sealed-secret.yaml goes into Git and secret.yaml goes into .gitignore.

Step 5: Do not forget the namespace
#

A sealed secret is bound to both name and namespace. That is not incidental, it is deliberate: otherwise someone could copy your sealed secret into a namespace they control and have it decrypted there.

In practice this means the namespace has to exist, and it should live in your repository as a manifest, not just in your head or in some old shell history.

apiVersion: v1
kind: Namespace
metadata:
  name: my-app

Step 6: Apply and verify
#

kubectl apply -f namespace.yaml -f sealed-secret.yaml

Within seconds the controller should produce a regular Secret:

kubectl get secret dockerhub-secret -n my-app
kubectl get sealedsecret dockerhub-secret -n my-app \
  -o jsonpath='{.status.conditions}'

Success shows as Synced=True. The more convincing test, though, is your workload: if the pod runs and pulled its image, the credentials are correct in substance too.

spec:
  containers:
    - name: app
      image: myuser/myimage:latest
  imagePullSecrets:
    - name: dockerhub-secret

Cluster binding: the part that bites later
#

A sealed secret can only be decrypted by the cluster whose public key encrypted it. Stand up a new cluster and its fresh controller generates a new key pair. Your ciphertext in Git is worthless there.

The symptom is quiet: the SealedSecret applies fine, but no Secret appears. The status and controller log say no key could decrypt secret. Meanwhile your pod sits in ImagePullBackOff and you go looking for the problem in your Deployment.

Three ways out:

With a key backup, the good case. Restore the saved key before the controller starts for the first time:

kubectl apply -f sealed-secrets-key-backup.yaml
kubectl delete pod -n kube-system -l name=sealed-secrets-controller

The controller picks up existing keys on startup and would not generate a new pair. Every sealed secret you have keeps working, untouched.

With the plaintext secret but no key backup reseal:

kubeseal --fetch-cert > pub-cert.pem       # the new cluster's key
kubeseal --format yaml --cert pub-cert.pem < secret.yaml > sealed-secret.yaml

The credentials stay the same; only the encryption is renewed.

With neither, the credentials are gone and have to be reissued: a new registry token, a new API key, a password reset. That is not a flaw in the mechanism, it is the point of it.

What else tends to go wrong
#

Secrets maintained in two places. If the same sealed secret lives in both deployment.yaml and sealed-secret.yaml, someone will eventually update only one of them. One object belongs in exactly one file.

Committing secret.yaml by accident. The plaintext file is an intermediate artifact and easy to forget. Set up .gitignore from the start, and if in doubt, rotate the token. Once pushed, it is in the history.

Forgetting about rotation. The controller periodically generates new keys and keeps the old ones for decryption. Existing sealed secrets keep working, but your backup quietly goes stale. Refresh it occasionally, or it will cover only part of your secrets when you need it.

Wrong namespace. Moving a secret to a different namespace means resealing it. Alternatively, kubeseal --scope namespace-wide or --scope cluster-wide allow looser bindings, which deliberately weakens the protection, so it stays a trade-off.

Checklist
#

  • Controller running, CLI version roughly matching
  • Sealing key backed up outside the repository
  • secret.yaml in .gitignore
  • Namespaces committed as manifests
  • Every object lives in exactly one file
  • Registry access via revocable tokens, not account passwords
  • Key backup refreshed after rotations

Takeaway
#

Sealed Secrets make full GitOps practical: your entire cluster state lives in the repository, secrets included. The price is one key you have to look after and the time to do that is at installation, not at the moment you need it.