Skip to main content

Configuration Management: ConfigMaps and Secrets

In a cloud-native architecture, code must be immutable. Configuration—database URLs, feature flags, and credentials—must be decoupled from the container image. Kubernetes provides ConfigMaps for non-sensitive data and Secrets for sensitive data.

1. CONFIGMAPS (The Configuration Hub)​

A ConfigMap is a dictionary of key-value pairs stored in etcd. It allows you to inject configuration into a Pod as environment variables or as files in a volume.

1.1 Injection Methods​

  1. Environment Variables: Best for simple flags. Note: These are static; if the ConfigMap changes, the Pod must be restarted to see the new value.
  2. Volumes: Best for configuration files (e.g., nginx.conf). These are dynamic; the Kubelet updates the files inside the Pod without a restart.
  3. envFrom: Bulk-injects all keys in a ConfigMap as environment variables.

1.2 Bible-Grade YAML: Production ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: prod
  labels:
    app: payment-api
# IMMUTABILITY: Prevents accidental changes. v1.21+
immutable: false 
data:
  # Simple key-value
  LOG_LEVEL: "DEBUG"
  # Multi-line file content
  ui_config.yaml: |
    color: blue
    retries: 5
    features:
      beta: true

2. SECRETS (Sensitive Data Management)​

Secrets are similar to ConfigMaps but are designed for sensitive data (passwords, SSH keys, TLS certs).

2.1 The "Base64" Warning​

Bible Rule: Base64 is NOT encryption. It is an encoding. Anyone with access to the YAML can decode it instantly: echo "S3Jldy1OaW5qYQ==" | base64 -d -> Krew-Ninja.

2.2 Internal Security Mechanics​

  1. Etcd Encryption at Rest: By default, Secrets are stored in plain text in etcd. You must enable EncryptionConfiguration at the API Server level to secure them.
  2. Memory-Backed Volumes: When a Secret is mounted as a volume, Kubelet stores it in a tmpfs (RAM-backed) filesystem. The data is never written to the node's physical disk.
  3. Namespace Isolation: Secrets are namespace-scoped. A Pod in Namespace A cannot reference a Secret in Namespace B.

2.3 Secret Types​

  • Opaque (Default): Generic key-value pairs.
  • kubernetes.io/tls: Specifically for public/private key pairs.
  • kubernetes.io/dockerconfigjson: Used for imagePullSecrets to authenticate with private registries.

2.4 Bible-Grade YAML: TLS Secret​

apiVersion: v1
kind: Secret
metadata:
  name: api-tls-certs
type: kubernetes.io/tls
data:
  # These must be base64 encoded strings
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t...
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0t...

One of the most complex internals in Kubernetes storage is how Kubelet updates mounted ConfigMaps without restarting the container.

3.1 How the Update Works​

When a ConfigMap is mounted as a volume, Kubelet uses a symlink-based atomic update:

  1. Kubelet creates a new directory (..2023_10_27_...) containing the updated data.
  2. Kubelet updates the ..data symlink to point to the new directory.
  3. The file seen by the application (e.g., config.yaml) is a symlink to ..data/config.yaml.
/etc/config/my-file.yaml (App reads this)
  -> ..data/my-file.yaml
      -> ..2023_10_27_10_00_00/my-file.yaml (The real file)

When the ConfigMap is updated, only the ..data pointer changes.

3.3 Propagation Delay​

Updates are not instantaneous. The delay is: Kubelet Sync Period (Default 1m) + ConfigMap Cache TTL. Typically, it takes 60-90 seconds for changes to appear inside the Pod.

4. BIBLE-GRADE WORKLOAD MANIFEST​

This Deployment demonstrates every way to consume configuration and secrets safely.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-api
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: api
        image: my-app:v1.2
        env:
        # 1. Single Key from ConfigMap
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: LOG_LEVEL
        # 2. Single Key from Secret
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
              
        volumeMounts:
        # 3. Mounting ConfigMap as File (Dynamic)
        - name: config-file
          mountPath: /etc/app/config.yaml
          subPath: ui_config.yaml # WARNING: subPath does NOT auto-update
        
        # 4. Mounting Secret as File (Memory-backed)
        - name: certs
          mountPath: /etc/tls
          readOnly: true
          
      volumes:
      - name: config-file
        configMap:
          name: app-config
      - name: certs
        secret:
          secretName: api-tls-certs

5. PRODUCTION BEST PRACTICES & PITFALLS​

5.1 The subPath Update Problem​

If you use subPath to mount a specific file from a ConfigMap, Kubernetes will NOT update that file if the ConfigMap changes.

  • The Fix: Mount the entire directory, or use a Sidecar to trigger a reload.

5.2 Immutable ConfigMaps/Secrets​

For high-scale clusters (thousands of pods), watching ConfigMaps for changes creates significant CPU load on the API Server.

  • Strategy: If your configuration doesn't change during the Pod's life, set immutable: true. This tells the Kubelet not to watch for updates, drastically improving performance.

5.3 Secrets Management (External Tools)​

As organizations grow, managing Kubernetes Secrets natively becomes difficult.

  • Architecture Recommendation: Use the Secrets Store CSI Driver. This allows Kubernetes to mount secrets directly from AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault without ever storing them in etcd.

6. TROUBLESHOOTING & NINJA COMMANDS​

Identify Missing Keys​

If a Pod is stuck in CreateContainerConfigError, it usually means a requested Secret or ConfigMap doesn't exist.

kubectl describe pod <pod-name>
# Error: secret "db-credentials" not found

Direct Decoding (Ninja Audit)​

Auditing all secrets in a namespace to check for plain-text leaks:

kubectl get secrets -o json | jq -r '.items[].data | map_values(@base64d)'

Verifying Volume Mount Sync​

# Check the timestamp of the symlink inside the pod
kubectl exec <pod-name> -- ls -la /etc/app/
# Look for the ..data symlink timestamp