Skip to main content

Project Lab 06: The Persistence Layer & Automated Maintenance

Applications are more than just code; they are a combination of Configuration, Secrets, and Persistent Data. In this lab, we will deploy a high-performance database stack that uses dynamic storage provisioning and automated background maintenance.

Reference Material:

  • docs/05-storage-config-jobs/2-persistent-storage-csi.md
  • docs/05-storage-config-jobs/4-configmaps-secrets.md
  • docs/05-storage-config-jobs/5-daemonsets-jobs-cronjobs.md

1. OBJECTIVE: DATA INTEGRITY & AUTOMATION​

We will build a "Log-Processing Database" with the following requirements:

  1. Dynamic Storage: Data must survive Pod restarts and land on high-performance SSDs.
  2. Externalized Config: DB tuning parameters must be managed via ConfigMaps.
  3. Encrypted Credentials: Passwords must never be in plain text.
  4. Scheduled Backups: A CronJob must perform a data snapshot every hour.
  5. Node Monitoring: A DaemonSet must run on every node to monitor the health of the host filesystem.

2. PHASE 1: THE STORAGE FOUNDATION​

We need to define a policy for how Kubernetes should request disks from the underlying Cloud/Infrastructure provider.

2.1 The StorageClass (fast-ssd-sc.yaml)​

Using WaitForFirstConsumer ensures the volume is created in the same Availability Zone where the pod is scheduled.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/no-provisioner # Change to ebs.csi.aws.com for EKS
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain # Safety: Don't delete physical disk if PVC is deleted
allowVolumeExpansion: true

2.2 The PersistentVolumeClaim (db-pvc.yaml)​

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-storage-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 10Gi

3. PHASE 2: CONFIGURATION & SECRETS​

We will decouple the database settings from the container image.

3.1 Create ConfigMap and Secret​

# 1. ConfigMap for DB Tuning (Tuning for log processing)
kubectl create configmap db-config \
  --from-literal=max_connections=200 \
  --from-literal=shared_buffers=256MB

# 2. Secret for Root Password
echo -n "Admin@123" > password.txt
kubectl create secret generic db-credentials --from-file=password.txt

4. PHASE 3: THE STATEFUL WORKLOAD​

We will deploy a database that consumes the storage, config, and secrets.

4.1 The Database Deployment (db-deploy.yaml)​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - name: database
        image: postgres:15-alpine
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password.txt
        # Injecting ConfigMap as Env Vars
        envFrom:
        - configMapRef:
            name: db-config
        volumeMounts:
        - name: data-vol
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: data-vol
        persistentVolumeClaim:
          claimName: database-storage-pvc

5. PHASE 4: INFRASTRUCTURE & MAINTENANCE​

5.1 The Node Monitor (DaemonSet)​

Ensures every node in the cluster has a log-exporter watching the filesystem.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-disk-monitor
spec:
  selector:
    matchLabels:
      name: disk-exporter
  template:
    metadata:
      labels:
        name: disk-exporter
    spec:
      containers:
      - name: monitor
        image: busybox
        command: ["sh", "-c", "while true; do df -h /host; sleep 60; done"]
        volumeMounts:
        - name: host-root
          mountPath: /host
          readOnly: true
      volumes:
      - name: host-root
        hostPath:
          path: /

5.2 The Scheduled Backup (CronJob)​

Simulates a snapshot of the database every hour.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-hourly-backup
spec:
  schedule: "0 * * * *" # Every hour at minute 0
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup-agent
            image: alpine
            command: ["sh", "-c", "echo 'Backing up database content...'; sleep 10; echo 'Backup Complete'"]
          restartPolicy: OnFailure

6. VERIFICATION & AUDIT​

6.1 Audit Storage Binding​

kubectl get pvc database-storage-pvc

Observation: Status should be Bound. If it stays Pending, check if the Pod is scheduled (StorageClass uses WaitForFirstConsumer).

6.2 Validate Secrets Inside Pod​

kubectl exec -it deployment/log-database -- sh -c 'echo $POSTGRES_PASSWORD'

Success: Should print Admin@123.

6.3 Trigger a Manual Backup​

To test the backup logic without waiting an hour:

kubectl create job --from=cronjob/db-hourly-backup test-backup-now
kubectl get pods # Watch for the short-lived backup pod to complete

7. TROUBLESHOOTING & NINJA COMMANDS​

7.1 Recovering from a Stuck Job​

If a Job fails too many times, it hits the backoffLimit.

kubectl describe job test-backup-now
# Look for: "Job has reached the specified backoff limit"

7.2 Inspecting Host-Level Mounts (DaemonSet)​

# Exec into the DaemonSet pod to see the real host disk usage
kubectl exec -it daemonset/node-disk-monitor -- df -h /host

7.3 Expansion Test​

If the 10Gi disk is full, you can expand it live:

kubectl patch pvc database-storage-pvc -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Observation: Kubelet will eventually trigger an online resize of the filesystem.

8. ARCHITECT'S KEY TAKEAWAYS​

  1. PersistentVolumes are Global: They are cluster-scoped, but PersistentVolumeClaims are Namespaced.
  2. ConfigMaps vs. Secrets: Use ConfigMaps for performance tuning and Secrets for any value that should be protected at-rest.
  3. DaemonSets bypass Scheduling: They ensure specific pods (monitoring, networking) are present even on specialized nodes.
  4. CronJobs are Release Managers for Jobs: They handle the scheduling and history management of short-lived tasks.