Skip to main content

Project Lab 11: Security Hardening & Resource Prioritization

Security in Kubernetes is a multi-layered approach. It involves securing the source (Images), the runtime (PSS), and the availability (Priority). This lab demonstrates how to harden a namespace to the highest standard while ensuring critical pods survive cluster-wide resource pressure.

Reference Material:

  • docs/10-security/1-pod-security-standards.md
  • docs/10-security/2-image-pull-secrets.md
  • docs/10-security/3-priority-classes.md

1. OBJECTIVE: THE SECURE ENCLAVE​

The goal is to build a namespace called secure-enclave with the following requirements:

  1. Strict Runtime Policy: No pod can run as root, use privileged escalation, or write to the root filesystem.
  2. Private Image Source: All images must be pulled from a private registry using credentials managed at the ServiceAccount level.
  3. Guaranteed Availability: These secure pods must have a higher priority than standard workloads, allowing them to preempt lower-priority pods if the node is full.

2. PHASE 1: WORKLOAD PRIORITIZATION​

We must define what "Mission Critical" means in our cluster before deploying the pods.

2.1 Create the PriorityClass​

High numeric values ensure that the Scheduler treats these pods as high-value assets.

# 01-priority.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: mission-critical-security
value: 1000000
globalDefault: false
description: "High priority for security-hardened workloads."

3. PHASE 2: NAMESPACE HARDENING (PSS)​

We use the built-in Pod Security Standard (PSS) to enforce a "Restricted" profile. This is the most secure level available in Kubernetes.

3.1 Create and Label the Namespace​

kubectl create namespace secure-enclave

# ENFORCE: Block any pod that doesn't meet 'Restricted'
kubectl label namespace secure-enclave \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest

4. PHASE 3: SUPPLY CHAIN SECURITY (IMAGE PULL SECRETS)​

We will configure the namespace to automatically use private registry credentials without requiring developers to add imagePullSecrets to every manifest.

4.1 Create the Registry Secret​

kubectl create secret docker-registry private-registry-key \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=<user> \
  --docker-password=<pass> \
  -n secure-enclave

4.2 Patch the ServiceAccount​

We will patch the default ServiceAccount in the secure-enclave namespace so all pods inherit the secret.

kubectl patch serviceaccount default -n secure-enclave \
  -p '{"imagePullSecrets": [{"name": "private-registry-key"}]}'

5. PHASE 4: THE HARDENED MANIFEST​

We will deploy a "Secure API" pod. This pod must satisfy the Restricted profile labels we applied to the namespace, or it will be rejected.

5.1 The Hardened Pod Manifest (secure-app.yaml)​

apiVersion: v1
kind: Pod
metadata:
  name: secure-api
  namespace: secure-enclave
spec:
  priorityClassName: mission-critical-security # Link to PriorityClass
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: api
    image: <your-private-repo>/secure-api:v1.0 # Requires the patched secret
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
    volumeMounts:
    - name: tmp-storage
      mountPath: /tmp
  volumes:
  - name: tmp-storage
    emptyDir: {}

6. VERIFICATION & PREEMPTION TEST​

6.1 Test PSS Enforcement (Negative Test)​

Try to deploy a standard Nginx pod (which runs as root by default) in the secure-enclave namespace.

kubectl run insecure-pod --image=nginx -n secure-enclave

Expected Output: Error from server (Forbidden): pods "insecure-pod" is forbidden: violates PodSecurity "restricted:latest"... This confirms your namespace is successfully hardened.

6.2 Test Priority Preemption​

  1. Fill the Cluster: Deploy a large number of low-priority pods in the default namespace that consume all available CPU.
  2. Deploy Secure App: Deploy the secure-api pod from Step 5.1.
  3. Observe: 
    kubectl get pods -A -w

Expected Observation: You will see some low-priority pods in the default namespace enter Terminating status. The Scheduler is Preempting them to make room for your mission-critical-security pod.

7. TROUBLESHOOTING & NINJA COMMANDS​

7.1 Inspecting Active Security Context​

If your pod is rejected by PSS, you can find the exact reason by checking the API server response or using kubectl explain:

# Check the specific fields required for Restricted profile
kubectl explain pod.spec.securityContext

7.2 Auditing Secret Inheritance​

Check if a pod actually received the imagePullSecret via the ServiceAccount:

kubectl get pod secure-api -n secure-enclave -o jsonpath='{.spec.imagePullSecrets}'

8. ARCHITECT'S KEY TAKEAWAYS​

  1. Restricted PSS is Non-Negotiable for Public Clouds: It prevents most container breakout techniques (e.g., writing to /host or escalating to root).
  2. Priority 0 is the Default: If you don't define PriorityClasses, your critical pods are just as "evictable" as a dev dashboard.
  3. ServiceAccount Patching simplifies GitOps: Instead of developers managing secrets in every YAML file, the Platform Team manages the ServiceAccount configuration.
  4. Read-Only Root Filesystem: This is the strongest defense against malware persistence; even if an attacker gets a shell, they cannot download tools to the local disk.