Skip to main content

Ephemeral Storage: Lifecycle and Architecture

Ephemeral storage is non-persistent disk space allocated to a Pod. Unlike Persistent Volumes (PVs), ephemeral storage is strictly bound to the Pod’s lifecycle. If a Pod is deleted, evicted, or relocated to another node, all data in ephemeral storage is purged.

1. EMPTYDIR: THE NODE-LOCAL SCRATCHPAD​

emptyDir is the most common form of ephemeral storage. It is physically located on the node's filesystem, typically under /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~empty-dir/.

1.1 Architecture & Inter-Container Communication​

All containers in a Pod share the same emptyDir volume. This is the primary mechanism for the Sidecar Pattern (e.g., a main app container writes logs, and a sidecar container tails those logs).

1.2 Memory-Backed (tmpfs) Storage​

By default, emptyDir is stored on the node's disk (SSD/HDD). You can force it to use the node's RAM by setting medium: Memory.

  • Architecture: This uses a tmpfs (temporary filesystem) mount.
  • Performance: Sub-millisecond latency.
  • Bible Warning: Data in a RAM-backed emptyDir counts against the Memory Limit of the container. If you fill the RAM-backed volume, your container will likely be OOMKilled.

1.3 Production Manifest: Multi-Container Data Sharing​

apiVersion: v1
kind: Pod
metadata:
  name: data-processor
spec:
  containers:
  - name: producer
    image: alpine
    command: ["sh", "-c", "while true; do date >> /data/shared.log; sleep 5; done"]
    volumeMounts:
    - name: shared-scratch
      mountPath: /data
  - name: consumer
    image: alpine
    command: ["sh", "-c", "tail -f /data/shared.log"]
    volumeMounts:
    - name: shared-scratch
      mountPath: /data
      readOnly: true # Sidecar only needs to read
  volumes:
  - name: shared-scratch
    emptyDir:
      medium: Memory # Store in RAM for high IOPS
      sizeLimit: 512Mi # Prevent runaway memory usage

2. DOWNWARD API: POD SELF-AWARENESS​

The Downward API allows a container to access its own metadata without having to query the Kubernetes API Server (avoiding RBAC complexities and network overhead).

2.1 Use Cases​

  • Logging: Tagging logs with the Pod Name and Node Name.
  • Config: Adjusting application behavior based on resource limits (e.g., Java Heap sizing based on limits.memory).

2.2 Fields Available​

  • metadata.name, metadata.namespace, metadata.uid, metadata.labels, metadata.annotations.
  • status.podIP, status.hostIP.
  • requests.cpu, limits.memory.

2.3 Production Manifest: Metadata Injection​

apiVersion: v1
kind: Pod
metadata:
  name: downward-api-demo
spec:
  containers:
  - name: app
    image: busybox
    env:
    - name: NODE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    volumeMounts:
    - name: pod-metadata
      mountPath: /etc/podinfo
  volumes:
  - name: pod-metadata
    downwardAPI:
      items:
      - path: "labels"
        fieldRef:
          fieldPath: metadata.labels

3. GENERIC EPHEMERAL VOLUMES (GEV)​

Introduced in K8s v1.23, GEVs allow you to use a StorageClass to provision temporary volumes. This allows you to leverage CSI features (like high-speed local NVMe or remote EBS) for scratch space.

3.1 Architecture vs. emptyDir​

  • emptyDir: Always uses the node's root partition or RAM.
  • GEV: Can use any CSI-supported backend. It creates a "per-pod PVC" that is automatically deleted when the Pod is deleted.

3.2 Production Manifest: Fast SSD Scratch Space​

kind: Pod
apiVersion: v1
metadata:
  name: fast-scratch-pod
spec:
  containers:
  - name: app
    image: alpine
    volumeMounts:
    - mountPath: /scratch
      name: tmp-nvme
  volumes:
  - name: tmp-nvme
    ephemeral:
      volumeClaimTemplate:
        spec:
          storageClassName: "fast-nvme-storage" # Provisioned via CSI
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 50Gi

4. MANAGEMENT & RESOURCE ENFORCEMENT​

Kubernetes manages ephemeral storage as a shared resource. If your Pods consume too much local disk, they risk crashing the Node (causing "No space left on device" errors for the Kubelet/Docker).

4.1 Requests vs. Limits​

  • Requests: The Scheduler ensures the node has enough local disk space on its root partition.
  • Limits: The Kubelet monitors the Pod’s disk usage (Container logs + writable layers + emptyDir).

4.2 The Eviction Loop​

If a Pod exceeds its ephemeral-storage limit, the Kubelet triggers an Eviction.

  1. Kubelet detects usage via du or project quotas.
  2. Pod is marked as Failed with reason Evicted.
  3. Pod is deleted and data is wiped.
spec:
  containers:
  - name: log-heavy-app
    resources:
      requests:
        ephemeral-storage: "2Gi"
      limits:
        ephemeral-storage: "4Gi" # Pod will be evicted if usage > 4Gi

5. DEBUGGING & REAL-WORLD COMMANDS​

5.1 Identifying Ephemeral Storage Pressure​

When a node runs out of disk space, it sets the DiskPressure condition.

kubectl get nodes -o custom-columns=NAME:.metadata.name,DISK_PRESSURE:.status.conditions[?(@.type==\"DiskPressure\")].status

5.2 Checking Pod-Level Consumption​

You cannot easily see ephemeral storage usage via kubectl top. You must check the node:

# Exec into the pod and check mounted space
kubectl exec -it <pod-name> -- df -h /cache

# Inspect the Kubelet metrics endpoint
kubectl get --raw /metrics | grep kubelet_evictions_total

5.3 Common Pitfall: Log Bloat​

Container logs (stdout/stderr) are stored on the node’s ephemeral storage. If your app logs excessively and you haven't set ephemeral-storage limits, the Pod will eventually crash the Node's filesystem.

Architectural Recommendation: Always set ephemeral-storage limits for any application that performs local file processing or produces heavy logs.