Skip to main content

Custom Resources, CRDs, and the Controller Pattern

Kubernetes is designed as an extensible platform. While it comes with built-in primitives (Pods, Services), its true power lies in the ability to define Custom Resources that represent domain-specific concepts (e.g., a Database, a Backup, or a ManagedMachine) and manage them using the same declarative patterns as native objects.

1. THE ARCHITECTURAL HIERARCHY​

To extend the API, you must understand the distinction between the data, the schema, and the logic.

  1. Custom Resource Definition (CRD): The blueprint. It registers a new "Kind" with the API Server and defines the validation schema (OpenAPI v3).
  2. Custom Resource (CR): The instance. A specific object created by a user based on the CRD.
  3. Custom Controller: The brain. A process (often running in a Pod) that watches the CR and reconciles the cluster's state to match the CR's .spec.

1.1 API Path Internal Logic​

Standard resources live in the Core or Named groups. Custom resources always live in a Named group.

  • Built-in Pod: /api/v1/namespaces/{ns}/pods
  • Custom Resource: /apis/{group}/{version}/namespaces/{ns}/{kind-plural}

When you apply a CRD, the API Server dynamically creates a new RESTful endpoint. It uses the OpenAPIV3Schema defined in the CRD to validate any incoming CRs before persisting them to etcd.

2. CUSTOM RESOURCE DEFINITION (CRD) INTERNALS​

A "Bible-grade" CRD is more than just a name; it includes validation, subresources, and structural schemas.

2.1 Subresources: Status and Scale​

Production CRDs should use Subresources:

  • /status: Allows the controller to update the status of an object without incrementing the metadata.generation field. This prevents unnecessary reconciliation loops.
  • /scale: Allows the Custom Resource to be targeted by the Horizontal Pod Autoscaler (HPA).

2.2 Finalizers (The Cleanup Logic)​

Finalizers are strings in the metadata.finalizers list. They prevent an object from being deleted from etcd until a controller performs cleanup (e.g., deleting a physical VM or a Cloud DB).

  1. User deletes CR.
  2. K8s sets deletionTimestamp.
  3. Controller sees the timestamp, cleans up external resources, and removes the finalizer string.
  4. K8s removes the object from etcd.

3. BIBLE-GRADE MANIFEST: THE CRD​

This example defines a BackupPolicy. It includes strict validation and status subresources.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # Name must follow the pattern <plural>.<group>
  name: backuppolicies.ops.io
spec:
  group: ops.io
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        # Enables the /status endpoint for better performance
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["schedule", "targetPVC"]
              properties:
                schedule:
                  type: string
                  pattern: '^(\*|([0-59](\-[0-59])?(\/[0-59])?)|(L|W)?|[?*])( (\*|([0-23](\-[0-23])?(\/[0-23])?)|(L|W)?|[?*]))( (\*|([1-31](\-[1-31])?(\/[1-31])?)|(L|W)?|[?*]))( (\*|([1-12](\-[1-12])?(\/[1-12])?)|(L|W)?|[?*]))( (\*|([0-6](\-[0-6])?(\/[0-6])?)|(L|W)?|[?*]))$'
                  description: "Must be a valid Cron expression."
                retentionDays:
                  type: integer
                  minimum: 1
                  maximum: 30
                  default: 7
                targetPVC:
                  type: string
            status:
              type: object
              properties:
                lastBackupTime:
                  type: string
                  format: date-time
                phase:
                  type: string
                  enum: ["Pending", "Running", "Completed", "Failed"]
      additionalPrinterColumns:
      - name: Schedule
        type: string
        jsonPath: .spec.schedule
      - name: Status
        type: string
        jsonPath: .status.phase
  scope: Namespaced
  names:
    plural: backuppolicies
    singular: backuppolicy
    kind: BackupPolicy
    shortNames:
    - bp

4. THE CUSTOM CONTROLLER: THE RECONCILIATION LOOP​

A controller is an infinite loop that watches for events (Add, Update, Delete) on a specific resource.

4.1 The Informer Pattern​

To avoid overwhelming the API Server, controllers use Informers.

  1. List-Watch: The Informer performs an initial LIST and then a long-lived WATCH for changes.
  2. Local Cache: It stores the objects in a local cache (Store).
  3. Event Handlers: It triggers functions (OnAdd, OnUpdate) that put the object's Key (namespace/name) into a WorkQueue.

4.2 The Reconciliation Function​

The "Worker" pulls a key from the queue and executes the logic:

Reconcile(key):
  1. Fetch current state from Cache (Informer).
  2. If object deleted (has DeletionTimestamp), handle Finalizers and Return.
  3. Compare Desired (Spec) vs Actual (System Status).
  4. Perform Action (e.g., Create a Job, Call a Cloud API).
  5. Update Status via the /status subresource.
  6. Requeue if error occurs.

5. BIBLE-GRADE MANIFEST: THE CUSTOM RESOURCE​

The end-user simply provides the intent.

apiVersion: ops.io/v1
kind: BackupPolicy
metadata:
  name: postgres-nightly-backup
  namespace: database-prod
  finalizers:
  - ops.io/cleanup-backups # Ensures controller cleans up remote storage on deletion
spec:
  schedule: "0 2 * * *"
  retentionDays: 14
  targetPVC: pg-data-pvc

6. PRODUCTION PITFALLS & REAL-WORLD WARNINGS​

6.1 Validation Logic​

  • The Trap: Defining a CRD without an openAPIV3Schema.
  • The Impact: The API Server will accept any garbage data. If your controller expects an integer but gets a string, it will crash or enter an infinite error loop.
  • Best Practice: Always use structural schemas with strict typing and defaults.

6.2 Versioning and Conversion​

  • The Challenge: Updating from v1alpha1 to v1beta1.
  • The Solution: You must implement a Conversion Webhook. When a user requests a v1alpha1 object, the API Server calls your webhook to convert it to the storage version (v1beta1) before returning it.

6.3 Controller Performance​

  • The Trap: Performing heavy synchronous work inside the reconciliation loop.
  • The Impact: The WorkQueue backs up, and the controller becomes unresponsive.
  • Best Practice: Use a WorkQueue with multiple workers and perform long-running tasks asynchronously.

7. TROUBLESHOOTING & NINJA COMMANDS​

7.1 Inspecting the Schema​

If you aren't sure what fields a custom resource supports, use explain:

kubectl explain backuppolicy.spec

7.2 Checking CRD Health​

# Verify the CRD is "Established" (Accepted by the API)
kubectl get crd backuppolicies.ops.io -o jsonpath='{.status.conditions[?(@.type=="Established")].status}'

7.3 Debugging the Controller​

If a Custom Resource isn't doing anything:

  1. Check Status: kubectl get bp postgres-nightly-backup -o yaml. Is the .status field being updated?
  2. Check Events: kubectl get events -n database-prod. Good controllers emit events for significant actions.
  3. Check Logs: Locate the controller pod and check for "Reconciliation Error" or "Rate Limiting" logs.