Skip to main content

Project Lab 09: Extending the API with CRDs and Operators

Standard Kubernetes resources (Pods, Services) are often too granular for complex applications like databases or message brokers. This lab demonstrates how to create a "Higher Level" abstraction called a ManagedDatabase using CRDs and details how an Operator automates its lifecycle.

Reference Material:

  • docs/08-crds-operators/crds.md
  • docs/08-crds-operators/operators.md
  • Pre-requisite Knowledge: RBAC (Chapter 07) and Storage (Chapter 05).

1. OBJECTIVE: THE DATABASE-AS-A-SERVICE (DBaaS)​

The goal is to extend the Kubernetes API so that a developer can simply say: "I want a PostgreSQL database with 20Gi storage," and the cluster automatically handles the Deployment, Service, and PVC.

  1. Define the Blueprint: Create a CRD for ManagedDatabase.
  2. Enforce Schema: Use OpenAPI v3 to ensure only valid data (e.g., specific versions) is accepted.
  3. Implement Status: Enable the /status subresource for real-time health reporting.
  4. Simulate the Brain: Walk through the Operator reconciliation logic required to manage these objects.

2. PHASE 1: DEFINING THE CUSTOM RESOURCE DEFINITION (CRD)​

The CRD tells the API Server that a new "Kind" of object exists.

2.1 The CRD Manifest (managed-db-crd.yaml)​

Note the use of served and storage versions, and the strict validation schema.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: manageddatabases.infra.io # Pattern: <plural>.<group>
spec:
  group: infra.io
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # Enables status updates without incrementing metadata.generation
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["engine", "version", "storageSize"]
              properties:
                engine:
                  type: string
                  enum: ["postgres", "mysql", "redis"] # Restrict to supported DBs
                version:
                  type: string
                  pattern: '^[0-9]+(\.[0-9]+)?$' # Regex for versioning
                storageSize:
                  type: string
            status:
              type: object
              properties:
                state:
                  type: string
                endpoint:
                  type: string
  scope: Namespaced
  names:
    plural: manageddatabases
    singular: manageddatabase
    kind: ManagedDatabase
    shortNames:
    - mdb

Apply and Verify:

kubectl apply -f managed-db-crd.yaml

# Verify the new API endpoint exists
kubectl api-resources | grep mdb

3. PHASE 2: DEFINING INTENT (CUSTOM RESOURCE)​

Now that the cluster knows what a ManagedDatabase is, a developer can create one.

3.1 The Custom Resource Manifest (prod-db.yaml)​

apiVersion: infra.io/v1
kind: ManagedDatabase
metadata:
  name: billing-db
  namespace: finance
spec:
  engine: "postgres"
  version: "15.0"
  storageSize: "20Gi"

4. PHASE 3: THE OPERATOR BRAIN (INTERNAL LOGIC)​

A CRD alone is just a data entry in Etcd. To make it "live," an Operator (Controller) must be running.

4.1 The Reconciliation Logic​

If you were writing this Operator in Go (using the Operator SDK), your Reconcile() function would follow these steps every time the billing-db object is changed:

  1. Observe: "Does a Deployment/PVC named billing-db exist?"
  2. Diff: "The CRD asks for 20Gi, but the current PVC is only 10Gi."
  3. Act (Reconcile):
    • Create a PersistentVolumeClaim of 20Gi (Ref: Chapter 05).
    • Create a Deployment with the postgres:15.0 image.
    • Create a Service to expose it.
  4. Update Status: Write the Service IP to status.endpoint and set status.state to Ready.

4.2 Handling Deletion (Finalizers)​

To prevent a user from deleting the CR before the actual cloud disk is cleaned up, the Operator adds a Finalizer.

# Check the CR for finalizers (The 'lock' that ensures clean deletion)
kubectl get mdb billing-db -n finance -o jsonpath='{.metadata.finalizers}'

5. VERIFICATION & AUDIT​

5.1 Test Schema Validation​

Try to create a DB with an unsupported engine (e.g., oracle).

kubectl patch mdb billing-db -n finance --type=merge -p '{"spec":{"engine":"oracle"}}'

Expected Output:

The ManagedDatabase "billing-db" is invalid: spec.engine: Unsupported value: "oracle": 
supported values: "postgres", "mysql", "redis"

Success: The API server used the CRD's OpenAPI schema to reject the invalid "desired state."

5.2 Inspecting the Status Subresource​

In a real Operator setup, you would monitor the status to confirm the Operator has finished its work.

kubectl get mdb billing-db -n finance -o wide
# OR
kubectl describe mdb billing-db -n finance

6. TROUBLESHOOTING & NINJA COMMANDS​

6.1 Stuck Deletion (The "Finalizer Hang")​

If you delete an Operator before deleting its Custom Resources, the CRs will get stuck in Terminating because the Operator isn't there to remove the Finalizer.

The "Principal's Fix":

# Manually remove the finalizer to force delete the object from Etcd
kubectl patch mdb billing-db -n finance -p '{"metadata":{"finalizers":null}}' --type=merge

6.2 Auditing the API Server Group​

Verify the internal versioning of your new API.

kubectl get --raw /apis/infra.io/v1/namespaces/finance/manageddatabases/billing-db | jq .

7. ARCHITECT'S KEY TAKEAWAYS​

  1. CRDs extend the API, Operators extend the Logic: A CRD without an Operator is just a static configuration file stored in the cluster.
  2. Subresources are Mandatory for Production: Using the /status subresource prevents "Reconciliation Loops" where updating status triggers a new "Update" event.
  3. OpenAPI v3 is the First Line of Defense: Validation at the API level is significantly faster and safer than validation inside the Operator code.
  4. Operator Maturity: A Level 1 Operator only installs the app; a Level 5 Operator (like we simulated here) handles scaling, health reporting, and complex versioning.