Skip to main content

Imperative Commands & Kubectl Ninja

Mastering kubectl is the primary differentiator between a novice and a Senior Kubernetes Engineer. This guide moves beyond basic get and describe to advanced filtering, debugging, and introspection.

1. DECLARATIVE VS. IMPERATIVE​

Understanding the workflow difference is critical for production stability.

1.1 Imperative ("Do this now")​

  • Commands: kubectl run, kubectl create, kubectl scale, kubectl expose.
  • Behavior: The API server executes the action immediately.
  • Use Case:
    • Disaster Recovery: Restarting workloads instantly.
    • Debugging: Launching temporary pods.
    • Bootstrapping: Generating initial YAML manifests.
  • Risk: Changes are not recorded in Git. The cluster state drifts from the repo.

1.2 Declarative ("Make it look like this")​

  • Commands: kubectl apply -f folder/.
  • Behavior: Client sends a patch. The Cluster calculates the difference between "Current State" and "Desired State" and reconciles.
  • Use Case: GitOps (ArgoCD, Flux). The yaml file is the source of truth.

2. GENERATORS (Dry Run)​

Never write YAML from scratch. Use imperative commands with --dry-run=client to generate the scaffold, then edit it.

Standard Boilerplates:

# 1. POD (with command arguments)
kubectl run tmp-shell --image=busybox --restart=Never --dry-run=client -o yaml \
  -- /bin/sh -c 'echo hello; sleep 3600' > pod.yaml

# 2. DEPLOYMENT (Scalable app)
kubectl create deploy nginx-app --image=nginx:1.23 --replicas=3 --dry-run=client -o yaml > deploy.yaml

# 3. SERVICE (Exposing the deployment)
kubectl expose deploy nginx-app --name=nginx-svc --port=80 --target-port=8080 --type=ClusterIP --dry-run=client -o yaml > svc.yaml

# 4. CRONJOB (Periodic tasks)
kubectl create cronjob backup-job --image=postgres:15 --schedule="0 2 * * *" --dry-run=client -o yaml > cron.yaml

3. KUBECTL DEBUGGING (The "Ninja" Tier)​

3.1 Ephemeral Containers (kubectl debug)​

Modern container images (Distroless) lack shells, ps, curl, and tar. You cannot kubectl exec into them meaningfully.

The Solution: Inject a "Sidecar on demand" that shares the Process Namespace (PID) of the target.

# Attach a "netshoot" container (loaded with net tools) to a running Pod
kubectl debug -it <target-pod-name> \
  --image=nicolaka/netshoot \
  --target=<main-container-name> \
  -- sh

# --target: Crucial. It makes the ephemeral container share the Process ID namespace.
# Once inside, run 'ps aux' to see the main container's processes.

3.2 Introspection (kubectl explain)​

Stop googling "Kubernetes Pod Spec API". The documentation is built into the CLI.

# View the hierarchy of the Pod Spec
kubectl explain pod.spec

# Drill down into specific fields (e.g., how to configure liveness probes)
kubectl explain pod.spec.containers.livenessProbe --recursive

3.3 Raw API Access​

Sometimes kubectl get hides details. You can hit the API server endpoints directly.

# Check cluster metrics (if Metrics Server is installed)
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq .

4. JSONPATH MASTERY​

Stop piping to grep and awk. Use standard JSONPath to extract data reliably for scripting.

Syntax: -o jsonpath='{expression}'

A. Security & Auditing​

1. List all container images running in the namespace:

kubectl get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort | uniq

2. Decode a Secret immediately:

kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

B. Networking​

1. Get Pod Name and IP Address:

kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'

2. Get the NodePort assigned to a Service:

kubectl get svc my-service -o jsonpath='{.spec.ports[0].nodePort}'

C. Troubleshooting​

1. Find Pods that are NOT Running:

kubectl get pods --field-selector=status.phase!=Running

2. Sort by Restart Count (Find the crash loops):

kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

3. Sort Events by Time (See what just happened):

kubectl get events --sort-by='.metadata.creationTimestamp'

5. HOW 'APPLY' WORKS (The Three-Way Merge)​

Understanding this is required to debug "Drift" in GitOps.

When you run kubectl apply -f file.yaml, Kubernetes performs a Three-Way Merge Patch:

  1. The Live Object: The current JSON in Etcd.
  2. The Local Config: The YAML file you are applying.
  3. The Last-Applied-Configuration: A specific annotation stored on the Live Object:
    • kubectl.kubernetes.io/last-applied-configuration

The Logic Logic​

ConditionAction Taken
Field in Local, not in LiveAdd (Update Live).
Field in Live, not in LocalCheck Last-Applied.
-> If present in Last-AppliedDelete (You removed it from YAML, so K8s removes it).
-> If MISSING in Last-AppliedIgnore/Keep (Another controller, like HPA or Istio, added this field. Do not touch).

The "Diff" Command​

Before applying to production, always check the delta.

kubectl diff -f deployment.yaml

Returns exit code 1 if there are differences, 0 if identical. Useful for CI/CD pipelines.

6. ESSENTIAL KREW PLUGINS​

krew is the plugin manager for kubectl. A Senior Engineer's toolkit typically includes:

  1. kubectl ctx / kubectl ns: Fast switching between contexts and namespaces.
  2. kubectl tree: Visualizes ownership (Deployment -> RS -> Pod).
    • kubectl tree deploy/my-app
  3. kubectl neat: Cleans up the "garbage" (managedFields, creationTimestamp) when exporting YAML.
    • kubectl get pod my-pod -o yaml | kubectl neat
  4. kubectl stern: Tail logs from multiple pods simultaneously using regex.
    • kubectl stern "backend-.*" -n default (Tails all backend pods at once).