Skip to main content

Project Lab 13: Layer 7 Traffic Orchestration & Gateway Migration

As clusters grow, the limitations of the legacy Ingress API (annotation bloat and lack of role-based control) become apparent. This lab demonstrates how to manage production traffic using the traditional Ingress model and how to modernize that infrastructure using the Gateway API.

Reference Material:

  • docs/12-ingress-gateway/1-Ingress.md
  • docs/12-ingress-gateway/2-Ingress-host-routing.md
  • docs/12-ingress-gateway/3-gateway-api.md

1. OBJECTIVE: EVOLVING THE EDGE​

The goal is to manage traffic for an enterprise application suite:

  1. Legacy Phase: Use NGINX Ingress to route api.example.com and app.example.com with TLS.
  2. Modernization Phase: Deploy a Gateway API infrastructure.
  3. Traffic Engineering: Implement a 90/10 Canary Split between v1 and v2 of the API using native Gateway API fields (no annotations).

2. PHASE 1: THE LEGACY INGRESS MODEL (NGINX)​

We will start by deploying two applications and a single Ingress controller to route traffic based on the Host header.

2.1 Setup Workloads and TLS Secret​

# 1. Deploy App V1
kubectl run app-v1 --image=hashicorp/http-echo --port=5678 --labels="app=web,version=v1" -- -text="Welcome to V1"
kubectl expose pod app-v1 --port=80 --target-port=5678 --name=web-v1-svc

# 2. Create self-signed TLS Secret for SNI testing
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt -subj "/CN=api.example.com"
kubectl create secret tls edge-tls-secret --key tls.key --cert tls.crt

2.2 The Host-Based Ingress (ingress-legacy.yaml)​

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: legacy-router
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts: ["api.example.com"]
    secretName: edge-tls-secret
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-v1-svc
            port:
              number: 80

3. PHASE 2: THE MODERN GATEWAY API​

Now, we will replicate and enhance this logic using the Gateway API. This separates the "Infrastructure" (Gateway) from the "Routing" (HTTPRoute).

3.1 Provision the Gateway (Infrastructure Layer)​

The Gateway defines where the Load Balancer listens. This is typically managed by the Platform/SRE team.

# 01-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
  namespace: infra-system
spec:
  gatewayClassName: nginx # Linked to the Nginx Gateway Fabric
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "*.example.com"
    tls:
      mode: Terminate
      certificateRefs:
      - name: edge-tls-secret
    allowedRoutes:
      namespaces:
        from: All # Allows dev teams to attach routes

3.2 Define the Canary Route (Developer Layer)​

The Developer team now creates an HTTPRoute to manage their own traffic logic, including a weighted split for a new version (v2).

# Deploy App V2
kubectl run app-v2 --image=hashicorp/http-echo --port=5678 --labels="app=web,version=v2" -- -text="Welcome to V2 (Canary)"
kubectl expose pod app-v2 --port=80 --target-port=5678 --name=web-v2-svc

Manifest: canary-route.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-service-route
  namespace: default
spec:
  parentRefs:
  - name: prod-gateway
    namespace: infra-system
  hostnames: ["api.example.com"]
  rules:
  - backendRefs:
    - name: web-v1-svc
      port: 80
      weight: 90 # 90% of production traffic
    - name: web-v2-svc
      port: 80
      weight: 10 # 10% for the Canary test

4. VERIFICATION & TRAFFIC AUDIT​

4.1 Test the Ingress SNI​

# Verify the Legacy Ingress is presenting the correct cert
curl -kvH "Host: api.example.com" https://<INGRESS_IP>

4.2 Validate Gateway API Status​

The Gateway API provides extensive feedback in the status field.

# Check if the Gateway is successfully programmed
kubectl get gateway prod-gateway -n infra-system -o jsonpath='{.status.conditions[?(@.type=="Programmed")]}'

# Check if the HTTPRoute successfully attached to the Gateway
kubectl describe httproute api-service-route

4.3 Verify the Canary Split​

Run a loop to see the 90/10 distribution in action.

for i in {1..10}; do curl -sH "Host: api.example.com" http://localhost:31000; done

Expected Observation: Approximately 9 out of 10 requests will return "Welcome to V1", and 1 will return "Welcome to V2 (Canary)".

5. TROUBLESHOOTING & NINJA COMMANDS​

5.1 The "Route Not Found" Debug​

If your Gateway is Ready but your Route returns 404:

  1. Check the parentRef: Does the name and namespace match the Gateway?
  2. Check allowedRoutes on the Gateway: Is the developer namespace permitted to attach?

5.2 Audit Nginx Realization​

If using an in-cluster Gateway (like Nginx Gateway Fabric), inspect the actual Nginx config generated by the Gateway API:

kubectl exec -it -n infra-system <ngf-pod-name> -- cat /etc/nginx/conf.d/api-service-route.conf

6. ARCHITECT'S KEY TAKEAWAYS​

  1. Ingress is Monolithic, Gateway API is Modular: Gateway API allows the SRE team to manage IP/Certificates while Developers manage only the routing logic.
  2. Native Traffic Splitting: Gateway API eliminates the need for proprietary annotations (like nginx.ingress.kubernetes.io/canary) by making weights a first-class citizen.
  3. SNI is the standard for Multi-Tenancy: Always use Host-based routing and SNI to consolidate multiple apps onto a single entry point.
  4. ReferenceGrant (Review): Remember that if your Route is in a different namespace than your Service, you must use a ReferenceGrant (Ref: Chapter 12 Notes).