Skip to main content

Services & Namespaces (Networking Layer)

In Kubernetes, Pods are ephemeral "cattle." They are created and destroyed dynamically, and their IP addresses change constantly. Services are the abstraction layer that provides a stable Virtual IP (VIP) and DNS entry to access a grouping of Pods.

1. SERVICES: ARCHITECTURE & INTERNALS​

1.1 The Components​

A "Service" is not a single process. It is a configuration rule implemented across the cluster.

  1. The Service Object: Defines the stable VIP and the Selector (app: payment) used to find backing Pods.
  2. EndpointSlices (The "New" Backend):
    • Legacy: Endpoints API object (scales poorly with >1000 pods).
    • Modern: EndpointSlices group backing Pod IPs/Ports into scalable chunks.
  3. Kube-Proxy (The Data Plane): Runs on every node. It watches the API Server for Service updates and programs the node's network rules to intercept traffic destined for the VIP and redirect it to a Pod.

1.2 Kube-Proxy Modes: Iptables vs. IPVS​

The performance of your networking depends heavily on the mode kube-proxy is running in.

ModeMechanismPerformanceComplexity
iptables (Default)Uses Linux netfilter rules.O(n). As Services grow to 5,000+, rule sync becomes slow. Packet processing is linear.Low. Standard Linux kernel feature.
IPVS (Production)Uses Linux IP Virtual Server (Kernel Load Balancer).O(1). Uses hash tables. Constant performance regardless of cluster size. Supports advanced algorithms (Least Connection, Locality).Medium. Requires kernel modules enabled on host.

Bible Rule: If your cluster will exceed 1,000 Services, ensure you are using IPVS mode.

2. SERVICE TYPES & TRAFFIC FLOW​

A. ClusterIP (Internal Only)​

  • Behavior: Allocates a stable VIP from the --service-cluster-ip-range (e.g., 10.96.0.0/12).
  • Access: Only reachable from within the cluster nodes/pods.
  • Flow: Client Pod to CoreDNS (resolve name) to VIP to Iptables (DNAT) to Target Pod IP.

B. NodePort (Static Entry)​

  • Behavior: Opens a static port (range 30000-32767) on every Node's physical IP.
  • Access: <NodeIP>:<NodePort>.
  • Flow: Client to Any Node IP:30005 to Iptables to Service VIP to Pod IP.
  • Warning: Security risk. Opens ports on the host. Not recommended for production internet-facing traffic.

C. LoadBalancer (Cloud Integration)​

  • Behavior: Provisions an external Cloud LB (AWS ELB, GCP LB, Azure LB).
  • Flow: Internet to Cloud LB to NodePort (hidden) to Service VIP to Pod IP.

D. ExternalName (DNS Alias)​

  • Behavior: No VIP. No Proxying. Just a CNAME record.
  • Use Case: Mapping a local service name (my-db) to an external RDS endpoint (db-prod.aws.com).

Production Manifest (Best Practices)​

apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: finance
spec:
  type: ClusterIP
  selector:
    app: payment-processor
  ports:
  - name: http           # ALWAYS name your ports
    port: 80             # The port clients hit (VIP Port)
    targetPort: 8080     # The port the Container is listening on
    protocol: TCP

3. ADVANCED TRAFFIC POLICIES​

3.1 ExternalTrafficPolicy: Local vs. Cluster​

This is the #1 reason for "Why is my client IP missing?" or "Why is my latency uneven?"

Default: Cluster​

Traffic hitting Node A can be routed to a Pod on Node B.

  • Hop: Node A to Node B.
  • SNAT: Node A applies Source Network Address Translation. The Pod on Node B sees the source IP as Node A, not the real Client IP.
  • Pros: Even load balancing.
  • Cons: Loss of Client IP; Extra network hop.

Config: Local​

Traffic hitting Node A must go to a Pod on Node A.

  • Hop: None. Direct to Pod.
  • SNAT: None. Client IP is preserved.
  • Pros: Lower latency, Visibility of real IP.
  • Cons: Imbalanced Load. If Node A has 1 pod and Node B has 0 pods, traffic hitting Node B is DROPPED (unless the external LB health checks fail Node B).
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local  # Preserve Client IP

3.2 Session Affinity (Sticky Sessions)​

By default, traffic is Round-Robin. If you need a user to stick to the same pod (e.g., in-memory shopping cart):

spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

4. HEADLESS SERVICES & DISCOVERY​

A Headless Service allows direct peer-to-peer discovery without a Virtual IP intermediate.

  • Configuration: clusterIP: None.
  • DNS Result: Instead of returning 1 VIP, CoreDNS returns multiple A records (one for every healthy Pod IP).
  • Use Case:
    • StatefulSets: Databases (Cassandra, Mongo) need to know exactly which peer is which (e.g., db-0, db-1).
    • gRPC Client-Side Load Balancing: The client wants to know all backend IPs to maintain its own connection pool.
apiVersion: v1
kind: Service
metadata:
  name: mongo-headless
spec:
  clusterIP: None   # Disables VIP allocation
  selector:
    app: mongo
  ports:
  - port: 27017

5. DNS IN KUBERNETES (CoreDNS)​

DNS is the glue of Kubernetes service discovery.

5.1 FQDN Anatomy​

Every object gets a DNS name. my-svc.my-ns.svc.cluster.local

  1. Service: my-svc
  2. Namespace: my-ns
  3. Type: svc (Service) or pod (Pod)
  4. Domain: cluster.local (Default)

5.2 The "ndots:5" Performance Pitfall​

By default, /etc/resolv.conf in a Pod has search default.svc.cluster.local svc.cluster.local cluster.local. This means if you curl google.com, the system tries:

  1. google.com.default.svc.cluster.local (Fail)
  2. google.com.svc.cluster.local (Fail)
  3. google.com.cluster.local (Fail)
  4. google.com (Success)

Optimization: If your app makes heavy external calls, use Fully Qualified Domain Names (end with a dot: google.com.) to skip the search path.

6. NAMESPACES (Logical Isolation)​

Namespaces allow multiple teams/projects to share a cluster.

A. What Is Isolated?​

  • Resources: Pods, Deployments, Services, PVCs, ConfigMaps, Secrets, ServiceAccounts.
  • Constraints: ResourceQuotas (CPU/RAM caps per namespace), LimitRanges.

B. What Is Global? (NOT Isolated)​

  • Compute: Nodes.
  • Storage: PersistentVolumes (PVs are global; PVCs are namespaced).
  • Networking: StorageClasses.
  • Identity: ClusterRoles, ClusterRoleBindings.

C. The Network Security Myth​

CRITICAL: By default, Namespaces DO NOT prevent network traffic. A Pod in dev can curl a database in prod unless you explicitly block it with NetworkPolicies.

D. Debugging Cross-Namespace​

Access a service in another namespace by appending the namespace to the DNS name:

curl http://db-service.production  # (If in same cluster)

7. TROUBLESHOOTING CHEATSHEET​

1. Service Discovery Check (DNS) Spawn a debug pod to test resolution.

kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup my-service

2. Endpoint Check Is the Service actually connected to Pods?

kubectl get endpoints -n my-ns my-service
# OR for modern K8s
kubectl get endpointslices -n my-ns

If output is empty/none, check your Selector labels.

3. Kube-Proxy / Node Level Debugging If a Service times out, the issue might be on the Node's iptables.

# SSH into the node and list NAT rules for a specific port
iptables -t nat -L KUBE-SERVICES | grep <PORT>

4. Check CoreDNS Logs If DNS fails cluster-wide:

kubectl logs -n kube-system -l k8s-app=kube-dns