Kubernetes Services, Networking and DNS: Following One Request

Overview#

A Service is not a proxy process and not a load balancer you can log into. It is a record in the API (a name, a virtual IP, a selector, a port mapping) that two other components act on: the endpoint controller, which lists the matching ready pods in an EndpointSlice, and kube-proxy on every node, which programs the node’s packet rules so that traffic to the virtual IP is rewritten to one of those pods. DNS sits in front and turns the name into the IP. Every networking problem in this part is one of those hops not doing what you assumed.

A client pod resolves the Service name through cluster DNS to the Service ClusterIP. kube-proxy on the node maps that IP and port to the addresses in the Service EndpointSlice, which the endpoint controller fills from pods matching the Service selector. Only pods whose readiness condition is true are eligible; a pod that fails its readiness probe stays listed with ready=false and receives no traffic, and a selector that matches no pods leaves the slice empty.

1. Cluster DNS. A pod asks for api; the resolver’s search path expands it to api.default.svc.cluster.local and CoreDNS answers with the Service’s ClusterIP.
2. Service. The ClusterIP and port are virtual: no process listens on them. kube-proxy rules on the node translate them.
3. EndpointSlice. The endpoint controller writes the addresses of pods that match the selector, with a ready condition per address.
4. A ready pod receives the connection on its targetPort.
5. A pod that fails readiness stays in the slice with ready=false and gets nothing.

Prerequisites

• The cluster, api Deployment, api Service and db StatefulSet from part 1
• kubectl 1.32 or newer

DNS: how api becomes an address#

Ask from inside a pod, because the cluster’s DNS is not reachable from your machine:

kubectl run dns --rm -i --restart=Never --image=busybox:1.37 -- nslookup api.default.svc.cluster.local
kubectl get svc api -o jsonpath='clusterIP={.spec.clusterIP}{"\n"}'

Server:    10.96.0.10
Address:   10.96.0.10:53
Name:      api.default.svc.cluster.local
Address:   10.96.223.236
clusterIP=10.96.223.236

10.96.0.10 is CoreDNS’s own Service, the first address in the pod’s /etc/resolv.conf. Applications use the short name api, which works because the pod’s search path is <namespace>.svc.cluster.local svc.cluster.local cluster.local: a bare name means “in my namespace”, a Service in another namespace is name.namespace, and the full form works from anywhere. Use the full form when testing with BusyBox’s nslookup; with a short name it also tries the other search domains and prints their NXDOMAIN answers, and on some networks a host resolver will invent an answer for a name like db-0.db before CoreDNS is consulted, which happened once during this run and is a good reason to prefer the full form for diagnosis.

Two DNS facts that matter operationally: the ClusterIP is stable for the life of the Service, so DNS caching is harmless; and a pod that cannot resolve any name has a resolv.conf or CoreDNS problem, while a pod that resolves kubernetes.default but not api has a Service problem.

The Service and its EndpointSlice#

kubectl get svc api -o yaml | sed -n '/^spec:/,$p'   # trimmed below to the fields that matter
kubectl get endpointslices -l kubernetes.io/service-name=api
kubectl get endpointslices -l kubernetes.io/service-name=api \
  -o jsonpath='{range .items[0].endpoints[*]}{.addresses[0]} ready={.conditions.ready} node={.nodeName}{"\n"}{end}'

spec:
  clusterIP: 10.96.223.236
  ports:
  - port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app: api
  type: ClusterIP

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                             AGE
api-gbt9q   IPv4          3000    10.244.2.42,10.244.1.33,10.244.2.55   20m

10.244.2.42 ready=true node=k8s-ops-lab-worker2
10.244.1.33 ready=true node=k8s-ops-lab-worker
10.244.2.55 ready=true node=k8s-ops-lab-worker2

Read the three pieces together. port: 80 is what clients connect to on the ClusterIP; targetPort: 3000 is the container port each endpoint receives on; the slice lists pod IPs (from the pod network, 10.244.x.x here, not the Service range) with their readiness. The EndpointSlice is the object to check first when a Service misbehaves, because it is the only one that shows what the Service actually resolved to.

Mistake one: the selector matches nothing#

Part 3 reproduced this. A selector of app: web against pods labelled app: api leaves the slice empty (endpoints=null) and connections fail in milliseconds:

curl: (7) Failed to connect to api port 80 after 10 ms: Could not connect to server

kubectl get pods -l <the Service's selector> is the one-line test: if it returns nothing, the Service has nothing. Label typos, a Deployment whose template labels drifted from its Service, and a Service copied from another app are the usual sources.

Mistake two: targetPort does not match the container#

The pods listen on 3000. Point targetPort at 80 and the slice is full, DNS works, and every connection is refused by the pod itself:

kubectl patch svc api -p '{"spec":{"ports":[{"port":80,"targetPort":80}]}}'
kubectl run curl --rm -i --restart=Never --image=curlimages/curl:8.14.1 -- sh -c 'curl -sS -m 3 http://api/version; echo "exit=$?"'
kubectl get endpointslices -l kubernetes.io/service-name=api -o jsonpath='ports={.items[0].ports}{"\n"}'
kubectl patch svc api -p '{"spec":{"ports":[{"port":80,"targetPort":3000}]}}'

curl: (7) Failed to connect to api port 80 after 2 ms: Could not connect to server
exit=7
ports=[{"name":"","port":80,"protocol":"TCP"}]

The client’s error is the same as for an empty selector, which is why it does not tell you which mistake you made; the EndpointSlice does. Its ports field is the target port (80 here, wrong), and its endpoints are present (right). Endpoints present plus connection refused means the pods are not listening where the Service sends traffic: compare targetPort with the container’s containerPort and with what the process actually binds (kubectl exec … -- netstat -tln or the application’s start-up log). Naming the container port (ports: [{ containerPort: 3000, name: http }]) and using targetPort: http avoids the number drifting.

Readiness and the slice#

Part 2 showed a pod with a failing readiness probe listed in its slice with ready=false. That is the mechanism by which a rolling update never sends traffic to a pod that is still starting, and by which a pod can take itself out of rotation (fail readiness when a dependency is down) without being restarted. If a Service has endpoints but they all say ready=false, the problem is the probe, not the network.

Headless Services: addresses for each pod#

clusterIP: None makes a headless Service: no virtual IP, and DNS returns the pod addresses directly. Part 1 created one for the StatefulSet, which is the main use:

kubectl run dns --rm -i --restart=Never --image=busybox:1.37 -- \
  sh -c 'nslookup db.default.svc.cluster.local; nslookup db-0.db.default.svc.cluster.local'

Name:      db.default.svc.cluster.local
Address:   10.244.1.6
Name:      db.default.svc.cluster.local
Address:   10.244.2.7

Name:      db-0.db.default.svc.cluster.local
Address:   10.244.2.7

Two answers for the Service name (one per pod, no load balancing by kube-proxy, the client picks), and a stable per-pod name db-0.db that follows the pod when it is recreated. Use headless Services for anything where the client needs to talk to a specific replica: databases, brokers, leader election.

Service types in one paragraph#

ClusterIP (the default) is reachable inside the cluster only, and is what every example above uses. NodePort additionally opens the same port on every node’s address, which is how kind and bare-metal clusters get traffic in without a cloud load balancer. LoadBalancer asks the cloud provider for an external address and points it at the NodePort. None of them understands HTTP; for host- and path-based routing and TLS termination there is Ingress.

Ingress, in outline#

An Ingress object declares HTTP routing rules (host: api.example.com, path: /, backend Service api port 80), and an ingress controller (ingress-nginx, Traefik, a cloud controller) watches those objects and configures a real proxy accordingly. The Gateway API is the newer, more expressive replacement, with Gateway and HTTPRoute resources and the same controller model.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

No ingress controller was installed on the lab cluster and this object was not applied; without a controller an Ingress is an inert record. When one is present, the same debugging order applies with one hop added: the controller’s own pod logs show what it did with the rule, and the Service behind it is checked exactly as above.

Security Considerations#

• NodePort and LoadBalancer Services expose ports outside the cluster. On a lab cluster that is convenient; on a shared network it is an unauthenticated entry point.
• The headless Service leaks pod IPs and per-pod names through DNS, which is its purpose. Keep the database’s own authentication on; DNS is not access control.
• Probe endpoints and /version endpoints answer anyone who can reach the pod. Do not put secrets or internal hostnames in them.

Troubleshooting#

Conclusion#

A request crosses four hops, and each has one object that shows its state: resolv.conf and CoreDNS for the name, the Service for the mapping, the EndpointSlice for the destinations and their readiness, the pod for the port it actually listens on. Read them in that order and the two common mistakes, a selector that matches nothing and a targetPort that matches nothing, are a one-line fix each. Part 5 changes how many destinations there are, by hand and automatically.

References#

• Kubernetes: Service
• Kubernetes: EndpointSlices
• Kubernetes: DNS for Services and Pods
• Kubernetes: Ingress
• Gateway API
• Kubernetes: debug Services