Fix Kubernetes Service Not Reachable: Step-by-Step Troubleshooting Guide

1. Introduction

A Kubernetes Service that isn't reachable can mean a dozen different things depending on where you're connecting from: inside the cluster (pod-to-service), from a node, or from outside the cluster (NodePort, LoadBalancer, Ingress). Each access path has its own failure modes, and the troubleshooting approach differs significantly between them.

This guide focuses on the most common scenarios: a Service that returns connection refused, times out, or gets no response. For cases where DNS resolution is the issue, see Fix DNS Resolution Issues in Kubernetes. For pod-to-pod communication without a Service, see Fix Internal Service Communication Failure.

2. What Service Not Reachable Means

A Kubernetes Service is a virtual IP and port that load-balances to a set of pods. Traffic hits the ClusterIP, and kube-proxy (via iptables or IPVS) redirects it to a ready pod endpoint. If the Service has no endpoints, all connections go nowhere. If kube-proxy is broken, the iptables rules don't exist. If the pods aren't ready, they're excluded from endpoints.

3. Common Causes

• Service selector doesn't match any pod labels — endpoints are empty
• All matched pods are failing readiness probes — excluded from endpoints
• Service port doesn't match pod's containerPort (targetPort mismatch)
• NetworkPolicy blocking traffic between the source and the Service/pods
• kube-proxy is not running on the node making the request
• DNS resolution of the Service name is failing
• For external access: firewall, security group, or cloud LB misconfiguration
• For NodePort: the node firewall blocks the NodePort range (30000-32767)

4. Step-by-Step Diagnosis and Fix

Step 1: Check Service endpoints

# The single most important check
kubectl get endpoints <service-name> -n <namespace>

# ENDPOINTS shows pod IPs and ports that will receive traffic
# If <none>: the selector matches no ready pods

# Check what the selector is
kubectl get service <service-name> -n <namespace> -o yaml | grep -A 5 selector

# Check which pods match the selector
kubectl get pods -n <namespace> -l <key=value> --show-labels

# Check pod readiness (not-ready pods are excluded from endpoints)
kubectl get pods -n <namespace> -l <key=value>
# READY column should show 1/1 or N/N

Step 2: Test the Service from inside the cluster

# Run a debug pod in the same namespace
kubectl run debug --image=curlimages/curl --rm -it --restart=Never -n <namespace> --   curl -v http://<service-name>:<port>/

# Test by ClusterIP (bypasses DNS)
CLUSTER_IP=$(kubectl get service <service-name> -n <namespace> -o jsonpath='{.spec.clusterIP}')
kubectl run debug --image=curlimages/curl --rm -it --restart=Never --   curl -v http://$CLUSTER_IP:<port>/

# If ClusterIP works but name doesn't = DNS issue
# If ClusterIP fails = endpoints or kube-proxy issue

Step 3: Check targetPort configuration

# View Service port config
kubectl get service <service-name> -n <namespace> -o yaml | grep -A 10 ports

# Example of misconfiguration:
# ports:
# - port: 80
#   targetPort: 8080   <-- must match what the container ACTUALLY listens on

# Verify what port the container listens on
kubectl exec -it <pod-name> -n <namespace> -- ss -tlnp
# or: netstat -tlnp

# Fix: update Service targetPort to match
kubectl patch service <service-name> -n <namespace>   --type='json' -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]'

Step 4: Check NetworkPolicy

# List policies in both source and target namespaces
kubectl get networkpolicy -n <source-namespace>
kubectl get networkpolicy -n <target-namespace>

# A default-deny-all policy blocks all traffic unless explicitly allowed
# Allow traffic to the Service's pods:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-service-traffic
  namespace: <target-namespace>
spec:
  podSelector:
    matchLabels:
      app: <target-app>
  ingress:
  - from:
    - namespaceSelector: {}   # allow from all namespaces
    ports:
    - port: 8080
  policyTypes:
  - Ingress

Step 5: Fix external access (NodePort/LoadBalancer)

# For NodePort: confirm the port is open on the node
NODE_IP=$(kubectl get node -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}')
NODE_PORT=$(kubectl get service <service-name> -o jsonpath='{.spec.ports[0].nodePort}')
curl http://$NODE_IP:$NODE_PORT/

# For AWS LoadBalancer: check Security Group allows inbound on the Service port
# The node security group needs inbound on NodePort range (30000-32767) from the LB
# Check: aws ec2 describe-security-groups --group-ids sg-xxxxx

# For Ingress-based access, see:
# /fix-kubernetes-ingress-not-working.html

5. Verification Steps

# Service should have endpoints
kubectl get endpoints <service-name> -n <namespace>
# Should show pod IPs, not <none>

# Direct curl test from inside the cluster should succeed
kubectl run verify --image=curlimages/curl --rm -it --restart=Never -n <namespace> --   curl -sf http://<service-name>:<port>/health
# Expected: HTTP 200

6. Common Mistakes

• Not checking endpoints first — empty endpoints explains 95% of "Service not reachable" issues
• Testing from outside the cluster and expecting ClusterIP to work — ClusterIP is cluster-internal only
• Forgetting that readiness probe failures remove pods from endpoints — a pod can be Running but not in the endpoint list
• Missing namespace in the Service DNS name for cross-namespace calls — use <service>.<namespace>.svc.cluster.local
• Not checking security groups for NodePort/LoadBalancer external access

7. Prevention Tips

• Test Service connectivity as part of your deployment pipeline — a smoke test that curls the Service confirms the selector and targetPort are correct
• Use kubectl get endpoints as your first command when debugging connectivity — it immediately tells you whether the Service can reach any pod
• Label pods consistently and test selectors with kubectl get pods -l <selector> before creating Services
• For external traffic, consider using Ingress instead of NodePort — Ingress provides TLS and host-based routing with better security defaults

8. FAQ

9. Summary