Fix Prometheus Targets Down (Context Deadline Exceeded & Connection Refused)

1. Introduction

You open Prometheus, go to Status → Targets, and a row is red: DOWN, with an error like context deadline exceeded or connection refused. Your dashboards go flat and alerts either fire on missing data or — worse — go blind. A down target means Prometheus tried to scrape an endpoint and failed; the message on that target row is the single most useful clue, and this guide maps each one to a fix.

We'll cover the whole path: reading the target's error and labels, checking the exporter is actually serving /metrics, fixing wrong ports and schemes, untangling relabeling that dropped or rewrote the address, and the Kubernetes-specific cases where a ServiceMonitor selector or a NetworkPolicy is the real problem.

2. What "Target DOWN" Means

Prometheus scrapes each target on an interval by making an HTTP GET to its /metrics endpoint. The target's health is up (1) or down (0), and the row shows the last error. The common errors and what they imply:

3. Common Causes

• Wrong port or path — scraping :8080 when the exporter serves :9100/metrics
• The exporter/app isn't running or isn't exposing /metrics
• Scheme mismatch — http vs https, or missing tls_config
• A firewall, security group, or Kubernetes NetworkPolicy blocking Prometheus from the target
• Relabeling rules that dropped the target or rewrote __address__ incorrectly
• Scrape timeout too low for a slow exporter (kube-state-metrics on big clusters)
• Kubernetes: a ServiceMonitor/PodMonitor selector that doesn't match, or the wrong port name
• Auth required (bearer token, basic auth) but not configured in the scrape job

4. Step-by-Step Diagnosis and Fix

Step 1: Read the exact error and the resolved address

Status → Targets shows the final __address__ Prometheus used and the last error. Query the built-in up metric to see which jobs are affected.

# In the Prometheus expression browser:
up == 0
# Group by job to see scope:
count by (job) (up == 0)

# Reachable via API too:
curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.health!="up") | {scrapeUrl, lastError}'

Step 2: Reproduce the scrape by hand

Curl the exact URL Prometheus is using, ideally from the Prometheus pod/host so the network path matches.

# From the Prometheus host/pod:
curl -v http://TARGET_IP:9100/metrics | head

# In Kubernetes, exec into the Prometheus pod:
kubectl exec -n monitoring deploy/prometheus-server -- \
  wget -qO- http://TARGET_IP:9100/metrics | head
# connection refused -> wrong port / not listening
# hang then timeout   -> NetworkPolicy / firewall

Step 3: Fix wrong port, path, or scheme in the scrape config

scrape_configs:
  - job_name: node
    scheme: http                  # use https + tls_config for TLS targets
    metrics_path: /metrics        # default; set if the exporter differs
    static_configs:
      - targets: ["10.0.1.10:9100"]   # host:PORT must match the exporter

# For a TLS endpoint with a private CA:
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/ca.crt
      insecure_skip_verify: false

# Validate and reload without restarting
promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload   # requires --web.enable-lifecycle

Step 4: Check relabeling didn't drop or rewrite the target

Relabeling is the most common "phantom" cause — the target vanishes or its __address__ becomes wrong. Compare Service Discovery vs Targets in the UI.

# A relabel that rewrites the scrape address to a pod's declared port:
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_container_port_number]
    action: keep
    regex: "9100"                 # too strict a keep = targets dropped
  - source_labels: [__address__, __meta_kubernetes_pod_ip]
    action: replace
    regex: (.+):\d+
    replacement: ${1}:9100
    target_label: __address__

Step 5: Fix Kubernetes ServiceMonitor / PodMonitor matching

# The ServiceMonitor's port must match the Service port NAME, not number
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app
  labels:
    release: kube-prometheus-stack   # must match Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: my-app                    # must match the Service's labels
  endpoints:
    - port: metrics                  # the NAMED port on the Service
      interval: 30s

# Why isn't my ServiceMonitor picked up? Check the selector Prometheus uses:
kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.serviceMonitorSelector}'; echo
# The ServiceMonitor's labels must satisfy this selector, and the Service
# must expose a named port matching endpoints[].port.

Step 6: Open the network path and raise timeouts

# context deadline exceeded from a NetworkPolicy? Allow Prometheus in:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
spec:
  podSelector: { matchLabels: { app: my-app } }
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { name: monitoring } }
      ports:
        - protocol: TCP
          port: 9100

# Slow exporter? give it more time (must be <= scrape_interval)
scrape_configs:
  - job_name: kube-state-metrics
    scrape_interval: 60s
    scrape_timeout: 30s

5. Verification Steps

# Target should flip to UP; confirm with the up metric
# up{job="node"} == 1

curl -s http://localhost:9090/api/v1/targets | \
  jq '.data.activeTargets[] | select(.labels.job=="node") | .health'
# "up"

# Confirm samples are actually landing
curl -s 'http://localhost:9090/api/v1/query?query=up{job="node"}' | jq '.data.result'

6. Common Mistakes

• Scraping the app's service port instead of the exporter's metrics port
• Setting scrape_timeout greater than scrape_interval (invalid config)
• Using the port number in a ServiceMonitor when it wants the port name
• Forgetting --web.enable-lifecycle, then wondering why /-/reload 404s
• Ignoring relabeling — assuming a missing target is a network issue when a keep rule dropped it

7. Prevention Tips

• Run promtool check config in CI so a broken scrape config never ships
• Alert on up == 0 and on scrape duration approaching scrape_timeout
• Standardize exporter ports and named Service ports so ServiceMonitors match predictably
• Label-allow the monitoring namespace in NetworkPolicies as a default
• If targets fail with no such host, it's DNS — cross-check Kubernetes DNS resolution
• For the Kubernetes metrics API (HPA/kubectl top), that's a different component — see Metrics Server not working

8. FAQ

9. Summary