nginx Active Health Checks Send Pod IPs as Host Headers — Here’s Why Your Backend Returns 502s

The Symptom: Intermittent 502s That Only Appear on Health-Check Traffic

The maddening part of this failure mode is that your backend looks healthy by every obvious metric. You curl the pod directly, you hit the service endpoint, everything returns 200. Real user traffic flows through fine — until nginx quietly decides the upstream is down and starts dropping requests entirely. The health check is failing, the pod gets ejected from the upstream pool, and nginx has no way to distinguish between “the pod crashed” and “the pod rejected a request with the wrong Host header”. Both look like an unavailable upstream from nginx’s perspective.

The specific pattern: active health checks fire against the pod IP directly — something like 10.244.3.17:8000 — and nginx sends that raw IP as the Host header. Your backend (uvicorn, vLLM, Ollama’s HTTP layer) sees a request for host 10.244.3.17, matches it against its configured virtual host list, finds nothing, and either returns a 400/444/503 or closes the connection. Nginx logs this as upstream failure, marks the peer down, and your legitimate traffic hits the dead-peer list.

The nginx error log will tell you exactly which variant of failure you’re dealing with if you read it carefully. Three messages to watch for:

• connect() failed (111: Connection refused) — the pod is genuinely unreachable. Network issue, container crashed, port not open. This is not the host-header problem.
• upstream sent invalid header — the upstream accepted the TCP connection and responded with something HTTP-shaped but malformed or with a status nginx considers fatal for health checks. A 400 Bad Request from a backend that rejected the unknown Host value lands here. This is the host-header problem.
• recv() failed (104: Connection reset by peer) — the upstream connected, then slammed the socket shut before completing an HTTP response. Some uvicorn configurations do exactly this when a request hits no matching route: connect, read headers, reject at the routing layer, RST the connection. Also the host-header problem.

The last two are frequently misread as pod instability. Engineers restart pods, increase resource limits, file tickets about flaky networking — none of it helps because the pod is working perfectly. It’s rejecting a request it was never configured to accept.

Self-hosted AI backends are particularly exposed to this because they tend to be opinionated about hostname matching in a way that managed cloud load balancers paper over. A vLLM server started with --host 0.0.0.0 but serving via a named endpoint, an Ollama instance behind a reverse proxy chain, a FastAPI app with uvicorn where --root-path and host validation are configured explicitly — these all commonly validate the Host header as a first-pass filter. Cloud teams using AWS ALB or GCP Cloud Load Balancing usually have that header rewritten automatically upstream before it ever reaches the backend. If you’re running your own nginx-to-pod path without a managed layer in between, the raw pod IP lands in the Host header, and strict backends refuse the connection on principle.

What nginx’s Active Health Check Module Actually Does

The behavior that causes most of the confusion: when nginx fires an active health check probe, it constructs a synthetic HTTP request from scratch — no client, no original headers, nothing inherited from real traffic. The Host header in that synthetic request gets populated with whatever address nginx stored when it resolved the upstream server entry. For a Kubernetes pod backing a service, that’s almost always a raw pod IP: 10.244.3.17, 172.20.8.4, something in that class. Your backend may work perfectly for every real client request and still fail health checks because it validates the Host header on arrival and a bare IP doesn’t match any expected virtual host.

This is the fundamental split between active and passive checks. Passive health checks don’t generate any requests — they ride along with real traffic and mark upstreams as failed when enough real responses come back with errors or timeouts. Those real requests already carry a Host header set by the client (or by an earlier proxy stage), so they pass backend validation naturally. Active checks get none of that context. They’re fired by nginx’s internal health check timer, built from the upstream block configuration alone, and they carry only what nginx synthesizes — which by default means a Host of the literal server address string.

The relevant directive stack looks like this in the nginx Plus / ngx_http_upstream_hc_module model:

upstream my_backend {
    zone backend_zone 64k;  # required for active checks in nginx Plus

    server 10.244.3.17:8080;  # pod IP — this string becomes the default Host header
    server 10.244.3.22:8080;

    keepalive 16;  # persistent connections to upstreams; does NOT affect synthetic request headers
}

server {
    location /api/ {
        proxy_pass http://my_backend;

        health_check interval=5s fails=2 passes=3 uri=/healthz;
        # ^ fires GET /healthz HTTP/1.1 with Host: 10.244.3.17 — no override unless you add one
    }
}

The keepalive directive is worth calling out specifically because it’s a common red herring during troubleshooting. Operators see a connection-reuse setting and assume it might affect how requests are formatted — it doesn’t. keepalive controls whether nginx maintains idle connections to upstream workers; it has zero influence on the headers placed inside those connections. The directives that do influence what the synthetic request looks like are health_check‘s own parameters (particularly uri and the match block it can reference), and any proxy_set_header directives in the location block — though whether those apply to health check traffic depends on the module version and configuration scope, which is exactly where the 502s start hiding.

One thing the nginx Plus docs understate: the server line inside upstream can be a hostname instead of an IP, and if it is, the resolved hostname becomes the default Host value. That sounds like an escape hatch, but in Kubernetes it rarely helps — pod IPs get registered directly in endpoint slices, so whatever feeds your upstream block (a DNS-based service discovery integration, manual config generated from the Endpoints API, or something like nginx Ingress Controller’s templating) typically emits raw IPs. The mismatch between what the health checker sends and what strict virtual-host validation expects is structural, not a misconfiguration you can fix by tweaking one knob.

Diagnosing the Root Cause: Tracing the Bad Host Header

The deceptive part of this failure mode is that nginx’s default combined log format makes a backend returning 400 Bad Request look identical to a TCP connection failure. Both show up as 502 in your access log. To actually see what’s happening, you need to add $upstream_status to your log format so the backend’s real response code survives into the nginx logs:

log_format upstream_debug '$remote_addr - $remote_user [$time_local] '
                          '"$request" $status $body_bytes_sent '
                          'upstream_status=$upstream_status '
                          'upstream_addr=$upstream_addr';

access_log /var/log/nginx/access.log upstream_debug;

After reloading nginx, a strict backend rejecting the IP-valued Host header will show upstream_status=400 (or 421) alongside the 502 your client sees. That single field is the difference between “investigate the network” and “investigate the HTTP layer.” Without it, you can spend an hour chasing phantom pod connectivity issues that don’t exist.

Once you suspect the Host header, replay exactly what nginx sends using curl from inside the cluster. Either kubectl exec into a debug pod or use tcpdump on the pod’s veth interface to capture the actual bytes. The curl replay is faster to iterate:

# Simulate what nginx active health check sends by default
curl -v -H 'Host: 10.244.3.17' http://10.244.3.17:8080/health

# Expected output from a strict backend:
# < HTTP/1.1 400 Bad Request
# < content-type: text/plain
# < x-reason: invalid host header

If that curl returns 400 or 421 and curl -v http://10.244.3.17:8080/health (which sends the IP as Host anyway via curl's default behavior) also fails, try explicitly passing the correct virtual hostname: curl -v -H 'Host: my-service.internal' http://10.244.3.17:8080/health. If that succeeds, you've isolated the exact problem — the backend cares about the Host value, and nginx is sending the pod IP.

For vLLM and Ollama specifically: vLLM's OpenAI-compatible server started with --host 0.0.0.0 doesn't enforce Host headers in its vanilla configuration — it'll respond to health checks regardless of what Host value nginx sends. The failure reappears if someone wraps vLLM behind a FastAPI middleware layer or adds request validation. Ollama's built-in HTTP server is similarly permissive. The dangerous scenario is when a platform team adds an API gateway or auth proxy in front of either, and that proxy enforces Host header validation without anyone updating the nginx health check config. To confirm the hypothesis without modifying nginx config at all, temporarily add a debug header on the backend side and inspect what comes back:

# In a FastAPI middleware or nginx location block on the backend side:
add_header X-Received-Host $http_host always;

# Then hit the health endpoint directly through nginx's upstream path
# and check the response headers — if you see:
# X-Received-Host: 10.244.3.17
# the pod IP is confirmed as the Host value nginx is sending

That echo technique is useful when you can't easily capture traffic but you can modify backend config temporarily. The always flag matters here — without it, nginx won't include the header on 4xx responses, which is exactly the response class you're trying to inspect.

The Fix: Forcing a Correct Host Header on Health-Check Requests

The instinct most people reach for first is wrong, and it's worth knowing exactly why before you write any config. proxy_set_header Host $upstream_addr; placed in the same location block as your traffic proxy does nothing for nginx Plus active health checks — those requests are generated internally by the nginx worker, not routed through your location's proxy directive chain. The header override simply doesn't apply. You need a different mechanism depending on whether you're running nginx Plus or open-source nginx, and they diverge significantly.

nginx Plus: Match Blocks + Dedicated Probe Location

nginx Plus active health checks use a health_check directive that optionally references a match block for response validation. The correct pattern for injecting a Host header is to route the synthetic probe through a separate internal location that sets its own proxy_set_header, then point the health check at that location via the uri parameter. The headers block some people assume exists inside health_check — it doesn't. Here's the working structure:

http {
    # Define what a healthy response looks like
    match ai_backend_healthy {
        status 200;
        header Content-Type ~ "application/json";
        body ~ '"status"\s*:\s*"ok"';
    }

    upstream ai_inference {
        zone ai_inference 64k;
        server 10.0.1.10:8080;
        server 10.0.1.11:8080;

        # Active health check fires against /internal-health location below
        health_check uri=/internal-health interval=5s fails=2 passes=1 match=ai_backend_healthy;
    }

    server {
        listen 443 ssl;
        server_name gateway.internal;

        # Real traffic — Host header comes from the client or your override
        location /api/ {
            proxy_pass http://ai_inference;
            proxy_set_header Host api.internal;
        }

        # Synthetic probe location — nginx Plus health checker hits this internally
        location /internal-health {
            internal;  # blocks external access entirely
            proxy_pass http://ai_inference/health;
            # This is the override that actually reaches the backend
            proxy_set_header Host api.internal;
        }
    }
}

The internal directive on the probe location means no external client can call /internal-health directly — only nginx's internal health checker and rewrite/try_files chains can reach it. Without that flag you've just exposed your raw health endpoint to the internet.

Open-Source nginx: Split the Location, Own the Header

Open-source nginx has no active health checks — the module was stripped to Plus long ago. What you get is passive health checks via proxy_next_upstream, which marks a peer down after real traffic failures. That's actually useful for the Kubernetes case (more below), but if you're self-hosting and genuinely need synthetic probes, the cleanest workaround is a dedicated location block that handles only health traffic and sets its own headers independently of the main traffic path:

upstream ai_inference {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;

    # Passive only — marks peer down after proxy_next_upstream triggers
    # No active health_check directive available in open-source builds
}

server {
    listen 80;
    server_name gateway.internal;

    location /api/ {
        proxy_pass http://ai_inference;
        proxy_set_header Host api.internal;
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
    }

    # External health probe endpoint — called by your load balancer's health check
    # or a cron/watchdog process, NOT by nginx itself
    location /probe/health {
        # Proxy to the upstream group, but force the Host header the backends expect
        proxy_pass http://ai_inference/health;
        proxy_set_header Host api.internal;  # critical — overrides the default $proxy_host
        proxy_connect_timeout 2s;
        proxy_read_timeout 3s;
        access_log off;
    }
}

If you're triggering this probe from an external system (an ALB, a Kubernetes liveness check against nginx itself, or a cron hitting /probe/health), add an allow/deny ACL or move it behind a different listen port so it's not publicly reachable. The proxy_connect_timeout and proxy_read_timeout values here are tight by design — a health probe that hangs for 30 seconds is worse than a fast failure.

Kubernetes: Skip nginx Active Checks Entirely

If you control the Pod spec — and in most self-hosted Kubernetes setups you do — the cleanest resolution to the Host header problem is to stop using nginx active health checks at all for readiness signaling. Let Kubernetes own that responsibility. Configure a readinessProbe directly on the container, and switch nginx to passive-only health checking. The kubelet sends HTTP probes directly to the Pod's port using the Pod IP, but critically, Kubernetes sets the Host header to the Pod IP by default too — so if your backend validates Host, you still need to handle this at the probe level. The better move is to expose a separate probe port that skips Host validation:

# Pod spec fragment — your inference container
containers:
- name: ai-inference
  image: your-registry/ai-server:1.4.2
  ports:
  - containerPort: 8080   # main traffic
  - containerPort: 9090   # probe port, bypasses Host validation in app config
  readinessProbe:
    httpGet:
      path: /healthz
      port: 9090           # hits the permissive probe port directly
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 2
    successThreshold: 1
  livenessProbe:
    httpGet:
      path: /healthz
      port: 9090
    initialDelaySeconds: 20
    periodSeconds: 10

On the nginx side in this topology, configure only passive health checks and let Kubernetes endpoints controller remove unready pods from the Service before nginx ever tries to proxy to them:

upstream ai_inference {
    server ai-inference-svc.default.svc.cluster.local:8080;

    # Passive: nginx marks a peer down after observed failures in real traffic
    # Kubernetes readiness probes keep unhealthy pods out of the Service endpoints
    # so nginx rarely sees those pods at all
}

server {
    location /api/ {
        proxy_pass http://ai_inference;
        proxy_set_header Host api.internal;

        # Retry on genuine failures, but don't hammer a pod that's mid-startup
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_timeout 5s;
        proxy_next_upstream_tries 2;
    }
}

The Kubernetes approach sidesteps the Host header problem structurally — you're not fighting nginx's health check behavior, you're removing the layer that causes it. The trade-off is that passive-only checking in nginx means the first failed real request to a degraded-but-still-listed pod gets eaten before failover triggers. For stateless inference endpoints where a retry is cheap, that's acceptable. For anything with side effects or strict SLOs, you want nginx Plus active checks with the explicit Host injection shown above.

Config Examples: nginx Open-Source vs nginx Plus vs Ingress-nginx

The fastest way to confirm you're hitting the Host-header 502 — before touching any config — is to curl the backend directly with an explicit Host header set to the IP vs. the expected hostname. If the IP gets a 4xx or immediate close and the hostname gets a 200, you've isolated it. Now here's exactly how to fix it in each deployment model.

Open-Source nginx (ngx_http_proxy_module)

The fix belongs in a dedicated location block scoped only to health checks. Putting proxy_set_header Host in the upstream-wide server block will override the header for all proxied traffic, which is usually not what you want — your application requests should still carry the original client Host. The pattern is to split health-check traffic into its own location and set the header only there:

upstream api_backend {
    # Use keepalive so health-check connections don't thrash sockets
    keepalive 32;

    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
}

server {
    listen 80;
    server_name api.example.com;

    # Normal application traffic — Host header passes through from client
    location / {
        proxy_pass http://api_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Health-check location nginx polls internally
    # Active checks require nginx Plus; open-source uses this as a passive
    # probe target or a dedicated poller location behind an internal allow
    location /healthz {
        allow 127.0.0.1;
        deny all;

        proxy_pass http://api_backend;

        # This is the fix: send the hostname the backend validates,
        # not the upstream IP nginx defaults to
        proxy_set_header Host api.example.com;

        # If the backend speaks HTTPS internally, these two lines must
        # travel together — see the HTTPS section below
        # proxy_ssl_server_name on;
        # proxy_ssl_name api.example.com;

        proxy_connect_timeout 2s;
        proxy_read_timeout 5s;
    }
}

Note the allow 127.0.0.1; deny all; guard. Without it, you're exposing an unthrottled health endpoint to the public, which is a separate problem. If you're using the commercial nginx Plus active health check directive (health_check inside the upstream block), the proxy_set_header Host in the matching location still applies — nginx Plus inherits it from the location context that owns the health_check directive.

Ingress-nginx (Kubernetes)

Ingress-nginx doesn't let you edit the generated nginx config directly — that file is owned by the controller and gets overwritten on every reconcile. The correct lever is the nginx.ingress.kubernetes.io/proxy-set-headers annotation, which points at a ConfigMap that lists headers to inject. Pair that with custom-http-errors to surface backend validation rejections distinctly from infrastructure 502s, otherwise you'll spend time chasing the wrong layer:

# Step 1: ConfigMap with the header override
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-headers
  namespace: ingress-nginx   # must match controller namespace
data:
  # Key is the header name, value is the desired value
  Host: "api.example.com"

---
# Step 2: Ingress resource referencing that ConfigMap
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: default
  annotations:
    # Points at namespace/configmap-name
    nginx.ingress.kubernetes.io/proxy-set-headers: "ingress-nginx/custom-headers"

    # 421 and 400 are common status codes strict backends return on
    # Host mismatch — listing them here routes them to a custom error
    # page instead of silently returning generic 502 to clients
    nginx.ingress.kubernetes.io/custom-http-errors: "400,421,502,503"

    # For HTTPS backends — controller won't negotiate SNI correctly without this
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 443

One thing the ingress-nginx docs understate: the ConfigMap must exist before the Ingress is created, or the controller will log a sync error and silently skip the header override entirely. The Ingress object will show as Synced in kubectl get ingress, but the header won't be applied. Check controller logs with kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --tail=50 to confirm the ConfigMap was actually loaded.

When the Backend Uses HTTPS Internally

This is where two separate bugs compound into one opaque 502. Fixing the Host header alone is not enough if the upstream listens on HTTPS, because TLS SNI is negotiated before the HTTP headers are sent. nginx will use the upstream IP as the SNI hostname during the TLS handshake unless you explicitly override it. The backend's TLS library rejects the handshake (cert CN doesn't match an IP), and you get a 502 before your Host header fix ever comes into play:

# Open-source nginx — these two directives must appear together
location /healthz {
    proxy_pass https://api_backend;   # note https://

    proxy_set_header Host api.example.com;

    # Tells nginx to use the value below as the SNI hostname in the TLS handshake
    proxy_ssl_server_name on;

    # Explicitly set the SNI name — defaults to proxy_pass host, which for
    # named upstreams resolves to a peer IP at connection time
    proxy_ssl_name api.example.com;

    # Only disable verification if the backend uses a self-signed cert AND
    # you've verified the host via another mechanism (mTLS, VPN, etc.)
    # proxy_ssl_verify off;  # avoid unless you have a real reason
}

For ingress-nginx with HTTPS backends, nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on" does the same job. Without it, even after setting the Host header ConfigMap correctly, you'll see TLS handshake failures in controller debug logs (SSL_do_handshake() failed) that look indistinguishable from the pod being down. The operational takeaway: any time you're chasing a 502 against an HTTPS upstream, check both the HTTP Host header and the TLS SNI hostname as two separate failure points.

What to Monitor After the Fix

The fix you applied to the host header is not self-verifying — the most insidious failure mode is a backend that accepts the corrected header for normal traffic but still rejects health checks through a second validation layer you haven't found yet. Before you declare the issue resolved, instrument the logging layer first. Add $upstream_status and $upstream_response_time to your log_format block:

log_format upstream_detail '$remote_addr - $remote_user [$time_local] '
    '"$request" $status $body_bytes_sent '
    'ups_status=$upstream_status '       # actual response code from backend, not the 502 nginx synthesizes
    'ups_rt=$upstream_response_time '    # latency per upstream; comma-separated if retried
    '"$http_referer" "$http_user_agent"';

access_log /var/log/nginx/access.log upstream_detail;

Without $upstream_status, a backend returning 400 or 421 looks identical in the access log to a refused TCP connection — both surface as 502 to the client. With it, you'll see ups_status=421 in the line, which tells you immediately that the backend is still enforcing strict host validation on the health check path even after you patched the proxy_set_header Host directive. The 421 (Misdirected Request) code is particularly common with HTTP/2 backends and TLS-terminating upstreams that check SNI against the Host header.

For the health check thresholds themselves, be conservative with AI inference backends. Ollama loading a model cold can hold a connection for several seconds before responding — a default fails=1 passes=1 configuration will evict that upstream almost immediately and then immediately re-admit it, causing flapping rather than a clean recovery. A saner baseline:

upstream ollama_backend {
    zone ollama 64k;
    server 127.0.0.1:11434;

    health_check interval=10s fails=3 passes=2 uri=/api/tags;
    # fails=3: tolerate two slow model-load responses before marking down
    # passes=2: require two consecutive successes before marking back up
    # /api/tags is lightweight — doesn't trigger a model load
}

If you have Prometheus scraping via nginx-prometheus-exporter, the metric to watch is nginx_upstream_peers_health with its state label. A peer toggling between healthy and unhealthy faster than your interval cadence should allow is a hard signal that the host header fix is incomplete or there's a load balancer or WAF further upstream doing its own header validation. Rapid flapping here will look like random transient errors at the application layer, which makes root cause diagnosis painful without this metric in view.

For pipelines wired through n8n or a TypeScript automation engine, upstream flapping doesn't produce clean error messages — it produces failed webhook deliveries, timed-out HTTP nodes, and n8n executions that error with generic fetch failures rather than anything that points at nginx. Cross-reference the timestamps of nginx_upstream_peers_health state transitions directly against n8n's execution error log. If they correlate, the nginx layer is still the problem. If n8n errors persist after upstream state stabilizes, the issue has moved downstream — check your n8n credential timeouts and HTTP Request node retry settings. The Workflow Automation in 2026: n8n, Zapier, and Self-Hosted Pipelines guide covers how these components interact across a full self-hosted stack, which is useful context if you're still mapping which failure belongs to which layer.

---

---