Skip to content
AppDues
Log in
Contact salesSign up

Fixing 400 Errors After Moving from NGINX to Envoy

When a proxy swap produces a sudden rise in 400 responses on POST endpoints, the cause is often not the application. This guide walks through a published case study from a core infrastructure team: how they narrowed the spike to Envoy, captured rare failing requests, and found that clients were closing connections before sending their payloads. It ends with the configuration change that removed the errors and the trade-off that change introduces.

The symptom: a spike in 400s on POST endpoints

The increase was roughly two orders of magnitude on one endpoint, though it remained a very small share of total traffic. The same underlying problem also showed up as a latency regression, because affected requests hung until a 30-second timeout. The spike lined up with the Envoy deployment rather than with an application release. A rollback was considered, but the failing requests genuinely lacked data, and the volume of successful requests was unchanged, so the team kept investigating instead of reverting.

First checks: where did the payload go?

Backend logs showed the POST payload arriving as null. That left two possibilities: Envoy never received the body, or it received it and failed to forward it. A Lua script dumped the request body inside Envoy and also found nothing, which was inconclusive because the same parsing problem could explain both the missing body and the missing forward. The next step had to be observing the raw traffic.

Capturing rare failures at scale

At close to 1 million requests per minute across roughly 100 servers at peak, catching a rare failure required sampling. A load balancer listener rule redirected a percentage of traffic for the affected endpoint into a capture environment. To avoid installing debugging tools on production hosts, a separate container joined the target container's network namespace and ran tcpdump there. The capture rotated files every 10 minutes across a full day, recording both the load-balancer-to-Envoy leg and the Envoy-to-backend leg.

Why a sidecar container instead of installing tools

Docker containers are isolated through Linux kernel namespaces, including PID, NET, IPC, MNT, and UTS. The NET namespace is the one that matters for packet capture. Using the --net container:ID option lets a debugging container share the target's network stack, so tcpdump and similar tools can observe its traffic. Keeping those tools out of the production image preserves a single responsibility per container and avoids adding vulnerabilities to the production surface.

Reading the packet captures

A normal POST appears as two segments: one flagged PSH and ACK that holds the HTTP headers, then one that holds the body. Envoy acknowledges the pair, handles the request, and replies. The failing traces diverged at the second segment: it was absent, and a FIN arrived from the load balancer to end the connection. With the body still outstanding, the backend sat until timeout and answered 400 because the request was malformed. Access logs from the load balancer, located by the X-Amzn-Trace-Id header, matched this pattern with a 460 status, which means the client closed the connection before the idle timeout elapsed.

Why NGINX tolerated this and Envoy did not

NGINX buffers requests by default and waits for the full request before proxying it. Envoy streams requests, so a partial request reaches the backend before the body is complete. Streaming can reduce latency, but it offers less protection for a backend that cannot handle incomplete input. For a fragile Flask service, buffering is the safer choice.

The fix and its trade-off

Enabling Envoy's buffer filter made Envoy wait for the complete request before forwarding it, and the 400 errors disappeared from the metrics. The filter requires a fixed buffer size and cannot switch to streaming for larger payloads. Requests above the configured limit receive a 413 status instead. The chosen limit did not fail any requests and did not increase memory consumption.

What remained open

The investigation also surfaced a client-side issue: some clients send incomplete requests. The write-up notes an ongoing investigation into that aspect, and into why Flask does not reject requests with a non-zero Content-Length header but no actual payload, since those should never reach application code.

Traffic capture command reference
Command
docker run -ti --net container:72ae33239d01 nicolaka/netshoot tcpdump -G 600 -W 144 -s 0 -i eth0 -w elb_envoy-%Y-%m-%d_%H.%M.%S.pcap port {{PORT}}
docker run -ti --net container:72ae33239d01 nicolaka/netshoot tcpdump -G 600 -W 144 -s 0 -i eth0 -w envoy_backend-%Y-%m-%d_%H.%M.%S.pcap port {{PORT}}

The two commands capture the load-balancer-to-Envoy leg and the Envoy-to-backend leg respectively. Replace {{PORT}} with the port in use for each leg.

Frequently asked questions

Why does Envoy return 400 instead of 413 for a partial request?

A 400 means the request was malformed, which is what the backend sees when the payload never arrives. A 413 comes from the buffer filter when a complete request exceeds the configured buffer size. They indicate different problems: incomplete input versus oversized input.

What does a 460 status from the load balancer mean?

It means the load balancer received a request from a client, but the client closed the connection before the idle timeout elapsed. In this investigation, that confirmed the client was ending the connection mid-request.

Can the Envoy buffer filter stream requests larger than its limit?

No. The buffer size is fixed, and requests above it receive a 413 response rather than being streamed. The limit must be chosen to fit the largest expected payload.