When a reverse proxy sits in front of your application and has proxy_request_buffering on (nginx's default),
the proxy owns the TCP connection to the client independently of the TCP connection to the upstream.
The upstream can die mid-request and the client notices nothing — the proxy already holds the bytes.
This post covers exactly how that works, down to the kernel socket buffers, and builds a reproducible demo:
a Node.js upload server, an nginx proxy, a 500 MB transfer, and a kill -9 on the backend at the halfway point.
Client (browser / curl)
│
│ TCP conn A ← proxy manages this independently
▼
┌──────────────────────┐
│ nginx (edge proxy) │ ← buffers to disk: /var/cache/nginx/client_temp
│ Raspberry Pi / VPS │
└──────────────────────┘
│
│ TCP conn B ← proxy manages this separately
▼
┌──────────────────────┐
│ Upstream app │ ← can go down; proxy holds the upload
│ (Node.js / Immich) │
└──────────────────────┘
Two independent TCP connections. The proxy is not a transparent passthrough — it is a terminating proxy that decouples the client's session from the upstream's session.
When a POST /upload arrives, nginx checks proxy_request_buffering.
With the default value of on:
client_body_buffer_size
(default 8k/16k depending on platform) of in-memory buffer.client_body_temp_path
(default /var/lib/nginx/tmp/client_body on Debian-based systems).The client's TCP connection is being fully serviced. From the client's perspective, the transfer is progressing normally because nginx is ACK-ing every segment.
Only after the entire client body is buffered does nginx open TCP conn B to the upstream, reconstruct the HTTP request with the body read from the temp file, and stream it upstream.
This is why a 9 GB upload keeps “uploading at realistic speeds” through a proxy even if the upstream is offline for most of the transfer: the bytes are going to the proxy's disk, not to the application.
When the upstream comes back up:
The client receives a single contiguous HTTP response. It never saw a TCP reset. It never got a 502.
On Linux, every TCP socket has a receive buffer managed by the kernel.
net.core.rmem_default / net.ipv4.tcp_rmem control the sizes.
When nginx recv()s data from the client socket, it drains the kernel receive buffer into userspace.
As long as nginx keeps calling recv(), the kernel keeps ACK-ing the client's segments and the
client's sliding window stays open — the upload proceeds at full speed.
Check current buffer sizes:
sysctl net.ipv4.tcp_rmem # output: net.ipv4.tcp_rmem = 4096 131072 6291456 # min / default / max in bytes
sendfile and splice
When nginx forwards the buffered temp file to the upstream, it uses sendfile(2)
(or splice(2) on Linux when sendfile is off) to transfer data directly between
a file descriptor and a socket without a userspace copy. This is a kernel zero-copy path:
disk ──(DMA)──▶ kernel page cache ──(splice)──▶ socket send buffer ──▶ NIC
No data traverses userspace during the upstream relay. This is why the replay is fast even on a low-power device like a Raspberry Pi.
nginx creates the client body temp file with O_TMPFILE (or open() + unlink())
so the file is unlinked from the directory immediately after creation. The inode persists via a file descriptor
reference held by the nginx worker. If the worker dies, the kernel drops the fd, the reference count hits zero,
and the inode is reclaimed. There is no cleanup step needed.
On a Raspberry Pi with a 32 GB SD card and a 9 GB upload, you are writing 9 GB to the SD card for every upload.
This matters for SD card wear. In production setups, point client_body_temp_path at a tmpfs or a
proper SSD mount.
# nginx sudo apt install nginx # Node.js (upload server) node --version # >= 18 # tools which curl pv # pv gives you a live transfer rate meter
// server.js
import http from 'http';
import fs from 'fs';
import path from 'path';
import { pipeline } from 'stream/promises';
const UPLOAD_DIR = '/tmp/uploads';
fs.mkdirSync(UPLOAD_DIR, { recursive: true });
const server = http.createServer(async (req, res) => {
if (req.method === 'POST' && req.url === '/upload') {
const filename = `upload-${Date.now()}.bin`;
const dest = path.join(UPLOAD_DIR, filename);
const out = fs.createWriteStream(dest);
let received = 0;
req.on('data', chunk => {
received += chunk.length;
process.stdout.write(`\r[upstream] received ${(received / 1024 / 1024).toFixed(1)} MB`);
});
try {
await pipeline(req, out);
console.log(`\n[upstream] write complete: ${dest}`);
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ ok: true, file: filename, bytes: received }));
} catch (err) {
console.error('\n[upstream] stream error:', err.message);
res.writeHead(500);
res.end();
}
return;
}
if (req.url === '/health') {
res.writeHead(200);
res.end('OK\n');
return;
}
res.writeHead(404);
res.end();
});
const PORT = 3000;
server.listen(PORT, '127.0.0.1', () => {
console.log(`[upstream] listening on 127.0.0.1:${PORT} (pid ${process.pid})`);
});
fs.writeFileSync('/tmp/upstream.pid', String(process.pid));
Start it:
node server.js # [upstream] listening on 127.0.0.1:3000 (pid 12345)
# /etc/nginx/sites-available/upload-proxy
client_max_body_size 0; # no size limit; default is 1m
client_body_buffer_size 128k; # in-memory before spilling to disk
client_body_temp_path /tmp/nginx_client_temp 1 2;
proxy_request_buffering on; # THE key directive
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
proxy_connect_timeout 75s;
upstream app_backend {
server 127.0.0.1:3000;
keepalive 4;
}
server {
listen 8080;
server_name localhost;
location /upload {
proxy_pass http://app_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Content-Type $content_type;
}
location /health {
proxy_pass http://app_backend;
}
error_page 502 503 504 /offline.html;
location = /offline.html {
root /var/www/html;
internal;
}
}
sudo mkdir -p /tmp/nginx_client_temp/{1,2}
sudo chown -R www-data:www-data /tmp/nginx_client_temp
sudo ln -s /etc/nginx/sites-available/upload-proxy /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
# 500 MB of pseudorandom data dd if=/dev/urandom bs=1M count=500 of=/tmp/testfile.bin status=progress
Terminal 1 — start the upload and watch with pv:
pv /tmp/testfile.bin | curl \ -X POST \ http://localhost:8080/upload \ -H "Content-Type: application/octet-stream" \ --data-binary @-
pv will print a live rate (e.g. 247MB 0:00:12 [19.8MB/s]).
Wait until roughly half the file has transferred.
Terminal 2 — kill the upstream:
kill -9 $(cat /tmp/upstream.pid)
Observe Terminal 1: the pv rate does not drop. The transfer continues. The bytes are going into nginx's client body temp path, not to the Node.js process.
Watch what nginx has on disk:
watch -n1 'ls -lah /tmp/nginx_client_temp/**/*' # You'll see a temp file growing in real time
Restart the upstream:
node server.js &
After a few seconds, nginx will open a new connection and replay the buffered body.
The curl command will eventually receive a 200 OK with the JSON response.
While the upload is running, trace the TCP connections:
ss -tnp | grep -E '8080|3000'
You will see two distinct connections:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process ESTAB 0 0 127.0.0.1:8080 127.0.0.1:XXXXX nginx ESTAB 0 0 127.0.0.1:YYYYY 127.0.0.1:3000 nginx
After kill -9 on the upstream, the second connection disappears but the first stays ESTAB.
nginx is still ACK-ing the client.
To watch the ACK stream directly:
sudo tcpdump -i lo -n 'tcp port 8080' -l | grep -E 'flags \[.\]' # You will see continuous ACK flags from nginx → client during the buffering phase
proxy_request_buffering off Case
With proxy_request_buffering off, nginx becomes a streaming proxy: it opens the upstream
connection immediately and pipes bytes in both directions concurrently. No temp file. Lower latency for the first byte,
lower memory/disk overhead.
But: if the upstream drops the connection before the client finishes sending, nginx receives a TCP RST or
ECONNRESET from the upstream, closes the client connection too, and the client gets an abrupt termination
mid-transfer.
# With this setting the isolation guarantee is gone proxy_request_buffering off;
This is the correct setting for streaming protocols (WebSockets, gRPC, SSE, large video streams where you
cannot afford to buffer the whole body). For regular file uploads where resilience matters, leave it at the default on.
| Setting | Latency to upstream | Disk usage on proxy | Upstream isolation |
|---|---|---|---|
on (default) | After full body received | Full body size | ✅ Complete |
off | Immediate | 0 | ❌ None |
When the upstream is down and nginx cannot open a connection at all, nginx returns 502 Bad Gateway.
The error_page 502 /offline.html directive intercepts this and serves a static page instead.
This is the other half of the story: nginx was running on a Raspberry Pi that stayed up during a power failure.
Even when the upstream was completely down, any new request got a clean 200 OK offline page rather than
a raw 502 or a TCP connection refused.
error_page 502 503 504 =200 /offline.html; # =200 rewrites the status code — useful for UX # Omit it if you want to preserve the 5xx for monitoring
Put a self-contained HTML file at /var/www/html/offline.html.
It must be served without hitting the upstream (hence internal;).
The default client_body_temp_path on most Linux systems is /var/lib/nginx/tmp/client_body.
For production:
# Fast SSD, survives reboots client_body_temp_path /mnt/fast-ssd/nginx/client_temp 1 2; # tmpfs — low-latency, but buffered uploads are lost on reboot client_body_temp_path /dev/shm/nginx_client_temp;
The 1 2 arguments create a two-level hashed directory structure to avoid thousands of files in a single
directory — a performance issue on most filesystems at scale.
Ensure the nginx worker user owns the temp directory:
install -d -o www-data -g www-data -m 0700 /mnt/fast-ssd/nginx/client_temp
nginx fully supports chunked request bodies. It dechunks the stream while buffering it, then re-frames the body as
a normal Content-Length request when forwarding to the upstream. The upstream never needs to know the
client used chunked encoding.
multipart/form-data)
For browser file uploads via <input type="file">, the browser sends multipart/form-data.
nginx buffers the entire multipart body the same way — it does not parse the parts. The temp file on disk is
the raw multipart MIME body. The upstream receives it intact.
TUS uses PATCH requests with Upload-Offset headers to resume at a byte offset.
Each PATCH is a small independent HTTP request. nginx buffers each PATCH individually.
This is actually more resilient than a single monolithic PUT: if the upstream dies in the
middle of a PATCH, that one patch fails, the client retries from the last acknowledged offset, and
the upstream only needs to have received the bytes from the previous PATCH calls.
When you buffer large files, the default nginx timeouts are too short:
# How long nginx waits for the client to send the body (per read, not total)
client_body_timeout 60s; # default; fine for fast LANs, too short for slow WAN
# How long nginx waits for the upstream to accept the connection
proxy_connect_timeout 75s; # fine
# How long nginx waits between successive reads from the upstream response
proxy_read_timeout 3600s; # default is 60s — INCREASE for large uploads
# covers the time the upstream spends writing to disk
# How long nginx waits between successive writes to the upstream
proxy_send_timeout 3600s; # default is 60s — INCREASE for the relay phase
The critical one is proxy_read_timeout. When nginx is relaying a 9 GB body to the upstream, the upstream
may spend minutes writing to disk. If proxy_read_timeout is hit before the upstream responds, nginx
terminates the connection and returns a 504 Gateway Timeout to the client — even though the upload
finished successfully. Increase this aggressively for upload endpoints.
Expose nginx stub status to watch active connections:
location /nginx_status {
stub_status;
allow 127.0.0.1;
deny all;
}
curl -s http://localhost/nginx_status # Active connections: 3 # server accepts handled requests # 1234 1234 1567 # Reading: 1 Writing: 2 Waiting: 0
Reading: N means nginx is currently reading request bodies from N clients. If this number is elevated
and the upstream is down, those are active uploads being buffered to disk.
Log the buffer size per request:
log_format upload_log '$remote_addr - $request_length bytes '
'[$time_local] "$request" $status '
'upstream_response_time=$upstream_response_time';
access_log /var/log/nginx/uploads.log upload_log;
$request_length is the total request size including body — useful for correlating temp file growth
with specific uploads.
client_body_temp_path to a USB SSD, not the SD card.
SD cards have limited write endurance and a 9 GB upload is a brutal write workload.client_max_body_size 0 to remove the default 1 MB body size cap.proxy_next_upstream error timeout to have nginx automatically retry the upstream
on the next healthy server in an upstream {} block if you have more than one backend.proxy_request_buffering on.proxy-request-buffering: "on" by default.
The temp path inside the ingress pod is ephemeral — if the ingress pod restarts mid-upload, the buffer is gone.| Layer | What happens |
|---|---|
| Client TCP (conn A) | nginx ACKs every segment; client sees a live connection regardless of upstream state |
| nginx client body buffer | First client_body_buffer_size bytes in memory, overflow spills to client_body_temp_path on disk |
| Temp file | Created with O_TMPFILE, unlinked immediately, kept alive by fd reference in nginx worker |
| Upstream TCP (conn B) | Opened only after full body is buffered; retried on reconnect if upstream was down |
| Kernel path for replay | sendfile(2) / splice(2) zero-copy from temp file fd → upstream socket |
| Client perception | Single continuous HTTP transaction; no TCP reset, no 5xx during upstream downtime |
The takeaway is that a proxy does not merely route requests — when configured correctly, it owns each side of the transaction independently. The client and the upstream are fully decoupled at the TCP level. Upstream restarts, OOM kills, and power failures become invisible to the client as long as the proxy stays up and has disk space to absorb the upload.