Hiding Downtime Behind a Proxy

My home setup puts a small always-on machine in front of a bigger server that I switch off when I’m not using it. At some point I noticed something odd: a large upload to a service on the big server kept going at normal speed even while that server was offline. Nothing was receiving the file and yet the progress bar kept moving.

The reason is a default in nginx called proxy_request_buffering. With it on, nginx doesn’t pass an upload through to the backend as it arrives. It accepts the whole request body itself, writes it to disk and only then talks to the backend. The client’s connection and the backend’s connection are two separate things and the client only ever sees the first one.

This post explains how that works, from the nginx config down to TCP and the kernel. Then it walks through a demo you can run yourself: a small Node.js upload server behind nginx, a 500 MB upload and a kill -9 on the backend halfway through.

Background: what a reverse proxy actually does#

A reverse proxy sits between clients and your application. Clients connect to the proxy and the proxy makes its own connection to the application (usually called the upstream) on their behalf. nginx, HAProxy, Caddy and cloud load balancers all do this.

The important word is its own. There are two TCP connections involved, not one:

• Connection A, from the client to nginx.
• Connection B, from nginx to the upstream.

Each TCP connection is kept alive by acknowledgements. When the client sends a chunk of data, the receiver replies with an ACK saying “got it, send more”. Each side also advertises a receive window, which is how much more data it is ready to accept. As long as the receiver keeps reading data out of its buffer, the window stays open and the sender keeps going at full speed.

On connection A, the receiver is nginx, not your application. So the client’s upload speed depends only on nginx keeping up and nothing that happens on connection B can reach the client unless nginx passes it along.

This kind of proxy terminates the client’s connection. It doesn’t just forward packets. That decoupling is what the rest of this post relies on.

What nginx does with a large upload#

With the default proxy_request_buffering on, an upload goes through three phases. The upstream isn’t involved in the first one at all.

Phase 1: buffer the body#

When a POST /upload arrives, nginx reads the request body from connection A into memory, up to client_body_buffer_size (16 KB by default on 64-bit systems). Once that fills, it spills the rest to a temporary file under client_body_temp_path. On Debian that is /var/lib/nginx/body.

nginx keeps reading from connection A as fast as the client sends, so the kernel keeps ACKing and the upload runs at full speed. The upstream hasn’t been contacted yet. It could be powered off and nothing about this phase would change.

Phase 2: send it upstream#

Only once the entire body has arrived does nginx open connection B and send the request, reading the body back from the temp file.

This is the part that explains what I saw. The bytes of that upload were going to the proxy’s disk, not to the application. The application only has to be reachable at the moment the upload finishes.

Phase 3: relay the response#

nginx reads the upstream’s response, sends it to the client over connection A and throws the temp file away. From the client’s side there was one continuous HTTP request that ended with a normal response.

Buffering doesn’t make nginx wait for a dead backend to come back. When the body is complete, nginx tries to connect once. If the upstream is still down at that moment, the client gets a 502 Bad Gateway (or whatever error_page turns it into) and the upload is lost. What buffering buys you is that the backend can be down for most of the upload. It just has to be back before the upload ends.

Under the hood#

Socket receive buffers#

Every TCP socket has a receive buffer in the kernel, sized by net.ipv4.tcp_rmem. Incoming data lands there and the application pulls it out with recv(). If the application stops reading, the buffer fills, the advertised window drops to zero and the sender has to pause.

nginx never stops reading during phase 1, because the data has somewhere to go: memory, then the temp file. So the window on connection A stays open for the whole upload, whatever state the upstream is in.

sysctl net.ipv4.tcp_rmem
# net.ipv4.tcp_rmem = 4096  131072  6291456
# min / default / max, in bytes

Sending the file without copying it#

In phase 2, nginx has to move the body from a file on disk to a socket. The naive way is to read() it into memory and write() it back out, which copies every byte into userspace and back. With sendfile on (Debian’s default nginx.conf sets it), nginx can use sendfile(2) instead, which tells the kernel to move the data from the file to the socket directly. Internally the kernel does this with the same machinery as splice(2).

Because nothing is copied into userspace, sending the body upstream stays cheap even on a small machine like a Raspberry Pi.

The temp file you can’t see#

nginx creates each temp file and then unlinks it immediately. The file has no name on disk any more, but it still exists, because the nginx worker holds an open file descriptor to it. When the request finishes, or if the worker crashes, the descriptor closes and the kernel frees the space. Nothing is ever left behind to clean up.

A side effect is that ls on the temp directory will always look empty. I’ll show how to see the file anyway in the demo.

Try it yourself#

You need a Linux machine with nginx, Node.js 18 or newer and curl. The whole demo runs on one machine.

sudo apt install nginx nodejs curl lsof
node --version   # 18 or newer

Step 1: a small upload server#

This server accepts a POST /upload, streams the body to a file and prints how much it has received as it goes. It also writes its PID to a file so it’s easy to kill later.

import http from 'http';
import fs from 'fs';
import path from 'path';
import { pipeline } from 'stream/promises';

const UPLOAD_DIR = '/tmp/uploads';
fs.mkdirSync(UPLOAD_DIR, { recursive: true });

const server = http.createServer(async (req, res) => {
  if (req.method === 'POST' && req.url === '/upload') {
    const filename = `upload-${Date.now()}.bin`;
    const dest = path.join(UPLOAD_DIR, filename);
    const out = fs.createWriteStream(dest);

    let received = 0;
    req.on('data', chunk => {
      received += chunk.length;
      process.stdout.write(`\r[upstream] received ${(received / 1024 / 1024).toFixed(1)} MB`);
    });

    try {
      await pipeline(req, out);
      console.log(`\n[upstream] write complete: ${dest}`);
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ ok: true, file: filename, bytes: received }));
    } catch (err) {
      console.error('\n[upstream] stream error:', err.message);
      res.writeHead(500);
      res.end();
    }
    return;
  }

  if (req.url === '/health') {
    res.writeHead(200);
    res.end('OK\n');
    return;
  }

  res.writeHead(404);
  res.end();
});

const PORT = 3000;
server.listen(PORT, '127.0.0.1', () => {
  console.log(`[upstream] listening on 127.0.0.1:${PORT} (pid ${process.pid})`);
});

fs.writeFileSync('/tmp/upstream.pid', String(process.pid));

Save it, add a package.json containing {"type": "module"} next to it so Node accepts the import syntax and start it:

$ node server.js
[upstream] listening on 127.0.0.1:3000 (pid 12345)

Step 2: the nginx config#

client_max_body_size        0;          # no size limit (the default is 1m)
client_body_buffer_size     128k;       # kept in memory before spilling to disk
client_body_temp_path       /var/lib/nginx/upload_tmp 1 2;

proxy_request_buffering     on;         # the key directive (and the default)
proxy_read_timeout          3600s;
proxy_send_timeout          3600s;

upstream app_backend {
    server 127.0.0.1:3000;
}

server {
    listen 8080;
    server_name localhost;

    location /upload {
        proxy_pass         http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /health {
        proxy_pass http://app_backend;
    }

    error_page 502 503 504 /offline.html;
    location = /offline.html {
        root /var/www/html;
        internal;
    }
}

Create the temp directory, enable the site and reload:

sudo install -d -o www-data -g www-data -m 0700 /var/lib/nginx/upload_tmp
sudo ln -s /etc/nginx/sites-available/upload-proxy /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
curl http://localhost:8080/health   # OK

Step 3: a test file#

# 500 MB of random data
dd if=/dev/urandom bs=1M count=500 of=/tmp/testfile.bin status=progress

Step 4: upload and watch the backend stay silent#

Over loopback, a 500 MB upload finishes in about a second, which is too fast to do anything in the middle of it. --limit-rate slows curl down to something closer to a real network, so the upload takes about 25 seconds:

curl -X POST --upload-file /tmp/testfile.bin --limit-rate 20M \
  -H "Content-Type: application/octet-stream" \
  -o - http://localhost:8080/upload

curl’s progress meter shows the upload climbing steadily. Now look at the terminal running server.js. Nothing is happening there. The received counter doesn’t move for the entire upload, because nginx is holding all of it. Then, the moment curl reaches 100%, the counter races to 500 MB almost instantly, as nginx replays the body from disk over loopback and curl prints the JSON response.

That silence is the clearest demonstration of phase 1.

Step 5: kill the backend halfway#

Start the same upload again. When curl shows roughly half the file sent, kill the upstream from another terminal:

kill -9 $(cat /tmp/upstream.pid)

Look back at curl: the rate doesn’t change. The upload carries on, because the bytes are going into nginx’s temp file and the dead Node.js process was never part of the path.

Now start the server again before the upload finishes:

node server.js

When curl reaches 100%, nginx connects to the new process, sends it the whole body and curl receives a normal 200 OK. From the client’s side, the backend was never gone.

Try it once more, but this time don’t restart the server. When the upload finishes, nginx can’t connect, gets a 502 and error_page serves /offline.html instead. That is the limit from the warning above.

Seeing it at the TCP level#

While an upload is running, list the TCP connections on both ports:

ss -tnp '( sport = :8080 or dport = :8080 or dport = :3000 )'

During the upload you’ll see connection A (curl on one end, nginx on port 8080 on the other) and nothing to port 3000. Connection B only appears right at the end, for the second or so it takes to send the body upstream.

To see the invisible temp file, ask for open files that have no name on disk:

sudo lsof -nP +L1 -c nginx
# ... /var/lib/nginx/upload_tmp/.../0000000001 (deleted)

Run it a few times during the upload and the SIZE/OFF column grows. That is the unlinked temp file from earlier, filling up as the client sends.

When you don’t want buffering#

With proxy_request_buffering off, nginx becomes a streaming proxy. It opens connection B straight away and passes bytes through as they arrive. There’s no temp file and the backend sees data sooner, but the isolation is gone. If the upstream dies mid-upload, the reset travels straight back to the client and the transfer fails.

# Streams the body upstream as it arrives. No isolation.
proxy_request_buffering off;

Streaming is the right choice for things that can’t wait for the whole body: WebSockets, gRPC streams, server-sent events, or media too large to hold on the proxy. For ordinary file uploads, where surviving a backend restart matters more, keep the default.

A friendly page when the upstream is down#

Buffering covers an upstream that dies during an upload. A new request that arrives while the upstream is fully down is a different case: nginx can’t connect and returns 502 Bad Gateway. error_page can catch that and serve a static page from the proxy instead.

This is the other half of how my setup behaves. When the big server is off, or lost power, the always-on proxy still answers every request with a clean offline page instead of a raw error.

error_page 502 503 504 =200 /offline.html;
# =200 rewrites the status code, which is friendlier to browsers.
# Leave it out if you want monitoring to still see the 5xx.

The page lives at /var/www/html/offline.html on the proxy and internal; in the location block means clients can’t request it directly.

Running this for real#

Where the temp files go#

Every buffered upload needs disk space on the proxy equal to its full size, so the temp path matters:

# Fast SSD with room to spare
client_body_temp_path /mnt/fast-ssd/nginx/client_temp 1 2;

# tmpfs: fast, but uses RAM, and limited by it
client_body_temp_path /dev/shm/nginx_client_temp;

The 1 2 creates two levels of hashed subdirectories, so a busy server doesn’t end up with thousands of files in one directory. The nginx worker user has to own the directory:

install -d -o www-data -g www-data -m 0700 /mnt/fast-ssd/nginx/client_temp

On a Raspberry Pi, point this at a USB SSD rather than the SD card. A few large uploads will wear through an SD card’s write endurance surprisingly quickly.

Chunked, multipart and resumable uploads#

• Transfer-Encoding: chunked: nginx accepts chunked bodies, buffers them the same way and sends them upstream with a normal Content-Length. The upstream never needs to know the client used chunked encoding.
• multipart/form-data (browser file inputs): nginx buffers the raw body without parsing it. The upstream receives exactly what the browser sent.
• Resumable uploads (the TUS protocol) send a file as a series of PATCH requests, each with an Upload-Offset. nginx buffers each small PATCH on its own. If the upstream dies, only that one piece fails and the client resumes from the last offset it got confirmed. For really large files this is more robust than one giant request.

Timeouts#

The defaults are tuned for ordinary requests, not multi-gigabyte ones:

# How long nginx waits between two reads from the client (per read, not in total)
client_body_timeout    60s;    # default; fine on a LAN, can be short on slow links

# How long nginx waits between two reads of the upstream's response
proxy_read_timeout     3600s;  # default is 60s; raise it for large uploads

# How long nginx waits between two writes to the upstream
proxy_send_timeout     3600s;  # default is 60s; raise it too

proxy_read_timeout is the one that bites. After receiving a very large body, the upstream might spend minutes writing it to its own disk before it answers. If the timeout fires first, the client gets a 504 Gateway Timeout even though the upload actually succeeded.

Watching it#

stub_status gives a quick view of what nginx is doing:

location /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

$ curl -s http://localhost/nginx_status
Active connections: 3
server accepts handled requests
 1234 1234 1567
Reading: 0 Writing: 2 Waiting: 1

Reading counts connections where nginx is still reading the request headers. A connection whose body is being buffered counts under Writing, so a jump in Writing with no matching traffic on the upstream is what a pile of in-progress uploads looks like.

For per-request detail, log the request size and how long the upstream took:

log_format upload_log '$remote_addr $request_length bytes '
                      '[$time_local] "$request" $status '
                      'upstream_response_time=$upstream_response_time';
access_log /var/log/nginx/uploads.log upload_log;

In Kubernetes#

The ingress-nginx controller buffers request bodies by default too. But the ingress pod’s temp directory is ephemeral: if the pod restarts during an upload, the buffered body is gone. For large uploads in a cluster, a resumable protocol like TUS, or uploading straight to object storage with a presigned URL, is the safer design.

Summary#

A proxy doesn’t just route requests. With buffering on, it owns each side of the conversation separately and that gap is where upstream restarts, crashes and power cuts disappear, as long as the proxy stays up, has disk to spare and the backend is back by the time the last byte arrives.

Further reading#

• nginx proxy_request_buffering
• nginx client_body_buffer_size
• Linux sendfile(2)
• TUS resumable upload protocol
• RFC 9112, HTTP/1.1