Pushing a Kubernetes Cluster to Its Limits

I wanted a Kubernetes cluster I could break on purpose. A managed cluster hides the interesting failures behind a control plane you never touch, so I built one from bare VMs instead, wired it up by hand and then pushed load at it until it bent.

The setup is small: two k3s nodes running as libvirt VMs on a laptop, a Go service that does nothing but burn CPU, a horizontal pod autoscaler and a k6 load test fired from a separate desktop. It topped out at 93 requests per second with zero errors. Getting there first cost me an evening of Docker, firewalld and libvirt fighting over who owns the host firewall.

This post walks through the whole thing in order: the layout, the networking problem, installing k3s, the autoscaler, the load test and what the numbers actually mean. If you have never touched Kubernetes, the short background section below covers the handful of ideas you need.

The layout#

Three machines, each with one job. The load generator needs to be at least as strong as the thing it is testing, otherwise you end up measuring the generator instead of the service.

The desktop can’t reach the pods directly. The VMs sit on libvirt’s default NAT network, which only exists inside the laptop. libvirt NAT only works one way: VMs can reach out, nothing outside can reach in. So the laptop forwards a port on its LAN address into the cluster, acting as a tiny load balancer.

The node sizing is tight on purpose. Small nodes put the scaling limit close enough to hit inside a five minute test.

Background: the pieces involved#

Skip this section if you already know Kubernetes. Everything later builds on these five ideas.

k3s is a small Kubernetes distribution packaged as a single binary. It runs the same API as full Kubernetes, but it bundles the pieces you would otherwise install separately, which makes it a good fit for machines with 2 GB of RAM.

Server and agent nodes. The server node runs the control plane: the API server, the scheduler, the controllers and the datastore. Agent nodes only run workloads. Here there is one of each.

Pods, Deployments and Services. A pod is one running copy of the application. A Deployment keeps a chosen number of pods alive. A Service gives those pods one stable address and spreads requests across them. A NodePort Service opens the same port (here 30080) on every node, so anything that can reach a node can reach the application.

Requests and limits. Every container can declare how much CPU it requests (what the scheduler reserves for it) and a limit it is never allowed to exceed. CPU is measured in millicores: 100m is a tenth of a core.

The Horizontal Pod Autoscaler (HPA) watches the average CPU usage of a Deployment’s pods and adds or removes pods to keep that average near a target. It gets its numbers from metrics-server, which k3s ships by default.

Step 1: Give the VMs addresses that don’t move#

My first runs addressed the VMs by whatever IP DHCP handed them. Leases expire, a VM comes back with a different address and the IP you have been treating as fixed now points at nothing. I lost real time SSHing into an address that an older incarnation of a VM used to hold.

The fix is to give each VM a fixed MAC address and pin a DHCP reservation to it:

# Fixed MAC per VM, pinned to a fixed IP, on both the live and persistent config.
sudo virsh net-update default add ip-dhcp-host \
  "<host mac='52:54:00:aa:bb:10' name='k3s-server' ip='192.168.122.10'/>" \
  --live --config

sudo virsh net-update default add ip-dhcp-host \
  "<host mac='52:54:00:aa:bb:11' name='k3s-agent' ip='192.168.122.11'/>" \
  --live --config

The --live --config pair turned out to be the lesson that kept repeating through this project. --live changes the running state and --config changes the saved definition. Apply only one and you get the classic “works until the next reboot” problem. The same thing applies to iptables rules and sysctls.

In the same spirit, run virsh net-autostart on the network and virsh autostart on both VMs. I rebooted the laptop once and the whole cluster was simply gone. Nothing was corrupted, it just never started again.

Step 2: The netfilter fight#

This took most of the evening and it’s the most useful part of the post.

The VMs came up, SSH worked, but they had no internet. The gateway answered, DNS resolved and yet every packet heading outside died:

$ ssh ubuntu@192.168.122.10 'ping -c2 192.168.122.1'   # gateway, replies fine
$ ssh ubuntu@192.168.122.10 'ping -c2 8.8.8.8'         # internet, 100% loss
$ ssh ubuntu@192.168.122.10 'ping -c2 google.com'      # resolves to an IP, still 100% loss

That combination is a strong hint. If the gateway answers and DNS works, the VM’s own network config is fine. What is failing is the step where the host rewrites the VM’s private address into its own before sending the packet out, which is NAT, or more precisely masquerading. So I looked at what the host’s firewall was doing with forwarded traffic:

$ sudo nft list ruleset | grep -i 'masquerade\|192.168.122'
  ip saddr 172.17.0.0/16 oifname != "docker0" counter masquerade
  ip saddr 172.18.0.0/16 oifname != "br-762974512e58" counter masquerade
  # rules for Docker's networks present, nothing for 192.168.122.0/24

There were masquerade rules for Docker’s networks and nothing at all for libvirt’s. Docker was installed on the laptop and when Docker starts it writes its own netfilter rules and sets the policy of the FORWARD chain to drop. In doing so it had pushed out libvirt’s masquerade rule. Packets from the VMs met a FORWARD policy of drop, had nothing to rewrite their source address and quietly disappeared.

The real cause is structural. Three things on this host all believe they manage the firewall: Docker, firewalld and libvirt. The more aggressive ones overwrite the others and every time I restarted networking they fought again and the NAT rule fell over again. I went around that loop a few times before I saw it for what it was.

When host networking breaks for no obvious reason, don’t start by asking which rule is wrong. Ask who else is writing rules on this machine and list every tool that manages netfilter before you touch anything.

I don’t need Docker on the host at all, because the cluster’s containers run inside the VMs under containerd. So the clean fix was to take the most aggressive player out of the fight:

# The cluster's containers run inside the VMs, not on the host.
# Take the most aggressive netfilter actor out of the fight.
sudo systemctl disable --now docker docker.socket

# Then let libvirt reinstall its NAT rules unopposed (a reboot is cleanest,
# since stopping Docker does not flush rules it already loaded).
sudo systemctl restart libvirtd

Step 3: Install k3s#

Install the server first, because it generates the token that the agent needs in order to join:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="\
  --node-ip=192.168.122.10 \
  --tls-san=192.168.122.10 \
  --write-kubeconfig-mode=644 \
  --disable=traefik \
  --disable=servicelb" sh -

# Then read the join token:
sudo cat /var/lib/rancher/k3s/server/node-token

Each flag is there for a reason:

• --node-ip pins the address k3s advertises. VMs often have more than one interface and k3s can pick the wrong one. It’s the same “don’t trust detected addresses” lesson as step 1.
• --tls-san adds that IP to the API server’s certificate, so kubectl doesn’t fail with a certificate mismatch.
• --disable=traefik --disable=servicelb drops the bundled ingress controller and load balancer. On a 2 GB node every megabyte the control plane uses is a megabyte the workload can’t.

Then join the agent:

# On the agent. K3S_URL makes the installer set up a worker that joins the server.
curl -sfL https://get.k3s.io | \
  K3S_URL="https://192.168.122.10:6443" \
  K3S_TOKEN="<the K10... token from the server>" \
  INSTALL_K3S_EXEC="--node-ip=192.168.122.11" sh -

$ sudo k3s kubectl get nodes -o wide
NAME         STATUS   ROLES           VERSION        INTERNAL-IP      OS-IMAGE
k3s-server   Ready    control-plane   v1.35.5+k3s1   192.168.122.10   Ubuntu 24.04.4 LTS
k3s-agent    Ready    <none>          v1.35.5+k3s1   192.168.122.11   Ubuntu 24.04.4 LTS

One thing that looks like an error but isn’t: running kubectl on the agent fails with connection to localhost:8080 refused. That is expected. Only the server runs an API server, so the agent has nothing local to talk to.

Step 4: The service under test#

The application is deliberately boring so the operational side stays the interesting part. It’s a small Go service with /healthz, /readyz, /metrics and the one endpoint that matters, /work?ms=200, which burns real CPU by hashing in a tight loop for the requested number of milliseconds:

// burnCPU does real, un-optimizable work for roughly ms milliseconds by hashing
// in a tight loop. Hitting /work pegs a CPU core, which is what HPA scales on.
func burnCPU(ms int) {
	deadline := time.Now().Add(time.Duration(ms) * time.Millisecond)
	var x [32]byte
	for time.Now().Before(deadline) {
		for i := 0; i < 1000; i++ {
			x = sha256.Sum256(x[:])
		}
	}
	_ = x
}

Making the work CPU-bound matters. The autoscaler scales on CPU, so a request that pins a core is the most direct way to make it react.

The service builds as a static binary on a distroless base image. The awkward part is getting that image onto the cluster with no registry and with Docker now gone from the host. Podman solves this: it builds images without a background daemon and doesn’t touch the firewall, so it doesn’t restart the fight from step 2. The image has to be imported on both nodes, because a pod can be scheduled onto either one. A node that doesn’t have the image can’t start it.

# Build with Podman (daemonless, no firewall rules), import into BOTH nodes' containerd.
podman build -t localhost/loadlab:v1 .
podman save -o /tmp/loadlab-v1.tar localhost/loadlab:v1

for node in ubuntu@192.168.122.10 ubuntu@192.168.122.11; do
  scp /tmp/loadlab-v1.tar "${node}:/tmp/loadlab-v1.tar"
  ssh "$node" 'sudo k3s ctr images import /tmp/loadlab-v1.tar'
done

Step 5: Requests and the autoscaler#

The Deployment sets resource requests and this isn’t optional tidiness. The HPA computes CPU utilization as a percentage of the request. With no request there is nothing to divide by and the autoscaler does nothing at all. It’s the most common reason a first HPA demo refuses to scale.

          resources:
            # REQUESTS = what the scheduler reserves, and what HPA measures against.
            # Omit requests and HPA has no denominator and will not scale.
            requests:
              cpu: "100m"
              memory: "32Mi"
            limits:
              cpu: "500m"
              memory: "64Mi"

With a request of 100m, a target of 50% means “add pods once the average pod uses more than 50 millicores”:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: loadlab
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: loadlab
  minReplicas: 2
  maxReplicas: 10        # on 2GB nodes this cap usually binds before node capacity
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50   # scale up when pods average above 50m (50% of request)
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0    # react immediately
    scaleDown:
      stabilizationWindowSeconds: 60   # wait 60s of calm before scaling down, avoids flapping

Once applied, the HPA shows a real reading instead of <unknown>, which means metrics-server is feeding it and it will respond to load. The pod IPs are worth a look too: each node owns a slice of the pod network (10.42.0.x and 10.42.1.x) and the cluster’s network plugin routes between them.

$ kubectl get pods -o wide
NAME                      READY   STATUS    IP          NODE
loadlab-b6ddd7b86-cchg5   1/1     Running   10.42.0.5   k3s-server
loadlab-b6ddd7b86-wpfwl   1/1     Running   10.42.1.2   k3s-agent

$ kubectl get svc loadlab
NAME      TYPE       CLUSTER-IP      PORT(S)        AGE
loadlab   NodePort   10.43.123.145   80:30080/TCP   98s

$ kubectl get hpa loadlab
NAME      REFERENCE            TARGETS       MINPODS   MAXPODS   REPLICAS
loadlab   Deployment/loadlab   cpu: 1%/50%   2         10        2

Step 6: Open a path in and fire load#

socat on the laptop listens on the LAN and forwards every connection to the NodePort. I tested the path one hop at a time, so that if something failed I would know which segment it was:

# On the laptop. Listens on the LAN, forwards every connection into the NodePort.
socat TCP-LISTEN:8080,fork,reuseaddr TCP:192.168.122.10:30080

# Three-hop test:
curl http://192.168.122.10:30080/work?ms=50   # laptop straight to NodePort
curl http://localhost:8080/work?ms=50          # laptop through socat
curl http://192.168.1.12:8080/work?ms=50       # desktop across the LAN

Calling it repeatedly shows the response switching between pod hostnames, which is the Service spreading requests across pods.

The load itself is a k6 test using the ramping arrival rate executor. Most load tests pick a fixed number of virtual users and let them loop. Arrival rate works the other way round: you choose how many requests per second should arrive, the way real traffic does and k6 adds workers as needed to keep up. The test also has pass/fail thresholds built in, so it produces a verdict instead of just a pile of numbers.

import http from 'k6/http';
import { check } from 'k6';

// Ramping arrival-rate: control REQUESTS PER SECOND, not just virtual users.
// k6 adds VUs as needed to sustain the rate, which models real arriving traffic.
export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 500,
      stages: [
        { target: 20,  duration: '30s' },  // warm up
        { target: 50,  duration: '1m'  },  // cross the 50% HPA threshold
        { target: 100, duration: '1m'  },  // expect scale-up
        { target: 200, duration: '2m'  },  // hammer, expect the ceiling
        { target: 0,   duration: '30s' },  // ramp down, watch HPA scale back
      ],
    },
  },
  // SLOs: pass/fail instead of vibes.
  thresholds: {
    http_req_duration: ['p(95)<500', 'p(99)<1000'],
    http_req_failed:   ['rate<0.01'],
  },
};

const BASE = __ENV.TARGET || 'http://192.168.1.12:8080';

export default function () {
  const res = http.get(`${BASE}/work?ms=200`);
  check(res, { 'status 200': (r) => r.status === 200 });
}

Results#

I ran it twice: first with k6 on the laptop itself, then from the desktop. I expected the desktop run to go higher, since the generator would no longer be competing with the VMs for the laptop’s CPU. It didn’t. The two runs were almost identical and that was the finding. It ruled out the load generator as the bottleneck and pointed straight at the cluster.

  THRESHOLDS
    http_req_duration
    x 'p(95)<500'  p(95)=671.05ms
    v 'p(99)<1000' p(99)=912.31ms
    http_req_failed
    v 'rate<0.01'  rate=0.00%

  TOTAL RESULTS
    checks_succeeded...: 100.00%  28015 out of 28015
    http_req_duration..: avg=375.19ms  med=355.02ms  max=1.29s  p(90)=575.09ms  p(95)=671.05ms
    http_req_failed....: 0.00%   0 out of 28015
    http_reqs..........: 28015   93.383226/s
    dropped_iterations.: 34      0.113333/s
    vus_max............: 84
  # HPA scaled 2 -> 10 pods, all Running, CPU pegged.

Reading the numbers#

Saturation, not failure#

There were zero errors, but the p95 latency target was missed. The system stayed correct under load and got slower. That’s graceful degradation and in practice it’s a more common shape than a hard crash.

The number that decides it is the gap between what I asked for and what I got. I asked for 200 requests per second and the cluster topped out around 93, with k6 unable to start some iterations on time (the dropped_iterations line). The HPA had already scaled to its maximum of 10 pods, all running and all pinned on CPU. So the cluster wasn’t sitting idle behind a slow funnel. It was genuinely out of CPU. Each request costs 200 ms of pure computation and two nodes with 2 vCPUs each can only hash so fast. A 16-thread desktop as the generator changed nothing because the generator was never the limit.

Throughput is not the same as concurrent users#

It’s tempting to read “peak 84 VUs” as “it handled about 100 concurrent users”. That mixes up two different things. The honest figures are 93 requests per second of throughput and a peak of 84 in-flight connections. Little’s Law connects them:

concurrency = throughput x latency
            = 93 req/s x 0.375 s
            ~ 35 requests in flight at any instant
            (k6 held up to 84 workers to sustain that through the slow tail)

k6 needed up to 84 workers to keep about 35 requests in flight, because latency was high and each worker spends most of its time waiting for a response. A virtual user with no pause between requests also stands in for many real people, who read and think between clicks. “Concurrent users” is the wrong unit for a result like this.

The number only holds for this workload#

93 requests per second is not this cluster’s capacity in general. It’s the capacity for requests that each burn 200 ms of CPU. A real service doing 5 ms database lookups would handle far more on the same hardware, because it isn’t pinning a core per request. A capacity number always comes attached to the workload that produced it.

The HPA scales pods, nothing scaled the nodes#

The autoscaler hit maxReplicas: 10 and stopped. That cap is what ended the test, not node capacity, because these pods are tiny. The HPA only ever adds pods. Nothing here adds nodes. In a cloud setup you would pair it with a cluster autoscaler that adds machines when pods can’t be scheduled. With a fixed two nodes, the ceiling is hard.

After scaling back down, the remaining pods mostly ended up on the agent. That’s expected: the server node also runs the API server, scheduler, controller manager, datastore, CoreDNS and metrics-server, so it has less room left over. In production you often mark the control plane node so workloads are never scheduled there at all.

What I would try next#

• Raise maxReplicas to force the node limit and watch pods go Pending with Insufficient cpu. That would complete the picture of pod scaling versus node scaling.
• Find the capacity that respects the SLO, not just the saturation point. Ramp slowly and find where p95 crosses 500 ms. That’s the number you would actually promise someone and it’s lower than 93.
• Replace socat with a real ingress and move the laptop off WiFi, to take both out of the latency budget.

Summary#

None of this is a production deployment and it was never meant to be. It’s a place to watch a system bend under load I generated, on hardware I can point at, with numbers I measured myself. The problems it threw at me along the way taught me more than a clean run would have.

Further reading#

• k3s documentation
• k6 ramping arrival-rate executor
• Kubernetes HorizontalPodAutoscaler
• Little’s Law