A Multi-Node k3s Cluster, Load Tested to Saturation

June 2026 · k3s · Kubernetes · libvirt · netfilter · k6

I wanted a Kubernetes setup I could break on purpose. Not a managed cluster where the interesting failures hide behind a control plane I never touch, but something I provisioned from bare VMs, wired up myself, and then hit hard enough to watch it bend.

Two k3s nodes on libvirt VMs, a deliberately CPU-heavy Go service, a horizontal pod autoscaler, and a k6 ramp fired from a separate desktop. It topped out at 93 req/s with zero errors. Getting there meant first losing an evening to a three-way fight between Docker, firewalld, and libvirt over who owns the host firewall.

The Shape of It

Three machines, each with a distinct job. The load generator has to be stronger than the target, or you end up measuring the generator instead of the service.

The desktop cannot reach the pods directly. Pods live on libvirt's NAT network, which is internal to the laptop, and libvirt NAT is one way: VMs reach out, nothing reaches in. So the laptop forwards a LAN port into the NodePort, acting as a small load balancer.

That node sizing is tight, and the tightness helps: it puts the scaling wall close enough to hit inside a five minute test.

Pinning Addresses

My first runs addressed the VMs by DHCP-assigned IPs. Leases expire, VMs reacquire different addresses, and an IP you have been treating as fixed points at nothing. I spent real time SSHing to a dead address a previous VM incarnation had held. Fix each VM to a fixed MAC and pin a DHCP reservation to it.

# Fixed MAC per VM, pinned to a fixed IP, on both the live and persistent planes.
sudo virsh net-update default add ip-dhcp-host \
  "<host mac='52:54:00:aa:bb:10' name='k3s-server' ip='192.168.122.10'/>" \
  --live --config

sudo virsh net-update default add ip-dhcp-host \
  "<host mac='52:54:00:aa:bb:11' name='k3s-agent' ip='192.168.122.11'/>" \
  --live --config

The --live --config pair is the recurring lesson of the whole project. One flag changes running state, the other changes the persistent definition. Apply only one and you get the classic "it works until reboot" confusion. Same thinking applies to iptables and sysctls.

The Netfilter Fight

This is the part that actually took the evening, and the most useful thing in the post. The VMs came up fine, SSH worked, but they had no internet. The gateway was reachable, DNS resolved, raw outbound packets died.

$ ssh ubuntu@192.168.122.10 'ping -c2 192.168.122.1'   # gateway, replies fine
$ ssh ubuntu@192.168.122.10 'ping -c2 8.8.8.8'         # internet, 100% loss
$ ssh ubuntu@192.168.122.10 'ping -c2 google.com'      # resolves to an IP, still 100% loss

Gateway reachable, plus DNS resolving, plus no route out, points squarely at host-side NAT rather than the VM. So I looked at what the host was doing to forwarded packets.

$ sudo nft list ruleset | grep -i 'masquerade\|192.168.122'
  ip saddr 172.17.0.0/16 oifname != "docker0" counter masquerade
  ip saddr 172.18.0.0/16 oifname != "br-762974512e58" counter masquerade
  # rules for Docker's networks present, nothing for 192.168.122.0/24

Masquerade rules for Docker's networks, nothing for libvirt's. Docker was installed on the host, and when Docker starts it installs its own netfilter rules and sets the FORWARD chain policy to drop. Doing so had evicted libvirt's masquerade rule. VM packets hit a FORWARD policy of drop with no masquerade to rewrite their source address, and quietly died.

The root cause is structural: three things on this host all want to manage netfilter, and the more aggressive ones stomp the others. Every time I cycled the network they re-fought and the NAT rule fell over again. I went around that loop more than once before naming it properly.

I don't need Docker on the host at all. The cluster's containers run inside the VMs under containerd. So the clean fix was to remove the most aggressive actor from the fight entirely.

# The cluster's containers run inside the VMs, not on the host.
# Take the most aggressive netfilter actor out of the fight.
sudo systemctl disable --now docker docker.socket

# Then let libvirt reinstall its NAT rules unopposed (a reboot is cleanest,
# since stopping Docker does not flush rules it already loaded).
sudo systemctl restart libvirtd

Installing k3s

Server first, because it generates the token the agent needs to join.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="\
  --node-ip=192.168.122.10 \
  --tls-san=192.168.122.10 \
  --write-kubeconfig-mode=644 \
  --disable=traefik \
  --disable=servicelb" sh -

# Then read the join token:
sudo cat /var/lib/rancher/k3s/server/node-token

• --node-ip pins the address k3s advertises. VMs are often multi-homed and k3s can pick the wrong interface, the same "don't trust auto-detected addresses" lesson as the DHCP pinning.
• --tls-san adds that IP to the API server certificate so kubectl doesn't hit a cert mismatch.
• --disable=traefik --disable=servicelb drops the bundled ingress and load balancer. On a 2 GB node every megabyte of control plane overhead is capacity taken from workloads.

# On the agent. K3S_URL makes the installer set up a worker that joins the server.
curl -sfL https://get.k3s.io | \
  K3S_URL="https://192.168.122.10:6443" \
  K3S_TOKEN="<the K10... token from the server>" \
  INSTALL_K3S_EXEC="--node-ip=192.168.122.11" sh -

$ sudo k3s kubectl get nodes -o wide
NAME         STATUS   ROLES           VERSION        INTERNAL-IP      OS-IMAGE
k3s-server   Ready    control-plane   v1.35.5+k3s1   192.168.122.10   Ubuntu 24.04.4 LTS
k3s-agent    Ready    <none>          v1.35.5+k3s1   192.168.122.11   Ubuntu 24.04.4 LTS

One thing that looks like an error and is not: kubectl on the agent fails with connection to localhost:8080 refused. That's correct. Only the control plane node runs an API server; the worker has nothing local to talk to.

The Service Under Test

The application is intentionally boring so the operational layer stays the interesting part. A small Go service with /healthz, /readyz, /metrics, and the one endpoint that matters, /work?ms=200, which burns real CPU by hashing in a tight loop.

// burnCPU does real, un-optimizable work for roughly ms milliseconds by hashing
// in a tight loop. Hitting /work pegs a CPU core, which is what HPA scales on.
func burnCPU(ms int) {
	deadline := time.Now().Add(time.Duration(ms) * time.Millisecond)
	var x [32]byte
	for time.Now().Before(deadline) {
		for i := 0; i < 1000; i++ {
			x = sha256.Sum256(x[:])
		}
	}
	_ = x
}

It builds as a static binary on a distroless base. The wrinkle is getting that image onto the cluster with no registry and Docker now gone from the host. Podman is daemonless and touches no firewall rules, so it doesn't reopen the fight. Import into both nodes, because a pod can be scheduled onto either and a node without the image fails to start it.

# Build with Podman (daemonless, no firewall rules), import into BOTH nodes' containerd.
podman build -t localhost/loadlab:v1 .
podman save -o /tmp/loadlab-v1.tar localhost/loadlab:v1

for node in ubuntu@192.168.122.10 ubuntu@192.168.122.11; do
  scp /tmp/loadlab-v1.tar "${node}:/tmp/loadlab-v1.tar"
  ssh "$node" 'sudo k3s ctr images import /tmp/loadlab-v1.tar'
done

Requests are the denominator

The deployment sets resource requests, and this is not optional hygiene. The HPA computes CPU utilization as a percentage of the request. With no request there is no denominator and the autoscaler does nothing. This is the single most common reason a first HPA demo quietly refuses to scale.

          resources:
            # REQUESTS = what the scheduler reserves, and what HPA measures against.
            # Omit requests and HPA has no denominator and will not scale.
            requests:
              cpu: "100m"
              memory: "32Mi"
            limits:
              cpu: "500m"
              memory: "64Mi"

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: loadlab
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: loadlab
  minReplicas: 2
  maxReplicas: 10        # on 2GB nodes this cap usually binds before node capacity
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50   # scale up when pods average above 50m (50% of request)
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0    # react immediately
    scaleDown:
      stabilizationWindowSeconds: 60   # wait 60s of calm before scaling down, avoids flapping

Applied, the cluster reports a real HPA reading rather than <unknown>, which means metrics-server is feeding it and it will react to load. Note the pod IPs: each node owns a slice of the pod network and the CNI routes between them.

$ kubectl get pods -o wide
NAME                      READY   STATUS    IP          NODE
loadlab-b6ddd7b86-cchg5   1/1     Running   10.42.0.5   k3s-server
loadlab-b6ddd7b86-wpfwl   1/1     Running   10.42.1.2   k3s-agent

$ kubectl get svc loadlab
NAME      TYPE       CLUSTER-IP      PORT(S)        AGE
loadlab   NodePort   10.43.123.145   80:30080/TCP   98s

$ kubectl get hpa loadlab
NAME      REFERENCE            TARGETS       MINPODS   MAXPODS   REPLICAS
loadlab   Deployment/loadlab   cpu: 1%/50%   2         10        2

Firing Load

# On the laptop. Listens on the LAN, forwards every connection into the NodePort.
socat TCP-LISTEN:8080,fork,reuseaddr TCP:192.168.122.10:30080

# Three-hop test:
curl http://192.168.122.10:30080/work?ms=50   # laptop straight to NodePort
curl http://localhost:8080/work?ms=50          # laptop through socat
curl http://192.168.1.12:8080/work?ms=50       # desktop across the LAN

Test the path in escalating hops, so that if something breaks you know which segment failed. Repeated calls show the response alternating between pod hostnames, the Service load balancing in real time.

The load is a k6 ramping arrival-rate test. Arrival rate, not a fixed number of virtual users, because the right model is requests per second the way real traffic arrives, with k6 adding workers as needed to sustain the rate. SLO thresholds are baked in so the run passes or fails outright instead of just printing numbers.

import http from 'k6/http';
import { check } from 'k6';

// Ramping arrival-rate: control REQUESTS PER SECOND, not just virtual users.
// k6 adds VUs as needed to sustain the rate, which models real arriving traffic.
export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 500,
      stages: [
        { target: 20,  duration: '30s' },  // warm up
        { target: 50,  duration: '1m'  },  // cross the 50% HPA threshold
        { target: 100, duration: '1m'  },  // expect scale-up
        { target: 200, duration: '2m'  },  // hammer, expect the ceiling
        { target: 0,   duration: '30s' },  // ramp down, watch HPA scale back
      ],
    },
  },
  // SLOs: pass/fail instead of vibes.
  thresholds: {
    http_req_duration: ['p(95)<500', 'p(99)<1000'],
    http_req_failed:   ['rate<0.01'],
  },
};

const BASE = __ENV.TARGET || 'http://192.168.1.12:8080';

export default function () {
  const res = http.get(`${BASE}/work?ms=200`);
  check(res, { 'status 200': (r) => r.status === 200 });
}

Results

I ran it twice. First with k6 on the laptop, then from the desktop, expecting the desktop run to go higher now that the generator wasn't competing with the VMs for CPU. It didn't. The two runs were nearly identical, which is the finding in itself: it ruled out the generator as the bottleneck and pointed at the cluster as the real limit.

  THRESHOLDS
    http_req_duration
    x 'p(95)<500'  p(95)=671.05ms
    v 'p(99)<1000' p(99)=912.31ms
    http_req_failed
    v 'rate<0.01'  rate=0.00%

  TOTAL RESULTS
    checks_succeeded...: 100.00%  28015 out of 28015
    http_req_duration..: avg=375.19ms  med=355.02ms  max=1.29s  p(90)=575.09ms  p(95)=671.05ms
    http_req_failed....: 0.00%   0 out of 28015
    http_reqs..........: 28015   93.383226/s
    dropped_iterations.: 34      0.113333/s
    vus_max............: 84
  # HPA scaled 2 -> 10 pods, all Running, CPU pegged.

Reading the Numbers

Saturation, not failure

Zero errors, but the p95 SLO was breached. The system stayed correct under load and got slow. That is graceful degradation, and it's a more common real-world shape than an outright fall over. The decisive number is the gap between requested and achieved throughput: I asked for 200 req/s and the system topped out around 93, with k6 unable to even launch some iterations on schedule.

The HPA scaled to its maximum of 10 pods, all Running, all CPU pegged, so the cluster wasn't idle behind a starving funnel, it was genuinely maxed. Each request is 200 ms of pure CPU. Two nodes of 2 vCPUs can only hash so fast. Moving the generator to a 16-thread desktop changed nothing because the generator was never the constraint.

Throughput is not concurrency

It's tempting to read "peak 84 VUs" as "we handled about 100 concurrent users." That conflates two things. The honest figures are 93 requests per second of throughput and a peak of 84 in-flight connections. The relationship is Little's Law:

concurrency = throughput x latency
            = 93 req/s x 0.375 s
            ~ 35 requests in flight at any instant
            (k6 held up to 84 workers to sustain that through the slow tail)

k6 needed up to 84 workers to sustain 35 in-flight requests because latency was high and each worker spends most of its time waiting. A virtual user with zero think-time also stands in for many real humans, who read and pause between clicks. "Concurrent users" is the wrong unit.

The number travels with its workload

93 req/s is not this cluster's capacity in general. It's its capacity for 200 ms of pure CPU per request. A real service doing 5 ms database lookups would push vastly higher rates on the same hardware, because it isn't burning a core per request.

HPA scales pods, nothing scaled nodes

The autoscaler hit maxReplicas of 10 and stopped. That cap bound the test, not node capacity, because these pods are tiny. The HPA only ever scales pods. Nothing here scales the cluster itself. In a cloud environment you'd pair it with a cluster autoscaler that adds nodes when pods can't be scheduled. With a fixed two nodes, the ceiling is hard.

After scale-down the surviving pods clustered on the agent, which is expected: the server node is also running the API server, scheduler, controller-manager, datastore, CoreDNS, and metrics-server, so it has less free capacity. In production you often taint the control plane so workloads never land on it at all.

What I Would Do Next

• Raise maxReplicas to force the node scheduling wall and watch pods go Pending with Insufficient cpu. That completes the pod-versus-node autoscaling picture.
• Find the SLO-respecting capacity, not just the saturation point. Ramp gently and find where p95 crosses 500 ms. That is the number you'd promise in an SLA, and it's lower than the saturation rate.
• Put it behind a real ingress instead of socat, and move off WiFi to remove that from the latency budget.

Summary

None of this is a production deployment, and it was never meant to be. It's a place to watch a system bend under load I generated, on hardware I can point at, with numbers I measured. The fights it picked along the way taught me more than a clean run would have.

Further Reading

• k3s documentation
• k6 ramping arrival-rate executor
• Kubernetes HorizontalPodAutoscaler
• Little's Law