A Multi-Node k3s Cluster, Load Tested to Saturation

June 2026 · k3s · Kubernetes · libvirt · netfilter · k6 · SRE · homelab

I wanted a Kubernetes setup I could break on purpose. Not a managed cluster where the interesting failures are hidden behind a control plane I never touch, but something I provisioned from bare VMs, wired up myself, and then hit hard enough to watch it bend. The goal was never a production deployment. It was the loop of building it, putting load on it, seeing where it gives, and working out why.

This post is the whole thing end to end. A two-node k3s cluster on libvirt VMs running on a laptop, a deliberately CPU-heavy Go service, a horizontal pod autoscaler, and a k6 ramp fired from a separate desktop. It also covers the part most tutorials skip: a nasty fight between Docker, firewalld, and libvirt over who owns the host firewall, which is what actually ate most of the evening. Every number here is real and measured on the hardware described below.

The Hardware

Three machines, each with a distinct job. The split matters: the load generator has to be stronger than the target, or you end up measuring the generator instead of the service.

The two cluster nodes are VMs on the laptop, each given 2 GB RAM and 2 vCPUs. That is tight, and the tightness turns out to help: it puts the scaling wall close enough that you hit it inside a five minute test.

k3s v1.35.5+k3s1 · containerd 2.2.3-k3s1 · Ubuntu 24.04 Noble (cloud image) · kernel 6.8.0-117-generic

The Whole Path

The desktop cannot reach the pods directly. Pods live on libvirt's NAT network (192.168.122.0/24), which is internal to the laptop, and libvirt NAT is one way: VMs reach out, nothing reaches in. So the laptop has to forward a LAN port into the cluster. That forward is the laptop acting as a small load balancer in front of the NodePort. Here is the full request path.

Provisioning the VMs

I drive the whole VM lifecycle from one idempotent script (in the sidebar as setup-lab.sh). It detects the distro, installs the libvirt and qemu stack with the right package manager, downloads the Ubuntu cloud image once, then tears down and rebuilds both VMs from a clean state on every run. Building by hand first and only then codifying it is the right order: you automate what you understand, not what you are hoping works.

Two decisions in there are worth calling out, because both came from getting burned.

Static addresses by MAC, not whatever DHCP feels like

My first runs addressed the VMs by their DHCP-assigned IPs. Leases expire, VMs reacquire different addresses, and suddenly an IP you have been treating as fixed points at nothing. I spent real time SSHing to a dead address that a previous VM incarnation had held. The fix is to assign each VM a fixed MAC and pin a DHCP reservation for it, so the node always lands on the same address.

The --live --config pair is the recurring lesson of the whole project. One flag changes the running state, the other changes the persistent definition. Apply only one and you get the classic "it works until reboot" or "my change vanishes on restart" confusion. Think about both planes every time, with iptables, with sysctls, with this.

Autostart everything, or a reboot eats your cluster

I rebooted the laptop at one point and everything was gone. Not corrupted, just off. The libvirt network had not been set to autostart and neither had the VMs, so a cold boot left the bridge down and both VMs powered off. A service that is not enabled to start on boot quietly disappears on the next reboot, and you spend the morning confused about where your infrastructure went. The script now sets autostart on the network and both domains.

The Netfilter Fight

This is the part that actually took the evening, and it is the most useful thing in the post. The VMs came up fine, SSH worked, but they had no internet. The gateway was reachable, DNS resolved, raw outbound packets died.

Gateway reachable plus DNS resolving plus no route to the wider internet points squarely at host side NAT, not the VM. So I looked at what the host was actually doing to forwarded packets. The answer was in the NAT table.

There were masquerade rules for Docker's networks (172.17.0.0/16, 172.18.0.0/16) and nothing for libvirt's 192.168.122.0/24. Docker was installed on the host, and when Docker starts it installs its own netfilter rules and sets the FORWARD chain policy to drop. In doing so it had evicted libvirt's own masquerade rule. VM packets hit a FORWARD policy of drop with no masquerade to rewrite their source address, and quietly died.

The root cause is structural: three different things on this host all want to manage netfilter. Docker, firewalld, and libvirt each install rules, and the more aggressive ones stomp the others. Every time I cycled the network, they re-fought and the bridge or the NAT rule fell over again. I went around this loop more than once before naming it properly.

I do not need Docker on the host at all. The cluster's containers run inside the VMs under containerd, not on the laptop. So the clean fix was to take the most aggressive actor out of the fight entirely, then let libvirt reassert its rules.

Stopping the daemon does not flush rules it already loaded, so a reboot (with Docker now disabled, so it does not come back) gave the cleanest slate: libvirt started unopposed and installed its NAT rules with nothing to clobber them. After that, outbound worked and stayed working. For completeness, the two host knobs that also have to be right are IP forwarding and not having firewalld quietly filter the bridge:

Installing k3s

Server first, because the server generates the token the agent needs to join. The flags are all deliberate.

• --node-ip pins the address k3s advertises. VMs are often multi-homed and k3s can pick the wrong interface. Same "do not trust auto-detected addresses" lesson as the DHCP pinning.
• --tls-san adds that IP to the API server certificate so kubectl from the laptop does not hit a cert mismatch.
• --write-kubeconfig-mode=644 makes the kubeconfig readable without sudo. Fine for a lab.
• --disable=traefik --disable=servicelb drops the bundled ingress and load balancer. I want to install those myself later, and on a 2 GB node every megabyte of control plane overhead is capacity taken from workloads.

Grab the join token, then join the agent by pointing it at the server's pinned address.

From the server, both nodes should report ready.

One thing that looks like an error and is not: running kubectl on the agent fails with connection to localhost:8080 refused. That is correct. Only the control plane node runs an API server. The worker has nothing local to talk to. You run cluster commands against the server, or from your laptop once the kubeconfig is set up.

kubeconfig hygiene

k3s writes a context literally named default, which is exactly how people end up running a destructive command against the wrong cluster. I already had another cluster in my kubeconfig, so I pulled k3s's config into a separate file, rewrote the server address from localhost to the pinned IP, renamed everything to k3s-lab, and merged it after backing up the original.

The Service Under Test

The application is intentionally boring so the operational layer is the interesting part. A small Go service (sidebar: main.go) with four endpoints:

• /healthz liveness, always cheap.
• /readyz readiness, with a toggle so I can simulate a pod going unready.
• /work?ms=200 burns CPU for the requested milliseconds by hashing in a tight loop, then reports which pod served it. This is what makes the service systems-heavy: hitting it pegs a core.
• /metrics Prometheus counters, a latency histogram, and an in-flight gauge.

It builds as a static binary on a distroless base, around 10 to 15 MB, no shell and no package manager in the final image. The wrinkle is getting that image onto the cluster with no registry, and with Docker now gone from the host. The answer is Podman, which is daemonless and touches no firewall rules, so it does not reopen the fight. Build, save to a tarball, and import into containerd on both nodes, because a pod can be scheduled onto either and a node without the image fails to start it.

The deployment sets resource requests, and this is not optional hygiene. The HPA computes CPU utilization as a percentage of the request. With no request there is no denominator and the autoscaler does nothing. This is the single most common reason a first HPA demo quietly refuses to scale.

The full manifests are in the sidebar: deployment.yaml, service.yaml, hpa.yaml. Apply and verify.

Note the pod IPs: 10.42.0.x on the server, 10.42.1.x on the agent. Each node owns a slice of the pod network and the CNI routes between them. One pod landed on each node, which is the scheduler bin-packing across available capacity, not balancing by name. The HPA reads cpu: 1%/50%, a real number rather than <unknown>, which means metrics-server is feeding it and it will actually react to load.

Exposing It and Firing Load

The laptop forwards its LAN port into the NodePort with socat. Leave it running in its own terminal: it is the load balancer hop.

Test the path in escalating hops so that if something breaks you know exactly which segment failed: laptop straight to the NodePort, laptop through socat, then the desktop across the LAN. Repeated calls show the response alternating between the two pod hostnames, which is the Service load balancing in real time.

The load itself is a k6 ramping arrival-rate test (sidebar: ramp.js). Arrival rate, not a fixed number of virtual users, because the right model is requests per second the way real traffic arrives, with k6 adding workers as needed to sustain the rate. It ramps to 200 requests per second of /work?ms=200 over five minutes, with SLO thresholds baked in so the run passes or fails outright instead of just printing numbers.

Results

I ran it twice. First with k6 on the laptop, then from the desktop, expecting the desktop run to go higher because the generator was no longer competing with the VMs for CPU. It didn't. The two runs were nearly identical, which is the finding in itself: it ruled out the generator as the bottleneck and pointed at the cluster as the real limit. Here is the desktop run.

Reading the Numbers

Saturation, not failure

Zero errors, but the p95 SLO was breached. The system stayed correct under load and got slow. That is graceful degradation, and it is the more common real-world shape than an outright fall over. The decisive number is the gap between requested and achieved throughput: I asked for a ramp to 200 req/s and the system topped out around 93, with k6 unable to even launch some iterations on schedule. Roughly 93 req/s is the saturation point for this cluster at this workload.

Why 93, and why the generator did not matter

The HPA scaled to its maximum of 10 pods, all of them Running, all CPU pegged. So the cluster was not idle behind a starving funnel: it was genuinely maxed. Each request is 200 ms of pure CPU work. Two nodes of 2 vCPUs each can only do so much hashing per second, and at ten replicas all bursting to their CPU limit, around 93 completed requests per second is about all the CPUs could produce. Moving the generator from the laptop to the 16-thread desktop changed nothing because the generator was never the constraint.

Throughput is not concurrency

It is tempting to read "peak 84 VUs" as "we handled about 100 concurrent users." That conflates two different things. The honest figures are: 93 requests per second of throughput, and a peak of 84 concurrent in-flight connections (VUs). The relationship between them is Little's Law:

93 req/s times 0.375 s average latency is about 35 requests in flight at any instant. k6 needed up to 84 workers to sustain that because latency was high and each worker spends most of its time waiting. A virtual user with zero think-time also stands in for many real humans, who read and pause between clicks. So "concurrent users" is the wrong unit. The figure that holds without extra assumptions is 93 req/s sustained, about 35 in flight at once.

The number only means anything with its workload

93 req/s is not "this cluster's capacity" in general. It is its capacity for this specific workload: 200 ms of pure CPU per request. A real service doing 5 ms database lookups would push vastly higher request rates on the same hardware, because it is not burning a core per request. The throughput number means nothing without the workload it was measured under.

HPA scales pods, nothing scaled nodes

The autoscaler hit its maxReplicas of 10 and stopped. That cap is what bound the test, not the node capacity, because these pods are tiny (100m CPU request). The HPA only ever scales pods. Nothing here scales the cluster itself. In a cloud environment you would pair the HPA with a cluster autoscaler that adds nodes when pods cannot be scheduled. With a fixed two nodes, the ceiling is hard. Raising maxReplicas past what the nodes can hold is the next experiment: pods pile up Pending with Insufficient cpu, which is the lesson the HPA cannot solve on its own.

The control plane node carries less workload

After scale-down, the surviving pods clustered on the agent. That is expected. The server node is also running the API server, scheduler, controller-manager, the datastore, CoreDNS, and metrics-server, so it has less free capacity, and the scheduler bin-packs onto whichever node has room. The server "working" means running the cluster, not necessarily running your app pods. In production you often taint the control plane node so workloads never land on it at all. k3s leaves it schedulable by default for small clusters where you want the capacity.

What I Would Do Next

• Raise maxReplicas to force the node scheduling wall and watch pods go Pending with Insufficient cpu. That completes the pod-versus-node autoscaling picture.
• Find the SLO-respecting capacity, not just the saturation point. Ramp more gently and find the rate where p95 crosses 500 ms. That crossover is the number you would actually promise in an SLA, and it is lower than the saturation rate.
• Put the cluster behind a real ingress controller instead of socat plus a NodePort, and move off WiFi to remove that variable from the latency budget.
• Bump the nodes to more RAM. At 2 GB the observability stack competes hard with the workload, and the scaling wall is artificially close.

Summary

None of this is a production deployment, and it was never meant to be. It is a place to watch a system bend under load that I generated, on hardware I can point at, with numbers I measured. The fights it picked along the way taught me more than a clean run would have.

Further Reading

• k3s documentation
• k6 ramping arrival-rate executor
• Kubernetes HorizontalPodAutoscaler
• libvirt network XML format
• Little's Law