How we benchmarked rustle against four proxies

A benchmark is only worth what you can check. rustle came first in all 41 tests of our campaign, which is a result that deserves suspicion. This post explains what we did to keep the comparison fair, what we did not do, and where the numbers stop meaning anything. The results themselves, with every value, are on the benchmarks page.

The machines

The campaign ran on three dedicated servers (Scaleway Elastic Metal, Paris), on one private 25 Gbit/s network:

• a load generator, with an AMD EPYC 8224P (24 cores);
• the proxy under test, on the same kind of server, pinned to a single core, with the sibling hyperthread switched off and the network interrupts moved to other cores;
• an origin that serves static files, and a sink that discards what it receives, on a third server.

We measured the network with iperf3: 23.5 Gbit/s, from the generator to the proxy and from the proxy to the origin. All three machines run Ubuntu 26.04.1 with the same kernel and the same system settings. Before a run, a script compares the three machines (settings, CPU governor, MTU, kernel, Docker, the version of the code) and refuses to start if anything differs. Proxies run in Docker containers on the host network, limited to their core.

The load generator is rustle’s own bench-client, written in Rust. It checks the size of every response body it receives. The origin is a small static server in Rust, which costs about 3 microseconds per request, so it does not limit the proxy.

The rules

• One core and one worker for every proxy. The main comparison asks what a proxy does with one core. We also ran a multi-core test, described below.
• Authentication on for everyone, with one key.
• A fixed seed and a shuffled order. Tests run in an order drawn from a seed, and the proxy is restarted before each one.
• The median of three runs of 15 seconds each, after a warm-up run.
• The best level of concurrency, not a fixed one. Each test runs at several numbers of connections and we keep the best result without a single client error. We changed to this after a fixed level of 64 connections made 3proxy look 5.8 times slower than it is: at 64 connections it used 19% of its CPU, at 1,024 it used 99%.
• A run is thrown away if the client sees any error, if a tunnel test opens fewer tunnels than requested, or if the load generator itself is saturated.
• A proxy that cannot do a test is shown as unsupported, never as zero. 3proxy and Tinyproxy have no HTTPS listener; Tinyproxy does not support WebSocket.

For HTTPS, the generator offers the same cipher first (AES-128-GCM) so every proxy negotiates TLS 1.3 with it. With the generator’s default order, Squid would have used another one, and the comparison would not have been like for like.

Giving the others a fair chance

A proxy with a default configuration is not always at its best. We tested each setting on a development machine before choosing it, and wrote the numbers down in FAIRNESS.md in the repository. Some examples:

• 3proxy, with its default configuration, served 1,510 requests per second. With TCP_NODELAY turned on for both connections, it served 5,490. That is a missing option, not the speed of 3proxy, so we turned it on.
• gost was tested with several values of the Go garbage collector setting. We kept GOGC=400, which gave about 15% more.
• Squid ran without cache, with the same 1 MB socket buffers as rustle, and with the results of its authentication helper cached.
• Tinyproxy was built without the features we do not use, with the client limit raised.

We also left some things out on purpose: no tricks around authentication, and no setting that trades much memory for a few percent.

The campaign is not neutral on every point, and we list them. rustle ran with 1 MB socket buffers, which Squid also got; the others use the kernel’s automatic sizing. Tinyproxy closes its connection to the server after every request. And rustle’s configuration for the benchmark differs from the default in several places: max_connections is raised, the access log is off, and private destinations are allowed, because the test network is private.

What the numbers say, and do not say

Where rustle came out well, we say it with a margin. The differences below 3 to 5% are noise and not conclusions. On the three tests that move the most data (1 MB, 10 MB and tunnels), rustle was limited by the network, not by its CPU, and the other proxies by their CPU. The lead there is therefore at least what we show, and probably more.

On the other side:

• One campaign. One seed, three repeats, no confidence intervals. The numbers cannot be compared with another campaign.
• One kind of traffic. A static origin with a known size. Uploads, chunked bodies, SOCKS5, IPv6 and many different origins are not in the matrix. DNS is resolved once, so its cost is not measured.
• Multi-core was measured on one test: plain HTTP, 1 KB, from 1 to 8 cores. TLS, HTTP/2 and tunnels were not.
• Latency is closed-loop: each client waits for its answer before sending the next request. We only publish a latency for WebSocket.
• The CPU per request includes the kernel work of the proxy’s container, but not the network work done on other cores.

Reproduce it, or tell us we are wrong

Everything published is generated from the raw results of each run, which are committed in the repository, and a check refuses a change that does not match them. You can rerun a proxy on your own machine with make bench, or the full three-machine run with the scripts of the benchmarks folder.

If you think a proxy was given a wrong setting, open an issue or a pull request with the configuration and the numbers. It goes in, and the campaign is run again.