docs: add benchmarking blog posts and performance reference page#254
Draft
SamBarker wants to merge 25 commits into
Draft
docs: add benchmarking blog posts and performance reference page#254SamBarker wants to merge 25 commits into
SamBarker wants to merge 25 commits into
Conversation
tombentley
reviewed
May 13, 2026
tombentley
reviewed
May 13, 2026
|
|
||
| The direct crypto cost is 13.3% (11.3% AES-GCM + 2.0% Kroxylicious filter logic). But encryption adds indirect costs too: | ||
|
|
||
| - **Buffer management (+5.8%)**: encrypted records need to be read into buffers, encrypted, and written to new buffers — more allocation, more copying |
Member
There was a problem hiding this comment.
Did we ever figure out how to reuse the buffers more? I think that was a TODO at one point.
Member
Author
There was a problem hiding this comment.
Correct — the TODO was never addressed. A BufferPool class existed at one point but was deleted as unused in early 2024. Cipher instances are still created fresh per operation. These remain genuine open optimisation opportunities.
tombentley
reviewed
May 13, 2026
PaulRMellor
reviewed
May 15, 2026
Contributor
PaulRMellor
left a comment
PaulRMellor
left a comment
There was a problem hiding this comment.
I read through the first post. Reads well overall, and the tone is nice and approachable. I left a few suggestions, particularly in places where the AI-assisted wording feels a bit noticeable.
|
|
||
| All good benchmarking stories start with a hunch. Mine was that Kroxylicious is cheap to run — I'd stake my career on it, in fact — but it turns out that "trust me, I wrote it" is not a widely accepted unit of measurement. People want proof. Sensibly. | ||
|
|
||
| There's a practical question underneath the hunch too. The most common thing operators ask us is some variation of: "How many cores does the proxy need?" Which is really just "is this thing going to slow down my Kafka?" in a polite engineering hat. We'd been giving the classic answer: "it depends on your workload and traffic patterns, so you'll need to test in your environment." Which is true. And also deeply unsatisfying for everyone involved, including us. |
Contributor
There was a problem hiding this comment.
Suggested change
| There's a practical question underneath the hunch too. The most common thing operators ask us is some variation of: "How many cores does the proxy need?" Which is really just "is this thing going to slow down my Kafka?" in a polite engineering hat. We'd been giving the classic answer: "it depends on your workload and traffic patterns, so you'll need to test in your environment." Which is true. And also deeply unsatisfying for everyone involved, including us. | |
| There's a practical question underneath the hunch too. The most common thing operators ask us is some variation of: "How many cores does the proxy need?" Which is really just another way of asking: "is this thing going to slow down my Kafka?" We'd been giving the classic answer: "it depends on your workload and traffic patterns, so you'll need to test in your environment." Which is true, but not especially useful. |
|
|
||
| There's a practical question underneath the hunch too. The most common thing operators ask us is some variation of: "How many cores does the proxy need?" Which is really just "is this thing going to slow down my Kafka?" in a polite engineering hat. We'd been giving the classic answer: "it depends on your workload and traffic patterns, so you'll need to test in your environment." Which is true. And also deeply unsatisfying for everyone involved, including us. | ||
|
|
||
| So we stopped saying "it depends", and got off the fence: we built something you can run **yourselves** on your own infrastructure with your own workload, and measured it. Here are some representative numbers from ours. |
Contributor
There was a problem hiding this comment.
Suggested change
| So we stopped saying "it depends", and got off the fence: we built something you can run **yourselves** on your own infrastructure with your own workload, and measured it. Here are some representative numbers from ours. | |
| So instead of saying "it depends", we built something measurable you can run **yourselves** on your own infrastructure with your own workload, and measured it. Here are some representative numbers from ours. |
Contributor
There was a problem hiding this comment.
"got off the fence" might be a bit colloquial for non-native speakers
|
|
||
| ## Record encryption: now we're doing real work | ||
|
|
||
| Ok, so let's make the proxy smarter — make it do something people actually care about! [Record encryption](https://kroxylicious.io/documentation/0.20.0/html/record-encryption-guide) uses AES-256-GCM to encrypt each record passing through the proxy. AES-256-GCM is going to ask the CPU to work relatively hard on its own, but it's also going to push the proxy to understand each record it receives, unpack it, copy it, encrypt it, and re-pack it before sending it on to the broker. With all that work going on we expect some impact to latency and throughput. To answer our original question we need to identify two things: the latency when everything is going smoothly, and the reduction in throughput all this work causes. Monitoring latency once we go past the throughput inflection point isn't very helpful — it's dominated by the throughput limits and their erratic impacts on the latency of individual requests (a big hello to batching and buffering effects). |
Contributor
There was a problem hiding this comment.
Suggested change
| Ok, so let's make the proxy smarter — make it do something people actually care about! [Record encryption](https://kroxylicious.io/documentation/0.20.0/html/record-encryption-guide) uses AES-256-GCM to encrypt each record passing through the proxy. AES-256-GCM is going to ask the CPU to work relatively hard on its own, but it's also going to push the proxy to understand each record it receives, unpack it, copy it, encrypt it, and re-pack it before sending it on to the broker. With all that work going on we expect some impact to latency and throughput. To answer our original question we need to identify two things: the latency when everything is going smoothly, and the reduction in throughput all this work causes. Monitoring latency once we go past the throughput inflection point isn't very helpful — it's dominated by the throughput limits and their erratic impacts on the latency of individual requests (a big hello to batching and buffering effects). | |
| [Record encryption](https://kroxylicious.io/documentation/0.20.0/html/record-encryption-guide) is a more representative workload because the proxy actively processes each record. Record encryption uses AES-256-GCM to encrypt each record passing through the proxy. AES-256-GCM is going to ask the CPU to work relatively hard on its own, but it's also going to push the proxy to understand each record it receives, unpack it, copy it, encrypt it, and re-pack it before sending it on to the broker. With all that work going on we expect some impact to latency and throughput. To answer our original question we need to identify two things: the latency when everything is going smoothly, and the reduction in throughput all this work causes. Monitoring latency once we go past the throughput inflection point isn't very helpful — it's dominated by the throughput limits and their erratic impacts on the latency of individual requests (a big hello to batching and buffering effects). |
Contributor
There was a problem hiding this comment.
Is understand the right word here: "push the proxy to understand each record it receives"?
Would parse suffice?
SamBarker
force-pushed
the
blog/benchmarking-the-proxy
branch
from
6d70f9c to
e814763
Compare
showuon
reviewed
May 21, 2026
Contributor
showuon
left a comment
showuon
left a comment
There was a problem hiding this comment.
Thanks for the blog post. It's good to see some real numbers about kroxy benchmark. Left some comments for the Does my proxy look big in this cluster? post.
Comment on lines
+67
to
+70
| | Publish latency avg | 2.66 ms | 2.82 ms | +0.16 ms (+6%) | | ||
| | Publish latency p99 | 5.54 ms | 6.07 ms | +0.53 ms (+10%) | | ||
| | E2E latency avg | 253.16 ms | 253.76 ms | +0.60 ms (+0.2%) | | ||
| | E2E latency p99 | 499.00 ms | 499.00 ms | 0 | |
Contributor
There was a problem hiding this comment.
It'd be good to explain the difference between publish latency and E2E latency.
|
|
||
| The transition wasn't a clean cliff edge — the proxy alternated between sustaining and saturating in a narrow band just above the ceiling. That pattern is characteristic of running right at a limit: it's not that it suddenly falls over, it's that small fluctuations (GC pauses, scheduling jitter) are enough to tip it either way. Stay below 14k and you're fine. Creep above it and you'll notice. The numbers are not absolute — they are just what we measured on our cluster; your mileage **will vary**. | ||
|
|
||
| ### The ceiling scales with CPU budget |
Contributor
There was a problem hiding this comment.
I'd be interested in knowing if memory increasing helps here, in addition to CPU? I believe so since you mentioned GC overhead somewhere?
| | 37,200 msg/s | Publish avg | 9.12 ms | 12.19 ms | +3.07 ms (+34%) | | ||
| | 37,200 msg/s | Publish p99 | 74.88 ms | 113.15 ms | +38.27 ms (+51%) | | ||
|
|
||
| So we know that somewhere above 34k we're hitting a limit. Time to hunt out exactly where — enter the rate-sweep. |
Contributor
There was a problem hiding this comment.
When reading here, I'm confusing about the numbers before 34,000 msg/s. Does that mean as long as our rate is 34000 msg/s below, then we don't need to worry about the latency? Or it stays a fixed latency percentage as what we see in 34k here?
| |-----------|---------| | ||
| | CPU | AMD EPYC-Rome, 2 GHz | | ||
| | Cluster | 8-node OpenShift (5 workers, 3 masters), RHCOS 9.6 | | ||
| | Kafka | 3-broker Strimzi cluster, replication factor 3 | |
Contributor
There was a problem hiding this comment.
It'd be to good know the kafka version you're running on.
| | Cluster | 8-node OpenShift (5 workers, 3 masters), RHCOS 9.6 | | ||
| | Kafka | 3-broker Strimzi cluster, replication factor 3 | | ||
| | Kroxylicious | 0.20.0, single proxy pod, 1000m CPU limit | | ||
| | KMS | HashiCorp Vault (in-cluster) | |
Comment on lines
+124
to
+128
| | CPU limit | Comfortable ceiling | Saturation point | | ||
| |-----------|--------------------|--------------------| | ||
| | 1000m | ~80k msg/s | ~126k msg/s | | ||
| | 2000m | ~80k msg/s | above 160k msg/s | | ||
| | 4000m | ~160k msg/s | above 321k msg/s | |
Contributor
There was a problem hiding this comment.
I know we use 1 KB sized messages, but it's hard to clearly know the real throughput here. It'd be good to have the size number also appended, ex: ~126k msg/s (~126MB/s). Same to other tables.
SamBarker
commented
May 21, 2026
Covers methodology, test environment, passthrough proxy results, encryption latency and throughput ceiling, the per-connection scaling insight, and sizing guidance. Includes a TODO placeholder for the connection sweep results before publication. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Covers why we chose OMB over Kafka's own tools, the benchmark harness we built (Helm chart, orchestration scripts, JBang result processors), workload design rationale, CPU flamegraphs with embedded interactive iframes, the per-connection ceiling discovery, bugs found in our own tooling, and the cluster recovery incident. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Adds /performance/ as a dedicated quick-reference page with headline benchmark numbers, comparison tables, and sizing guidance, linked from both blog posts. Updates the existing Performance section in overview.markdown with the key headline numbers and a link to the full reference page. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
…aming - Shift publication dates to May 21 and May 28 - Replace speculative per-connection ceiling explanation with empirical finding: encryption throughput ceiling scales linearly with CPU budget (validated at 1000m, 2000m, 4000m) - Add sizing formula: CPU (mc) = 20 × produce_MB_per_s, with worked example - Add RF=3 masking caveat: initial 1-topic sweeps conflated Kafka replication ceiling with proxy CPU ceiling; coefficient derived from RF=1 multi-topic workloads - Post 2: add full investigation narrative — workload isolation approach, coefficient derivation, 4-core confirmation, and 2-core prediction/validation - Drop stale "future work" items that are now complete Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
The proxy is selectively L7: default infrastructure filters do genuine Kafka protocol work (address rewriting, API version negotiation, metadata caching) while high-volume produce/consume traffic bypasses full deserialisation via the decode predicate. The 1.4% proxy CPU share validates this design, not just reflects it. Also drop the Fyre cluster upgrade section — OCP-internal incident with no relevance to readers. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Warm up test environment intro: realistic deployment framing - Add conversational lead-in to sizing guidance in both documents - Improve caveats opener in Post 1 - Add caveats section to performance page (RF=3 masking, message size, horizontal scaling) Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- New opening: laptop/codebase/confidence → harness/cluster/nuance - Why not Kafka tools: add coordinated omission bullet with voice - What we built: reframe around two experimental questions (rate sweep, connection sweep) before tooling details; add two-dimensions framing - Banishing click-ops: replace dry Helm section with Red Hat/operator motivation and all-your-CRs joke - JSON always comes in megabytes: replace docs dump with signal/noise framing; sharpen Comparator vs Summariser distinction - Following the ceiling: rewrite as investigation arc (spare CPU → what were we hitting? → RF=3 masking → connection sweep → coefficient) - Rename Post 2 title to "How hard can it be??? Maxing out a Kroxylicious instance" - Revert slug rename (benchmarking-the-proxy-under-the-hood stays) - Update performance.markdown cross-links to match Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Replaces dry methodology notes with a fuller narrative arc: - Opens with the representative vs repeatable tension in benchmarking - Explains the single-partition choice and why it makes the author wince - Justifies RF=3: proxy adds one real hop, but RF=1 would double the hop count — not a fair production comparison - Multi-topic runs reconnect to representative: baseline tax at normal load - Rate sweep methodology explained as technique, not run-specific numbers Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Format all narrator asides as *(italic brackets)* to distinguish narrator voice from main text - Fix coordinated omission bullet missing bold formatting - Fix "tracking...tracking" redundancy in OMB paragraph - "it made me wince" → "*(I had to squirm to type it)*" — more honest, author reached for single-partition deliberately Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Reframe the takeaway: the proxy boils the latency-sensitive path to near-TCP-stack overhead while operating at Layer 7 — that's the win - Add paragraph explaining why overhead holds across 10/100 topics: the proxy doesn't contend between topics (unlike a broker which juggles disk I/O, partition leaders, and replication); the connection sweep validates linear throughput scaling Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Full investigation arc: spare CPU shock → NIC elimination → 4-producer test → anti-affinity attempt (3 nodes, 3 brokers, nowhere to go) → new cluster → baseline shock → RTT math reveals co-location → second penny drops on OMB scheduling → RF=1 unlocks proxy CPU ceiling → coefficient → prediction. Corrects several issues in the prior draft: Netty theory discarded (proxy metrics showed minimal back pressure); co-location framed at pod/node level not VM level; 37k flagged as the only figure from the original cluster; all coefficient and sweep numbers confirmed as coming from the new distributed cluster. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Applies accurate numbers from the distributed 8-node cluster (5 workers, 3 masters) across all three files, replacing figures from the original co-located cluster: - Cluster description: 6-node → 8-node (5 workers, 3 masters) - RF=3 throughput ceiling: 37.2k→14,600 msg/s (encryption), 50-52k→19,400 msg/s (baseline), 26%→25% reduction - Coefficient: 12.5 mc/MB/s → 9.7 measured / 10 mc/MB/s operator formula - Formula: expose general form (10 × total proxy MB/s) with fan-out explanation; 20 × produce MB/s remains the 1:1 shorthand - 1-core RF=1: ~40k ceiling replaced with safe at 80k (91ms p99), saturating at ~126k - 4-core validation: 447ms→247ms at 160k; catastrophic→elevated at 321k (1,706ms); saturation above 321k - 2-core: comfortable at 80k (850ms), sustaining at 160k (720ms) — saturation not yet measured, consistent with model - Netty aside corrected: thread count scales with availableProcessors() (CPU limit), not fixed at 4 Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Rewrites flamegraph intro with personal motivation: hot path minimalism, Amdahl's law framing, and honest admission that the full sweep story didn't come together - Adds forward reference to bugs section to stitch the structure together - Moves OSS transparency point into "Run it yourself" where it naturally belongs, with a TODO placeholder for the raw data link - Drops duplicate "we share our workings" phrase from flamegraph prose Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Fix punctuation on OMB methodology comparability sentence - Fix repeated "We leaned towards repeatable" in workload design section - Fix tense: "will make" -> "makes" for workload design aside - Fix typo: "died in the wool" -> "dyed in the wool" - Add closing paragraph to flamegraph section: proxy wins are real but we aren't going to make AES faster - Replace stale 36k msg/s flamegraph references with FIXME pending new profiler runs Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Change "All good" to "Every good benchmarking story starts" (Bob's suggestion) - Add TL;DR paragraph with key numbers and sizing formula; flagged with FIXME comment pending final benchmark run Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Fix lone space-hyphen-space to em dash in OMB description - Add runtime warning (~14 hours) before benchmark commands with link to the full blog post reproduction script as a gist Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Explains why MWU testing was added (PhD teammate asked "is the difference real?"), how check-significance.sh works (per-window p99, ~30 samples, p < 0.05), and the honest caveat that per-window samples aren't fully uncorrelated. Distinguishes clearly between what MWU covers (latency delta realness) and what the coefficient derivation doesn't (n=4, no significance test, untested across message sizes). Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Move p99 explanation before first passthrough table where percentiles are first encountered; remove duplicate from encryption section - Expand Layer 7 point with one sentence of context for non-technical readers: most Kafka proxies operate at L4, Kroxylicious parses every message yet still adds only 0.2 ms - Add distribution board analogy for independent connection handling vs broker shared resource contention - Simplify replication factor caveat to one sentence, linking to companion post for detail - Fix "Most proxies" → "Most proxies operate on Kafka" for accuracy Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Post 2 is dated 2026-05-28 — Jekyll skips future posts by default, causing post_url resolution to fail at build time. Replace linked references with plain "companion post" text; links will be restored via a follow-up PR when Post 2 goes live. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Convert TL;DR from prose to bulleted list (S1) - Soften "dominated by Kafka consumer fetch timeouts" to "likely dominated by" — this is an inference, not a measured fact (S5) - Inline definition of rate sweep at first use in sizing guidance (S9) - Broaden "With record encryption" to "With filters (record encryption is the representative example here)" (S10) Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Rename files and update front matter dates; update post_url reference in Post 2 to match Post 1's new filename. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Fix "opevators" → "operators" - Add .DS_Store and .op/ to .gitignore - Remove accidentally committed macOS metadata and 1Password plugin files Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
These files are generated by running Jekyll locally and should not be committed. Add glob patterns to .gitignore to prevent recurrence. Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
SamBarker
force-pushed
the
blog/benchmarking-the-proxy
branch
from
7c62876 to
6c40ee4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/performance/reference page summarising key numbers and linking to both postsoverview.markdownwith headline performance figures and a link to the reference pageStatus
Draft — the posts are first drafts. Known open items:
Test plan
./run.shand verify site renders athttp://127.0.0.1:4000//performance/page renders with correct tables/performance/work🤖 Generated with Claude Code