Add span-derived primary tags (CSS v1.3.0)#11402
Open
dougqh wants to merge 10 commits into
Open
Conversation
dougqh
added
type: enhancement
dougqh
force-pushed
the
dougqh/metrics-memory-efficiency
branch
from
46c04bd to
823a5d4
Compare
dougqh
force-pushed
the
dougqh/metrics-arbitrary-tags
branch
from
c552e73 to
42947dd
Compare
Contributor
|
dougqh
added a commit
that referenced
this pull request
May 21, 2026
Ports the adversarial JMH benchmark from #11402 down to this branch so we can compare #11381 vs master on a high-cardinality, high-throughput workload. Adapted to use ConflatingMetricsAggregator (pre-rename) and the FixedAgentFeaturesDiscovery / NullSink helpers already in ConflatingMetricsAggregatorBenchmark. 8 producer threads hammer publish() with unique (service, operation, resource, peer.hostname) per op so the aggregate cache fills+evicts continuously and the inbox saturates. tearDown prints the drop counters (inboxFull vs aggregateDropped) so the test verifies the subsystem stayed bounded under attack. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
May 26, 2026
Trim per-span work on metrics aggregator publish path
ConflatingMetricsAggregator.publish does a handful of redundant operations on
every span. None individually is large; together they show as ~2.5% on the
existing JMH benchmark once the benchmark actually exercises span.kind.
- dedup span.isTopLevel(): publish() reads it into a local, then shouldComputeMetric
read it again. Pass the cached value in.
- resolve spanKind to String once: master called toString() twice per span (once
inside spanKindEligible, once at the getPeerTags call site) and used HashSet
contains on a CharSequence (which routes through equals on String). Normalize
to String up front and reuse.
- lazy-allocate the peer-tag list: getPeerTags() always allocated an ArrayList
sized to features.peerTags() even when the span had none of those tags set.
Defer allocation until the first match; return Collections.emptyList() when
none hit. MetricKey already treats null/empty peerTags as emptyList, so no
behavior change.
Drop the spanKindEligible helper — the HashSet.contains call inlines fine in
shouldComputeMetric.
Update the JMH benchmark to set span.kind=client on every span. Without it the
filter path short-circuits before the peer-tag and toString work, so the wins
above aren't measurable. With it:
baseline 6.755 us/op (CI [6.560, 6.950], stdev 0.129)
optimized 6.585 us/op (CI [6.536, 6.634], stdev 0.033)
2 forks x 5 iterations x 15s. ~2.5% mean improvement and much tighter variance
fork-to-fork.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add SpanKindFilter and CoreSpan.isKind for bitmask-based kind checks
Introduce SpanKindFilter -- a tiny builder-built immutable filter whose state
is an int bitmask indexed by the span.kind ordinals already cached on
DDSpanContext. Each include* on the builder sets one bit (1 << ordinal); the
runtime check is a single AND against (1 << span's ordinal).
CoreSpan.isKind(SpanKindFilter) is the new entry point. DDSpan overrides it
to do the bit-test directly against the cached ordinal -- no virtual call,
no tag-map lookup. The two existing test-only CoreSpan impls (SimpleSpan
and TraceGenerator.PojoSpan, the latter in two source sets) implement isKind
by reading the span.kind tag and delegating to SpanKindFilter.matches(String),
which converts via DDSpanContext.spanKindOrdinalOf and does the same AND.
Refactor: DDSpanContext.setSpanKindOrdinal(String) now delegates to a new
package-private static spanKindOrdinalOf(String) so the same string-to-ordinal
mapping serves both the tag interceptor path and SpanKindFilter.matches.
This is groundwork -- nothing in the codebase calls isKind yet. The next
commit will replace the HashSet-based eligibility checks in
ConflatingMetricsAggregator with SpanKindFilter instances.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use SpanKindFilter in ConflatingMetricsAggregator
Replace the two ELIGIBLE_SPAN_KINDS_FOR_* HashSet<String> constants and the
SPAN_KIND_INTERNAL.equals check with three SpanKindFilter instances:
METRICS_ELIGIBLE_KINDS, PEER_AGGREGATION_KINDS, INTERNAL_KIND. Eligibility
checks now go through span.isKind(filter), which on DDSpan is a volatile
byte read against the already-cached span.kind ordinal plus a single bit-test.
Also defer the span.kind tag read: previously read at the top of the publish
loop and threaded through both shouldComputeMetric and the inner publish.
isKind no longer needs the string, so the read can move down into the inner
publish where it's still needed for the SPAN_KINDS cache key / MetricKey.
Supporting changes:
- DDSpanContext.spanKindOrdinalOf(String) is now public so non-DDSpan CoreSpan
impls can compute the ordinal at tag-write time.
- SpanKindFilter gains a public matches(byte) fast-path overload that callers
with a pre-computed ordinal use directly.
- SimpleSpan caches the ordinal in setTag(SPAN_KIND, ...), mirroring what
TagInterceptor does for DDSpanContext, and its isKind now hits the byte
fast path. Without this, the JMH benchmark (which uses SimpleSpan) would
re-derive the ordinal on every isKind call and overstate the cost.
Benchmark on the bench updated last commit (kind=client on every span,
4 forks x 5 iter x 15s):
prior commit 6.585 ± 0.049 us/op
this commit 6.903 ± 0.096 us/op
The slight regression is a SimpleSpan-via-groovy-dispatch artifact -- the
interface call to isKind through CoreSpan, then through SimpleSpan, then
through SpanKindFilter.matches, doesn't fold as aggressively as a HashSet
contains on a static field. In production DDSpan.isKind inlines to a context
field read + ordinal byte read + bit-test, so the production path is faster
than the prior HashSet approach. A DDSpan-based benchmark would show this;
the existing SimpleSpan-based one doesn't.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add DDSpan-based variant of ConflatingMetricsAggregator JMH benchmark
The existing ConflatingMetricsAggregatorBenchmark uses SimpleSpan, a groovy
mock. That's enough for measuring queue/CHM/MetricKey work, but it conceals
the production cost of CoreSpan.isKind: SimpleSpan's isKind goes through
groovy interface dispatch into SpanKindFilter.matches, while DDSpan.isKind
inlines to a context byte-read + bit-test.
This new benchmark uses real DDSpan instances created through a CoreTracer
(with a NoopWriter so finishing doesn't reach the agent). Same shape as the
SimpleSpan bench (64-span trace, span.kind=client, peer.hostname set).
Numbers (2 forks x 5 iter x 15s):
master: 6.428 +- 0.189 us/op (HashSet eligibility checks)
this branch: 6.343 +- 0.115 us/op (SpanKindFilter bitmask)
About 1.3% faster on the production path. The SimpleSpan benchmark in the
same conditions shows a ~2.2% slowdown -- the mock's dispatch shape gives a
misleading signal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten SpanKindFilter encapsulation
Make SpanKindFilter.kindMask and its constructor private now that DDSpan.isKind
no longer needs direct field access -- it delegates to SpanKindFilter.matches(byte).
The Builder.build() in the same outer class still constructs instances via the
private constructor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Defer MetricKey construction and cache lookups to the aggregator thread
Replace the producer-side conflation pipeline with a thin per-span SpanSnapshot
posted to the existing aggregator thread. The aggregator now builds the
MetricKey, does the SERVICE_NAMES / SPAN_KINDS / PEER_TAGS_CACHE lookups, and
updates the AggregateMetric directly -- all off the producer's hot path.
What the producer does now, per span:
- filter (shouldComputeMetric, resource-ignored, longRunning)
- collect tag values into a SpanSnapshot (1 allocation per span)
- inbox.offer(snapshot) + return error flag for forceKeep
What moved off the producer:
- MetricKey construction and its hash computation
- SERVICE_NAMES.computeIfAbsent (UTF8 encoding of service name)
- SPAN_KINDS.computeIfAbsent (UTF8 encoding of span.kind)
- PEER_TAGS_CACHE lookups (peer-tag name+value UTF8 encoding)
- pending/keys ConcurrentHashMap operations
- Batch pooling, batch atomic ops, batch contributeTo
Removed entirely:
- Batch.java -- the conflation primitive is no longer needed; the
aggregator's existing LRUCache<MetricKey, AggregateMetric> IS the
conflation point now.
- pending ConcurrentHashMap<MetricKey, Batch>
- keys ConcurrentHashMap<MetricKey, MetricKey> (canonical dedup)
- batchPool MessagePassingQueue<Batch>
- The CommonKeyCleaner role of tracking keys.keySet() on LRU eviction --
AggregateExpiry now just reports drops to healthMetrics.
Added:
- SpanSnapshot: immutable value carrying the raw MetricKey inputs + a
tagAndDuration long (duration | ERROR_TAG | TOP_LEVEL_TAG).
- AggregateMetric.recordOneDuration(long tagAndDuration) -- the single-hit
equivalent of the existing recordDurations(int, AtomicLongArray).
- Peer-tag values flow through the snapshot as a flattened String[] of
[name0, value0, name1, value1, ...]; the aggregator encodes them through
PEER_TAGS_CACHE on its own thread.
Benchmark results (2 forks x 5 iter x 15s):
ConflatingMetricsAggregatorDDSpanBenchmark
prior commit 6.343 +- 0.115 us/op
this commit 2.506 +- 0.044 us/op (~60% faster)
ConflatingMetricsAggregatorBenchmark (SimpleSpan)
prior commit 6.585 +- 0.049 us/op
this commit 3.116 +- 0.032 us/op (~53% faster)
Caveat on the benchmark: without conflation, the producer pushes 1 inbox
item per span instead of ~1 per 64. At the benchmark's synthetic rate the
consumer can't keep up and inbox.offer silently drops. The numbers measure
producer publish() latency only; consumer throughput at realistic span rates
is a follow-up to validate. Tuning maxPending matters more in this design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Report aggregator inbox-full drops via health metrics
With the per-span SpanSnapshot inbox path, the producer can lose snapshots
when the bounded MPSC queue is full -- silently, since inbox.offer() returns
a boolean we previously ignored. The conflating-Batch design used to absorb
~64x more producer pressure per inbox slot, so this is a new failure mode
worth surfacing.
Wire it through the existing HealthMetrics path:
- HealthMetrics.onStatsInboxFull() (no-op default).
- TracerHealthMetrics gets a statsInboxFull LongAdder and a new reason tag
reason:inbox_full reported under the same stats.dropped_aggregates metric
used for LRU evictions. Two LongAdders, two tagged time series.
- ConflatingMetricsAggregator.publish increments the counter when
inbox.offer(snapshot) returns false.
This doesn't fix the drop -- tuning maxPending and/or building producer-side
batching are the actual fixes. But it makes the failure visible in the same
place ops already watches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-producer-wins
Merge branch 'dougqh/conflating-metrics-producer-wins' into dougqh/conflating-metrics-background-work
Resize previousCounts for inbox-full health metric
The new reason:inbox_full reportIfChanged call advances countIndex to 51,
but previousCounts was still sized for 51 counters (max index 50), so the
metric never emitted and the resize warning fired every flush. Bump the
array to 52 and add a regression test that exercises the flush path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skip SpanSnapshot allocation when the inbox is already at capacity
publish() previously did all of the tag extraction (peer-tag pairs,
HTTP method/endpoint, span kind, gRPC status) and the SpanSnapshot
allocation before calling inbox.offer; on a full inbox the offer
failed and everything became garbage.
Early-out with an approximate size() vs capacity() check up front. The
jctools MPSC queue's size() is best-effort but that's fine: under-
estimation falls through to the existing offer-as-source-of-truth
path, over-estimation drops a snapshot that would have fit (and
onStatsInboxFull was about to fire on the next span anyway).
error is computed first so the force-keep return is correct whether
or not the snapshot is built.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge remote-tracking branch 'origin/master' into dougqh/conflating-metrics-background-work
Introduce slim PeerTagSchema; capture peer-tag values not pairs
Addresses sarahchen6's review comment on ConflatingMetricsAggregator
extractPeerTagPairs: replaces the worst-case-allocation + trim-and-copy
flat-pairs layout with a parallel-array carrier.
- New PeerTagSchema: minimal carrier of String[] names. Two flavors -- a
static INTERNAL singleton (one entry: base.service) for internal-kind
spans, and per-discovery built schemas for client/producer/consumer
spans. Deliberately no cardinality limiters or per-cycle state; that
layers on top in a later PR.
- ConflatingMetricsAggregator: caches the peer-aggregation schema keyed
on reference equality of features.peerTags() -- a single volatile read
+ a long compare on the steady-state producer hot path, no allocation.
The producer now captures only a String[] of values parallel to the
schema's names; the schema reference is carried on SpanSnapshot. The
prior "build worst-case pairs then trim" code is gone.
- SpanSnapshot: replaces String[] peerTagPairs with PeerTagSchema +
String[] peerTagValues. Producer drops the schema reference if no
values fired so the consumer short-circuits on null.
- Aggregator.materializePeerTags: now reads name/value pairs at the same
index from (schema.names, snapshot.peerTagValues). Counts hits once
for exact-size allocation; preserves the singletonList fast path for
the common one-entry case (e.g. internal-kind base.service).
Producer-side cost goes from "allocate String[2n] + walk + maybe trim"
to "single volatile read + walk + lazy String[n] only on first hit".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address PR #11381 review (round 2)
- Aggregator.materializePeerTags: fold the firstHit-discovery nested if
into a single guarded post-increment (amarziali, #3279243138). One
body line: `if (values[i] != null && hitCount++ == 0) firstHit = i;`.
- Drop redundant isKind(SpanKindFilter) overrides in both
TraceGenerator.groovy files (amarziali, #3279264553 / #3279382648).
CoreSpan.java:84 already supplies a default implementation that reads
the same span.kind tag.
- Bump TRACER_METRICS_MAX_PENDING default from 2048 -> 131072 to address
the capacity regression amarziali flagged (#3279378375). Without
producer-side conflation, the inbox now holds 1 SpanSnapshot per
metrics-eligible span instead of 1 conflated Batch per ~64 spans;
restoring effective capacity parity (~2048 * ~64 = 131072) prevents a
~64x rise in inbox-full drops at the same span rate. ~100 B per
SpanSnapshot puts the worst-case heap floor at ~13 MB -- bounded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover inbox-full fast-path in ConflatingMetricsAggregator.publish
Addresses PR #11381 review (amarziali, #3279325340 -- "Are the existing
tests covering this case?").
New ConflatingMetricsAggregatorInboxFullTest constructs the aggregator
with a small inbox (queueSize=8), deliberately does NOT call start() so
the consumer thread never drains, then publishes enough spans to
overflow the inbox. Verifies that healthMetrics.onStatsInboxFull() is
called at least once -- the fast-path's `inbox.size() >= inbox.capacity()`
short-circuit triggers when the producer-side queue is at capacity.
Test is Java + JUnit 5 + Mockito per the project convention for new
tests; uses a CoreSpan Mockito mock rather than the SimpleSpan Groovy
fixture so we don't depend on Groovy-then-Java compile order from the
test source set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reconcile PeerTagSchema once per reporting cycle on the aggregator thread
Addresses amarziali's review comment #3279340181 ("It would be more
efficient to trigger from the other side"). The producer-side reference
compare on every publish goes away; the aggregator thread reconciles
the cached schema against feature discovery once per reporting cycle.
- DDAgentFeaturesDiscovery: expose getLastTimeDiscovered() so callers
can detect a discovery refresh without copying the peerTags Set.
- PeerTagSchema: add `long lastTimeDiscovered` (plain, aggregator-only)
and `hasSameTagsAs(Set)`. of(Set, long) takes the timestamp; INTERNAL
uses a -1L sentinel since it's never reconciled.
- ConflatingMetricsAggregator:
* Drop the cachedPeerTagsSource volatile and the per-publish reference
compare.
* Producer fast path is now `cachedPeerTagSchema` volatile read +
null-check; first publish takes the one-time synchronized bootstrap.
* Add reconcilePeerTagSchema() that runs once per cycle on the
aggregator thread: fast-path timestamp compare, slow-path set
compare, bump-in-place when the set is unchanged.
- Aggregator: new `Runnable onReportCycle` constructor parameter, run at
the start of report() (before the flush, so any test awaiting
writer.finishBucket() observes the schema in its post-reconcile state
and so the next publish sees the new schema without a handoff).
- Update "should create bucket for each set of peer tags" to drive two
reporting cycles separated by a report() that triggers reconcile. The
old test relied on per-publish reference detection, which the new
design intentionally doesn't preserve -- the schema is now stable
within a cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add bootstrap + reconcile coverage for PeerTagSchema
Addresses round-3 review nice-to-haves on PR #11381.
- PeerTagSchemaTest: unit coverage for hasSameTagsAs() (the predicate
that drives the reconcile fast/slow path split), the of(Set, long)
factory, and the INTERNAL singleton. The hasSameTagsAs cases include
same-content-different-Set-reference (the case the reconcile fast path
relies on after a discovery refresh) and content-mismatch in either
direction.
- ConflatingMetricsAggregatorBootstrapTest: integration coverage for
the producer-side bootstrap + aggregator-thread reconcile flow.
* bootstrapHappensOnceOnFirstPublish -- three publishes against an
un-started aggregator (no consumer thread, no reconciles); verifies
features.peerTags() and features.getLastTimeDiscovered() are each
called exactly once.
* reconcileSkipsDeepCompareWhenTimestampMatches -- two cycles with
constant features.getLastTimeDiscovered(); each post-report
reconcile short-circuits on the timestamp fast path, so peerTags()
is called only by bootstrap (1 total).
* reconcileSurvivesTimestampBumpWhenTagsUnchanged -- timestamps bump
every reconcile, forcing the slow set-compare path; the tag set
stays identical, so the schema is preserved and continues to flush
buckets correctly across cycles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Preserve TRACER_METRICS_MAX_PENDING semantic + drop stale imports
TRACER_METRICS_MAX_PENDING previously counted conflating Batch slots
(~64 spans each). The inbox now holds 1 SpanSnapshot per slot, so
multiply the configured value by LEGACY_BATCH_SIZE (64) to keep
pre-existing customer overrides delivering the same effective
span-throughput capacity. Default stays at 2048 logical -> 131072
snapshot slots, identical to the prior 2048 batches * 64 spans.
Also drops two unused datadog.trace.core.SpanKindFilter imports left
behind in TraceGenerator.groovy after the isKind() override was removed
in favor of the CoreSpan default implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add AdversarialMetricsBenchmark for capacity-bound stress testing
Ports the adversarial JMH benchmark from #11402 down to this branch so
we can compare #11381 vs master on a high-cardinality, high-throughput
workload. Adapted to use ConflatingMetricsAggregator (pre-rename) and
the FixedAgentFeaturesDiscovery / NullSink helpers already in
ConflatingMetricsAggregatorBenchmark.
8 producer threads hammer publish() with unique (service, operation,
resource, peer.hostname) per op so the aggregate cache fills+evicts
continuously and the inbox saturates. tearDown prints the drop
counters (inboxFull vs aggregateDropped) so the test verifies the
subsystem stayed bounded under attack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Trim AdversarialMetricsBenchmark counters and clarify printout
Drop traceComputedCalls / totalSpansCounted: under 8-way contention
the volatile-long ++/+= pattern was losing ~20% of updates (296M
counted vs 245M reported), and the numbers duplicate signal JMH's
ops/s already provides.
Switch inboxFull / aggregateDropped to LongAdder so the printed drop
shape (the order-of-magnitude story the bench is built to tell) is
accurate under contention.
Replace the stale "both forks combined for this run" string with text
that matches the actual @fork(value=1) config and notes that counters
accumulate across warmup + measurement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Close PeerTagSchema reconcile race + cover the swap branch
buildPeerTagSchema previously read features.peerTags() before
features.getLastTimeDiscovered(). DDAgentFeaturesDiscovery exposes
those as two separate accessors against its volatile State -- a
state-swap interleaving could leave the cached schema tagged with a
NEWER timestamp than its names, after which the next reconcile
short-circuits on the timestamp compare and misses the tag-set update
until the next discovery refresh (~minute later).
Swap the read order so timestamp is captured first. With this
ordering, an interleaving leaves the schema OLDER than its names
instead -- the next reconcile sees a timestamp mismatch, runs the
deep compare, and self-heals on the very next cycle.
Also adds reconcileSwapsSchemaWhenTagSetChanges, which closes the
test gap on the slow-path swap branch
(cachedPeerTagSchema = PeerTagSchema.of(...)). End-to-end check via
the writer's captured MetricKeys: pre-swap snapshot carries only
peer.hostname, post-swap snapshot carries both peer.hostname and
peer.service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clarify materializePeerTags hit-counting loop
Splits the `if (values[i] != null && hitCount++ == 0)` conjunction
into nested ifs. Same semantics, no codegen impact after JIT --
just visibly says what the loop is doing rather than relying on
post-increment-inside-conjunction. Closes amarziali's review thread
on this block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop unused Tags imports flagged by codenarc
Leftover from removing the isKind() override in TraceGenerator earlier
in this session -- I dropped the SpanKindFilter import but missed
datadog.trace.bootstrap.instrumentation.api.Tags, which is no longer
referenced in either file.
Resolves codenarcTest and codenarcTraceAgentTest UnusedImport
violations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Update dd-trace-core/src/main/java/datadog/trace/common/metrics/PeerTagSchema.java
Co-authored-by: Sarah Chen <sarah.chen@datadoghq.com>
Address sarahchen6's review pass
PeerTagSchema.java: drop the duplicate Javadoc line that the GitHub UI
suggestion accept inadvertently added (it added rather than replaced),
collapsing back to the single intended line per sarahchen6's
suggestion. Original line said "no cardinality limiters or per-cycle
state" which was misleading since lastTimeDiscovered IS per-cycle
state; suggestion rightly drops that clause.
Config.java: wrap the TRACER_METRICS_MAX_PENDING * LEGACY_BATCH_SIZE
multiplication in Math.multiplyExact to fail fast on absurd customer
overrides (>= ~33M) rather than silently wrap to a negative int and
explode the MPSC queue allocation with a confusing downstream error.
Per sarahchen6's suggestion citing the codex bot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clamp TRACER_METRICS_MAX_PENDING instead of throwing on overflow
The previous Math.multiplyExact approach would fail the agent startup
with ArithmeticException on absurd customer overrides (>= ~33M for
the configured value). Clamping is gentler -- the agent starts
successfully and just runs with a capped inbox.
Long-promote the multiplication to a long so the product can't wrap,
then clamp to MAX_SAFE_ARRAY_SIZE (Integer.MAX_VALUE - 8, the JDK's
own SOFT_MAX_ARRAY_LENGTH convention for array allocations).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Suppress forbiddenApis for tearDown's System.err diagnostics
AdversarialMetricsBenchmark.tearDown prints drop counters via
System.err so a benchmark run shows how saturated each capacity bound
was (inbox-full drops, aggregate-cache drops). forbiddenApisJmh
disallows System.err by default to prevent excess logging in
production code -- not a concern for a JMH benchmark, where stderr is
the conventional channel for diagnostic output and matches the
existing pattern in ExtractorBenchmark / InjectorBenchmark.
Annotates tearDown with @SuppressForbidden (method-scoped, not class-
scoped) so the suppression is narrowly targeted to the three
println calls and any future hot-path code that lands in the
benchmark stays gated by the check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Use DDAgentFeaturesDiscovery.state() hash for PeerTagSchema reconcile
Addresses amarziali's review on getLastTimeDiscovered(): the existing
state() accessor returns a SHA-256 of the discovery response, which is
a more precise change key than the timestamp. Timestamp advances on
every successful refresh regardless of content; the hash only advances
when something actually changed -- so reconcile fast-path now fires
only on real change, not every cycle.
- PeerTagSchema: long lastTimeDiscovered -> String state. Factory
signature of(Set, long) -> of(Set, String). INTERNAL carries null
(it is never reconciled).
- ConflatingMetricsAggregator: read features.state() first then
peerTags() (same defensive ordering rationale -- if a discovery
refresh interleaves, leave the schema with stale state rather than
stale tags so the next reconcile re-runs the deep compare).
Objects.equals for null-tolerant comparison (state can be null
before discovery has produced a response).
- DDAgentFeaturesDiscovery: drop the public getLastTimeDiscovered()
accessor added on this branch -- the field stays private for the
existing throttling logic in discoverIfOutdated().
- Tests updated to mock state() instead of getLastTimeDiscovered().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Convert TRACER_METRICS_MAX_PENDING rationale to /* */ block comment
Addresses amarziali's readability nit (#3289149416) -- multi-line
prose reads better as a single block comment than as a stack of //
lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge remote-tracking branch 'origin/dougqh/conflating-metrics-background-work' into dougqh/conflating-metrics-background-work
Add cardinality-isolation companions to AdversarialMetricsBenchmark
Two new JMH benches that hold every dimension constant except one,
to attribute throughput deltas to a specific axis:
- HighCardinalityResourceMetricsBenchmark: ~1M distinct resource
values; service/operation/peer.hostname pinned. Exercises the
aggregate-cache LRU on the resource axis specifically.
- HighCardinalityPeerMetricsBenchmark: ~32K distinct peer.hostname
values; service/operation/resource pinned. Isolates the peer-tag
encoding hot path (PEER_TAGS_CACHE lookups, UTF8 encoding,
parallel-array capture in SpanSnapshot).
Same shape as AdversarialMetricsBenchmark (8 threads, 2x15s warmup +
5x15s measurement, 1 fork) and reuse its CountingHealthMetrics so the
inbox-full vs aggregate-dropped counters print on teardown for an
apples-to-apples comparison.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
May 29, 2026
Trim per-span work on metrics aggregator publish path
ConflatingMetricsAggregator.publish does a handful of redundant operations on
every span. None individually is large; together they show as ~2.5% on the
existing JMH benchmark once the benchmark actually exercises span.kind.
- dedup span.isTopLevel(): publish() reads it into a local, then shouldComputeMetric
read it again. Pass the cached value in.
- resolve spanKind to String once: master called toString() twice per span (once
inside spanKindEligible, once at the getPeerTags call site) and used HashSet
contains on a CharSequence (which routes through equals on String). Normalize
to String up front and reuse.
- lazy-allocate the peer-tag list: getPeerTags() always allocated an ArrayList
sized to features.peerTags() even when the span had none of those tags set.
Defer allocation until the first match; return Collections.emptyList() when
none hit. MetricKey already treats null/empty peerTags as emptyList, so no
behavior change.
Drop the spanKindEligible helper — the HashSet.contains call inlines fine in
shouldComputeMetric.
Update the JMH benchmark to set span.kind=client on every span. Without it the
filter path short-circuits before the peer-tag and toString work, so the wins
above aren't measurable. With it:
baseline 6.755 us/op (CI [6.560, 6.950], stdev 0.129)
optimized 6.585 us/op (CI [6.536, 6.634], stdev 0.033)
2 forks x 5 iterations x 15s. ~2.5% mean improvement and much tighter variance
fork-to-fork.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add SpanKindFilter and CoreSpan.isKind for bitmask-based kind checks
Introduce SpanKindFilter -- a tiny builder-built immutable filter whose state
is an int bitmask indexed by the span.kind ordinals already cached on
DDSpanContext. Each include* on the builder sets one bit (1 << ordinal); the
runtime check is a single AND against (1 << span's ordinal).
CoreSpan.isKind(SpanKindFilter) is the new entry point. DDSpan overrides it
to do the bit-test directly against the cached ordinal -- no virtual call,
no tag-map lookup. The two existing test-only CoreSpan impls (SimpleSpan
and TraceGenerator.PojoSpan, the latter in two source sets) implement isKind
by reading the span.kind tag and delegating to SpanKindFilter.matches(String),
which converts via DDSpanContext.spanKindOrdinalOf and does the same AND.
Refactor: DDSpanContext.setSpanKindOrdinal(String) now delegates to a new
package-private static spanKindOrdinalOf(String) so the same string-to-ordinal
mapping serves both the tag interceptor path and SpanKindFilter.matches.
This is groundwork -- nothing in the codebase calls isKind yet. The next
commit will replace the HashSet-based eligibility checks in
ConflatingMetricsAggregator with SpanKindFilter instances.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use SpanKindFilter in ConflatingMetricsAggregator
Replace the two ELIGIBLE_SPAN_KINDS_FOR_* HashSet<String> constants and the
SPAN_KIND_INTERNAL.equals check with three SpanKindFilter instances:
METRICS_ELIGIBLE_KINDS, PEER_AGGREGATION_KINDS, INTERNAL_KIND. Eligibility
checks now go through span.isKind(filter), which on DDSpan is a volatile
byte read against the already-cached span.kind ordinal plus a single bit-test.
Also defer the span.kind tag read: previously read at the top of the publish
loop and threaded through both shouldComputeMetric and the inner publish.
isKind no longer needs the string, so the read can move down into the inner
publish where it's still needed for the SPAN_KINDS cache key / MetricKey.
Supporting changes:
- DDSpanContext.spanKindOrdinalOf(String) is now public so non-DDSpan CoreSpan
impls can compute the ordinal at tag-write time.
- SpanKindFilter gains a public matches(byte) fast-path overload that callers
with a pre-computed ordinal use directly.
- SimpleSpan caches the ordinal in setTag(SPAN_KIND, ...), mirroring what
TagInterceptor does for DDSpanContext, and its isKind now hits the byte
fast path. Without this, the JMH benchmark (which uses SimpleSpan) would
re-derive the ordinal on every isKind call and overstate the cost.
Benchmark on the bench updated last commit (kind=client on every span,
4 forks x 5 iter x 15s):
prior commit 6.585 ± 0.049 us/op
this commit 6.903 ± 0.096 us/op
The slight regression is a SimpleSpan-via-groovy-dispatch artifact -- the
interface call to isKind through CoreSpan, then through SimpleSpan, then
through SpanKindFilter.matches, doesn't fold as aggressively as a HashSet
contains on a static field. In production DDSpan.isKind inlines to a context
field read + ordinal byte read + bit-test, so the production path is faster
than the prior HashSet approach. A DDSpan-based benchmark would show this;
the existing SimpleSpan-based one doesn't.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add DDSpan-based variant of ConflatingMetricsAggregator JMH benchmark
The existing ConflatingMetricsAggregatorBenchmark uses SimpleSpan, a groovy
mock. That's enough for measuring queue/CHM/MetricKey work, but it conceals
the production cost of CoreSpan.isKind: SimpleSpan's isKind goes through
groovy interface dispatch into SpanKindFilter.matches, while DDSpan.isKind
inlines to a context byte-read + bit-test.
This new benchmark uses real DDSpan instances created through a CoreTracer
(with a NoopWriter so finishing doesn't reach the agent). Same shape as the
SimpleSpan bench (64-span trace, span.kind=client, peer.hostname set).
Numbers (2 forks x 5 iter x 15s):
master: 6.428 +- 0.189 us/op (HashSet eligibility checks)
this branch: 6.343 +- 0.115 us/op (SpanKindFilter bitmask)
About 1.3% faster on the production path. The SimpleSpan benchmark in the
same conditions shows a ~2.2% slowdown -- the mock's dispatch shape gives a
misleading signal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten SpanKindFilter encapsulation
Make SpanKindFilter.kindMask and its constructor private now that DDSpan.isKind
no longer needs direct field access -- it delegates to SpanKindFilter.matches(byte).
The Builder.build() in the same outer class still constructs instances via the
private constructor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Defer MetricKey construction and cache lookups to the aggregator thread
Replace the producer-side conflation pipeline with a thin per-span SpanSnapshot
posted to the existing aggregator thread. The aggregator now builds the
MetricKey, does the SERVICE_NAMES / SPAN_KINDS / PEER_TAGS_CACHE lookups, and
updates the AggregateMetric directly -- all off the producer's hot path.
What the producer does now, per span:
- filter (shouldComputeMetric, resource-ignored, longRunning)
- collect tag values into a SpanSnapshot (1 allocation per span)
- inbox.offer(snapshot) + return error flag for forceKeep
What moved off the producer:
- MetricKey construction and its hash computation
- SERVICE_NAMES.computeIfAbsent (UTF8 encoding of service name)
- SPAN_KINDS.computeIfAbsent (UTF8 encoding of span.kind)
- PEER_TAGS_CACHE lookups (peer-tag name+value UTF8 encoding)
- pending/keys ConcurrentHashMap operations
- Batch pooling, batch atomic ops, batch contributeTo
Removed entirely:
- Batch.java -- the conflation primitive is no longer needed; the
aggregator's existing LRUCache<MetricKey, AggregateMetric> IS the
conflation point now.
- pending ConcurrentHashMap<MetricKey, Batch>
- keys ConcurrentHashMap<MetricKey, MetricKey> (canonical dedup)
- batchPool MessagePassingQueue<Batch>
- The CommonKeyCleaner role of tracking keys.keySet() on LRU eviction --
AggregateExpiry now just reports drops to healthMetrics.
Added:
- SpanSnapshot: immutable value carrying the raw MetricKey inputs + a
tagAndDuration long (duration | ERROR_TAG | TOP_LEVEL_TAG).
- AggregateMetric.recordOneDuration(long tagAndDuration) -- the single-hit
equivalent of the existing recordDurations(int, AtomicLongArray).
- Peer-tag values flow through the snapshot as a flattened String[] of
[name0, value0, name1, value1, ...]; the aggregator encodes them through
PEER_TAGS_CACHE on its own thread.
Benchmark results (2 forks x 5 iter x 15s):
ConflatingMetricsAggregatorDDSpanBenchmark
prior commit 6.343 +- 0.115 us/op
this commit 2.506 +- 0.044 us/op (~60% faster)
ConflatingMetricsAggregatorBenchmark (SimpleSpan)
prior commit 6.585 +- 0.049 us/op
this commit 3.116 +- 0.032 us/op (~53% faster)
Caveat on the benchmark: without conflation, the producer pushes 1 inbox
item per span instead of ~1 per 64. At the benchmark's synthetic rate the
consumer can't keep up and inbox.offer silently drops. The numbers measure
producer publish() latency only; consumer throughput at realistic span rates
is a follow-up to validate. Tuning maxPending matters more in this design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Report aggregator inbox-full drops via health metrics
With the per-span SpanSnapshot inbox path, the producer can lose snapshots
when the bounded MPSC queue is full -- silently, since inbox.offer() returns
a boolean we previously ignored. The conflating-Batch design used to absorb
~64x more producer pressure per inbox slot, so this is a new failure mode
worth surfacing.
Wire it through the existing HealthMetrics path:
- HealthMetrics.onStatsInboxFull() (no-op default).
- TracerHealthMetrics gets a statsInboxFull LongAdder and a new reason tag
reason:inbox_full reported under the same stats.dropped_aggregates metric
used for LRU evictions. Two LongAdders, two tagged time series.
- ConflatingMetricsAggregator.publish increments the counter when
inbox.offer(snapshot) returns false.
This doesn't fix the drop -- tuning maxPending and/or building producer-side
batching are the actual fixes. But it makes the failure visible in the same
place ops already watches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-producer-wins
Merge branch 'dougqh/conflating-metrics-producer-wins' into dougqh/conflating-metrics-background-work
Add Hashtable and LongHashingUtils to datadog.trace.util
Two general-purpose utilities used by the client-side stats aggregator
work (PR #11382 and follow-ups), extracted into their own change so the
metrics-specific PRs can build on a smaller, reviewable foundation.
- Hashtable: a generic open-addressed-ish bucket table abstraction
keyed by a 64-bit hash, with a public abstract Entry type so client
code can subclass it for higher-arity keys. The metrics aggregator
uses it to back its AggregateTable.
- LongHashingUtils: chained 64-bit hash combiners with primitive
overloads (boolean, short, int, long, Object). Used in place of
varargs combiners to avoid Object[] allocation and boxing on the
hot path.
No callers within internal-api itself yet -- the metrics aggregator PR
will introduce the first usages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add AggregateTable + AggregateEntry backed by Hashtable
Standalone classes for swapping the consumer-side LRUCache<MetricKey,
AggregateMetric> with a multi-key Hashtable in the next commit. No call sites
use them yet.
- AggregateEntry extends Hashtable.Entry, holds the canonical MetricKey, the
mutable AggregateMetric, and copies of the 13 raw SpanSnapshot fields for
matches(). The 64-bit lookup hash is computed via chained
LongHashingUtils.addToHash calls (no varargs, no boxing of short/boolean).
- AggregateTable wraps a Hashtable.Entry[] from Hashtable.Support.create.
findOrInsert(SpanSnapshot) walks the bucket comparing raw fields, falling
back to MetricKeys.fromSnapshot on a true miss. On cap overrun, it scans
for an entry with hitCount==0 and unlinks it; if none, it returns null and
the caller drops the data point.
- MetricKeys.fromSnapshot extracts the canonicalization logic (DDCache
lookups + UTF8 encoding) from Aggregator.buildMetricKey, so the helper can
be called from AggregateTable on miss.
This also commits Hashtable and LongHashingUtils (added earlier, previously
uncommitted) and lifts Hashtable.Entry / Hashtable.Support visibility so
client code outside datadog.trace.util can build higher-arity tables -- the
case the javadoc describes but the original visibility didn't actually
support. Specifically: Entry is now public abstract with a protected ctor;
keyHash, next(), and setNext() are public; Support's create / clear /
bucketIndex / bucketIterator / mutatingBucketIterator methods are public.
Tests: AggregateTableTest covers hit, miss, distinct-by-spanKind, peer-tag
identity (including null vs non-null), cap overrun with stale victim, cap
overrun with no victim (returns null), expungeStaleAggregates, forEach,
clear, and that the canonical MetricKey is built at insert.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap LRUCache for AggregateTable in Aggregator + route disable() clear
Replace LRUCache<MetricKey, AggregateMetric> with the AggregateTable added
in the prior commit. The hot path in Drainer.accept becomes:
AggregateMetric aggregate = aggregates.findOrInsert(snapshot);
if (aggregate != null) {
aggregate.recordOneDuration(snapshot.tagAndDuration);
dirty = true;
} else {
healthMetrics.onStatsAggregateDropped();
}
On the steady-state hit path the lookup is a 64-bit hash compute + bucket
walk + matches(snapshot) -- no MetricKey allocation, no SERVICE_NAMES /
SPAN_KINDS / PEER_TAGS_CACHE lookups. The canonical MetricKey is now built
once per unique key at insert time, in MetricKeys.fromSnapshot.
Behavioral change in the cap-overrun path
-----------------------------------------
The old LRUCache evicted least-recently-used: at cap, a new insert would
push out the oldest entry regardless of whether it was live or stale.
AggregateTable instead scans for a hitCount==0 entry to recycle, and drops
the new key if none exists. Practical impact: in the common case where
the table holds a stable set of recurring keys, an unrelated burst of new
keys is dropped (and reported via onStatsAggregateDropped) rather than
evicting the established keys. The existing test that asserted "service0
evicted in favor of service10" is updated to assert the new semantics.
The other cap-related test ("should not report dropped aggregate when
evicted entry was already flushed") still passes unchanged: after report()
clears all entries to hitCount=0, the next wave of inserts recycles them.
Threading fix
-------------
ConflatingMetricsAggregator.disable() used to call aggregator.clearAggregates()
and inbox.clear() directly from the Sink's IO event thread, racing with the
aggregator thread mid-write. The race was tolerable for LinkedHashMap; it
is not for AggregateTable (chain corruption can NPE or loop). disable()
now offers a ClearSignal to the inbox so the aggregator thread itself
performs the table clear and the inbox.clear(). Adds one SignalItem
subclass + one branch in Drainer.accept; preserves the single-writer
invariant for AggregateTable end-to-end.
Removed: LRUCache import, AggregateExpiry inner class, the static
buildMetricKey / materializePeerTags / encodePeerTag helpers (now in
MetricKeys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eliminate MetricKey: inline its fields onto AggregateEntry
MetricKey existed for two reasons -- the prior LRUCache key role (now handled
by AggregateTable's Hashtable.Entry mechanics) and as the labels argument
to MetricWriter.add. The first is gone; the second is the only thing keeping
MetricKey alive. Fold its UTF8-encoded label fields onto AggregateEntry,
change MetricWriter.add to take AggregateEntry directly, and delete
MetricKey + MetricKeys.
What AggregateEntry now holds
-----------------------------
- 10 UTF8BytesString label fields (resource, service, operationName,
serviceSource, type, spanKind, httpMethod, httpEndpoint, grpcStatusCode,
and a List<UTF8BytesString> peerTags for serialization).
- 3 primitives (httpStatusCode, synthetic, traceRoot).
- AggregateMetric (the value being accumulated).
- The raw String[] peerTagPairs is retained alongside the encoded peerTags
-- matches() compares it positionally against the snapshot's pairs; the
encoded form is only consumed by the writer.
matches(SpanSnapshot) compares the entry's UTF8 forms to the snapshot's raw
String / CharSequence fields via content-equality (UTF8BytesString.toString()
returns the underlying String in O(1)). This closes a latent bug in the
prior raw-vs-raw matches(): if one snapshot delivered a tag value as String
and a later snapshot delivered the same content as UTF8BytesString, the old
Objects.equals would return false and the table would split into two
entries. Content-equality matching collapses them into one.
Consolidated caches
-------------------
The static UTF8 caches that used to live partly on MetricKey (RESOURCE_CACHE,
OPERATION_CACHE, SERVICE_SOURCE_CACHE, TYPE_CACHE, KIND_CACHE,
HTTP_METHOD_CACHE, HTTP_ENDPOINT_CACHE, GRPC_STATUS_CODE_CACHE, SERVICE_CACHE)
and partly on ConflatingMetricsAggregator (SERVICE_NAMES, SPAN_KINDS,
PEER_TAGS_CACHE) are all now on AggregateEntry. The split was duplicating
work -- SERVICE_NAMES and SERVICE_CACHE both cached service-name to
UTF8BytesString. One cache per field now.
API change: MetricWriter.add
----------------------------
Was: add(MetricKey key, AggregateMetric aggregate)
Now: add(AggregateEntry entry)
The aggregate lives on the entry. Single-arg.
SerializingMetricWriter reads the same UTF8 fields off AggregateEntry that it
previously read off MetricKey; the wire format is byte-identical.
Test impact
-----------
AggregateEntry.of(...) takes the same 13 positional args new MetricKey(...)
took, so test diffs are mostly mechanical:
new MetricKey(args) -> AggregateEntry.of(args)
writer.add(key, _) -> writer.add(entry)
ValidatingSink in SerializingMetricWriterTest now iterates List<AggregateEntry>
directly. ConflatingMetricAggregatorTest's Spock matchers (~36 sites) rely
on AggregateEntry.equals comparing the 13 label fields (not the aggregate)
so the mock matches by labels regardless of the aggregate state at call time;
post-invocation closures verify aggregate state.
Benchmarks (2 forks x 5 iter x 15s)
-----------------------------------
The change is consumer-thread only; producer publish() is unchanged.
SimpleSpan bench: 3.123 +- 0.025 us/op (prior: 3.119 +- 0.018)
DDSpan bench: 2.412 +- 0.022 us/op (prior: 2.463 +- 0.041)
Both within noise -- the win is structural (one less class, one less
allocation per miss, one fewer cache layer) rather than benchmarked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add unit tests for Hashtable and LongHashingUtils
LongHashingUtilsTest (14 cases):
- hashCodeX null sentinel + non-null pass-through
- all primitive hash() overloads match the boxed Java hashCodes
- hash(Object...) 2/3/4/5-arg overloads match the chained addToHash
formula they are documented to constant-fold to
- addToHash(long, primitive) overloads match the Object-version
- linear-accumulation invariant (31 * h + v) holds across a sequence
- iterable / deprecated int[] / deprecated Object[] variants match
chained addToHash
- intHash treats null as 0 (observable via hash(null, "x"))
HashtableTest (24 cases across 5 nested classes):
- D1: insert/get/remove/insertOrReplace/clear/forEach, in-place value
mutation, null-key handling, hash-collision chaining with disambig-
uating equals, remove-from-collided-chain leaves siblings intact
- D2: pair-key identity, remove(pair), insertOrReplace matches on
both parts, forEach
- Support: capacity rounds up to a power of two, bucketIndex stays
in range across a wide hash sample, clear nulls every slot
- BucketIterator: walks only matching-hash entries in a chain, throws
NoSuchElementException when exhausted
- MutatingBucketIterator: remove from head-of-chain unlinks, replace
swaps the entry while preserving chain, remove() without prior
next() throws IllegalStateException
Tests live in internal-api/src/test/java/datadog/trace/util and use the
already-present JUnit 5 setup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply spotless formatting to Hashtable and LongHashingUtils
Bring the new util/ files in line with google-java-format
(tabs → spaces, line wrapping, javadoc list markup) so
spotlessCheck passes in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add JMH benchmarks for Hashtable.D1 and D2
Compares Hashtable.D1 and Hashtable.D2 against equivalent HashMap
usage for add, update, and iterate operations. Each benchmark thread
owns its own map (Scope.Thread), but @Threads(8) is used so the
allocation/GC pressure that Hashtable is designed to avoid surfaces
in the throughput numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add benchmark results to HashtableBenchmark header
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Guard Support.sizeFor against overflow and use Integer.highestOneBit;
reject capacities above 1 << 30 instead of looping forever.
- Add braces around single-statement while bodies in BucketIterator.
- Split HashtableBenchmark into HashtableD1Benchmark / HashtableD2Benchmark.
- Add regression tests for Support.sizeFor bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix dropped argument in HashingUtils 5-arg Object hash
The 5-arg Object overload was forwarding only obj0..obj3 to the int
overload, silently dropping obj4. Also align LongHashingUtils.hash 3-arg
signature with its 2/4/5-arg siblings (int parameters) and strengthen
the 5-arg HashingUtilsTest to detect the missing-arg regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Split D1Tests and D2Tests into HashtableD1Test and HashtableD2Test;
extract shared test entry classes into HashtableTestEntries.
- Reduce visibility of LongHashingUtils.hash(int...) chaining overloads
to package-private; they are internal building blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop reflection in iterator tests via package-private D1.buckets
The iterator tests need a populated Hashtable.Entry[] to drive
Support.bucketIterator / mutatingBucketIterator. Relaxing D1.buckets
from private to package-private lets the same-package tests read it
directly, removing the reflection helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resize previousCounts for inbox-full health metric
The new reason:inbox_full reportIfChanged call advances countIndex to 51,
but previousCounts was still sized for 51 counters (max index 50), so the
metric never emitted and the resize warning fired every flush. Bump the
array to 52 and add a regression test that exercises the flush path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fold AggregateMetric into AggregateEntry
The label fields and the mutable counters/histograms are 1:1 with each
entry; carrying them on a separate object meant one extra allocation per
unique key plus an indirection on every hot-path update. Merging them
puts the counters directly on AggregateEntry, drops the entry.aggregate
hop, and consolidates ERROR_TAG / TOP_LEVEL_TAG onto the same class the
consumer uses to decode them.
AggregateTable.findOrInsert now returns AggregateEntry. Callers in
Aggregator and SerializingMetricWriter updated. Migrated
AggregateMetricTest.groovy to AggregateEntryTest.java per project policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Avoid capturing lambda in Aggregator.report
Add a context-passing forEach(T, BiConsumer) overload to AggregateTable,
mirroring TagMap's pattern. Aggregator.report now hands the writer in as
context to a static BiConsumer so no fresh Consumer is allocated each
report cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add context-passing forEach to Hashtable.D1 and D2
Mirrors the TagMap pattern: pairs the existing forEach(Consumer) with a
forEach(T context, BiConsumer<T, TEntry>) overload so callers can hand
side-band state to a non-capturing lambda and avoid the
fresh-Consumer-per-call allocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move forEach loop body to Support helper
Factors the unchecked (TEntry) cast out of D1.forEach / D2.forEach (and
the BiConsumer variants) into Support.forEach(buckets, ...). The cast
now lives in one place, mirroring how Entry.next() handles it, and the
D1/D2 methods become one-liners. Downstream higher-arity tables built
on Support gain the same helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Delegate AggregateTable.forEach to Support.forEach
Now that Hashtable.Support exposes the parameterized forEach helpers,
AggregateTable's own forEach methods can drop their duplicated loop body
and the (AggregateEntry) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move bucket-head cast to Support.bucket helper
Adds Support.bucket(buckets, keyHash) which returns the bucket head
already cast to the caller's concrete entry type. D1.get and D2.get
now drop the raw-Entry intermediate variable and walk the chain via
Entry.next() directly. The unchecked cast lives in one place,
consistent with Entry.next() and Support.forEach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Use Support.bucket and type chain walks as AggregateEntry
- findOrInsert: walks via Support.bucket(buckets, keyHash) instead of
Hashtable.Entry + intermediate cast; bucketIndex is only computed on
the miss path now.
- evictOneStale / expungeStaleAggregates: chain variables typed as
AggregateEntry from the head down, leveraging Entry.next()'s generic
inference, so the per-iteration getHitCount() checks drop their
(AggregateEntry) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop d1_/d2_ prefix from per-table benchmark methods
Holdover from when both lived in a shared HashtableBenchmark; redundant
now that each lives in its own class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTableIterator
Three consumer-facing helpers that callers building higher-arity tables on
top of Hashtable.Support kept open-coding:
- MAX_RATIO_NUMERATOR / _DENOMINATOR: the 4/3 multiplier for sizing a
bucket array from a target working-set under a 75% load factor.
- insertHeadEntry(buckets, bucketIndex, entry): the (setNext + array-store)
pair for splicing a new entry at the head of a bucket chain.
- MutatingTableIterator + Support.mutatingTableIterator(buckets): walks
every entry in the table (not filtered by hash) with remove() support,
for sweeps like eviction and expunge that aren't keyed to a specific
hash. Sibling of MutatingBucketIterator.
Tests cover the table-wide iterator at head-of-bucket and mid-chain
removal, empty buckets between live entries, exhaustion, and
remove-without-next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Simplify AggregateTable via new Hashtable.Support helpers
- Constructor sizing now uses Support.MAX_RATIO_NUMERATOR / _DENOMINATOR
instead of an open-coded * 4 / 3.
- findOrInsert delegates the chain-head splice to Support.insertHeadEntry.
- evictOneStale and expungeStaleAggregates both rewritten in terms of
Support.mutatingTableIterator. Drops the bespoke head-vs-mid-chain
branching that read as more complicated than the operation actually is.
Net -28 lines in AggregateTable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap MAX_RATIO numerator/denominator pair for a single float + scaled create()
Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float
MAX_RATIO constant, and add a Support.create(int, float) overload that
takes a scale factor. Callers now write Support.create(n, MAX_RATIO)
instead of stitching together the int arithmetic at the call site.
The scaled size is truncated (not ceiled) before going through sizeFor.
sizeFor already rounds up to the next power of two, so truncation just
absorbs float fuzz that would otherwise push a result like 12 * 4/3 =
16.0000005f past 16 and double the bucket array size for no reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Address second-round review on AggregateTable / Aggregator
- AggregateTable: switch to Support.create(maxAggregates, Support.MAX_RATIO)
now that the load-factor scaling is a Support concern.
- AggregateTable: replace open-coded "keyHash == X && matches(s)" with a
new AggregateEntry.matches(long keyHash, SpanSnapshot) overload that
bundles the hash gate.
- AggregateTable: rename local iterator var "it" -> "iter".
- Aggregator: drop WRITE_AND_CLEAR static field, inline as a non-capturing
lambda; the JIT reuses non-capturing lambdas, no need for the static
until a profile says otherwise.
- Aggregator: comment the ClearSignal branch with the thread-safety
rationale (single-writer invariant for AggregateTable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
Five small cleanups from a design re-review pass:
1. Support javadoc: drop the stale "methods are package-private" sentence;
most of them were made public in earlier commits for higher-arity
callers. Also drop the "nested BucketIterator" framing (iterators are
peers of Support inside Hashtable, not nested inside Support).
2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float)
deliberately truncates and is the canonical pathway.
3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so
the behavior difference is explicit: D1 uses Long.MIN_VALUE as a
sentinel that's collision-free against any int-valued hashCode(); D2
has no such sentinel and relies on matches() to resolve null/null vs
hash-0 collisions.
4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to
requestedSize. The cap is on the bucket-array length, not entry count;
the new name reflects that. Error messages updated to match.
5. Drop the `abstract` modifier on Hashtable in favor of `final` with a
private constructor. Nothing actually subclasses Hashtable -- the
abstract was a namespace device that read as "intended for extension."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that
derives the bucket index itself. Callers that already have a hash but
not the index (the common case) now avoid the redundant bucketIndex(...)
hop.
- D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the
new overload, drop the (thisBuckets local, bucketIndex compute,
setNext, store) sequence at each call site.
- D2.buckets: drop the `private` modifier to match D1.buckets. Both are
package-private so iterator tests in the same package can drive
Support.bucketIterator against the table's bucket array. Added a short
comment on both fields documenting the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
Three follow-ups from the design review:
- Make Hashtable.Entry.next private. All same-package readers
(BucketIterator) already had a next() accessor; the leftover direct
field reads now route through it. Closes the "mixed encapsulation"
gap where some readers used the accessor and same-package ones
reached for the field.
- BucketIterator and MutatingBucketIterator now document that chain-walk
work happens in next() (and the constructor for the first match);
hasNext() is an O(1) field read.
- Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction).
Both reuse the lookup hash for the insert on miss, avoiding the
double-hash that "get; if null then insert" callers would otherwise
pay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Use keyHash insertHeadEntry overload in AggregateTable.findOrInsert
Picks up the Support.insertHeadEntry(buckets, long keyHash, entry)
overload added on the util-hashtable branch; saves the redundant
Support.bucketIndex(buckets, keyHash) hop at the call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace // nullable comments with @Nullable annotations on AggregateEntry
Use javax.annotation.Nullable (the codebase's convention -- see DDSpan,
TagInterceptor, ScopeContext, etc.) on the four nullable label fields
(serviceSource, httpMethod, httpEndpoint, grpcStatusCode), their
getters, and the corresponding parameters of AggregateEntry.of.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop redundant load-factor comment from AggregateTable ctor
Support.MAX_RATIO and the scaled create(int, float) overload already
convey the 75% load-factor intent at the call site -- the inline
comment was duplicating their self-documentation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Import java.util.Objects in AggregateEntry instead of fully qualifying
Style nit -- the equals() method had eight fully-qualified references
to java.util.Objects.equals; add the import and drop the qualifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document evictOneStale cost and disable() best-effort offer
Two design-review trade-offs that won't change in this PR but should be
explicit at the call sites:
- AggregateTable.evictOneStale: O(N) scan per call (vs LRUCache's O(1)),
acceptable because the new policy drops the *new* key on cap-overrun
rather than evicting an established one -- so eviction is expected to
be rare. Cursor-caching is the future optimization if a workload runs
persistently at cap.
- ConflatingMetricsAggregator.disable: single inbox.offer(CLEAR) is
best-effort. If the inbox is full the clear is dropped, but the
system self-heals (supportsMetrics() is already false, the next
report-sink-rejection retries disable). Worst case is one extra cycle
of stale data, not a leak.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skip SpanSnapshot allocation when the inbox is already at capacity
publish() previously did all of the tag extraction (peer-tag pairs,
HTTP method/endpoint, span kind, gRPC status) and the SpanSnapshot
allocation before calling inbox.offer; on a full inbox the offer
failed and everything became garbage.
Early-out with an approximate size() vs capacity() check up front. The
jctools MPSC queue's size() is best-effort but that's fine: under-
estimation falls through to the existing offer-as-source-of-truth
path, over-estimation drops a snapshot that would have fit (and
onStatsInboxFull was about to fire on the next span anyway).
error is computed first so the force-keep return is correct whether
or not the snapshot is built.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hashtable: add missing braces and detach removed/replaced entries
Addresses PR #11409 review comments:
- #3267164119 / #3267165525: wrap every single-line if/break body in
braces (7 sites across BucketIterator, MutatingBucketIterator, and the
full-table Iterator).
- #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced
entry's next pointer after splicing it out of the chain in
MutatingBucketIterator.remove / .replace. Applied the same fix to the
full-table Iterator.remove for consistency.
Rationale: detaching prevents accidental traversal through a removed
entry via a stale reference and lets the GC reclaim a chain tail that
the removed entry was the last referrer to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Add Hashtable and LongHashingUtils to datadog.trace.util
Two general-purpose utilities used by the client-side stats aggregator
work (PR #11382 and follow-ups), extracted into their own change so the
metrics-specific PRs can build on a smaller, reviewable foundation.
- Hashtable: a generic open-addressed-ish bucket table abstraction
keyed by a 64-bit hash, with a public abstract Entry type so client
code can subclass it for higher-arity keys. The metrics aggregator
uses it to back its AggregateTable.
- LongHashingUtils: chained 64-bit hash combiners with primitive
overloads (boolean, short, int, long, Object). Used in place of
varargs combiners to avoid Object[] allocation and boxing on the
hot path.
No callers within internal-api itself yet -- the metrics aggregator PR
will introduce the first usages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add unit tests for Hashtable and LongHashingUtils
LongHashingUtilsTest (14 cases):
- hashCodeX null sentinel + non-null pass-through
- all primitive hash() overloads match the boxed Java hashCodes
- hash(Object...) 2/3/4/5-arg overloads match the chained addToHash
formula they are documented to constant-fold to
- addToHash(long, primitive) overloads match the Object-version
- linear-accumulation invariant (31 * h + v) holds across a sequence
- iterable / deprecated int[] / deprecated Object[] variants match
chained addToHash
- intHash treats null as 0 (observable via hash(null, "x"))
HashtableTest (24 cases across 5 nested classes):
- D1: insert/get/remove/insertOrReplace/clear/forEach, in-place value
mutation, null-key handling, hash-collision chaining with disambig-
uating equals, remove-from-collided-chain leaves siblings intact
- D2: pair-key identity, remove(pair), insertOrReplace matches on
both parts, forEach
- Support: capacity rounds up to a power of two, bucketIndex stays
in range across a wide hash sample, clear nulls every slot
- BucketIterator: walks only matching-hash entries in a chain, throws
NoSuchElementException when exhausted
- MutatingBucketIterator: remove from head-of-chain unlinks, replace
swaps the entry while preserving chain, remove() without prior
next() throws IllegalStateException
Tests live in internal-api/src/test/java/datadog/trace/util and use the
already-present JUnit 5 setup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply spotless formatting to Hashtable and LongHashingUtils
Bring the new util/ files in line with google-java-format
(tabs → spaces, line wrapping, javadoc list markup) so
spotlessCheck passes in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add JMH benchmarks for Hashtable.D1 and D2
Compares Hashtable.D1 and Hashtable.D2 against equivalent HashMap
usage for add, update, and iterate operations. Each benchmark thread
owns its own map (Scope.Thread), but @Threads(8) is used so the
allocation/GC pressure that Hashtable is designed to avoid surfaces
in the throughput numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add benchmark results to HashtableBenchmark header
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Guard Support.sizeFor against overflow and use Integer.highestOneBit;
reject capacities above 1 << 30 instead of looping forever.
- Add braces around single-statement while bodies in BucketIterator.
- Split HashtableBenchmark into HashtableD1Benchmark / HashtableD2Benchmark.
- Add regression tests for Support.sizeFor bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix dropped argument in HashingUtils 5-arg Object hash
The 5-arg Object overload was forwarding only obj0..obj3 to the int
overload, silently dropping obj4. Also align LongHashingUtils.hash 3-arg
signature with its 2/4/5-arg siblings (int parameters) and strengthen
the 5-arg HashingUtilsTest to detect the missing-arg regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Split D1Tests and D2Tests into HashtableD1Test and HashtableD2Test;
extract shared test entry classes into HashtableTestEntries.
- Reduce visibility of LongHashingUtils.hash(int...) chaining overloads
to package-private; they are internal building blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop reflection in iterator tests via package-private D1.buckets
The iterator tests need a populated Hashtable.Entry[] to drive
Support.bucketIterator / mutatingBucketIterator. Relaxing D1.buckets
from private to package-private lets the same-package tests read it
directly, removing the reflection helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add context-passing forEach to Hashtable.D1 and D2
Mirrors the TagMap pattern: pairs the existing forEach(Consumer) with a
forEach(T context, BiConsumer<T, TEntry>) overload so callers can hand
side-band state to a non-capturing lambda and avoid the
fresh-Consumer-per-call allocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move forEach loop body to Support helper
Factors the unchecked (TEntry) cast out of D1.forEach / D2.forEach (and
the BiConsumer variants) into Support.forEach(buckets, ...). The cast
now lives in one place, mirroring how Entry.next() handles it, and the
D1/D2 methods become one-liners. Downstream higher-arity tables built
on Support gain the same helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move bucket-head cast to Support.bucket helper
Adds Support.bucket(buckets, keyHash) which returns the bucket head
already cast to the caller's concrete entry type. D1.get and D2.get
now drop the raw-Entry intermediate variable and walk the chain via
Entry.next() directly. The unchecked cast lives in one place,
consistent with Entry.next() and Support.forEach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop d1_/d2_ prefix from per-table benchmark methods
Holdover from when both lived in a shared HashtableBenchmark; redundant
now that each lives in its own class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTableIterator
Three consumer-facing helpers that callers building higher-arity tables on
top of Hashtable.Support kept open-coding:
- MAX_RATIO_NUMERATOR / _DENOMINATOR: the 4/3 multiplier for sizing a
bucket array from a target working-set under a 75% load factor.
- insertHeadEntry(buckets, bucketIndex, entry): the (setNext + array-store)
pair for splicing a new entry at the head of a bucket chain.
- MutatingTableIterator + Support.mutatingTableIterator(buckets): walks
every entry in the table (not filtered by hash) with remove() support,
for sweeps like eviction and expunge that aren't keyed to a specific
hash. Sibling of MutatingBucketIterator.
Tests cover the table-wide iterator at head-of-bucket and mid-chain
removal, empty buckets between live entries, exhaustion, and
remove-without-next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap MAX_RATIO numerator/denominator pair for a single float + scaled create()
Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float
MAX_RATIO constant, and add a Support.create(int, float) overload that
takes a scale factor. Callers now write Support.create(n, MAX_RATIO)
instead of stitching together the int arithmetic at the call site.
The scaled size is truncated (not ceiled) before going through sizeFor.
sizeFor already rounds up to the next power of two, so truncation just
absorbs float fuzz that would otherwise push a result like 12 * 4/3 =
16.0000005f past 16 and double the bucket array size for no reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
Five small cleanups from a design re-review pass:
1. Support javadoc: drop the stale "methods are package-private" sentence;
most of them were made public in earlier commits for higher-arity
callers. Also drop the "nested BucketIterator" framing (iterators are
peers of Support inside Hashtable, not nested inside Support).
2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float)
deliberately truncates and is the canonical pathway.
3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so
the behavior difference is explicit: D1 uses Long.MIN_VALUE as a
sentinel that's collision-free against any int-valued hashCode(); D2
has no such sentinel and relies on matches() to resolve null/null vs
hash-0 collisions.
4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to
requestedSize. The cap is on the bucket-array length, not entry count;
the new name reflects that. Error messages updated to match.
5. Drop the `abstract` modifier on Hashtable in favor of `final` with a
private constructor. Nothing actually subclasses Hashtable -- the
abstract was a namespace device that read as "intended for extension."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that
derives the bucket index itself. Callers that already have a hash but
not the index (the common case) now avoid the redundant bucketIndex(...)
hop.
- D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the
new overload, drop the (thisBuckets local, bucketIndex compute,
setNext, store) sequence at each call site.
- D2.buckets: drop the `private` modifier to match D1.buckets. Both are
package-private so iterator tests in the same package can drive
Support.bucketIterator against the table's bucket array. Added a short
comment on both fields documenting the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
Three follow-ups from the design review:
- Make Hashtable.Entry.next private. All same-package readers
(BucketIterator) already had a next() accessor; the leftover direct
field reads now route through it. Closes the "mixed encapsulation"
gap where some readers used the accessor and same-package ones
reached for the field.
- BucketIterator and MutatingBucketIterator now document that chain-walk
work happens in next() (and the constructor for the first match);
hasNext() is an O(1) field read.
- Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction).
Both reuse the lookup hash for the insert on miss, avoiding the
double-hash that "get; if null then insert" callers would otherwise
pay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hashtable: add missing braces and detach removed/replaced entries
Addresses PR #11409 review comments:
- #3267164119 / #3267165525: wrap every single-line if/break body in
braces (7 sites across BucketIterator, MutatingBucketIterator, and the
full-table Iterator).
- #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced
entry's next pointer after splicing it out of the chain in
MutatingBucketIterator.remove / .replace. Applied the same fix to the
full-table Iterator.remove for consistency.
Rationale: detaching prevents accidental traversal through a removed
entry via a stale reference and lets the GC reclaim a chain tail that
the removed entry was the last referrer to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename LongHashingUtils.hashCodeX(Object) to hash(Object) for API consistency
Addresses PR #11409 review comment #3276167001. The method parallels the
primitive hash(boolean) / hash(int) / hash(long) / ... family, so naming
it hash(Object) -- with null collapsing to Long.MIN_VALUE as a sentinel
distinct from any real hashCode -- matches the rest of the public surface.
Test call sites that pass a literal null now disambiguate against
hash(int[]) / hash(Object[]) / hash(Iterable) via an (Object) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge remote-tracking branch 'origin/master' into dougqh/optimize-metric-key
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Merge remote-tracking branch 'origin/master' into dougqh/conflating-metrics-background-work
Introduce slim PeerTagSchema; capture peer-tag values not pairs
Addresses sarahchen6's review comment on ConflatingMetricsAggregator
extractPeerTagPairs: replaces the worst-case-allocation + trim-and-copy
flat-pairs layout with a parallel-array carrier.
- New PeerTagSchema: minimal carrier of String[] names. Two flavors -- a
static INTERNAL singleton (one entry: base.service) for internal-kind
spans, and per-discovery built schemas for client/producer/consumer
spans. Deliberately no cardinality limiters or per-cycle state; that
layers on top in a later PR.
- ConflatingMetricsAggregator: caches the peer-aggregation schema keyed
on reference equality of features.peerTags() -- a single volatile read
+ a long compare on the steady-state producer hot path, no allocation.
The producer now captures only a String[] of values parallel to the
schema's names; the schema reference is carried on SpanSnapshot. The
prior "build worst-case pairs then trim" code is gone.
- SpanSnapshot: replaces String[] peerTagPairs with PeerTagSchema +
String[] peerTagValues. Producer drops the schema reference if no
values fired so the consumer short-circuits on null.
- Aggregator.materializePeerTags: now reads name/value pairs at the same
index from (schema.names, snapshot.peerTagValues). Counts hits once
for exact-size allocation; preserves the singletonList fast path for
the common one-entry case (e.g. internal-kind base.service).
Producer-side cost goes from "allocate String[2n] + walk + maybe trim"
to "single volatile read + walk + lazy String[n] only on first hit".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Address PR #11381 review (round 2)
- Aggregator.materializePeerTags: fold the firstHit-discovery nested if
into a single guarded post-increment (amarziali, #3279243138). One
body line: `if (values[i] != null && hitCount++ == 0) firstHit = i;`.
- Drop redundant isKind(SpanKindFilter) overrides in both
TraceGenerator.groovy files (amarziali, #3279264553 / #3279382648).
CoreSpan.java:84 already supplies a default implementation that reads
the same span.kind tag.
- Bump TRACER_METRICS_MAX_PENDING default from 2048 -> 131072 to address
the capacity regression amarziali flagged (#3279378375). Without
producer-side conflation, the inbox now holds 1 SpanSnapshot per
metrics-eligible span instead of 1 conflated Batch per ~64 spans;
restoring effective capacity parity (~2048 * ~64 = 131072) prevents a
~64x rise in inbox-full drops at the same span rate. ~100 B per
SpanSnapshot puts the worst-case heap floor at ~13 MB -- bounded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover inbox-full fast-path in ConflatingMetricsAggregator.publish
Addresses PR #11381 review (amarziali, #3279325340 -- "Are the existing
tests covering this case?").
New ConflatingMetricsAggregatorInboxFullTest constructs the aggregator
with a small inbox (queueSize=8), deliberately does NOT call start() so
the consumer thread never drains, then publishes enough spans to
overflow the inbox. Verifies that healthMetrics.onStatsInboxFull() is
called at least once -- the fast-path's `inbox.size() >= inbox.capacity()`
short-circuit triggers when the producer-side queue is at capacity.
Test is Java + JUnit 5 + Mockito per the project convention for new
tests; uses a CoreSpan Mockito mock rather than the SimpleSpan Groovy
fixture so we don't depend on Groovy-then-Java compile order from the
test source set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reconcile PeerTagSchema once per reporting cycle on the aggregator thread
Addresses amarziali's review comment #3279340181 ("It would be more
efficient to trigger from the other side"). The producer-side reference
compare on every publish goes away; the aggregator thread reconciles
the cached schema against feature discovery once per reporting cycle.
- DDAgentFeaturesDiscovery: expose getLastTimeDiscovered() so callers
can detect a discovery refresh without copying the peerTags Set.
- PeerTagSchema: add `long lastTimeDiscovered` (plain, aggregator-only)
and `hasSameTagsAs(Set)`. of(Set, long) takes the timestamp; INTERNAL
uses a -1L sentinel since it's never reconciled.
- ConflatingMetricsAggregator:
* Drop the cachedPeerTagsSource volatile and the per-publish reference
compare.
* Producer fast path is now `cachedPeerTagSchema` volatile read +
null-check; first publish takes the one-time synchronized bootstrap.
* Add reconcilePeerTagSchema() that runs once per cycle on the
aggregator thread: fast-path timestamp compare, slow-path set
compare, bump-in-place when the set is unchanged.
- Aggregator: new `Runnable onReportCycle` constructor parameter, run at
the start of report() (before the flush, so any test awaiting
writer.finishBucket() observes the schema in its post-reconcile state
and so the next publish sees the new schema without a handoff).
- Update "should create bucket for each set of peer tags" to drive two
reporting cycles separated by a report() that triggers reconcile. The
old test relied on per-publish reference detection, which the new
design intentionally doesn't preserve -- the schema is now stable
within a cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Add bootstrap + reconcile coverage for PeerTagSchema
Addresses round-3 review nice-to-haves on PR #11381.
- PeerTagSchemaTest: unit coverage for hasSameTagsAs() (the predicate
that drives the reconcile fast/slow path split), the of(Set, long)
factory, and the INTERNAL singleton. The hasSameTagsAs cases include
same-content-different-Set-reference (the case the reconcile fast path
relies on after a discovery refresh) and content-mismatch in either
direction.
- ConflatingMetricsAggregatorBootstrapTest: integration coverage for
the producer-side bootstrap + aggregator-thread reconcile flow.
* bootstrapHappensOnceOnFirstPublish -- three publishes against an
un-started aggregator (no consumer thread, no reconciles); verifies
features.peerTags() and features.getLastTimeDiscovered() are each
called exactly once.
* reconcileSkipsDeepCompareWhenTimestampMatches -- two cycles with
constant features.getLastTimeDiscovered(); each post-report
reconcile short-circuits on the timestamp fast path, so peerTags()
is called only by bootstrap (1 total).
* reconcileSurvivesTimestampBumpWhenTagsUnchanged -- timestamps bump
every reconcile, forcing the slow set-compare path; the tag set
stays identical, so the schema is preserved and continues to flush
buckets correctly across cycles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Preserve TRACER_METRICS_MAX_PENDING semantic + drop stale imports
TRACER_METRICS_MAX_PENDING previously counted conflating Batch slots
(~64 spans each). The inbox now holds 1 SpanSnapshot per slot, so
multiply the configured value by LEGACY_BATCH_SIZE (64) to keep
pre-existing customer overrides delivering the same effective
span-throughput capacity. Default stays at 2048 logical -> 131072
snapshot slots, identical to the prior 2048 batches * 64 spans.
Also drops two unused datadog.trace.core.SpanKindFilter imports left
behind in TraceGenerator.groovy after the isKind() override was removed
in favor of the CoreSpan default implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Add AdversarialMetricsBenchmark for capacity-bound stress testing
Ports the adversarial JMH benchmark from #11402 down to this branch so
we can compare #11381 vs master on a high-cardinality, high-throughput
workload. Adapted to use ConflatingMetricsAggregator (pre-rename) and
the FixedAgentFeaturesDiscovery / NullSink helpers already in
ConflatingMetricsAggregatorBenchmark.
8 producer threads hammer publish() with unique (service, operation,
resource, peer.hostname) per op so the aggregate cache fills+evicts
continuously and the inbox saturates. tearDown prints the drop
counters (inboxFull vs aggregateDropped) so the test verifies the
subsystem stayed bounded under attack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Trim AdversarialMetricsBenchmark counters and clarify printout
Drop traceComputedCalls / totalSpansCounted: under 8-way contention
the volatile-long ++/+= pattern was losing ~20% of updates (296M
counted vs 245M reported), and the numbers duplicate signal JMH's
ops/s already provides.
Switch inboxFull / aggregateDropped to LongAdder so the printed drop
shape (the order-of-magnitude story the bench is built to tell) is
accurate under contention.
Replace the stale "both forks combined for this run" string with text
that matches the actual @Fork(value=1) config and notes that counters
accumulate across warmup + measurement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Close PeerTagSchema reconcile race + cover the swap branch
buildPeerTagSchema previously read features.peerTags() before
features.getLastTimeDiscovered(). DDAgentFeaturesDiscovery exposes
those as two separate accessors against its volatile State -- a
state-swap interleaving could leave the cached schema tagged with a
NEWER timestamp than its names, after which the next reconcile
short-circuits on the timestamp compare and misses the tag-set update
until the next discovery refresh (~minute later).
Swap the read order so timestamp is captured first. With this
ordering, an interleaving leaves the schema OLDER than its names
instead -- the next reconcile sees a timestamp mismatch, runs the
deep compare, and self-heals on the very next cycle.
Also adds reconcileSwapsSchemaWhenTagSetChanges, which closes the
test gap on the slow-path swap branch
(cachedPeerTagSchema = PeerTagSchema.of(...)). End-to-end check via
the writer's captured MetricKeys: pre-swap snapshot carries only
peer.hostname, post-swap snapshot carries both peer.hostname and
peer.service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Adapt reconcileSwapsSchemaWhenTagSetChanges to AggregateEntry shape
#11382 collapses MetricWriter.add(MetricKey, AggregateMetric) into
add(AggregateEntry). Re-target the captor and accessors on this branch
so the test compiles and the same end-to-end peer-tag verification
holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clarify materializePeerTags hit-counting loop
Splits the `if (values[i] != null && hitCount++ == 0)` conjunction
into nested ifs. Same semantics, no codegen impact after JIT --
just visibly says what the loop is doing rather than relying on
post-increment-inside-conjunction. Closes amarziali's review thread
on this block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
# Conflicts:
# dd-trace-core/src/main/java/datadog/trace/common/metrics/Aggregator.java
Fix MetricsIntegrationTest entry recording call site
AggregateEntry consolidated MetricKey + AggregateMetric so recordDurations
lives directly on AggregateEntry now. The previous entry1.aggregate.
recordDurations(...) form compiles under Groovy's dynamic dispatch but
would throw MissingPropertyException at runtime since there is no
`aggregate` property. Resolves chatgpt-codex-connector's review comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make ConflatingMetricAggregatorTest counter checks actually verify
The `1 * writer.add(value) >> { closure }` pattern treats the closure
as a stubbed return value -- Spock evaluates it but discards the
result, so `e.getHitCount() == X && ...` was a silent no-op across
31 occurrences. Wrapping the expression in `assert` makes Groovy's
power-assert throw on mismatch, which Spock surfaces as a real
failure. Resolves chatgpt-codex-connector's review comment.
All 41 tests still pass, so the previously-unverified assertions
happened to hold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop dead recordDurations(int, AtomicLongArray) batch API
This method was a vestige of master's Batch design where multiple
producer threads wrote into an AtomicLongArray slot concurrently and
the aggregator drained ~64 durations per Batch in one call. The new
producer/consumer split publishes one SpanSnapshot per span, so
production only ever calls recordOneDuration(long).
Migrate the three remaining callers (AggregateEntryTest,
SerializingMetricWriterTest, MetricsIntegrationTest) to a loop of
recordOneDuration(long) calls, then delete the batched method and its
AtomicLongArray imports.
Drops the recordDurationsIgnoresTrailingZeros test -- that behavior
was a specific quirk of the batched API (count parameter shorter than
the array length) and doesn't apply to recordOneDuration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Warn about colon split in AggregateEntry.of test factory
The factory recovers (name, value) pairs from pre-encoded "name:value"
strings by splitting at the FIRST colon. Test-only, but worth being
explicit so callers don't hand it a peer-tag value containing a colon
(URLs, IPv6, service:env) and get a silently wrong (name, value) pair.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add coverage for disable() -> ClearSignal threading path
The bundled fix in this PR routes the agent-downgrade clear through
the inbox so the aggregator thread stays the sole writer to
AggregateTable. Prior to this test, there was no regression coverage
for that routing.
The test fires DOWNGRADED from the test thread (production-like
OkHttpSink callback path), waits for the immediate no-flush window,
then publishes a marker span with a distinct resource name. The
subsequent report's writer.add captor must see only the marker -- if
CLEAR didn't actually wipe the original entry, the original
"resource" would still be present and the assertion would catch it.
Cannot directly verify thread identity of the clear from inside this
test (CLEAR's inbox.clear() drops any latch signal we'd queue behind
it), so this is an observable-contract test rather than a strict
thread-id test. Still catches both the missing-clear regression and
the …
Implements the span-derived primary tags feature on the new producer/
consumer architecture: users configure DD_TRACE_STATS_ADDITIONAL_TAGS
(comma-separated tag keys); the tracer extracts the matching span tag
values and includes them as additional aggregation dimensions on
ClientGroupedStats.AdditionalMetricTags.
Design choices, matched to the PoC where reasonable:
- Wire format: repeated string of "<key>:<value>" entries, in
schema (alphabetical-by-key) order; field omitted when no slots
are populated. Customers who don't configure additional tags pay
zero payload overhead.
- Cardinality protection:
MAX_ADDITIONAL_TAG_KEYS = 10 -- configured-key count cap;
MAX_ADDITIONAL_TAG_VALUE_LENGTH = 250 -- per-value length cap;
DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT = 100 (config-
urable, <=0 -> warn + fallback) -- per-bucket stat-entry cap.
- Single-global counter for the per-bucket cap, single-threaded
(aggregator thread is the sole writer of the table + limiter), so
a plain int suffices -- no AtomicInteger.
- All canonicalization stays on the aggregator thread, consistent
with the rest of the post-redesign pipeline: producer just
captures raw String values into SpanSnapshot.additionalTagValues
parallel to the schema; Canonical.populate applies the length cap
and builds the per-slot UTF8BytesString "key:value" form;
AggregateTable.findOrInsert applies the bucket cap by rebuilding
the canonical with per-key blocked sentinels if needed.
- Acknowledged spec deviation: single-global counter rather than
per-tag isolation. A misconfigured tag can starve another tag's
admission of new entries within a bucket, but every span still
gets emitted with its dimension keys preserved (values masked).
Adds onAdditionalTagValueCardinalityBlocked(String tagKey) callback on
HealthMetrics and TracerHealthMetrics's "stats.additional_tag.cardin-
ality_blocked" counter (length-blocks + bucket-cap blocks).
Test coverage:
- AdditionalTagsSchemaTest: empty-config sentinel, sort+dedupe+cap,
per-key blocked sentinels.
- AdditionalTagsCardinalityLimiterTest: length cap behavior, counter
+ cap + reset, recordCardinalityBlock health-metric firing.
- AggregateTableAdditionalTagsTest: distinct/same identity, overlong
values collapse to one entry, cardinality cap collapses new
entries to the blocked sentinel while existing entries continue.
- SerializingMetricWriterAdditionalTagsTest: AdditionalMetricTags
wire field shape, omission when empty, null-slot skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the fixed-size additionalTagsBuffer sized at Canonical construction time. The buffer is now growable, and Canonical tracks additionalTagsCount = snapshot.additionalTagsSchema.size() per populate -- length-aware hash, match, and toEntry use the (buffer, count) pair, mirroring how peer tags already work. AggregateTable and Aggregator drop their schema parameters since Canonical no longer needs one; schema lives where it's used (ClientStatsAggregator + the snapshot). AdditionalTagsMetricsBenchmark mirrors AdversarialMetricsBenchmark for the additional-tags hot path: two configured keys with a per-key cardinality cap of 100, unique values per op so the cap saturates fast. Catches future regressions on producer-side capture, schema.register, and the per-cycle block-counter flush. Adds an onTagCardinalityBlocked override to the shared CountingHealthMetrics so both benchmarks observe the new flush counter. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skips the closed dougqh/metrics-memory-efficiency (#11389); #11387 now serves as the direct base. Conflict resolution: took HEAD (combined cardinality + additional-tags) for all conflicts. Restored @nullable annotations on populatePeerTags/populateAdditionalTags params that were lost during rebase. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dougqh
force-pushed
the
dougqh/metrics-arbitrary-tags
branch
from
592f240 to
a4af772
Compare
dougqh
changed the base branch from
dougqh/metrics-memory-efficiency
to
dougqh/control-tag-cardinality
…ardinality AdditionalTagsSchema now holds one TagCardinalityHandler per key, matching the pattern used for peer tags in PeerTagSchema. Benefits: - UTF8 caching: "key:value" UTF8BytesStrings are reused across populate() calls via the two-table rolling design; no fresh allocation on the hit path - Per-tag value cardinality limiting at MetricCardinalityLimits.ADDITIONAL_TAG_VALUE (512, same as peer tags), in addition to the existing global stat-entry cap - Block counts flushed to HealthMetrics.onTagCardinalityBlocked via AdditionalTagsSchema.resetHandlers() each cycle Also: validate tag keys at schema construction (reject empty or colon-containing keys), dedup sorted keys, and add per-tag global stat-entry cap (addressing global-counter-vs-per-tag spec deviation). Length-based capping removed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… is sufficient The per-tag TagCardinalityHandler (512 values/tag/cycle) already provides value-level cardinality control. The table-level maxAggregates cap bounds total entry count regardless. The stat-entry limiter was a workaround for the previous design that lacked per-tag handlers; it is now redundant. Removes: AdditionalTagsCardinalityLimiter class, rebuildAdditionalTagsWithBlockedSentinels, hasAdditionalTags, trace.stats.additional.tags.cardinality.limit config key, and all related constructor parameters. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dougqh
commented
Jun 3, 2026
| timeUnit, | ||
| healthMetric, | ||
| this::resetCardinalityHandlers); | ||
| this::resetCardinalityHandlers, |
Contributor
Author
There was a problem hiding this comment.
I'd like the reset handler to come last in the parameter list
- AggregateEntry.Canonical.toEntry(): use EMPTY_ADDITIONAL_TAGS when all additionalTagsBuffer slots are null (span had no additional-tag values) instead of cloning an all-null array. Avoids one UTF8BytesString[N] allocation per new entry on workloads that don't use additional tags. - Update populateAdditionalTags Javadoc to drop length-cap reference - AdditionalTagsMetricsBenchmark: fix from() call (drop removed cardinality limit param), update comments to reflect TagCardinalityHandler design, fix tearDown to use LongAdder.sum() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both false/true variants now run within the same fork so they see equivalent thermal conditions, giving a fair limits-on vs limits-off comparison without requiring separate JVM invocations. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ences The limitsEnabled=true/false arms are not a fair "cost of limiting" comparison: with unbounded distinct values, limits=true collapses overflow to one hot sentinel entry (recordOneDuration runs every span) while limits=false saturates the table and drops overflow (never reaches the histogram). GC profiling (3 forks, 2026-06-03) confirmed limits=true allocates MORE per op (888 vs 820 B/op) — the histogram recording, not the sentinel path. Documents this so the arm isn't misread as a regression. Also fixes two stale "cap of 100" references (actual is ADDITIONAL_TAG_VALUE=512). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AdditionalTagsMetricsBenchmark.tearDown uses System.err (matches the sibling AdversarialMetricsBenchmark); add @SuppressForbidden so forbiddenApisJmh passes - AdditionalTagsSchemaTest: cover the new key validation — empty keys and ':'-containing keys are dropped; all-invalid input returns EMPTY Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
additionalTags was a sparse schema-width UTF8BytesString[] with null slots for absent tags. Switch to a compact array of only the present values (schema order preserved), matching the peerTags representation. Each value carries its "key:" prefix so packing loses no information. Eliminates the null-handling this required: - SerializingMetricWriter: drop countNonNull() pre-scan and the null-skip in the write loop; startArray(additionalTags.length) + tight loop - hashOf: take (array, count) and loop over count, like peer tags - matches: compact additionalTagsEqual() instead of Arrays.equals over the full sparse buffer (which would have compared stale trailing slots) - toEntry: Arrays.copyOf(buffer, size) instead of clone + allNull scan Pure cleanup/consistency; wire format unchanged. No measurable throughput change (these paths are flush-time or feature-gated) but removes a class of null-handling and makes additional tags parallel to peer tags. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dougqh
requested review from
amarziali and
sarahchen6
and removed request for
a team
Contributor
Author
Petclinic load test (multi-endpoint, 2026-06-03)Ran the multi-endpoint petclinic load test (32 threads, 10 endpoint templates, 90s measure) on this branch's agent jar to confirm no macro-level regression from the additional-tags plumbing. No additional tags configured — this measures the cost of the feature being threaded through the pipeline but unused (the realistic default-off case). n=2 interleaved at each heap, 3-min cooling between runs:
† 4th run of a sequential sweep — thermal dip, not a code effect. Against the rest of the CSS stack on the same workload/methodology:
‡ thermal bombs from sequential-sweep heat on those earlier sessions, not design — see the stack's perf notes. Conclusion: #11402 is at parity with the rest of the stack. The 256m pair is the cleanest in the comparison (no variance bomb on either pass). The additional-tags feature carries no measurable macro cost when unconfigured — consistent with the JMH baseline (per-snapshot path is a null check + early return when no schema is set). (Results by Claude Sonnet 4.6) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack: master → #11382 → #11478 → #11387 → this PR. (#11389 was closed — its memory-efficiency work was folded into #11478's lazy-histogram +
namesHashcaching.) Implements Client-Side Stats v1.3.0 span-derived primary tags on the new producer/consumerClientStatsAggregatorarchitecture. Users configureDD_TRACE_STATS_ADDITIONAL_TAGS(comma-separated tag keys); the tracer extracts the matching span tag values and includes them as additional aggregation dimensions onClientGroupedStats.AdditionalMetricTags.All canonicalization (
UTF8BytesStringinterning, per-tag cardinality cap) runs on the aggregator thread; the producer just captures raw values into aString[]parallel to the schema.Design
Wire format: new
AdditionalMetricTagsfield onClientGroupedStats, emitted asrepeated stringof"<key>:<value>"entries (mirrorsPeerTags). Schema-ordered (alphabetical by key); null slots skipped; field omitted when no slots are populated so customers who don't configure additional tags pay zero payload overhead.Configuration:
DD_TRACE_STATS_ADDITIONAL_TAGS/dd.trace.stats.additional.tags— comma-separated tag keys.Cardinality protection —
AdditionalTagsSchemaholds oneTagCardinalityHandlerper configured key, exactly mirroringPeerTagSchema:MAX_ADDITIONAL_TAG_KEYS = 10— configured-key count cap. Excess keys dropped at startup with a warn log.MetricCardinalityLimits.ADDITIONAL_TAG_VALUE = 512— per-key distinct-value cap per reporting cycle. Once a key's budget is exhausted, further new values get the per-key"<key>:blocked_by_tracer"sentinel (when limits are enabled) so they collapse into one bucket instead of fragmenting the table.:are rejected with a warn log; the key set is sorted + deduped."key:value"string is re-allocated.Threading: the per-key
TagCardinalityHandlers and their block counters are aggregator-thread-only.AdditionalTagsSchema.resetHandlers()runs once per reporting cycle alongside the property/peer handler resets, flushing each key's block count toHealthMetrics.onTagCardinalityBlocked.What's new vs. PoC (#11358)
unsafeGetTag(name)per configured key →String[]parallel to the schema.TagCardinalityHandler(same component as peer tags) rather than a bespoke limiter — gets UTF8 caching for free and removes the separate global stat-entry counter the PoC used.UTF8BytesString[]on the entry, writing each non-null slot directly — no per-write byte composition.New files
dd-trace-core/src/main/java/datadog/trace/common/metrics/AdditionalTagsSchema.javadd-trace-core/src/test/java/datadog/trace/common/metrics/AdditionalTagsSchemaTest.javadd-trace-core/src/test/java/datadog/trace/common/metrics/AggregateTableAdditionalTagsTest.javadd-trace-core/src/test/java/datadog/trace/common/metrics/SerializingMetricWriterAdditionalTagsTest.javadd-trace-core/src/jmh/java/datadog/trace/common/metrics/AdditionalTagsMetricsBenchmark.javaHealth metric
HealthMetrics.onTagCardinalityBlocked(String tag, long count)— fires per key per cycle for any key whose per-cycle value budget was exceeded (shared with peer tags and property fields).TracerHealthMetricssurfaces this asstats.cardinality_blocked(tagged by field).Benchmarks
No-additional-tags baseline (8 producer threads, 2×15s warmup + 5×15s)
HighCardinalityResourceMetricsBenchmark/HighCardinalityPeerMetricsBenchmarkset no additional tags, so they measure the cost of the plumbing being threaded through but not populated. On a clean machine (2026-06-03) this PR was at parity with #11387 and master (38–43M ops/s band), confirming the per-snapshot path is a single null check + early return inCanonical.populateAdditionalTagswhen no schema is configured.Additional-tags path (
AdditionalTagsMetricsBenchmark, new)Configures two keys and generates unbounded distinct values so the per-key budget saturates immediately. Runs both
limitsEnabledarms via@Param. 3-fork run with-prof gc(2026-06-03):limitsEnabledThe two arms are not a fair "cost of limiting" comparison and should not be read as a regression. With unbounded distinct values they diverge into different downstream behavior, not just a different branch in
register:limitsEnabled=true: every over-budget span collapses to the single"<key>:blocked_by_tracer"sentinel entry →findOrInsertalways hits a live entry →recordOneDuration(a DDSketch histogram insert) runs for every drained span.limitsEnabled=false: every over-budget value canonicalizes to a distinct entry → the table saturates atmaxAggregates→ most subsequent spans are dropped atfindOrInsert, never reaching the histogram.So
limitsEnabled=truemeasures lower throughput here precisely because it does more useful work per span (keeps the masked data and records it) where the disabled arm drops the overflow. The GC profile confirms this: limits=true allocates more per op (888 vs 820 B/op) — that delta is the histogram recording, not the sentinel path (which allocates nothing). My initial hypothesis (that the disabled arm would allocate more from freshUTF8BytesStringcreation) was wrong — the drop path short-circuits before those allocations matter.A production workload with a bounded value set never saturates the budget and sees neither arm's overflow behavior, which is why
HighCardinalityResourceis at parity with limits on/off. Throughput CIs here are wide (>20%, single-machine thermal drift across the sequential@Paramarms); the per-op allocation figures are the reliable signal.Test plan
:dd-trace-core:test --tests "datadog.trace.common.metrics.*"— all pass (existing + three new test files for the feature).:-containing keys.ADDITIONAL_TAG_VALUEcollapse to the per-key sentinel; existing entries still hit normally.AdditionalMetricTagsfield present with schema-ordered"key:value"entries; omitted when nothing matches; null slots skipped.Notes for reviewers
masterandConflatingMetricsAggregator. This branch sits on Per-component / tag cardinality limits in client-side stats #11387 and ships against the producer/consumer-splitClientStatsAggregator.setMeasuredTag(key, value)programmatic API,DD_TAGShandling, per-value length cap (removed from this revision — revisit if production shows oversized values).🤖 Generated with Claude Code