Skip to content

fix(skills): misleading DCE advice in setup-harness#350

Merged
SuperMuel merged 1 commit into
mainfrom
fix/skill-pedantic-not-for-dce
May 26, 2026
Merged

fix(skills): misleading DCE advice in setup-harness#350
SuperMuel merged 1 commit into
mainfrom
fix/skill-pedantic-not-for-dce

Conversation

@SuperMuel
Copy link
Copy Markdown
Contributor

@SuperMuel SuperMuel commented May 14, 2026

The skill told users to use benchmark.pedantic() in Python to prevent dead code elimination, like black_box() in Rust. Two problems with that.

benchmark.pedantic does not prevent dead code elimination. It just lets you set rounds, iterations, and a setup function for the benchmark.

CPython does not delete unused function calls. It can't know if a function has side effects, so it always runs it. Pure-Python benchmarks do not need a black_box.

The new wording only mentions compiled languages and uses the right name for each one.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 14, 2026

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 7 untouched benchmarks


Comparing fix/skill-pedantic-not-for-dce (8bf3d7d) with main (0496dcd)

Open in CodSpeed

SuperMuel added a commit that referenced this pull request May 14, 2026
Caught by the trailing-whitespace pre-commit hook on PR #350.
@SuperMuel SuperMuel requested a review from art049 May 14, 2026 21:29
@SuperMuel SuperMuel force-pushed the fix/skill-pedantic-not-for-dce branch from d395403 to 27bd3d0 Compare May 16, 2026 21:03
@SuperMuel SuperMuel changed the base branch from main to fix-ci-rustup-macos May 16, 2026 21:03
@SuperMuel SuperMuel force-pushed the fix/skill-pedantic-not-for-dce branch from 27bd3d0 to c12d934 Compare May 16, 2026 21:26
@GuillaumeLagrange GuillaumeLagrange force-pushed the fix-ci-rustup-macos branch 3 times, most recently from e332482 to 78b800a Compare May 20, 2026 08:57
Base automatically changed from fix-ci-rustup-macos to main May 22, 2026 12:29
Comment thread .github/actions/install-rust/action.yml
`benchmark.pedantic` is not a dead-code-elimination tool; it controls
the measurement loop. CPython doesn't perform the kind of DCE that
`black_box` guards against in compiled languages, so pure-Python
benchmarks don't need an equivalent. List the actual primitives for
the languages that do (C++: DoNotOptimize, JMH: Blackhole.consume).
@SuperMuel SuperMuel force-pushed the fix/skill-pedantic-not-for-dce branch from c12d934 to 8bf3d7d Compare May 26, 2026 08:51
@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 26, 2026

Greptile Summary

Corrects a single misleading line in the codspeed-setup-harness skill that wrongly suggested benchmark.pedantic() prevents dead code elimination in Python. The replacement swaps it for accurate compiled-language examples — benchmark::DoNotOptimize (Google Benchmark / C++) and Blackhole.consume (JMH / Java).

  • Removes the Python benchmark.pedantic() DCE claim, which was doubly wrong: CPython never elides live function calls, and pedantic controls rounds/iterations, not optimization barriers.
  • Adds benchmark::DoNotOptimize (C++) and Blackhole.consume (JMH) as replacement examples; both are correct but JMH isn't covered elsewhere in the skill document.

Confidence Score: 5/5

This is a one-line documentation fix removing factually incorrect advice; it introduces no code or behavioural changes and the new text is accurate.

The change only touches documentation, removes genuinely wrong guidance, and replaces it with correct compiler-barrier references. The only minor note is that JMH appears in a skill that has no Java harness section, which could mildly confuse readers.

No files require special attention; the single changed line in skills/codspeed-setup-harness/SKILL.md is straightforward documentation.

Important Files Changed

Filename Overview
skills/codspeed-setup-harness/SKILL.md Removes incorrect benchmark.pedantic() Python DCE advice; replaces with accurate C++ (benchmark::DoNotOptimize) and JMH (Blackhole.consume) references. Single-line documentation fix.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Skill: Prevent DCE advice] --> B{Language?}
    B --> C[Rust]
    B --> D[C++]
    B --> E[Java / JMH]
    B --> F[Python]
    B --> G[Go]
    C --> C1["`black_box()`"]
    D --> D1["`benchmark::DoNotOptimize`"]
    E --> E1["`Blackhole.consume`"]
    F --> F2["No DCE — CPython always executes calls (no special wrapper needed)"]
    G --> G2["Assign to package-level sink (idiomatic Go workaround)"]
    C1 --> OK["✅ Mentioned in updated line"]
    D1 --> OK
    E1 --> OK2["⚠️ Mentioned but Java/JMH harness not covered in this skill"]
    F2 --> FIX["✅ Correctly removed from advice"]
    G2 --> MISS["❓ Go is covered in skill but not mentioned in DCE guidance"]
Loading

Reviews (1): Last reviewed commit: "fix(skills): misleading DCE advice in se..." | Re-trigger Greptile

Comment thread skills/codspeed-setup-harness/SKILL.md
@SuperMuel SuperMuel requested a review from art049 May 26, 2026 09:20
@SuperMuel SuperMuel merged commit 4693eed into main May 26, 2026
31 of 33 checks passed
@SuperMuel SuperMuel deleted the fix/skill-pedantic-not-for-dce branch May 26, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants