Skip to content

TestTokenizer: reduced amount of settingsBuilder().build() calls#8367

Merged
chrchr-github merged 2 commits intodanmar:mainfrom
firewave:test-tokenize
Mar 23, 2026
Merged

TestTokenizer: reduced amount of settingsBuilder().build() calls#8367
chrchr-github merged 2 commits intodanmar:mainfrom
firewave:test-tokenize

Conversation

@firewave
Copy link
Collaborator

No description provided.

@firewave
Copy link
Collaborator Author

Debug - before

Benchmark 1: bin/testrunner TestTokenizer
  Time (mean ± σ):     10.709 s ±  0.054 s    [User: 10.635 s, System: 0.011 s]
  Range (min … max):   10.636 s … 10.823 s    10 runs

Debug - after

Benchmark 1: bin/testrunner TestTokenizer
  Time (mean ± σ):      2.170 s ±  0.019 s    [User: 2.144 s, System: 0.012 s]
  Range (min … max):    2.146 s …  2.198 s    10 runs

Release - before

Benchmark 1: bin/testrunner TestTokenizer
  Time (mean ± σ):      2.339 s ±  0.297 s    [User: 2.316 s, System: 0.008 s]
  Range (min … max):    2.181 s …  3.156 s    10 runs

Release - after

Benchmark 1: bin/testrunner TestTokenizer
  Time (mean ± σ):     254.9 ms ±   6.1 ms    [User: 241.7 ms, System: 11.2 ms]
  Range (min … max):   251.7 ms … 272.1 ms    10 runs

@firewave
Copy link
Collaborator Author

With tokenizeAndStringifyWindows() removed

Debug

Benchmark 1: bin/testrunner TestTokenizer
  Time (mean ± σ):      2.019 s ±  0.012 s    [User: 1.990 s, System: 0.013 s]
  Range (min … max):    2.005 s …  2.049 s    10 runs

Release

Benchmark 1: bin/testrunner TestTokenizer
  Time (mean ± σ):     187.3 ms ±   1.1 ms    [User: 173.9 ms, System: 12.0 ms]
  Range (min … max):   185.8 ms … 189.9 ms    15 runs

@sonarqubecloud
Copy link

@firewave
Copy link
Collaborator Author

Running an optimized build in valgrind reduces the time from almost 2 minutes to 15 seconds.

@chrchr-github chrchr-github merged commit bb948d2 into danmar:main Mar 23, 2026
72 checks passed
@firewave firewave deleted the test-tokenize branch March 23, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants