perf: implement NextGEQ by cheb0 · Pull Request #345 · ozontech/seq-db

cheb0 · 2026-02-06T13:21:03Z

Description

A new operation which boosts search and aggregation performance. It's implemented on top of faster cmp branch because fast compares are vital for this operation. It's also a base for future improvements like skipping disk reads.

Currently, NextGeq works more like a hint - some nodes do not support it and NextGeq behaviour is same as Next. Therefore, some operations still have loops even when calling NextGeq (i.e. scrolling past lower values), like in aggregator.go. Might be improved in future.

Measurements

Performance improvement depends on a particular operation and search request (and even tokens). If we can skip a lot, then there is a lot of improvement.

For example, service:X AND level:3 speeds up by 60% when level:3 is used, and speeds up only by 6% when level:4 is used as second predicate. Because we reduce total number of results with level:3 by a lot, while almost all service logs has level:4.

The performance improvement on typical user aggregation is good, since we skip a lot in agg tree.
service:xyz | group by k8s_pod count(*) (prod fracs)

master: 370-456 ms (hot-cold)
fast cmp: 320-402 ms
next geq: 73-153 ms (-50-70%)

There are also downsides. It's not zero-cost. For example, the case when we iterate the whole tree and there is nothing to skip (i.e. we traverse exactly same paths but there is an extra comparison in NextGeq operation) :
exists:service | group by k8d_pod count(*)
master: 690-821 ms
fast cmp: 600-730 ms
next geq: 662-800 ms (+10% skipping overhead)

However, thanks to faster cmp, it will still be faster than current main branch, i.e. there is no performance downgrade. The next steps are probably to implement some sort of cost based planning (and cardinality estimation) where we can evaluate if there is a skipping potential and disable it altogether, or maybe even avoid constructing an aggregation tree.

Fixes #332

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

codecov-commenter · 2026-02-07T13:01:12Z

Codecov Report

❌ Patch coverage is 96.32353% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.57%. Comparing base (1acfc59) to head (0b0a5ba).

Files with missing lines	Patch %	Lines
frac/sealed/lids/iterator_desc.go	77.77%	4 Missing ⚠️
frac/sealed/lids/iterator_asc.go	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
+ Coverage   71.29%   71.57%   +0.28%     
==========================================
  Files         206      206              
  Lines       15156    15290     +134     
==========================================
+ Hits        10805    10944     +139     
+ Misses       3565     3560       -5     
  Partials      786      786

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-02-15T09:01:34Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`And/size=1000000-4`	`8e3eca`	`f18094`
	`4.58 ns/op`	`5.12 ns/op`	`1.12`	`🔴`
`OrTreeNextGeq/size=1000-4`	`------`	`f18094`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.79 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`f18094`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.89 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`f18094`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.36 ns/op`	`NaN`	`🔴`

# Conflicts: # frac/processor/aggregator.go # node/bench_test.go # node/node_and.go

# Conflicts: # frac/processor/aggregator.go # frac/processor/aggregator_test.go # frac/sealed/lids/iterator_asc.go # frac/sealed/lids/iterator_desc.go # node/lid.go # node/node.go # node/node_and.go # node/node_nand.go # node/node_or.go # node/node_range.go # node/node_static.go # node/node_test.go # node/sourced_node_wrapper.go

github-actions · 2026-03-19T13:33:53Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/large-4`	`8bbe78`	`a237ba`
	`14227.40 MB/s`	`12678.05 MB/s`	`0.89`	`🔴`
`Not/size=1000-4`	`8bbe78`	`a237ba`
	`4.64 ns/op`	`5.48 ns/op`	`1.18`	`🔴`
`Not/size=10000-4`	`8bbe78`	`a237ba`
	`4.65 ns/op`	`5.42 ns/op`	`1.17`	`🔴`
`Not/size=1000000-4`	`8bbe78`	`a237ba`
	`4.76 ns/op`	`5.64 ns/op`	`1.19`	`🔴`
`OrTreeNextGeq/size=1000-4`	`------`	`a237ba`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.73 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`a237ba`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.75 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`a237ba`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.19 ns/op`	`NaN`	`🔴`

github-actions · 2026-03-19T17:30:22Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`8bbe78`	`8b826a`
	`6735.60 MB/s`	`4864.17 MB/s`	`0.72`	`🔴`
	`39.57 ns/op`	`52.63 ns/op`	`1.33`	`🔴`
`Not/size=1000-4`	`8bbe78`	`8b826a`
	`4.64 ns/op`	`5.40 ns/op`	`1.16`	`🔴`
`Not/size=10000-4`	`8bbe78`	`8b826a`
	`4.65 ns/op`	`5.41 ns/op`	`1.16`	`🔴`
`Not/size=1000000-4`	`8bbe78`	`8b826a`
	`4.76 ns/op`	`5.56 ns/op`	`1.17`	`🔴`
`OrTreeNextGeq/size=1000-4`	`------`	`8b826a`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.65 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`8b826a`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.69 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`8b826a`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.11 ns/op`	`NaN`	`🔴`

github-actions · 2026-03-19T17:51:13Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`8bbe78`	`62d2cd`
	`6735.60 MB/s`	`5114.94 MB/s`	`0.76`	`🔴`
	`39.57 ns/op`	`50.05 ns/op`	`1.26`	`🔴`
`Not/size=1000-4`	`8bbe78`	`62d2cd`
	`4.64 ns/op`	`5.38 ns/op`	`1.16`	`🔴`
`Not/size=10000-4`	`8bbe78`	`62d2cd`
	`4.65 ns/op`	`5.41 ns/op`	`1.16`	`🔴`
`Not/size=1000000-4`	`8bbe78`	`62d2cd`
	`4.76 ns/op`	`5.53 ns/op`	`1.16`	`🔴`
`OrTreeNextGeq/size=1000-4`	`------`	`62d2cd`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.71 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`62d2cd`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.67 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`62d2cd`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.29 ns/op`	`NaN`	`🔴`

github-actions · 2026-03-20T07:08:11Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Not/size=1000-4`	`4e7a41`	`768c67`
	`4.68 ns/op`	`5.30 ns/op`	`1.13`	`🔴`
`Not/size=10000-4`	`4e7a41`	`768c67`
	`4.71 ns/op`	`5.33 ns/op`	`1.13`	`🔴`
`Not/size=1000000-4`	`4e7a41`	`768c67`
	`4.82 ns/op`	`5.48 ns/op`	`1.14`	`🔴`
`OrTreeNextGeq/size=1000-4`	`------`	`768c67`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.52 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`768c67`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`4.60 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`768c67`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.15 ns/op`	`NaN`	`🔴`

dkharms

Merge this bad boy.

github-actions · 2026-03-20T10:34:21Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`OrTreeNextGeq/size=1000-4`	`------`	`7608fc`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.03 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`7608fc`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.17 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`7608fc`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.85 ns/op`	`NaN`	`🔴`

github-actions · 2026-03-20T10:42:42Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`OrTreeNextGeq/size=1000-4`	`------`	`b2fef6`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.25 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`b2fef6`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.26 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`b2fef6`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.92 ns/op`	`NaN`	`🔴`

github-actions · 2026-03-23T06:45:54Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`OrTreeNextGeq/size=1000-4`	`------`	`0b246d`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.12 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`0b246d`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.18 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`0b246d`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.79 ns/op`	`NaN`	`🔴`

github-actions · 2026-03-23T06:57:59Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`OrTreeNextGeq/size=1000-4`	`------`	`9345c7`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.16 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=10000-4`	`------`	`9345c7`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.16 ns/op`	`NaN`	`🔴`
`OrTreeNextGeq/size=1000000-4`	`------`	`9345c7`
	`NaN B/op`	`0.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`0.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`5.87 ns/op`	`NaN`	`🔴`

cheb0 changed the title ~~perf: Next GEQ~~ perf: implement NextGEQ Feb 9, 2026

dkharms added the performance Features or improvements that positively affect seq-db performance label Feb 10, 2026

cheb0 force-pushed the 332-next-geq branch from b41fba2 to 564ec15 Compare February 12, 2026 13:22

ozontech deleted a comment from github-actions Bot Feb 13, 2026

ozontech deleted a comment from seqbenchbot Feb 13, 2026

ozontech deleted a comment from github-actions Bot Feb 13, 2026

ozontech deleted a comment from seqbenchbot Feb 15, 2026

ozontech deleted a comment from github-actions Bot Feb 15, 2026

ozontech deleted a comment from seqbenchbot Feb 15, 2026

ozontech deleted a comment from dkharms Feb 15, 2026

ozontech deleted a comment from seqbenchbot Feb 15, 2026

ozontech deleted a comment from dkharms Feb 15, 2026

ozontech deleted a comment from seqbenchbot Feb 15, 2026

ozontech deleted a comment from seqbenchbot Feb 16, 2026

fast lid representation for optimized comparison

6b42af1

cheb0 mentioned this pull request Feb 17, 2026

perf: optimize LID comparison in iterators #358

Merged

2 tasks

cheb0 added 4 commits February 17, 2026 15:10

add tests

c4d671e

remove reverse flag from node_range.go, delete unneeded null checks

ad98152

Merge branch 'refs/heads/main' into 0-fast-cmp-xor

6d5bbab

rebase onto fast cmp

e9401ba

cheb0 added 6 commits March 18, 2026 14:51

review fixes

eadd789

review fixes

bcbf323

review fixes: make masks private

3678a3a

Merge branch 'main' into 0-fast-cmp-xor

5925841

rename

1e602f6

Merge remote-tracking branch 'origin/0-fast-cmp-xor' into 0-fast-cmp-xor

45611fb

Base automatically changed from 0-fast-cmp-xor to main March 18, 2026 16:33

cheb0 added 3 commits March 18, 2026 20:45

Merge remote-tracking branch 'origin/0-fast-cmp-xor' into 332-next-geq

48d45b5

# Conflicts: # frac/processor/aggregator.go # node/bench_test.go # node/node_and.go

fix build

8be3a21

eguguchkin approved these changes Mar 19, 2026

View reviewed changes

review fixes: retire gallop search, honest NextGEQ

e31ba10

review fixes

53e51d8

review fixes: call Next()

2bd2a94

review fixes: add TODO

db23bd2

review fix: delete gallop search

045ae6a

dkharms approved these changes Mar 20, 2026

View reviewed changes

linter fixes

7c7eb95

Merge branch 'main' into 332-next-geq

3e12a0a

PR review: add TODO comment

0b0a5ba

cheb0 merged commit ec0eb71 into main Mar 23, 2026
8 checks passed

cheb0 deleted the 332-next-geq branch March 23, 2026 07:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: implement NextGEQ#345

perf: implement NextGEQ#345
cheb0 merged 42 commits intomainfrom
332-next-geq

cheb0 commented Feb 6, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Feb 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Feb 15, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 19, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

dkharms left a comment

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 20, 2026

Uh oh!

github-actions Bot commented Mar 23, 2026

Uh oh!

github-actions Bot commented Mar 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

cheb0 commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Measurements

Uh oh!

codecov-commenter commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Feb 15, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 19, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 19, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 19, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 20, 2026

🔴 Performance Degradation

Uh oh!

dkharms left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 20, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 20, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 23, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Mar 23, 2026

🔴 Performance Degradation

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cheb0 commented Feb 6, 2026 •

edited

Loading

codecov-commenter commented Feb 7, 2026 •

edited

Loading