Open
Conversation
The surrounding code already logs enough on success
The log in stream_aggregator can become very spammy with a small block range The log in monitor isn't really needed since surrounding code already logs enough information
Refactor the loop body in new_data_stream into focused helper functions to improve readability and prepare for lazy stream chaining: - build_query_streams: builds SQL queries and fires them via the client - build_data_stream: creates StreamAggregator with metrics and reorg check - Refactor next_block_ranges/next_block_range to take specific fields instead of &Context<AC> - Simplify min_start_block/max_end_block as one-liner aggregates No behavior change.
Replace the eager loop that builds all StreamAggregator instances upfront and chains them together with stream::unfold + flatten. Each block-range iteration's stream is now produced on-demand only when the previous one is exhausted, avoiding a large upfront chain structure when there are many block ranges to cover.
When a stream returns Poll::Ready but the aggregator can't produce output yet (empty batch or data buffered without completed groups), the waker is consumed and never re-registered. This causes the aggregator to stop being polled, deadlocking the subgraph. Fix by calling cx.waker().wake_by_ref() before returning Pending whenever any stream returned Ready during the polling cycle.
Rename GRAPH_AMP_MAX_BLOCK_RANGE to just GRAPH_AMP_BLOCK_RANGE Lower the default to 100k blocks from 2M blocks
Entity::merge_remove_null_fields iterates via the raw Object iterator which, unlike every other Entity API, does not filter the VID field. Since EntityCache::set stamps a fresh VID on every save, the VID always differs from the stored entity's VID, causing merge_remove_null_fields to report a change even when the user-visible data is identical. This produces a spurious Overwrite modification (and therefore a new row in the database) each time an unchanged entity is re-saved. Skip the VID when computing the changed flag while still merging it into the entity so downstream code (e.g. store insert queries) sees the latest value.
This adds a new section "as_modifications_load" that shows us how much time we spend reading from the database
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.