feat(drive-abci,dashmate): seed Orchard shielded pool at SDK_TEST_DATA devnet genesis#3732
Draft
shumkov wants to merge 29 commits into
Draft
feat(drive-abci,dashmate): seed Orchard shielded pool at SDK_TEST_DATA devnet genesis#3732shumkov wants to merge 29 commits into
shumkov wants to merge 29 commits into
Conversation
…A devnet genesis
Pre-populates the shielded pool with 500_000 Orchard notes (8 owned by two
deterministic test wallets) when a local devnet binary is built with
`--cfg=create_sdk_test_data`. Closes the gap for benchmarking wallet sync at
scale without paying per-note Halo 2 proof time at chain bring-up.
Seeder (drive-abci):
- `create_data_for_shielded_pool` runs inside `create_sdk_test_data` and
emits 500k `ShieldedPoolOperationType::InsertNote` ops through the
production `commitment_tree_insert_op` path. GroveDB's
`preprocess_commitment_tree_ops` batches them into a single
Sinsemilla-frontier load / `append_with_mem_buffer` loop / Merk
propagation.
- Two-tier note generator: filler (random valid Pallas-base `cmx` + 216
bytes of opaque ciphertext) + owned (real `Note::from_parts` encrypted
to a fixed ZIP-32-derived address via `OrchardNoteEncryption<DashMemo>`).
- Genesis-time anchor recording at `block_height=1` matches production's
end-of-block-1 anchor — single recorded anchor suffices for spends via
the wallet's one-checkpoint-at-post-sync-tree-size invariant.
- Determinism: single `StdRng::seed_from_u64(0xDEAD_BEEF)` threaded
through every loop; `seed_a`/`seed_b` test wallets derived via
`SpendingKey::from_zip32_seed(seed, coin_type=1, account=0)`.
dashmate config plumbing:
- New `buildArgs: Record<string,string>` field on `dockerBuild` schema —
generic per-image build-arg map. Dashmate becomes the single source of
truth for `SDK_TEST_DATA` and `CARGO_BUILD_PROFILE`; shell-env
passthrough is dropped.
- `scripts/setup_local_network.sh` writes
`buildArgs.SDK_TEST_DATA="true"` + `CARGO_BUILD_PROFILE="release"` to
each `local_N` after `dashmate setup local`. Release profile is
mandatory — debug-Sinsemilla pushes InitChain past tenderdash's timeout.
(Marked TODO/temporary in the script — removable once Option B
precomputed-snapshot lands, or N drops low enough for debug seeding.)
- `generateEnvsFactory` flattens both `platform.drive.abci.docker.build.buildArgs`
and `platform.dapi.rsDapi.docker.build.buildArgs` into the
docker-compose env so `${KEY}` substitution in the compose `build.args`
blocks picks them up.
`dashmate config set` bug fix:
- The old `config.get(path)` pre-check rejected legal sets of new keys
inside `additionalProperties: <schema>` maps (e.g. `…buildArgs.X`).
Replaced with `Config.isSchemaPathAllowed(path)` which walks the JSON
schema descending through `properties`, `additionalProperties` value
schemas, and `$ref` references. 15 unit tests pin the walker.
Tests:
- 23 in-process tests in `rs-drive-abci`: wallet derivation, note
generator (filler + owned + ρ uniqueness + ciphertext layout +
determinism + per-wallet decrypt + cross-wallet privacy + aggregate
balance), Drive-level integration (count + anchor + cross-platform
byte-identical determinism).
- 15 dashmate unit tests for the schema walker.
- One `#[ignore]`-d functional test in `rs-platform-wallet` that drives
the full `PlatformWalletManager → bind_shielded → coordinator.sync →
shielded_balances` flow against a live SDK_TEST_DATA devnet.
Cost (release profile, 500_000 notes, Apple Silicon Docker Desktop):
~3h 41m wall-clock for the seeder. CPU work is ~95s; the rest is
GroveDB writes through the macOS Docker VM. See
`docs/shielded-seeder-performance.md` for the breakdown and the
Option-B follow-up.
Refs #3714.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…t InitChain
Cuts SDK_TEST_DATA shielded-pool seeding from ~3h41m (Docker on macOS at
N=500k) or ~65 min (native release at N=500k) to ~134 ms at InitChain by
moving the seed work to a one-shot bake during docker image build and
loading the result via `IngestExternalFile` + parent-Merk leaf patch.
End-to-end proven on local devnet: bake-inside-Dockerfile produces a
snapshot file with `combined_root=b682f38442c8...8144e3c3` byte-for-byte
identical to a local macOS bake; InitChain applies it in 134 ms; wallet-A
sync against the snapshot-loaded chain recovers the expected 400 000
balance (4 owned × 100 000) — proving proof verification works against
snapshot-loaded state.
Components
----------
- `packages/rs-drive-abci/src/shielded_snapshot/mod.rs` (NEW):
- `dump_shielded_subtree(grove, w)` — writes one SST per CF + header +
blake3 checksum.
- `apply_shielded_snapshot(grove, r, txn)` — validates header,
`ingest_subtree_sst`, cross-validates `combined_root` against the
reconstructed CommitmentTree, patches parent Merk leaf via
`replace_commitment_tree_subtree_root`. No fallback — any mismatch
is FATAL so the operator notices.
- `drive-abci snapshot-bake --out <path>` subcommand:
- Self-contained: opens fresh tempdir, runs `create_genesis_state`
(which under `cfg(create_sdk_test_data)` seeds the shielded pool),
then dumps the resulting subtree. Uses a NoopCoreRPC stub because
genesis doesn't talk to Core.
- `DRIVE_SHIELDED_SNAPSHOT` env var read in `create_data_for_shielded_pool`:
takes the snapshot fast-path when set, runs the runtime seeder
otherwise. Failure during apply is fatal (no silent fallback).
- Dockerfile: new `bake-shielded-snapshot` stage runs `drive-abci
snapshot-bake` against an in-container tempdir when `SDK_TEST_DATA=true`,
embeds the snapshot at `/opt/dashmate/snapshots/shielded-pool.snap` in
the runtime image, sets `ENV DRIVE_SHIELDED_SNAPSHOT=...`.
- `docs/genesis-snapshot-design.md` — design doc covering format,
cross-validation, threat model, compatibility policy.
Side-effect changes
-------------------
- `ShieldedSeedConfig::sdk_test_data().total_notes`: 500_000 → 5_000 for
fast iteration while the snapshot path is fresh. Bump back when needed.
- `scripts/setup_local_network.sh`: `CARGO_BUILD_PROFILE` set to `dev`
for fast docker rebuilds during snapshot dev. Flip to `release` when
going back to large N for stress testing.
- grovedb dep rev bumped to `04f2d4243872b65fbec33650e15d85571df385e1`
(branch `feat/snapshot-apply-public-api` — DON'T MERGE; adds three
public methods: `ingest_subtree_sst`, `replace_commitment_tree_subtree_root`,
`raw_storage`).
- New deps in `rs-drive-abci`: `rocksdb` (SstFileWriter), `blake3`
(snapshot checksum). Plus `grovedb-path` + `grovedb-storage` as dev-deps
for the data-location test.
Tests
-----
- `dump_only_default_and_aux_cfs_under_shielded_subtree_prefix` — pins
the CF layout the dump enumerates (default CF only, contrary to design's
original §3 guess of default + aux).
- `snapshot_dump_apply_preserves_anchor` — in-process roundtrip; A seeds,
dumps, B applies, asserts anchors match byte-for-byte.
- `bench_native_seed_full` — bake-feasibility benchmark; measures
N=500k seed in release locally.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
shumkov
changed the title
…ake at N=1M Builds on the Phase 1 snapshot bake + apply design with the crypto-level optimization from grovedb PR #751: skip the Sinsemilla frontier append for filler notes (which no wallet owns and nobody needs spend proofs for), keeping the full Sinsemilla path only for the 8 owned notes. Drops bake time at N=1M from a projected 8+ hours (Phase 1, hit during the prior overnight run) to ~12 minutes in-process on native macOS release. Roundtrip test passes: anchor matches byte-for-byte between the bake-side seed and the snapshot apply on a fresh DB. Changes ------- - Cherry-picked grovedb PR #751 onto our existing `feat/snapshot-apply-public-api` branch — exposes `append_raw_without_frontier` / `append_many_without_frontier` on `CommitmentTree` under the `test-seeding-ct` Cargo feature, plus an MMR-root cache that makes append O(1) instead of O(N). grovedb rev bumped: 04f2d424 → 60d121900. - `rs-drive-abci/Cargo.toml`: enable `test-seeding-ct` feature on the grovedb-commitment-tree dep. - `OwnedLayout::compute`: moved owned positions to the **tail** `[N-owned_count, N)` instead of striped. Forced by the frontier-less constraint (PR #751 hard-rejects a non-empty frontier in the no-frontier append path) — must bulk-seed ALL filler first while frontier is empty, then append owned through the regular path. - `seed_shielded_pool_with_config`: replaced the `apply_drive_operations(InsertNote × N)` per-note path with: 1. Open CommitmentTree directly via grovedb's `raw_storage().get_transactional_storage_context(...)`. 2. Bulk-seed filler via `append_many_without_frontier(iter)` — blake3 only, no Pallas math. 3. Per-note `append_raw` + `save` + `commit_mmr` for the 8 owned (mirrors grovedb's existing commitment_tree_insert pattern). 4. Commit subtree's StorageBatch through the transaction. 5. Patch parent Merk leaf via `replace_commitment_tree_subtree_root` using the combined_root from the final `append_raw`'s result (compute_current_state_root hit "Inconsistent store" after intermediate commit_mmr flushes; reading the result-returned roots avoids that). 6. Post-bake assert `count == cfg.total_notes` — catches silent truncation from a panic mid-bake. - Progress logging: emit `seed phase A progress` every 30s with appended/total/pct/elapsed/rate/ETA from inside the bulk iterator. Previously the seeder was silent between start and end, making N=1M bakes invisible. - `docs/genesis-snapshot-design.md` §15 expanded with the F1 constraint, the bake/apply asymmetry, anchor non-spendability caveat, F6 rejection-sampling-skip caveat, and updated test plan. §8 scoped to Phase 1; §10 test #6 (equivalence with runtime seeder) marked retired under Phase 2. Consequences ------------ - Anchor recorded at height 1 reflects only the 8 owned cmx at frontier-positions 0..7 (Sinsemilla frontier has its own counter, independent of BulkAppendTree's total_count). Wallets attempting to construct spend proofs would fail. Devnet-only; gated by cfg(create_sdk_test_data) + SDK_TEST_DATA=true at image build. - combined_root produced by a Phase-2 bake ≠ combined_root a runtime-seeded chain would produce. Cross-validation between bake and apply still works (both read the same stored frontier). - Wallet sync still recovers the 8 owned notes correctly because sync uses chunk proofs authenticated by bulk_state_root (transitively authenticated by the grovedb root), not by the Sinsemilla anchor. Performance ----------- N=1M in-process roundtrip on native macOS release: - seed (filler bulk + owned full): ~10 min - dump (267 MB SST): ~30 s - apply on fresh DB: ~1 min - total: 12 min wall-clock Anchors match byte-for-byte: d1f7ed699e0b0a7741ca91dbe7513abf7c5a53d418206ba7bde3b0a3dd974631 Phase 1 at N=1M (didn't complete after 8+ hours in docker on macOS) projected to ~2-3 hours native release. Phase 2 is roughly 40× faster end-to-end at this N. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…API change bind_shielded now takes &[u8] instead of [u8; 32] after v3.1-dev merge (part of IdentityManager refactor in PR #3651). One-character fix. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The original Regtest-only check made sense for local dashmate setups but blocked the intended use case from issue #3714: stress-testing wallet sync on real Dev networks with N=1M pre-seeded shielded notes via the snapshot image. Real Devnet chains run with Network::Devnet, not Network::Regtest. Mainnet + Testnet still rejected — they must never carry SDK test fixtures (random identities + seeded shielded pool + the junk Sinsemilla anchor that isn't a valid spend anchor). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the iOS-side measurement surface for shielded sync against a pre-populated note pool (paloma devnet / local snapshot image): - ShieldedService publishes lastSyncDuration + a 1Hz currentSyncElapsed ticker driven by the false→true edge of shieldedSyncIsSyncing. CoreContentView renders both in the ZK Shielded Sync Status section, with monospaced digits so the live ticker doesn't reflow. - New FFI platform_wallet_manager_bind_shielded_with_raw_seed accepts a raw 32-byte ZIP-32 seed, bypassing the MnemonicResolver. Required to bind the chain-side test wallet A (`[0x73; 32]`) that the snapshot bake seeds — no BIP-39 mnemonic can produce that seed. Swift wrapper bindShieldedRawSeed mirrors the existing bindShielded shape. - New orange "Bind Test Wallet A (Shielded)" debug button under the same Sync Status section hardcodes the seed and starts the sync loop. All raw-seed / button code is tagged `TODO(shielded-snapshot-devnet-test)` and is meant to be removed once SwiftExampleApp has a real test-wallet import flow (tracked: #3714). Spec: packages/swift-sdk/SwiftExampleApp/docs/shielded-sync-timing-spec.md Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…DAPI + quorum URLs Devnet had no end-to-end Platform path on the iOS app — `SDK.init` ignored the `platformDAPIAddresses` UserDefaults on non-regtest networks, and even when addresses were provided, the trusted context provider panicked on `Devnet` without an explicit quorum-list URL (no built-in default exists; `quorums.devnet.<name>.networks.dash.org` is templated on a devnet name we don't carry across FFI). Result: switching to Devnet always showed Disconnected. This wires it end-to-end: - DashSDKConfig gains a `quorum_url: *const c_char` field. When set, `dash_sdk_create_trusted` constructs the TrustedHttpContextProvider via `new_with_url(...)` regardless of network — overrides the hardcoded network default so devnet (and non-default mainnet/testnet shards) work. - SDK.init now reads `platformDAPIAddresses` and a new `platformQuorumURL` UserDefaults key and forwards both to the FFI on devnet + regtest unconditionally, and on mainnet/testnet when `useDockerSetup` is set. Helper `withOptionalCStrings` keeps the two-string lifetime contract clean. - OptionsView gets a dedicated devnet branch with three text fields (SPV Peers, DAPI URL, Quorum URL); no toggle since all three are always custom on devnet. The picker's onChange force-enables `useLocalhostCore` and seeds the SPV peers default for devnet so the Sync tab's `startSpv()` path picks up peers without a hidden toggle. - WalletManagerStore.activate compared cached managers by network only, returning the cached one even when AppState handed in a freshly-built SDK. That kept the manager pointing at the old SDK clone (with stale DAPI / quorum endpoints) so proof verification failed forever with "no available addresses to use". Now we cache the configured SDK handle and rebuild the manager when it doesn't match — the FFI `configure` is single-shot (precondition !isConfigured) so we can't swap in place. - SDKLogger.log mirrors to NSLog in addition to stdout so `xcrun simctl spawn booted log stream` and Console.app capture the timing log lines even when no Xcode debugger is attached. Token FFI test fixtures get the new `quorum_url: ptr::null()` field to keep compiling. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…stics
Adds the diagnostic surface for the iOS balance-of-zero investigation
(P0.2) and the followup punch list that came out of the live test
session.
- `tests/shielded_sync_paloma.rs` — new integration test that binds
wallet A (`SEED_A = [0x73; 32]`) against a remote SDK_TEST_DATA
devnet via TrustedHttpContextProvider, fans the gRPC traffic across
all 13 paloma masternodes' Platform gateways (`AddressList` picks
randomly per request), runs one sync pass, asserts balance =
EXPECTED_BALANCE_A = 400_000. Verified PASS against paloma in
1299s: total_scanned=1_000_000, new_notes_total=4, balances={0: 400000}
— confirms decryption + persistence work end-to-end at the Rust
layer, isolates iOS balance=0 to iOS-specific persister callback /
display path.
- `PlatformWalletPersistenceHandler.swift` — temporary NSLog
instrumentation on `persistShieldedNotes` and
`persistShieldedSyncedIndices` so the next iOS sync run surfaces
whether the callbacks fire (via `simctl spawn booted log stream`).
Both tagged `TODO(shielded-snapshot-devnet-test)` for removal once
P0.2 closes. Uses the safer `NSLog("%@", message)` pattern after
the multi-arg variadic form SIGBUSed (verified, May 2026).
- `docs/shielded-sync-devnet-followups.md` — punch list captured from
the live test session (P0.1 cold-sync duration preservation, P0.2
balance investigation, P1.1 auto-discover DAPI from /masternodes,
P1.2 per-chunk progress, P1.3 node-count indicator). Status table
+ prioritized order of attack.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…-out
Two iOS UX improvements that came out of the live paloma devnet test
session — both Swift-only.
**P0.1 — Preserve longest-pass duration**
The existing "Last sync duration" row gets clobbered every time a sync
completes. After a 20-minute cold sync of paloma's 1M-note pool, a
subsequent 3-second steady-state pass would overwrite the headline
number, leaving no way to recover what the cold sync actually took.
ShieldedService now publishes `longestSyncDuration` alongside
`lastSyncDuration`. Set to `max(prev, elapsed)` at every completion;
reset on bind / reset / clearLocalState (the four sites the existing
`lastSyncDuration = nil` pattern already covers).
CoreContentView renders a "Longest pass: N.NN s" row beneath "Last
sync duration", but only when meaningfully longer than the most recent
pass (`longest > last + 0.05s`) so steady-state runs don't render two
identical lines.
**P1.1 — Auto-populate DAPI list from `/masternodes`**
Previously, devnet usage required pasting all DAPI URLs by hand. With
only one URL in the field the SDK funnels every gRPC request through
one masternode (`AddressList::pick_address` picks randomly per request);
on a 1M-note sync that becomes the bottleneck.
`SDK.discoverDAPIAddresses(quorumBase:)` synchronously hits
`{quorumURL}/masternodes`, filters `status == "ENABLED"`, and builds
a comma-separated `https://<ip>:<platformHTTPPort>,…` list. 5-second
URLSession timeout via DispatchSemaphore (init can't be async without
a deeper refactor; the call runs off the main thread inside
`AppState.switchNetwork`'s Task).
SDK init triggers discovery when:
- the network uses overrides (devnet, regtest, or `useDockerSetup`),
- a quorum URL is set,
- and `platformDAPIAddresses` is empty (manual entry wins).
Resolved list is written back to `platformDAPIAddresses` UserDefaults
so OptionsView displays the actual fanned-out URLs. Subsequent SDK
builds skip the network round-trip; clearing the field re-triggers
discovery on the next switch.
OptionsView shows a small "N nodes" label above the DAPI URL field —
"auto (from /masternodes)" when empty, "13 nodes" when populated.
TextField switches to vertical-axis with `lineLimit(1...4)` so the
13-URL list is readable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Surface live cumulative-notes-scanned during a long shielded sync pass so the iOS UI can render a ticking counter / `ProgressView` instead of staring at a spinner for the 20+ min cold sync of a 1M-note pool. Previously `sync_shielded_notes` was a single async call that returned only at completion — between start and end there was no signal anything was happening. Touches every layer: - **rs-sdk** (`platform/shielded/notes_sync/types.rs`, `sync_shielded_notes.rs`): new `ProgressCallback` type (`Arc<dyn Fn(u64, u64) + Send + Sync>`). Added as an optional field on `ShieldedSyncConfig`; fired once per chunk inside the sliding-window fetch loop with `(cumulative_scanned, latest_block_height)`. Default `None` preserves prior behavior. - **rs-platform-wallet** (`events.rs`, `wallet/shielded/coordinator.rs`, `wallet/shielded/sync.rs`, `manager/mod.rs`): new trait method `PlatformEventHandler::on_shielded_sync_progress`. Coordinator holds an optional progress handler in a `Mutex<Option<…>>` (ArcSwap can't store `dyn Fn` — needs `T: Sized`); installed by the manager in `configure_shielded` with a closure that forwards into the event manager. `sync_notes_across` reads the handler, wraps it in a `ShieldedSyncConfig.on_chunk_completed`, and passes to the SDK. Network-scoped event (no wallet_id) — a single coordinator pass covers every bound IVK. - **rs-platform-wallet-ffi** (`event_handler.rs`): new `on_shielded_sync_progress_fn` slot on `EventHandlerCallbacks` (C-ABI-stable regardless of `shielded` feature). FFIEventHandler dispatches when shielded compiled in. - **swift-sdk** (`PlatformWalletManagerAddressSync.swift`, `PlatformWalletManagerShieldedSync.swift`, `PlatformWalletManager.swift`): wire new `shieldedSyncProgressCallback` C trampoline. PlatformWalletManager publishes `currentShieldedSyncScanned` and `currentShieldedSyncBlockHeight` (cleared on every completion). - **SwiftExampleApp** (`ShieldedService.swift`, `CoreContentView.swift`): ShieldedService republishes via `combineLatest` of the two manager publishers into `currentSyncScanned` / `currentSyncBlockHeight`. CoreContentView's "Syncing… elapsed" row gets a "Scanned this pass: N notes" sub-row + a linear `ProgressView` (indeterminate — chain commitment count isn't separately queried; absolute number is more useful than a fake bar). Both reset across all four bind/clear paths. - **WalletManagerStore.swift**: drive-by fix for the stale-SDK rebuild path — `activeManager` is non-optional so `= nil` failed to compile; let the post-rebuild `activeManager = manager` line overwrite cleanly. Release-mode paloma test (1M-note cold sync) baseline established at 1022 s wall clock; per-chunk callback fires ~once per 2048 notes inside the SDK fetch phase (~6 min in release). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
**P0.2 — wallet A balance=0 after sync, root-caused and fixed**
Rebinding shielded keys on a wallet that already has a persisted
watermark left the next sync starting at near-tree-tip, so the new
IVK never visited the positions where its owned notes lived.
`bind_shielded` calls `unregister_wallet` → `register_wallet` →
`restore_for_wallet`, and that last step rehydrates the watermark
from disk. For the mnemonic-bind path that's correct (same IVK
across restarts). For the raw-seed test path (different IVK) it
silently breaks scanning.
End-to-end validation against paloma (release-Rust integration test
+ debug iOS sim):
Rust test: total_scanned=1_000_000, decrypted_for_driver=4,
balances={0: 400_000}, 1022 s
iOS app: Scanned 1,000,000, New 4, Spent 0,
balance=0.000004 DASH (= 400_000 credits), 539.80 s
Fix: `NetworkShieldedCoordinator::force_rescan_subwallets(wallet_id,
accounts)` zeros the in-memory watermark for the bound subwallets.
Called from `platform_wallet_manager_bind_shielded_with_raw_seed`
right after `wallet.bind_shielded()` returns.
**Test-only scaffolding — strong DELETE-BEFORE-MERGE banner**
Every site that exists only to support the devnet sync-timing test
now carries an explicit `⚠️ TEST-ONLY CODE — DELETE BEFORE MERGE ⚠️ `
header with the full 5-site cleanup checklist:
- `platform_wallet_manager_bind_shielded_with_raw_seed` FFI entry
- `NetworkShieldedCoordinator::force_rescan_subwallets` helper
- Swift `PlatformWalletManager.bindShieldedRawSeed` wrapper
- `ShieldedService.bindWithRawSeed`
- "Bind Test Wallet A (Shielded)" button HStack in CoreContentView
- ATS exception in Info.plist (auto-cleaned with the rest)
Tag: TODO(shielded-snapshot-devnet-test). Tracked: #3714.
**Devnet UX simplification**
Reduced the devnet user-input surface to a single field — the
quorum list service URL — by deriving everything else from
`{quorum}/masternodes`. New shared helper
`SDK.discoverActiveMasternodes` returns each ENABLED masternode's
SPV peer (`ip:CoreP2PPort` from the `address` field) and DAPI URL
(`https://ip:platformHTTPPort`). Both are fetched fresh on every
SDK init / SPV start — self-healing on node churn.
OptionsView devnet branch: drops the manual SPV Peers TextField
and DAPI URL TextField; only the Quorum URL field remains.
SDK.init: always auto-discovers DAPI from /masternodes on devnet,
no longer writes back to UserDefaults.
`CoreContentView.spvPeerOverride`: devnet branch fetches the same
endpoint and extracts each masternode's `address` field verbatim
(paloma reports `ip:20001`, not the canonical 29999 — using
masternode-reported port is correct).
**Bundled Info.plist + ATS exception**
iOS's App Transport Security blocks plain-HTTP requests by default,
which prevented the SDK init from reaching the HTTP-only paloma
quorum-list-server. Switched the SwiftExampleApp project from
auto-generated to a real `Info.plist` (at project root so it stays
out of `PBXFileSystemSynchronizedRootGroup`'s auto-resource
inclusion) with `NSAllowsArbitraryLoads = true`. Plist carries the
same bundle/version/scene-manifest keys Xcode would have synthesized
plus the ATS dict. Test-only, must be removed before any production
release.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…t_data # Conflicts: # Cargo.lock
…t_data # Conflicts: # packages/rs-sdk-ffi/src/types.rs
shumkov
changed the title
Deletes the entire raw-seed bind scaffolding now that P0.2 is resolved and the iOS devnet sync-timing test has been validated end-to-end (0.000004 DASH on paloma matching the chain-side bake spec). Removed sites (the 5 banner-flagged deletions plus the P0.2 diagnostic NSLogs): - `platform_wallet_manager_bind_shielded_with_raw_seed` (rs-platform-wallet-ffi) - `NetworkShieldedCoordinator::force_rescan_subwallets` (rs-platform-wallet) - `PlatformWalletManager.bindShieldedRawSeed` (swift-sdk) - `ShieldedService.bindWithRawSeed` (SwiftExampleApp) - "Bind Test Wallet A (Shielded)" orange button HStack (CoreContentView) - Diagnostic NSLogs on `persistShieldedNotes` and `persistShieldedSyncedIndices` (PlatformWalletPersistenceHandler) What's kept (broader devnet path, not wallet-A-specific): - `SDK.discoverActiveMasternodes` + auto-discovery in `init` - Info.plist + ATS exception (needed for plain-HTTP /masternodes) - Devnet OptionsView simplification (single Quorum URL input) - All P0.1 / P1.1 / P1.2 timing + progress UI - `rs-platform-wallet/tests/shielded_sync_paloma.rs` — the canonical Rust integration test that proves the fix; keeps SEED_A by design Verified: Rust workspace + iOS framework + SwiftExampleApp all build clean. The non-wallet-A devnet flow (mnemonic bind on devnet against the auto-discovered DAPI nodes) is unaffected. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two `#[ignore]`-gated benchmarks that drove the diagnostic work behind the multi-chunk query change (#3756). `tests/shielded_chunk_timing_bench.rs` — issues N sequential single- chunk `ShieldedEncryptedNotes` fetches against a live SDK_TEST_DATA devnet (paloma by default) and reports per-chunk wall-clock distribution plus the per-chunk net/verify split (from `tracing::info!` inside `fetch_with_metadata_and_proof`). The flat per-chunk cost across count=64 vs count=2048 (1.57s vs 1.52s avg) is what proved the bottleneck is fixed per-request overhead, not bandwidth — directly justifying batching multiple chunks into one proof. `tests/shielded_decrypt_bench.rs` — measures trial-decrypt throughput over generated `ShieldedEncryptedNote` fixtures, single-threaded vs rayon-parallel. Showed decrypt is <1% of cold-sync wall-clock (~1.3s for 1M notes single-threaded on M-series), ruling out decrypt as the target for the multi-chunk PR. Run: cargo test -p platform-wallet --release --features shielded \ --test shielded_chunk_timing_bench -- --ignored --nocapture cargo test -p platform-wallet --release --features shielded \ --test shielded_decrypt_bench -- --ignored --nocapture New dev-deps to keep the benches self-contained: - drive-proof-verifier (for the wire types) - rayon (for parallel decrypt comparison) `rs-sdk-trusted-context-provider` was already a dev-dep for the existing paloma sync test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Imports re-sorted (std first, then external, then internal — alphabetized within each group) and over-wrapped function signatures collapsed back to one line per `rustfmt`'s defaults. Pure formatting; no behaviour change. Surfaced when running cargo fmt while staging the chunk-timing / decrypt benchmark commit — a handful of files in rs-drive-abci, rs-platform-wallet, and rs-platform-wallet-ffi had drifted out of fmt-clean state. Folding the cleanup into its own commit keeps the bench commit minimal and gives the next person clean diffs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps grovedb pin to feat/snapshot-apply-public-api tip (bdc0d6e9d7a5f7cec47b2701d08d6e35f558c7b8), which incorporates the rewritten PR #751: the old `_without_frontier` API (which skipped the Sinsemilla frontier entirely) is replaced with batched `CommitmentTree::append_many_raw`. The old API produced an anchor that didn't reflect the actual leaf set — wallets reconstructing the tree locally got a root the chain had never recorded and spend proofs failed. `append_many_raw` is byte-for-byte equivalent to N × `append_raw` (same frontier, same bulk MMR, same final roots) but the Sinsemilla anchor and bulk state root are each computed once at the end. Replaces the two-phase seed in `create_genesis_state/test/shielded.rs` (Phase A: bulk-seed filler via `append_many_without_frontier`; Phase B: per-note `append_raw` for owned, with per-note `save()` + `commit_mmr()`) with a single `append_many_raw(chain(filler, owned))` call followed by one `save()` at the end. The two-phase structure only existed to work around the old API's frontier divergence; with the new API the split is unnecessary. Owned notes still land at positions `[filler.len(), filler.len() + owned.len())`, matching the previous layout downstream test-data construction relies on. Drops the `test-seeding-ct` feature flag (deleted in PR #751 — the `open()` frontier/bulk consistency check is now unconditional).
…t_data # Conflicts: # packages/rs-sdk-ffi/src/types.rs # packages/swift-sdk/Sources/SwiftDashSDK/SDK.swift # packages/swift-sdk/SwiftExampleApp/Info.plist
Two cleanups on top of the prior `append_many_raw` migration: **1. Removed hardcoded test wallets.** Deleted `shielded_test_wallets.rs` entirely (SEED_A/SEED_B and TestWallet), along with `owned_count`/`owned_value` fields on ShieldedSeedConfig, the OwnedLayout struct, generate_owned_note, and every test that relied on test-wallet decryption (8 in `mod tests` + the try_decrypt helper). The seeder is now a pure stress test: N random filler notes with no special-position owned tier. Wallet-balance flows in tests should go through real shielded-fund transitions post-genesis (supported via PR #3753 / shield-from-asset-lock), not via the seeder. **2. Batched seed loop.** The single up-front allocation of all `total_notes` followed by a single `append_many_raw` call is replaced with a loop: - Generate 10_000 filler notes from the seeded RNG - `ct.append_many_raw(iter)` - `ct.save()` to persist the Sinsemilla frontier - `ct.commit_mmr()` to flush the MMR overlay (required between repeated `append_many_raw` calls — without it, the next batch hits "MMR get_root failed: Inconsistent store") - Log running anchor + bulk-state root for observability Memory: peak allocation goes from O(total_notes) × ~280 bytes (216 ciphertext + 32 cmx + 32 rho + Vec overhead) to O(10k) × 280 bytes ≈ 2.7 MB regardless of N. For N=1M that's ~280 MB → ~3 MB. Determinism: `batched_generator_matches_single_call` test pins that chained `generate_filler_batch` calls produce byte-identical output to a single `generate_notes(cfg)` call sized to total_notes, so the seeded GroveDB state is invariant to BATCH_SIZE. Observability: each batch logs `sinsemilla_root` + `bulk_state_root` in hex. Comparing roots between bake runs (or between batched and non-batched paths) immediately shows where divergence happens. Verified: - `cargo check -p drive-abci` clean with and without `--cfg create_sdk_test_data` - 12 tests pass (4 unit-tests + 6 integration + 2 new batched tests): generate_notes_*, batched_generator_matches_single_call, generated_cmx_values_are_unique, empty_config_*, seeded_pool_count_*, seeded_anchor_*, seeding_with_same_config_*, different_rng_seeds_*, seeding_zero_notes_*.
QuantumExplorer
added a commit
to dashpay/grovedb
that referenced
this pull request
May 28, 2026
GroveDB's `StorageContext::get` reads straight from the transaction and never consults its `StorageBatch` (see `storage/src/rocksdb_storage/storage_context/ context_tx.rs:296-310`). So as soon as `commit_mmr` drains the MMR overlay into the batch, the freshly-flushed peaks become invisible to reads until `commit_multi_context_batch` lands the batch in the tx — which only happens at the very end of a session. For chained `append_many_raw` calls on the same `CommitmentTree` (the 500k-note Drive seeder pattern in dashpay/platform#3732), the internal `commit_mmr` at the end of batch N drained the overlay; batch N+1's first compaction then read a peak via `MmrStore::element_at_position` → `ctx.get` → tx (peak not there) → `MMR::push` raised `InconsistentStore`. Make `commit_mmr` the caller's responsibility — same as `append_raw`, which also never flushes on its own. The overlay now stays alive across chained batches; the caller flushes it once, right before committing the surrounding batch. The docs spell this out and explicitly warn against mid-session `commit_mmr`. Updated the seeding bench to call `commit_mmr` explicitly before dropping `ct`. The byte-for-byte equivalence, spend-usable anchor, and cost-accounting tests are unaffected (they compare in-memory frontier/state-root values that don't depend on commit ordering). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shumkov
pushed a commit
to dashpay/grovedb
that referenced
this pull request
May 28, 2026
GroveDB's `StorageContext::get` reads straight from the transaction and never consults its `StorageBatch` (see `storage/src/rocksdb_storage/storage_context/ context_tx.rs:296-310`). So as soon as `commit_mmr` drains the MMR overlay into the batch, the freshly-flushed peaks become invisible to reads until `commit_multi_context_batch` lands the batch in the tx — which only happens at the very end of a session. For chained `append_many_raw` calls on the same `CommitmentTree` (the 500k-note Drive seeder pattern in dashpay/platform#3732), the internal `commit_mmr` at the end of batch N drained the overlay; batch N+1's first compaction then read a peak via `MmrStore::element_at_position` → `ctx.get` → tx (peak not there) → `MMR::push` raised `InconsistentStore`. Make `commit_mmr` the caller's responsibility — same as `append_raw`, which also never flushes on its own. The overlay now stays alive across chained batches; the caller flushes it once, right before committing the surrounding batch. The docs spell this out and explicitly warn against mid-session `commit_mmr`. Updated the seeding bench to call `commit_mmr` explicitly before dropping `ct`. The byte-for-byte equivalence, spend-usable anchor, and cost-accounting tests are unaffected (they compare in-memory frontier/state-root values that don't depend on commit ordering). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multi-batch `append_many_raw` was failing on the second call with "MMR get_root failed: Inconsistent store". Root cause is upstream grovedb behavior: `append_many_raw`'s internal MMR overlay accumulates compaction state across consecutive calls and is only safe to flush once at the end of the whole bake. Calling `ct.commit_mmr()` between batches corrupted the in-memory overlay so the next call read inconsistent state. Fixes: 1. **Bump grovedb pin** to `5eb7a5380a6e974513343352acfd6b30a8c1f87c` — picks up upstream commit `1340db71 fix(commitment-tree): don't commit_mmr inside append_many_raw`. 2. **Restructure the seeder loop** in `shielded.rs`: remove the per-batch `ct.commit_mmr()` introduced earlier, call it exactly once after the loop completes (immediately before `compute_commitment_tree_state_root`). Keep per-batch `ct.save()` — it persists the Sinsemilla frontier and is cheap (gives durable mid-bake checkpoints + keeps the per-batch root logs accurate). 3. **Bump the round-trip test** `snapshot_dump_apply_preserves_anchor` from N=5_000 (single-batch path) to N=30_000 (three batches at `BATCH_SIZE = 10_000`) so it actually pins the inter-batch MMR-overlay handoff that this fix targets.
QuantumExplorer
added a commit
to dashpay/grovedb
that referenced
this pull request
May 28, 2026
…ntier API (#751) * feat(commitment-tree): frontier-less bulk seeding for sync/scale testing Add `test-seeding`-gated `append_raw_without_frontier` and `append_many_without_frontier` on `CommitmentTree`. These populate the underlying BulkAppendTree at blake3 speed WITHOUT updating the Sinsemilla frontier, so a devnet shielded pool can be pre-loaded with a large N of filler notes without paying the per-note Pallas/Sinsemilla hashing (or the full Drive insert path). A frontier-less seeded tree has no valid Orchard anchor (so seeded notes aren't spendable), but BulkAppendTree chunk proofs — authenticated by the blake3 bulk state root, not the frontier — still verify, which is exactly what client wallet sync exercises. Under `test-seeding`, `CommitmentTree::open` tolerates the empty-frontier/populated-bulk shape so seeded state round-trips; production behavior is unchanged when the feature is off. The grovedb crate forwards this via `commitment_tree_test_seeding`. For devnet/benchmark seeding only — never enable in production. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(commitment-tree): reject frontier-less seeding on a non-empty frontier Guard both `append_raw_without_frontier` and `append_many_without_frontier`: advancing the bulk tree while the frontier already has leaves would leave `frontier_size < total_count`, a mismatch `open` cannot tolerate (it only accepts an empty frontier), producing unreopenable state. Reject it instead. Addresses CodeRabbit review on #751. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(commitment-tree): rename feature to test-seeding-ct; drop open() check under it Rename the seeding feature `test-seeding` -> `test-seeding-ct` (and the grovedb forwarding feature to match). Under the feature, drop the frontier/bulk consistency check in `CommitmentTree::open` entirely rather than tolerating only the empty-frontier case. This lets a frontier-less-seeded tree have real, frontier-tracked notes added on top via the normal `append` path and still reopen at any `(frontier_size, total_count)` pair. Production behavior is unchanged when the feature is off. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(commitment-tree): cover frontier-less seeding error paths for patch coverage Add fault-injection to the test storage mock (toggleable get/put failures) and three tests covering the previously-uncovered error branches in the frontier-less seeding methods: the wrapped bulk-append storage error, per-entry error propagation out of the bulk loop, and the empty-input state-root recomputation failure. The remaining commit_mmr flush error is annotated codecov:ignore (unreachable via the seeding API, which always flushes in-call). Raises patch coverage above the 90% gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(bulk-append-tree): cache MMR root so append is O(1), not O(N) per call `BulkAppendTree::append` called `get_mmr_root()` on every non-compaction append, and `get_mmr_root()` clones the `mmr_overlay` — which accumulates the chunk leaf nodes (each carrying a ~573 KB blob) across all compaction cycles until the session-end flush. Cloning that growing, blob-bearing overlay on every append made bulk seeding O(N^2): a 1M frontier-less seed degraded from ~2400 notes/s to sub-1100 and never finished in reasonable time. The MMR is only mutated on compaction (every `epoch_size` appends), so its root is unchanged in between. Cache it (`last_mmr_root`), refreshing only on compaction. `from_state` keeps it lazy (`None`) because a restored MMR may not be readable until the first append; the first append computes it once, exactly as before. Bulk seeding is now linear at a constant ~2435 notes/s (1M in ~6.85 min vs. 30+ min / non-terminating before). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bench(commitment-tree): add 1M frontier-less seeding throughput benchmark Standalone (harness=false) bench that seeds N random filler notes into a real RocksDB-backed CommitmentTree via append_many_without_frontier and reports wall-clock time, splitting compute from the disk commit. Defaults to 1M; override with SEED_N. Gated on test-seeding-ct. cargo bench -p grovedb-commitment-tree --bench seeding --features test-seeding-ct Measured 1M random notes in ~6.85 min (linear ~2435 notes/s) after the MMR-root caching fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(commitment-tree): batched append_many_raw — replace _without_frontier API Per-leaf `CommitmentTree::append_raw` was slow on bulk seeds not because of Sinsemilla-per-leaf (amortized ~1 hash via the carry chain) but because `CommitmentFrontier::append` called `self.root_hash()` after every insert, walking the depth-32 Sinsemilla path for ~32 hashes per leaf. For 1M leaves that's ~32M extra hashes. The upstream `incrementalmerkletree::Frontier` already separates append (cheap) from root (expensive); our wrapper conflated them. The earlier `_without_frontier` shape side-stepped this by skipping the frontier entirely, producing a Sinsemilla anchor that reflected only a handful of cmx — wallets reconstructing the tree locally got a root the chain had never recorded, and spend proofs failed. This commit fixes the actual problem. Added: * `CommitmentFrontier::append_no_root(cmx)` — upstream `Frontier::append`, no depth-32 walk; carry-chain cost only. * `CommitmentFrontier::root_hash_with_cost()` — pure accessor that attributes the deferred 32-hash depth walk for batched callers. * `BulkAppendTree::append_no_state_root(value)` — `append` minus the per-leaf `compute_state_root` blake3. `append` now delegates to it + one final `compute_current_state_root`. * `BulkAppendTree::append_many<I: IntoIterator<Item = Vec<u8>>>` — batched bulk-tree appends, one state-root computation at the end. * `CommitmentTree::append_many_raw<I: IntoIterator<Item = ([u8;32],[u8;32], Vec<u8>)>>` — byte-for-byte equivalent to N × `append_raw` (same dense-buffer, MMR, `CommitmentFrontier::serialize()`, final state and Sinsemilla roots) but the Sinsemilla anchor and bulk state root are each computed exactly once at the end. Flushes the MMR overlay before return. * `compute_current_state_root` now uses the `last_mmr_root` cache when populated (O(1) on the hot path). Removed: * `test-seeding-ct` Cargo feature in `grovedb-commitment-tree` and the forwarding feature in `grovedb`. * `append_raw_without_frontier`, `append_many_without_frontier`, `FrontierLessAppendResult`, `BulkSeedSummary`, and their lib.rs re-exports. * The `#[cfg(not(feature = "test-seeding-ct"))]` gating around the frontier/bulk consistency check in `CommitmentTree::open` — the check is now unconditional. * Fault-injection mock state and storage-fault tests that only existed to cover the deleted branches. Tests: * `append_many_raw_byte_for_byte_matches_per_leaf` — for N ∈ {0,1,2,3,100, 2048,10_000}, the per-leaf and batched paths produce identical `frontier.root_hash()`, `CommitmentFrontier::serialize()`, bulk state root, and `total_count`. * `append_many_raw_anchor_is_spend_usable` — independent Sinsemilla auth-path recomputation; verifies `orchard::MerklePath::from_parts(pos, path).root(cmx) == ct.anchor()`. * `append_no_root_cost_omits_per_leaf_depth_walk` — confirms the savings are exactly the per-leaf depth walks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(commitment-tree): add --features server to seeding bench example commands The bench is gated by required-features = ["server"], so the example invocations in the file-level docs need --features server or they run the fallback `main` and skip the actual benchmark. Addresses CodeRabbit review on #751. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(commitment-tree): don't commit_mmr inside append_many_raw GroveDB's `StorageContext::get` reads straight from the transaction and never consults its `StorageBatch` (see `storage/src/rocksdb_storage/storage_context/ context_tx.rs:296-310`). So as soon as `commit_mmr` drains the MMR overlay into the batch, the freshly-flushed peaks become invisible to reads until `commit_multi_context_batch` lands the batch in the tx — which only happens at the very end of a session. For chained `append_many_raw` calls on the same `CommitmentTree` (the 500k-note Drive seeder pattern in dashpay/platform#3732), the internal `commit_mmr` at the end of batch N drained the overlay; batch N+1's first compaction then read a peak via `MmrStore::element_at_position` → `ctx.get` → tx (peak not there) → `MMR::push` raised `InconsistentStore`. Make `commit_mmr` the caller's responsibility — same as `append_raw`, which also never flushes on its own. The overlay now stays alive across chained batches; the caller flushes it once, right before committing the surrounding batch. The docs spell this out and explicitly warn against mid-session `commit_mmr`. Updated the seeding bench to call `commit_mmr` explicitly before dropping `ct`. The byte-for-byte equivalence, spend-usable anchor, and cost-accounting tests are unaffected (they compare in-memory frontier/state-root values that don't depend on commit ordering). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(commitment-tree): cover append_many_raw + frontier error paths Address patch-coverage failures on PR #751: * Drop unused `BulkAppendTree::append_many` + `AppendManyResult`. The user spec said "or the equivalent" and `CommitmentTree::append_many_raw` goes through `append_no_state_root` in a loop — `append_many` had no real caller and was inflating the patch as dead code. * Re-add fault-injection toggles to the test storage mock and exercise: - `append_many_raw_rejects_invalid_cmx_mid_batch` - `append_many_raw_rejects_wrong_payload_size_mid_batch` - `append_many_raw_surfaces_bulk_storage_error` - `frontier_append_no_root_rejects_invalid_cmx` - `commit_mmr_success` (drives the bulk-tree `commit_mmr` flush path from the commitment-tree side, where `append_many_raw` purposely leaves the overlay in place per #751's chained-batch fix). * Annotate genuinely unreachable error branches with `codecov:ignore`: TreeFull on both `append` and `append_no_root` (requires 2^32 leaves), the frontier-append-error branch in `append_many_raw` (cmx already pre-validated upstream), and the end-of-batch state-root error branch (the dense-tree write-through cache + in-session MMR cache mean a fault-injecting mock can't trip it from inside `append_many_raw`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`FileBackedShieldedStore::open_path` now sets `journal_mode=WAL`, `synchronous=NORMAL`, and `temp_store=MEMORY` on the SQLite connection before handing it to `ClientPersistentCommitmentTree`. Prior default (DELETE journal + sync=FULL) forced a per-cmx fsync on hosts that honor it strictly, dominating cold-sync wall-clock on iOS Simulator and macOS. Measured with `tests/shielded_tree_append_bench.rs` at 100k appends: default (DELETE + sync=FULL) 71.67 s 716.7 µs/append WAL + sync=NORMAL 2.75 s 27.5 µs/append :memory: (CPU only) 0.88 s 8.8 µs/append FileBackedShieldedStore (wallet) 2.30 s 23.0 µs/append Projected for 1M leaves: 717 s → 23 s. The :memory: column shows the remaining cost is pure shardtree + Sinsemilla CPU; SQLite I/O is no longer the bottleneck. Crash-safety: NORMAL retains WAL fsync at checkpoint. The commitment tree holds no user funds — every cmx is chain-side authenticated and fully recoverable by resyncing from `last_synced_note_index`, so a torn WAL on power loss imposes at worst the same cost as a fresh install. Rationale documented inline on `open_path`. Adds `rusqlite = "0.38"` as an optional dep behind the `shielded` feature (matches the version grovedb-commitment-tree resolves to). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New lightweight unproved query returning the current leaf count of the shielded notes MMR. Wallets call it once at the start of a shielded sync to seed a determinate progress-bar denominator (notes_total → mmr_chunks_total → both download/checked bars). Why a new RPC instead of extending `GetShieldedPoolState`: the two metadata are unrelated semantically (pool credit balance vs MMR leaf count), have different access patterns (pool state is consensus- relevant; notes count is a UI hint), and the wallet may want to call either independently. Keeping them separate avoids coupling future versioning of one to the other. No proof variant: `Drive::shielded_pool_notes_count` reads the leaf counter off the CommitmentTree node — tree metadata, not a stored GroveDB key, so there's no `PathQuery` that could produce a proof. Same pattern as `GetStatus` / `GetCurrentQuorumsInfo`. SDK helper: `fetch_shielded_notes_count(&sdk) -> Result<u64, Error>` via `FetchUnproved`. Wires it end-to-end: - proto + dapi-grpc codegen registration (request-only) - drive-abci query handler at `query::shielded::notes_count` + tests - `service.rs` gRPC route - `notes_count: FeatureVersionBounds` slot on `DriveAbciQueryShieldedVersions` (v0, v1, v2_test mock all initialized to (0,0,0); no protocol bump) - rs-dapi-client transport route - rs-dapi platform_service drive_method! arm - drive-proof-verifier `ShieldedNotesCount(pub u64)` + unproved impl - rs-sdk `FetchUnproved` impl + `Query<…>` impl + mock registration Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the sequential fetch-all → decrypt-all → append-all shielded sync pipeline with an interleaved one: the wallet appends commitments to its local Merkle tree while the SDK is still fetching later chunks. Total cold-sync wall-clock drops from network + tree to ~max(network, tree), and the UI can now show two distinct progress signals. SDK (rs-sdk): - New `sync_shielded_notes_stream` yields `ShieldedChunkBatch`es in strictly ascending tree-position order via a pull-based `futures::stream::unfold`. Out-of-order network completions wait in an internal `ReorderBuffer` until their predecessor emits. - Backpressure is pull-based: the `FuturesUnordered` only advances when the consumer polls, so a slow tree-append caps in-flight fetches at `max_concurrent` and bounds the reorder buffer to the same window — no spawned producer task (also keeps it wasm32-safe). - `sync_shielded_notes` is now a thin wrapper that drives the stream to completion and assembles the same `ShieldedSyncResult`; every existing caller is unchanged. `next_start_index` rewind semantics preserved exactly. - Trial decryption moved to per-chunk (just before emission). - 3 unit tests on the extracted `ReorderBuffer` pin the ordering invariant (ascending under scrambled arrival, hold-back until predecessor, non-zero resume watermark). - `platform::types::shielded` made `pub` so the wallet can call `fetch_shielded_notes_count`. Wallet (platform-wallet): - `sync_notes_across` consumes the stream batch-by-batch: append → fire tree-progress → trial-decrypt (driver + other subwallets) → stage notes, all per batch. All documented invariants preserved: the append idempotency gate still compares against the pre-stream `tree_size` snapshot (NOT the live, actively-growing size), the monotonic checkpoint id is still the true post-append leaf count and still hard-fails past u32, mark-every-position and per-subwallet save gating are unchanged, watermark/`total_scanned` accumulate from batches identically to the one-shot path. - Denominator for the "checked" bar comes from a single `fetch_shielded_notes_count` at pass start; a failure degrades to 0 (indeterminate total) rather than failing the sync. - New coordinator `ShieldedTreeProgressCallback` + `install_tree_progress_handler`/`tree_progress_handler()`, threaded into `sync_notes_across` alongside the existing download callback. Behavioral note: the consumer holds the store write lock across the whole interleaved fetch+append (previously the fetch ran lock-free). Correct — sync is the sole writer and the stream's producer never touches the store — but store readers block for the full sync duration instead of just the append phase. Acceptable: cold sync balance isn't meaningful yet, incremental syncs are sub-second. Can move to per-batch locking if it ever matters. FFI/Swift wiring for the second progress bar is a deliberate follow-up (TODO left at the manager install site); the coordinator hook and the per-batch firing already exist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the second "checked / committed-to-tree" progress signal from the coordinator through events → FFI → Swift → a dual ProgressView, mirroring the existing "downloaded" signal at every layer. The interleaved sync already fires a per-batch tree-progress callback; this lights it up end-to-end. Rust: - events.rs: `PlatformEventHandler::on_shielded_tree_progress( leaves_committed, total_target)` trait method (default no-op) + `PlatformEventManager` dispatch. `total_target == 0` ⇒ indeterminate. - rs-platform-wallet-ffi event_handler.rs: `on_shielded_tree_progress_fn` appended at the END of `EventHandlerCallbacks` (preserves existing C-ABI field offsets), plumbed unconditionally; trait impl gated on the shielded feature. - manager/mod.rs: replaced the follow-up TODO with the real `coordinator.install_tree_progress_handler(...)` wiring, forwarding to the new event via a dedicated `Arc::clone` of the event manager. Swift: - PlatformWalletManager: `currentShieldedTreeCommitted` / `currentShieldedTreeTotal` published mirrors. - PlatformWalletManagerShieldedSync: `handleShieldedTreeProgress` + `shieldedTreeProgressCallback` C trampoline; both vars cleared on pass completion. - PlatformWalletManagerAddressSync: registers the new callback. - ShieldedService: `currentTreeCommitted` / `currentTreeTotal` published + a second combineLatest subscription, cancelled/reset at every teardown site. - CoreContentView: `ShieldedDualProgressRows` — "Downloaded" over "Checked", both using the Rust-carried total as a shared denominator (total notes == total leaves). Determinate once total > 0; honest spinner + raw count when indeterminate. Value clamped to total so a batch landing before the denominator refreshes can't overshoot 1.0. Per swift-sdk/CLAUDE.md the denominator arrives entirely from Rust via the callback — no Swift-side notes-count fetch or chunk math. Verified: both Rust crates check clean, C header regenerates with the new field, xcframework + SwiftExampleApp build succeed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shielded progress-bar denominator now comes "for free" from the
proof that already delivers each note chunk — no separate RPC, no
server change, no proof-size change, and crucially no dependency on
GetShieldedNotesCount being deployed (so the bars work against
currently-deployed nodes).
The note-fetch proof is a GroveDB layered proof that subqueries INTO
the shielded notes CommitmentTree, so the parent
`Element::CommitmentTree(total_count, ..)` is necessarily present in
the proof bytes (it chains to the root hash and feeds total_count into
the inner CommitmentTree subquery verification — the
`add_parent_tree_on_subquery: false` flag only suppresses sibling
subtree walks, not the queried tree's own element). We were simply
discarding it.
`verify_shielded_encrypted_notes` now extracts it by running an
additional `verify_subset_query` against the SAME proof bytes with a
single-key PathQuery for the CommitmentTree element (no subquery), then
decoding `total_count` (field 0). A round-trip test inserts N notes,
builds a real note-fetch proof via the server-side PathQuery, and
asserts the extracted count == N — proving the parent element is in the
proof.
Plumbing:
- verifier returns `(root_hash, notes, total_count)`.
- `ShieldedEncryptedNotes` becomes `{ notes, total_count }`; `FromProof`
threads it and now yields `Some(..)` on empty chunks so the count
still flows on the final empty chunk.
- `fetch_chunk` returns `total_count`; `ShieldedChunkBatch` and
`ShieldedSyncResult` carry it.
- wallet `sync_notes_across` sources `total_target` from
`batch.total_count` (max-seen) instead of calling
`fetch_shielded_notes_count` — that up-front RPC is removed.
The standalone GetShieldedNotesCount query (PR #3769) stays as the
primitive for showing the total WITHOUT syncing; the sync path no
longer needs it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves conflicts created when #3769 (GetShieldedNotesCount) merged to v3.1-dev as the PROVED query, superseding this branch's earlier UNPROVED version (eabb145). Conflict resolution: - All GetShieldedNotesCount surface (proto, notes_count handler, types.rs ShieldedNotesCount doc, query.rs Query impl, and the SDK unproved/fetch surface) taken from v3.1-dev — the proved version is authoritative. The branch's stale unproved impls (FromUnproved, FetchUnproved) are dropped; query.rs keeps the proved-only recoverable error on prove=false. - proto + regenerated gRPC clients now match v3.1-dev exactly. - Kept this branch's separate note-fetch total_count work: the ShieldedEncryptedNotes { notes, total_count } struct and the verify_shielded_encrypted_notes extraction are preserved (they don't conflict with the proved standalone query). Verified: dapi-grpc, drive-proof-verifier, dash-sdk, drive-abci, platform-wallet (--features shielded), and wasm-sdk all check clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the gap from #3714 for benchmarking wallet sync at scale. When a local devnet's drive-abci is built with
--cfg=create_sdk_test_data,InitChainnow pre-populates the Orchard shielded pool with 500_000 notes (8 owned across two deterministic ZIP-32 test wallets) before block 1 is proposed. No per-note Halo 2 proving at bring-up; deterministic root hash for byte-identical re-runs.Adopts Option A from the design audit: seeder runs through the production
commitment_tree_insert_oppath, batched into a singleapply_drive_operationscall so GroveDB'spreprocess_commitment_tree_opscollapses 500k inserts into one Sinsemilla-frontier load +append_with_mem_bufferloop + one Merk propagation.What's in scope
Drive-abci seeder
create_data_for_shielded_poolinsidecreate_sdk_test_data(regtest-only runtime check kept).cmx, opaque 216-byte ciphertext) + owned (realNote::from_partsencrypted to a fixed wallet viaOrchardNoteEncryption<DashMemo>).block_height = 1(matches production's first end-of-block anchor).StdRngthreaded through every randomness call (seed_a = [0x73;32],seed_b = [0x74;32], derived viaSpendingKey::from_zip32_seed(seed, coin_type=1, account=0)).orchard+zip32deps onrs-drive-abci(gated for use only inside the cfg).dashmate generic
buildArgsconfigdockerBuildschema:buildArgs: Record<string,string>. Replaces ad-hoc shell-env passthrough as the single source of truth for image build args.scripts/setup_local_network.shwritesbuildArgs.SDK_TEST_DATA = "true"+buildArgs.CARGO_BUILD_PROFILE = "release"to eachlocal_Nafterdashmate setup local. Marked TODO/temporary — release profile is mandatory while the seeder is slow.generateEnvsFactoryforwards both drive-abci and rs-dapi build-args into compose's env so${KEY}substitution inbuild.args:resolves.dashmate config setbug fixconfig.get(path)rejected legal sets of new keys underadditionalProperties: <schema>maps. Replaced withConfig.isSchemaPathAllowed(path)— a schema walker that descendsproperties,additionalPropertiesvalue schemas, and$ref. 15 unit tests pin it.What's NOT in scope (follow-ups)
docs/shielded-seeder-performance.mdfor the breakdown and removal-conditions of the TODO.Drive::validate_anchor_existsvia constructed but unsubmitted spend) — design doc §9 acceptance, deemed tautological given current sync test (we drop it here, can revisit).SEED_A/SEED_B— currently duplicated between drive-abci side and the platform-wallet functional test with// keep in synccomment.Test plan
cargo test -p drive-abci --lib 'create_genesis_state::test::shielded'withRUSTFLAGS='--cfg create_sdk_test_data --cfg tokio_unstable'): 23/23 passing — wallet derivation, generator (filler + owned + ρ uniqueness + ciphertext + determinism + per-wallet decrypt + cross-wallet privacy + aggregate balance), Drive integration (count + anchor + cross-platform byte-identical determinism).yarn mocha test/unit/config/Config.spec.js): 15/15 passing.PlatformWalletManager → bind_shielded → coordinator.sync → shielded_balancesrecovers400_000per wallet against an SDK_TEST_DATA-seeded chain (earlier run at N=500). Functional test ships inpackages/rs-platform-wallet/tests/shielded_sync.rsas#[ignore]-d; run withcargo test -p platform-wallet --test shielded_sync --features shielded -- --ignored --nocaptureafteryarn startbrings a seeded devnet up.🤖 Generated with Claude Code