From 66e0d6d217378dc1e7891002504405565ef962bd Mon Sep 17 00:00:00 2001 From: Ivan Shumkov Date: Thu, 28 May 2026 19:56:37 +0700 Subject: [PATCH 1/2] feat(dashmate): configure docker build args via config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a `dockerBuild.buildArgs` map on the dashmate config schema so per-image Docker build args can be pinned in the config file (and overridden per-invocation through shell env), without having to bake them into the compose YAML by hand or into yarn scripts. Pieces: - **Schema** (`src/config/configJsonSchema.js`): `dockerBuild.buildArgs` is `{ [string]: string }`. Common keys today are `CARGO_BUILD_PROFILE` (Rust profile) and `SDK_TEST_DATA` (compile-time cfg flag), but it's open-ended. - **Config helpers** (`src/config/Config.js`, `src/commands/config/set.js`): traverse the schema correctly when the user does `dashmate config set platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA true` (the original pre-check rejected map-shaped paths; this PR fixes that and pins it with `Config.spec.js::buildArgs`). - **Env forwarding** (`src/config/generateEnvsFactory.js`): both `platform.drive.abci.docker.build.buildArgs` and `platform.dapi.rsDapi.docker.build.buildArgs` are merged and exported as env vars at compose-spawn time so the compose `${KEY:-default}` substitution picks them up. Dashmate config is the single source of truth — `process.env[key]` is not consulted as fallback. - **Compose YAML** (`docker-compose.build.drive_abci.yml`, `docker-compose.build.rs-dapi.yml`): forward `CARGO_BUILD_PROFILE` to the Dockerfile `ARG` (`${CARGO_BUILD_PROFILE:-dev}`). - **Default config** (`configs/defaults/getBaseConfigFactory.js`): `buildArgs: {}` on both rs-dapi and drive-abci build blocks so the schema accepts overrides. Yarn-script side (`scripts/setup_local_network.sh`): - The local-network setup script pins `SDK_TEST_DATA=true` on each `local_N` config so the genesis SDK test data runs at devnet bring-up. `CARGO_BUILD_PROFILE` is deliberately NOT pinned in the script — operators set release per-invocation via `CARGO_BUILD_PROFILE=release yarn start`, or persist it via `yarn dashmate config set` directly. Keeping the profile out of the script means the same `yarn setup` works for both dev iteration loops (default `dev` profile) and SDK_TEST_DATA bakes (release). Docs: - `docs/shielded-seeder-performance.md` documents why release profile is recommended when SDK_TEST_DATA is on at the default N=1_000_000 bake (~13× faster Sinsemilla on release vs debug). Referenced from `configJsonSchema.js` and `getBaseConfigFactory.js`. Tests: `Config.spec.js` adds 15 cases covering the map-shaped path traversal — including a regression test for the original `buildArgs.SDK_TEST_DATA` failure. --- docs/shielded-seeder-performance.md | 132 ++++++++++++++++ .../configs/defaults/getBaseConfigFactory.js | 6 + .../docker-compose.build.drive_abci.yml | 6 + .../dashmate/docker-compose.build.rs-dapi.yml | 2 + packages/dashmate/src/commands/config/set.js | 14 +- packages/dashmate/src/config/Config.js | 68 +++++++++ .../dashmate/src/config/configJsonSchema.js | 18 +++ .../src/config/generateEnvsFactory.js | 26 ++++ .../dashmate/test/unit/config/Config.spec.js | 143 ++++++++++++++++++ scripts/setup_local_network.sh | 15 ++ 10 files changed, 428 insertions(+), 2 deletions(-) create mode 100644 docs/shielded-seeder-performance.md create mode 100644 packages/dashmate/test/unit/config/Config.spec.js diff --git a/docs/shielded-seeder-performance.md b/docs/shielded-seeder-performance.md new file mode 100644 index 0000000000..25d7372721 --- /dev/null +++ b/docs/shielded-seeder-performance.md @@ -0,0 +1,132 @@ +# Shielded test-data seeder — performance notes + +The `SDK_TEST_DATA` shielded-pool seeder runs at `InitChain` and inserts +`ShieldedSeedConfig::sdk_test_data().total_notes` notes (currently +`500_000`) into the chain's commitment tree before the first block is +proposed. This is CPU-heavy work that scales linearly with `N`. This +document tracks the options for making it fast enough to use on a real +devnet. + +## Cost breakdown for N = 500 000 + +| Step | Cost shape | Dominant in | +|---|---|---| +| Pallas-base rejection sampling | ~1 ChaCha12 draw + 1 field check per filler `cmx` | release: negligible; debug: ~5–10% | +| Owned-note Orchard encryption | ~8 ops total (`Note::from_parts` + `OrchardNoteEncryption`) | always negligible | +| Random ciphertext bytes for filler | `N × 216` ChaCha12 bytes (~108 MB) | always negligible | +| `BulkAppendTree` append | `N` buffer writes + `⌈N/2048⌉` chunk compactions (dense Merkle + MMR + blake3) | release: ~5–10%; debug: ~15% | +| **Sinsemilla frontier append** | **`N × O(log N)` Pallas hashes (~9.5M for N=500k)** | **release: ~85%; debug: ~75%** | +| GroveDB Merk propagation | Once per `apply_drive_operations` call (batched) | always negligible | + +The Sinsemilla appends dominate. Each Pallas hash is the inner loop: +~20k cycles in release, ~200k–2M cycles in debug. + +## Wall-clock measurements + +Apple M-series, single SDK_TEST_DATA dashmate node, fresh `yarn reset`: + +| N | Profile | Wall clock | Source | +|---|---|---|---| +| 500 | dev | ~1 s | observed | +| 16 | dev | <100 ms | unit test | +| 500 000 | dev | **30+ min and not done** | observed 2026-05-23 | +| 500 000 | release | ~1–3 min | extrapolated from release Pallas timing (~10 μs/hash × 9.5M ÷ 1 core ≈ 95 s) | +| 1 000 000 | release | ~3–6 min | linear extrapolation | +| 1 000 000 | dev | ~60–120 min | unusable | + +## Options researched, ranked by impact / cost + +### 1. Release build (`CARGO_BUILD_PROFILE=release`) — **recommended first step** + +- **Speedup:** ~20–50× for the Sinsemilla phase; ~10× overall. +- **Effort:** one config option in dashmate (see below) + one Docker build arg. +- **Tradeoff:** slower image builds (release optimizations take longer to compile), larger debug-info-stripped binary remains ~50–100 MB vs ~700 MB for debug. +- **Status:** plumbed through dashmate as `platform.drive.abci.docker.build.cargoBuildProfile` — set to `release` when you want fast seeding. + +### 2. Parallelize note generation + +- **Speedup:** ~4–8× on top of #1. +- **Effort:** moderate. Two-stage pipeline: + - Worker pool: per note, sample `cmx` + build owned ciphertext or random filler. + - Single consumer thread: feed `(cmx, rho, encrypted_note)` tuples sequentially into `commitment_tree_insert_op` (Sinsemilla append MUST stay sequential — frontier state depends on previous step). +- **Tradeoff:** new concurrency surface, harder determinism guarantees (need deterministic per-worker RNG seeding). +- **Status:** not implemented. Consider only if #1 isn't enough. + +### 3. Option B — precomputed GroveDB snapshot baked into the image + +- **Speedup:** seeding cost → **0** at every `yarn reset` (one-time precomputation cost when the snapshot is generated). +- **Effort:** significant. + - Standalone tool that opens a fresh GroveDB, writes BulkAppendTree chunk blobs (`e{u64}` keys), tail buffer (`b{u32}`), chunk-MMR nodes (`m{u64}`), metadata (`M`, `mmr_size`), Sinsemilla frontier (`__ct_data__`), and the parent Merk's `Element::CommitmentTree(...)` element — all consistently. + - Dockerfile change: bundle the precomputed `db/` directory into the drive-abci image. + - Genesis code: detect the bundled snapshot and skip the seeder. +- **Tradeoff:** big code surface, but the runtime cost completely vanishes. +- **Status:** not implemented. The right answer if scaling to 5M+ notes or doing repeated benchmarks where seed time matters. + +### 4. Skip Pallas rejection sampling for filler + +- **Speedup:** ~5–10% (saves one `from_repr` field check per filler). +- **Effort:** small refactor — replace the rejection loop with `Nullifier::dummy(&mut rng)` which uses `extract_p(&pallas::Point::random(rng))` (always valid by construction). +- **Tradeoff:** depends on `Nullifier::dummy` being `pub` (it's `pub(crate)` in upstream `orchard` unless we enable `unstable-voting-circuits`). +- **Status:** not implemented. Marginal win; skip unless every second matters. + +### 5. Deterministic-bytes filler ciphertext + +- **Speedup:** ~5–10% (saves `rng.fill_bytes(216)` per filler). +- **Effort:** trivial — derive 216 bytes from `blake3(rng_seed || position)` instead. +- **Tradeoff:** changes byte layout slightly (still valid 216-byte payload; wallet treats it as opaque garbage). +- **Status:** not implemented. Marginal. + +### 6. GPU / SIMD-accelerated Sinsemilla + +- **Speedup:** 100×+ in theory for the Pallas-hash inner loop. +- **Effort:** way out of scope. Out-of-tree dependency on a GPU Sinsemilla crate; integration is non-trivial. +- **Tradeoff:** breaks portability (requires NVIDIA hardware) and determinism guarantees become tricky. +- **Status:** not pursued. + +## How build args reach the binary + +Dashmate's `dockerBuild` config exposes a generic `buildArgs` map that flows +through `generateEnvsFactory` → `docker compose build` → Dockerfile `ARG`s. +This is the **only** supported way to set build args — shell env vars are +not wired through. + +`scripts/setup_local_network.sh` (run automatically by `yarn setup` after +`dashmate setup local` creates the per-node configs) sets two args on each +`local_1/2/3` config: + +| arg | value | why | +|---|---|---| +| `SDK_TEST_DATA` | `"true"` | activates the `create_sdk_test_data` cfg flag → genesis seeder runs | +| `CARGO_BUILD_PROFILE` | `"release"` | optimised binary — without it, 500k-note seeding takes 30+ min in debug | + +Both are mandatory together. Debug-profile builds with N=500_000 push +InitChain past tenderdash's timeout window; release-profile finishes in 1–3 +minutes. + +## Switching back to debug for faster compile iteration + +If you're iterating on drive-abci itself and don't care about seeding speed, +swap CARGO_BUILD_PROFILE back to `dev` per-config: + +```bash +for cfg in local_1 local_2 local_3; do + yarn dashmate config set platform.drive.abci.docker.build.buildArgs.CARGO_BUILD_PROFILE dev --config $cfg +done +``` + +Note: any `yarn reset` will rerun `scripts/setup_local_network.sh` and put +`CARGO_BUILD_PROFILE=release` back. Edit that script if you want a different +default permanently. + +The same `buildArgs` field works for any other build arg the Dockerfile +declares. The schema is `Record`. + +## When to revisit Option B + +Re-evaluate the precomputed-snapshot path (#3) when any of these are true: + +- Seeding takes more than ~5 minutes even in release mode (i.e. N ≥ 2M). +- The benchmark workflow does many resets per day and seed time becomes the + per-iteration bottleneck. +- The chain config changes shape such that even release-mode seeding becomes + uneconomical to repeat. diff --git a/packages/dashmate/configs/defaults/getBaseConfigFactory.js b/packages/dashmate/configs/defaults/getBaseConfigFactory.js index e21b2111e8..56ca45c224 100644 --- a/packages/dashmate/configs/defaults/getBaseConfigFactory.js +++ b/packages/dashmate/configs/defaults/getBaseConfigFactory.js @@ -268,6 +268,7 @@ export default function getBaseConfigFactory() { context: path.join(PACKAGE_ROOT_DIR, '..', '..'), dockerFile: path.join(PACKAGE_ROOT_DIR, '..', '..', 'Dockerfile'), target: 'rs-dapi', + buildArgs: {}, }, }, metrics: { @@ -293,6 +294,11 @@ export default function getBaseConfigFactory() { context: path.join(PACKAGE_ROOT_DIR, '..', '..'), dockerFile: path.join(PACKAGE_ROOT_DIR, '..', '..', 'Dockerfile'), target: 'drive-abci', + // Extra docker build args (see `dockerBuild` schema). Common + // override: `CARGO_BUILD_PROFILE: "release"` for SDK_TEST_DATA + // shielded seeding at N > a few thousand + // (`docs/shielded-seeder-performance.md`). + buildArgs: {}, }, }, logs: { diff --git a/packages/dashmate/docker-compose.build.drive_abci.yml b/packages/dashmate/docker-compose.build.drive_abci.yml index 480f49eda2..1d618b7012 100644 --- a/packages/dashmate/docker-compose.build.drive_abci.yml +++ b/packages/dashmate/docker-compose.build.drive_abci.yml @@ -16,6 +16,12 @@ services: SCCACHE_REGION: ${SCCACHE_REGION} SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX} SDK_TEST_DATA: ${SDK_TEST_DATA} + # Forwarded by dashmate from + # `platform.drive.abci.docker.build.buildArgs.CARGO_BUILD_PROFILE` + # (overridable per-invocation via shell env). "release" is required + # for SDK_TEST_DATA shielded seeding at N > a few thousand — see + # docs/shielded-seeder-performance.md. + CARGO_BUILD_PROFILE: ${CARGO_BUILD_PROFILE:-dev} secrets: - GITHUB_TOKEN cache_from: diff --git a/packages/dashmate/docker-compose.build.rs-dapi.yml b/packages/dashmate/docker-compose.build.rs-dapi.yml index 82192f13bc..d36eb2001b 100644 --- a/packages/dashmate/docker-compose.build.rs-dapi.yml +++ b/packages/dashmate/docker-compose.build.rs-dapi.yml @@ -15,6 +15,8 @@ services: SCCACHE_BUCKET: ${SCCACHE_BUCKET} SCCACHE_REGION: ${SCCACHE_REGION} SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX} + # See drive_abci compose for the buildArgs forwarding contract. + CARGO_BUILD_PROFILE: ${CARGO_BUILD_PROFILE:-dev} secrets: - GITHUB_TOKEN cache_from: diff --git a/packages/dashmate/src/commands/config/set.js b/packages/dashmate/src/commands/config/set.js index aae44f9ef7..4f6e27992c 100644 --- a/packages/dashmate/src/commands/config/set.js +++ b/packages/dashmate/src/commands/config/set.js @@ -1,5 +1,7 @@ import { Args } from '@oclif/core'; import ConfigBaseCommand from '../../oclif/command/ConfigBaseCommand.js'; +import Config from '../../config/Config.js'; +import InvalidOptionPathError from '../../config/errors/InvalidOptionPathError.js'; export default class ConfigSetCommand extends ConfigBaseCommand { static description = `Set config option @@ -38,8 +40,16 @@ Sets a configuration option in the default config flags, config, ) { - // check for existence - config.get(optionPath); + // Validate the path against the schema, not against the currently-set + // value. `config.get(...)` would throw `InvalidOptionPathError` for any + // key inside a map-shaped property (e.g. `…buildArgs.SDK_TEST_DATA`) + // because the value doesn't exist yet — that gate is the wrong shape for + // schemas that use `additionalProperties: ` to model open maps. + // `Config.isSchemaPathAllowed` walks the schema and permits descent into + // both typed `properties` and `additionalProperties` value schemas. + if (!Config.isSchemaPathAllowed(optionPath)) { + throw new InvalidOptionPathError(optionPath); + } let value; diff --git a/packages/dashmate/src/config/Config.js b/packages/dashmate/src/config/Config.js index 2644bef698..80dbb95b29 100644 --- a/packages/dashmate/src/config/Config.js +++ b/packages/dashmate/src/config/Config.js @@ -45,6 +45,74 @@ export default class Config { return lodashGet(this.options, path) !== undefined; } + /** + * Check whether a path is reachable per the config JSON schema (regardless + * of whether a value is currently set there). + * + * Use this when checking the legality of a `set` to a path that doesn't yet + * have a value — notably under map-shaped properties whose schema uses + * `additionalProperties: ` (e.g. `…build.buildArgs.`), where + * `config.has(...)` will return `false` even though `config.set(...)` is + * semantically legal. + * + * `configJsonSchema` IS the per-config schema — the top-level + * `properties: { description, group, docker, core, platform, … }` describes + * one config entry. Walks it descending through: + * - `properties[segment]` (typed field), + * - `additionalProperties` (variable-key map, only when the value is a + * schema object — `additionalProperties: false` blocks the descent), + * - `$ref` references into `#/definitions/...`. + * + * @param {string} path - dot-separated option path (e.g. + * `'platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA'`). + * @return {boolean} true when the path is allowed by the schema. + */ + static isSchemaPathAllowed(path) { + if (typeof path !== 'string' || path.length === 0) return false; + + const resolveRef = (node) => { + if (!node || typeof node !== 'object') return node; + if (typeof node.$ref !== 'string') return node; + const ref = node.$ref; + if (!ref.startsWith('#/')) return null; + const segments = ref.slice(2).split('/'); + let resolved = configJsonSchema; + for (const seg of segments) { + if (!resolved || typeof resolved !== 'object') return null; + resolved = resolved[seg]; + } + return resolveRef(resolved); + }; + + let node = resolveRef(configJsonSchema); + if (!node) return false; + + for (const segment of path.split('.')) { + node = resolveRef(node); + if (!node || typeof node !== 'object') return false; + + // Typed property. + if (node.properties && Object.prototype.hasOwnProperty.call(node.properties, segment)) { + node = node.properties[segment]; + continue; + } + + // Map with a schema for extra keys — descend into the value schema. + if ( + node.additionalProperties + && typeof node.additionalProperties === 'object' + ) { + node = node.additionalProperties; + continue; + } + + // No match and no permissive additionalProperties — path not allowed. + return false; + } + + return true; + } + /** * Get config option * diff --git a/packages/dashmate/src/config/configJsonSchema.js b/packages/dashmate/src/config/configJsonSchema.js index ecf969bde6..6f42b4d4b7 100644 --- a/packages/dashmate/src/config/configJsonSchema.js +++ b/packages/dashmate/src/config/configJsonSchema.js @@ -32,6 +32,24 @@ export default { target: { type: ['string', 'null'], }, + // Extra build args forwarded to `docker compose build` for this image. + // Each key becomes an env var with the same name, picked up by the + // `${KEY:-default}` substitutions in `docker-compose.build.*.yml` + // which mirror them into the Dockerfile `ARG`s. Shell env overrides + // any value set here (so `CARGO_BUILD_PROFILE=release yarn start` + // wins per-invocation). + // + // Common keys (when meaningful for the image's target): + // - CARGO_BUILD_PROFILE: "dev" | "release" — Rust profile for + // drive-abci / rs-dapi. Release is required for SDK_TEST_DATA + // shielded seeding at N > a few thousand; see + // `docs/shielded-seeder-performance.md`. + // - SDK_TEST_DATA: "true" — enable the SDK test-data cfg flag in + // the binary at compile time. + buildArgs: { + type: 'object', + additionalProperties: { type: 'string' }, + }, }, required: ['enabled', 'context', 'dockerFile', 'target'], additionalProperties: false, diff --git a/packages/dashmate/src/config/generateEnvsFactory.js b/packages/dashmate/src/config/generateEnvsFactory.js index 4992cd524f..d4aa203570 100644 --- a/packages/dashmate/src/config/generateEnvsFactory.js +++ b/packages/dashmate/src/config/generateEnvsFactory.js @@ -103,6 +103,32 @@ export default function generateEnvsFactory(configFile, homeDir, getConfigProfil ...convertObjectToEnvs(config.getOptions()), }; + // Forward extra docker `build.args` declared per-image in dashmate config + // (`platform.drive.abci.docker.build.buildArgs`, + // `platform.dapi.rsDapi.docker.build.buildArgs`) as env vars under the + // arg name. `docker-compose.build.*.yml` reads them via `${NAME}` + // substitution and forwards them into the Dockerfile `ARG`. + // + // Dashmate config is the single source of truth for build args — do NOT + // fall back to `process.env[key]`. Operators who need a per-invocation + // override should `yarn dashmate config set ...` rather than `FOO=bar + // yarn start`. Keeping this single-source matches the `yarn setup` flow + // that writes SDK_TEST_DATA into the local config automatically. + // + // drive-abci and rs-dapi share the workspace so a shared key like + // CARGO_BUILD_PROFILE typically wants the same value in both; the merge + // order below means drive-abci's value wins on collision. + const getBuildArgs = (configPath) => ( + config.has(configPath) ? (config.get(configPath) || {}) : {} + ); + const mergedBuildArgs = { + ...getBuildArgs('platform.dapi.rsDapi.docker.build.buildArgs'), + ...getBuildArgs('platform.drive.abci.docker.build.buildArgs'), + }; + for (const [key, value] of Object.entries(mergedBuildArgs)) { + envs[key] = value; + } + const configuredAccessLogPath = config.get('platform.dapi.rsDapi.logs.accessLogPath'); const hasConfiguredPath = typeof configuredAccessLogPath === 'string' && configuredAccessLogPath.trim() !== ''; diff --git a/packages/dashmate/test/unit/config/Config.spec.js b/packages/dashmate/test/unit/config/Config.spec.js new file mode 100644 index 0000000000..0d10b89d6f --- /dev/null +++ b/packages/dashmate/test/unit/config/Config.spec.js @@ -0,0 +1,143 @@ +import { expect } from 'chai'; +import Config from '../../../src/config/Config.js'; + +describe('Config', () => { + describe('.isSchemaPathAllowed', () => { + // The bug that triggered this method: `dashmate config set + // platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA true` + // was failing because the old `config.get(path)` pre-check rejected + // any path whose value isn't already stored — including new keys + // inside map-shaped properties whose schema legally accepts them + // via `additionalProperties: `. Each test pins one slice + // of the schema-walk so a regression surfaces fast. + + describe('typed properties', () => { + it('accepts a deeply-nested path that traverses only `properties`', () => { + expect( + Config.isSchemaPathAllowed('platform.drive.abci.docker.build.enabled'), + ).to.be.true(); + }); + + it('accepts a top-level property', () => { + expect(Config.isSchemaPathAllowed('network')).to.be.true(); + }); + + it('rejects a top-level typo', () => { + // `platfom` (typo) is not in top-level `properties` and the schema + // has `additionalProperties: false`, so it must not be allowed. + expect( + Config.isSchemaPathAllowed('platfom.drive.abci.docker.build.enabled'), + ).to.be.false(); + }); + + it('rejects a typo in the middle of the path', () => { + expect( + Config.isSchemaPathAllowed('platform.drive.abc.docker.build.enabled'), + ).to.be.false(); + }); + }); + + describe('map-shaped properties (additionalProperties: )', () => { + it('accepts a new key inside a value-keyed map', () => { + // The original failure. `buildArgs` is defined as + // `{ type: 'object', additionalProperties: { type: 'string' } }` + // so any key with a string value is schema-legal even if not yet set. + expect( + Config.isSchemaPathAllowed( + 'platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA', + ), + ).to.be.true(); + }); + + it('accepts an arbitrary key inside the map (the whole point)', () => { + expect( + Config.isSchemaPathAllowed( + 'platform.drive.abci.docker.build.buildArgs.ANY_KEY_AT_ALL', + ), + ).to.be.true(); + }); + + it('accepts the map property itself', () => { + expect( + Config.isSchemaPathAllowed('platform.drive.abci.docker.build.buildArgs'), + ).to.be.true(); + }); + }); + + describe('$ref traversal', () => { + it('descends through a $ref to `#/definitions/dockerBuild`', () => { + // `platform.drive.abci.docker.build` resolves via $ref to the shared + // `dockerBuild` definition — buildArgs is defined there. + expect( + Config.isSchemaPathAllowed( + 'platform.drive.abci.docker.build.buildArgs.X', + ), + ).to.be.true(); + }); + + it('descends through a $ref for a sibling Rust build (rs-dapi)', () => { + expect( + Config.isSchemaPathAllowed( + 'platform.dapi.rsDapi.docker.build.buildArgs.X', + ), + ).to.be.true(); + }); + }); + + describe('edge cases', () => { + it('rejects an empty path', () => { + expect(Config.isSchemaPathAllowed('')).to.be.false(); + }); + + it('rejects a non-string path', () => { + expect(Config.isSchemaPathAllowed(null)).to.be.false(); + expect(Config.isSchemaPathAllowed(undefined)).to.be.false(); + expect(Config.isSchemaPathAllowed(42)).to.be.false(); + }); + + it('rejects descending past a leaf primitive', () => { + // `network` is a string at top level; you cannot index further. + expect(Config.isSchemaPathAllowed('network.something')).to.be.false(); + }); + }); + }); + + describe('regression: paths the buggy pre-check rejected are now permitted', () => { + // Before the fix, `dashmate config set` did a value-existence pre-check + // (`config.get(path)`) that threw `InvalidOptionPathError` for any path + // whose value wasn't already stored. That blocked legal sets under + // map-shaped properties (`additionalProperties: `). The fix + // replaced the pre-check with `isSchemaPathAllowed`. These tests pin + // each path the original failure surfaced through. + + it('permits the original failing path (`…buildArgs.SDK_TEST_DATA`)', () => { + expect( + Config.isSchemaPathAllowed( + 'platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA', + ), + ).to.be.true(); + }); + + it('permits the same shape for rs-dapi (parallel Rust build)', () => { + expect( + Config.isSchemaPathAllowed( + 'platform.dapi.rsDapi.docker.build.buildArgs.CARGO_BUILD_PROFILE', + ), + ).to.be.true(); + }); + + it('still permits the canonical typed paths that the pre-check used to handle', () => { + // Sanity: paths whose values DO exist after `dashmate setup local` — + // the original pre-check used to gate these via `config.get`. The + // schema walker must accept them too, or the CLI breaks for everyone. + for (const path of [ + 'platform.drive.abci.docker.build.enabled', + 'platform.drive.abci.docker.image', + 'platform.dapi.rsDapi.docker.build.enabled', + 'core.insight.enabled', + ]) { + expect(Config.isSchemaPathAllowed(path), `path: ${path}`).to.be.true(); + } + }); + }); +}); diff --git a/scripts/setup_local_network.sh b/scripts/setup_local_network.sh index 543262b3ab..b506c18afe 100755 --- a/scripts/setup_local_network.sh +++ b/scripts/setup_local_network.sh @@ -16,3 +16,18 @@ yarn run dashmate setup local --verbose \ # enable insight yarn dashmate config set core.insight.enabled true --config local_seed + +# Bake SDK_TEST_DATA=true into the drive-abci docker build for each masternode +# so the genesis shielded-pool seeder + identity/contract fixtures run on every +# local devnet bring-up. Production / release builds explicitly do NOT set this. +# +# Note: the seeder runs at `docker build` time inside the Dockerfile's bake +# stage, so a release-mode binary is highly recommended when SDK_TEST_DATA is +# on (see `docs/shielded-seeder-performance.md`). Set it per-invocation via +# `CARGO_BUILD_PROFILE=release yarn start`, or pin per config with +# `yarn dashmate config set --config=local_N +# platform.drive.abci.docker.build.buildArgs.CARGO_BUILD_PROFILE release`. +for i in $(seq 1 ${MASTERNODES_COUNT}); do + yarn dashmate config set --config=local_${i} \ + platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA "true" +done From 868ae5212214c109f1b65130dd5d5b32fdf37a56 Mon Sep 17 00:00:00 2001 From: Ivan Shumkov Date: Thu, 28 May 2026 21:56:20 +0700 Subject: [PATCH 2/2] fix(dashmate): forward buildArgs via dynamic-compose, drop env passthrough MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the env-var-based forwarding (which hardcoded every supported arg key in `docker-compose.build.*.yml` as `${KEY:-default}` and re-exported them from `generateEnvsFactory`) with a generic dynamic-compose render path. Now `dockerBuild.buildArgs` is open-ended on the schema and any key/value the operator pins via `dashmate config set platform.{drive.abci|dapi.rsDapi}.docker.build.buildArgs.X Y` is emitted as a `services..build.args.X: "Y"` entry in the per-config `dynamic-compose.yml` and merged with the static build compose by docker compose. Changes: - **dynamic-compose.yml.dot**: iterate over `platform.drive.abci.docker.build.buildArgs` and `platform.dapi.rsDapi.docker.build.buildArgs` and emit a `build.args` block per service. The drive_abci service block is only emitted when either driveLogs or driveBuildArgs has entries; rs_dapi gains a conditional build block alongside its existing expose stanza. - **docker-compose.build.drive_abci.yml** / **docker-compose.build.rs-dapi.yml**: drop the hardcoded `CARGO_BUILD_PROFILE: ${CARGO_BUILD_PROFILE:-dev}` and (for drive_abci) `SDK_TEST_DATA: ${SDK_TEST_DATA}` lines. Static compose now only carries args sourced from shell env (sccache config); all config-sourced args flow through dynamic-compose. - **generateEnvsFactory.js**: drop the env-var passthrough block. - **configJsonSchema.js** / **getBaseConfigFactory.js**: trim the comment to one line — the schema doesn't need example keys or doc references. - **getConfigFileMigrationsFactory.js**: 3.1.0 migration backfills `buildArgs: {}` onto both build blocks for pre-existing configs. - **scripts/setup_local_network.sh**: replace the verbose multi-line comment with one sentence. Removed `docs/shielded-seeder-performance.md` — it was added by mistake to this PR (lives with the shielded-pool work, not the dashmate plumbing). Tests: - `dynamicComposeBuildArgs.spec.js`: 3 new tests covering drive_abci emission, rs_dapi emission, and the empty-buildArgs no-op path. - `Config.spec.js`: 15 tests still passing. - 101/101 unit tests pass on `yarn mocha test/unit/**/*.spec.js`. --- docs/shielded-seeder-performance.md | 132 ------------------ .../configs/defaults/getBaseConfigFactory.js | 6 +- .../configs/getConfigFileMigrationsFactory.js | 11 ++ .../docker-compose.build.drive_abci.yml | 11 +- .../dashmate/docker-compose.build.rs-dapi.yml | 4 +- .../dashmate/src/config/configJsonSchema.js | 19 +-- .../src/config/generateEnvsFactory.js | 26 ---- .../templates/dynamic-compose.yml.dot | 20 ++- .../templates/dynamicComposeBuildArgs.spec.js | 65 +++++++++ scripts/setup_local_network.sh | 12 +- 10 files changed, 110 insertions(+), 196 deletions(-) delete mode 100644 docs/shielded-seeder-performance.md create mode 100644 packages/dashmate/test/unit/templates/dynamicComposeBuildArgs.spec.js diff --git a/docs/shielded-seeder-performance.md b/docs/shielded-seeder-performance.md deleted file mode 100644 index 25d7372721..0000000000 --- a/docs/shielded-seeder-performance.md +++ /dev/null @@ -1,132 +0,0 @@ -# Shielded test-data seeder — performance notes - -The `SDK_TEST_DATA` shielded-pool seeder runs at `InitChain` and inserts -`ShieldedSeedConfig::sdk_test_data().total_notes` notes (currently -`500_000`) into the chain's commitment tree before the first block is -proposed. This is CPU-heavy work that scales linearly with `N`. This -document tracks the options for making it fast enough to use on a real -devnet. - -## Cost breakdown for N = 500 000 - -| Step | Cost shape | Dominant in | -|---|---|---| -| Pallas-base rejection sampling | ~1 ChaCha12 draw + 1 field check per filler `cmx` | release: negligible; debug: ~5–10% | -| Owned-note Orchard encryption | ~8 ops total (`Note::from_parts` + `OrchardNoteEncryption`) | always negligible | -| Random ciphertext bytes for filler | `N × 216` ChaCha12 bytes (~108 MB) | always negligible | -| `BulkAppendTree` append | `N` buffer writes + `⌈N/2048⌉` chunk compactions (dense Merkle + MMR + blake3) | release: ~5–10%; debug: ~15% | -| **Sinsemilla frontier append** | **`N × O(log N)` Pallas hashes (~9.5M for N=500k)** | **release: ~85%; debug: ~75%** | -| GroveDB Merk propagation | Once per `apply_drive_operations` call (batched) | always negligible | - -The Sinsemilla appends dominate. Each Pallas hash is the inner loop: -~20k cycles in release, ~200k–2M cycles in debug. - -## Wall-clock measurements - -Apple M-series, single SDK_TEST_DATA dashmate node, fresh `yarn reset`: - -| N | Profile | Wall clock | Source | -|---|---|---|---| -| 500 | dev | ~1 s | observed | -| 16 | dev | <100 ms | unit test | -| 500 000 | dev | **30+ min and not done** | observed 2026-05-23 | -| 500 000 | release | ~1–3 min | extrapolated from release Pallas timing (~10 μs/hash × 9.5M ÷ 1 core ≈ 95 s) | -| 1 000 000 | release | ~3–6 min | linear extrapolation | -| 1 000 000 | dev | ~60–120 min | unusable | - -## Options researched, ranked by impact / cost - -### 1. Release build (`CARGO_BUILD_PROFILE=release`) — **recommended first step** - -- **Speedup:** ~20–50× for the Sinsemilla phase; ~10× overall. -- **Effort:** one config option in dashmate (see below) + one Docker build arg. -- **Tradeoff:** slower image builds (release optimizations take longer to compile), larger debug-info-stripped binary remains ~50–100 MB vs ~700 MB for debug. -- **Status:** plumbed through dashmate as `platform.drive.abci.docker.build.cargoBuildProfile` — set to `release` when you want fast seeding. - -### 2. Parallelize note generation - -- **Speedup:** ~4–8× on top of #1. -- **Effort:** moderate. Two-stage pipeline: - - Worker pool: per note, sample `cmx` + build owned ciphertext or random filler. - - Single consumer thread: feed `(cmx, rho, encrypted_note)` tuples sequentially into `commitment_tree_insert_op` (Sinsemilla append MUST stay sequential — frontier state depends on previous step). -- **Tradeoff:** new concurrency surface, harder determinism guarantees (need deterministic per-worker RNG seeding). -- **Status:** not implemented. Consider only if #1 isn't enough. - -### 3. Option B — precomputed GroveDB snapshot baked into the image - -- **Speedup:** seeding cost → **0** at every `yarn reset` (one-time precomputation cost when the snapshot is generated). -- **Effort:** significant. - - Standalone tool that opens a fresh GroveDB, writes BulkAppendTree chunk blobs (`e{u64}` keys), tail buffer (`b{u32}`), chunk-MMR nodes (`m{u64}`), metadata (`M`, `mmr_size`), Sinsemilla frontier (`__ct_data__`), and the parent Merk's `Element::CommitmentTree(...)` element — all consistently. - - Dockerfile change: bundle the precomputed `db/` directory into the drive-abci image. - - Genesis code: detect the bundled snapshot and skip the seeder. -- **Tradeoff:** big code surface, but the runtime cost completely vanishes. -- **Status:** not implemented. The right answer if scaling to 5M+ notes or doing repeated benchmarks where seed time matters. - -### 4. Skip Pallas rejection sampling for filler - -- **Speedup:** ~5–10% (saves one `from_repr` field check per filler). -- **Effort:** small refactor — replace the rejection loop with `Nullifier::dummy(&mut rng)` which uses `extract_p(&pallas::Point::random(rng))` (always valid by construction). -- **Tradeoff:** depends on `Nullifier::dummy` being `pub` (it's `pub(crate)` in upstream `orchard` unless we enable `unstable-voting-circuits`). -- **Status:** not implemented. Marginal win; skip unless every second matters. - -### 5. Deterministic-bytes filler ciphertext - -- **Speedup:** ~5–10% (saves `rng.fill_bytes(216)` per filler). -- **Effort:** trivial — derive 216 bytes from `blake3(rng_seed || position)` instead. -- **Tradeoff:** changes byte layout slightly (still valid 216-byte payload; wallet treats it as opaque garbage). -- **Status:** not implemented. Marginal. - -### 6. GPU / SIMD-accelerated Sinsemilla - -- **Speedup:** 100×+ in theory for the Pallas-hash inner loop. -- **Effort:** way out of scope. Out-of-tree dependency on a GPU Sinsemilla crate; integration is non-trivial. -- **Tradeoff:** breaks portability (requires NVIDIA hardware) and determinism guarantees become tricky. -- **Status:** not pursued. - -## How build args reach the binary - -Dashmate's `dockerBuild` config exposes a generic `buildArgs` map that flows -through `generateEnvsFactory` → `docker compose build` → Dockerfile `ARG`s. -This is the **only** supported way to set build args — shell env vars are -not wired through. - -`scripts/setup_local_network.sh` (run automatically by `yarn setup` after -`dashmate setup local` creates the per-node configs) sets two args on each -`local_1/2/3` config: - -| arg | value | why | -|---|---|---| -| `SDK_TEST_DATA` | `"true"` | activates the `create_sdk_test_data` cfg flag → genesis seeder runs | -| `CARGO_BUILD_PROFILE` | `"release"` | optimised binary — without it, 500k-note seeding takes 30+ min in debug | - -Both are mandatory together. Debug-profile builds with N=500_000 push -InitChain past tenderdash's timeout window; release-profile finishes in 1–3 -minutes. - -## Switching back to debug for faster compile iteration - -If you're iterating on drive-abci itself and don't care about seeding speed, -swap CARGO_BUILD_PROFILE back to `dev` per-config: - -```bash -for cfg in local_1 local_2 local_3; do - yarn dashmate config set platform.drive.abci.docker.build.buildArgs.CARGO_BUILD_PROFILE dev --config $cfg -done -``` - -Note: any `yarn reset` will rerun `scripts/setup_local_network.sh` and put -`CARGO_BUILD_PROFILE=release` back. Edit that script if you want a different -default permanently. - -The same `buildArgs` field works for any other build arg the Dockerfile -declares. The schema is `Record`. - -## When to revisit Option B - -Re-evaluate the precomputed-snapshot path (#3) when any of these are true: - -- Seeding takes more than ~5 minutes even in release mode (i.e. N ≥ 2M). -- The benchmark workflow does many resets per day and seed time becomes the - per-iteration bottleneck. -- The chain config changes shape such that even release-mode seeding becomes - uneconomical to repeat. diff --git a/packages/dashmate/configs/defaults/getBaseConfigFactory.js b/packages/dashmate/configs/defaults/getBaseConfigFactory.js index 56ca45c224..eb75c00722 100644 --- a/packages/dashmate/configs/defaults/getBaseConfigFactory.js +++ b/packages/dashmate/configs/defaults/getBaseConfigFactory.js @@ -294,10 +294,8 @@ export default function getBaseConfigFactory() { context: path.join(PACKAGE_ROOT_DIR, '..', '..'), dockerFile: path.join(PACKAGE_ROOT_DIR, '..', '..', 'Dockerfile'), target: 'drive-abci', - // Extra docker build args (see `dockerBuild` schema). Common - // override: `CARGO_BUILD_PROFILE: "release"` for SDK_TEST_DATA - // shielded seeding at N > a few thousand - // (`docs/shielded-seeder-performance.md`). + // Extra Docker build args — see the `buildArgs` field on + // `dockerBuild` in the config schema. buildArgs: {}, }, }, diff --git a/packages/dashmate/configs/getConfigFileMigrationsFactory.js b/packages/dashmate/configs/getConfigFileMigrationsFactory.js index adaa072685..df42492a30 100644 --- a/packages/dashmate/configs/getConfigFileMigrationsFactory.js +++ b/packages/dashmate/configs/getConfigFileMigrationsFactory.js @@ -1433,6 +1433,17 @@ export default function getConfigFileMigrationsFactory(homeDir, defaultConfigs) .get('platform.drive.tenderdash.docker.image'); } + // Backfill the new `buildArgs: {}` field on each build block — + // forwarded into `dynamic-compose.yml` as `build.args` entries. + if (options.platform?.drive?.abci?.docker?.build + && typeof options.platform.drive.abci.docker.build.buildArgs === 'undefined') { + options.platform.drive.abci.docker.build.buildArgs = {}; + } + if (options.platform?.dapi?.rsDapi?.docker?.build + && typeof options.platform.dapi.rsDapi.docker.build.buildArgs === 'undefined') { + options.platform.dapi.rsDapi.docker.build.buildArgs = {}; + } + if (options.platform?.drive?.tenderdash?.p2p && typeof options.platform.drive.tenderdash.p2p.allowlistOnly === 'undefined') { options.platform.drive.tenderdash.p2p.allowlistOnly = defaultConfig diff --git a/packages/dashmate/docker-compose.build.drive_abci.yml b/packages/dashmate/docker-compose.build.drive_abci.yml index 1d618b7012..aec9498aca 100644 --- a/packages/dashmate/docker-compose.build.drive_abci.yml +++ b/packages/dashmate/docker-compose.build.drive_abci.yml @@ -15,13 +15,10 @@ services: SCCACHE_BUCKET: ${SCCACHE_BUCKET} SCCACHE_REGION: ${SCCACHE_REGION} SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX} - SDK_TEST_DATA: ${SDK_TEST_DATA} - # Forwarded by dashmate from - # `platform.drive.abci.docker.build.buildArgs.CARGO_BUILD_PROFILE` - # (overridable per-invocation via shell env). "release" is required - # for SDK_TEST_DATA shielded seeding at N > a few thousand — see - # docs/shielded-seeder-performance.md. - CARGO_BUILD_PROFILE: ${CARGO_BUILD_PROFILE:-dev} + # Additional build args (SDK_TEST_DATA, CARGO_BUILD_PROFILE, …) are + # injected per-config by dashmate from + # `platform.drive.abci.docker.build.buildArgs`, rendered into + # `dynamic-compose.yml`, and merged with this file at compose time. secrets: - GITHUB_TOKEN cache_from: diff --git a/packages/dashmate/docker-compose.build.rs-dapi.yml b/packages/dashmate/docker-compose.build.rs-dapi.yml index d36eb2001b..95920dcadf 100644 --- a/packages/dashmate/docker-compose.build.rs-dapi.yml +++ b/packages/dashmate/docker-compose.build.rs-dapi.yml @@ -15,8 +15,8 @@ services: SCCACHE_BUCKET: ${SCCACHE_BUCKET} SCCACHE_REGION: ${SCCACHE_REGION} SCCACHE_S3_KEY_PREFIX: ${SCCACHE_S3_KEY_PREFIX} - # See drive_abci compose for the buildArgs forwarding contract. - CARGO_BUILD_PROFILE: ${CARGO_BUILD_PROFILE:-dev} + # Additional build args are injected per-config by dashmate from + # `platform.dapi.rsDapi.docker.build.buildArgs` via dynamic-compose. secrets: - GITHUB_TOKEN cache_from: diff --git a/packages/dashmate/src/config/configJsonSchema.js b/packages/dashmate/src/config/configJsonSchema.js index 6f42b4d4b7..aa34d870e9 100644 --- a/packages/dashmate/src/config/configJsonSchema.js +++ b/packages/dashmate/src/config/configJsonSchema.js @@ -32,20 +32,11 @@ export default { target: { type: ['string', 'null'], }, - // Extra build args forwarded to `docker compose build` for this image. - // Each key becomes an env var with the same name, picked up by the - // `${KEY:-default}` substitutions in `docker-compose.build.*.yml` - // which mirror them into the Dockerfile `ARG`s. Shell env overrides - // any value set here (so `CARGO_BUILD_PROFILE=release yarn start` - // wins per-invocation). - // - // Common keys (when meaningful for the image's target): - // - CARGO_BUILD_PROFILE: "dev" | "release" — Rust profile for - // drive-abci / rs-dapi. Release is required for SDK_TEST_DATA - // shielded seeding at N > a few thousand; see - // `docs/shielded-seeder-performance.md`. - // - SDK_TEST_DATA: "true" — enable the SDK test-data cfg flag in - // the binary at compile time. + // Extra build args forwarded to `docker compose build` for this + // image. Each key/value pair becomes a `build.args` entry rendered + // into the per-config `dynamic-compose.yml` and picked up by + // compose at build time. Open-ended — image-specific keys live + // here (`CARGO_BUILD_PROFILE`, `SDK_TEST_DATA`, …). buildArgs: { type: 'object', additionalProperties: { type: 'string' }, diff --git a/packages/dashmate/src/config/generateEnvsFactory.js b/packages/dashmate/src/config/generateEnvsFactory.js index d4aa203570..4992cd524f 100644 --- a/packages/dashmate/src/config/generateEnvsFactory.js +++ b/packages/dashmate/src/config/generateEnvsFactory.js @@ -103,32 +103,6 @@ export default function generateEnvsFactory(configFile, homeDir, getConfigProfil ...convertObjectToEnvs(config.getOptions()), }; - // Forward extra docker `build.args` declared per-image in dashmate config - // (`platform.drive.abci.docker.build.buildArgs`, - // `platform.dapi.rsDapi.docker.build.buildArgs`) as env vars under the - // arg name. `docker-compose.build.*.yml` reads them via `${NAME}` - // substitution and forwards them into the Dockerfile `ARG`. - // - // Dashmate config is the single source of truth for build args — do NOT - // fall back to `process.env[key]`. Operators who need a per-invocation - // override should `yarn dashmate config set ...` rather than `FOO=bar - // yarn start`. Keeping this single-source matches the `yarn setup` flow - // that writes SDK_TEST_DATA into the local config automatically. - // - // drive-abci and rs-dapi share the workspace so a shared key like - // CARGO_BUILD_PROFILE typically wants the same value in both; the merge - // order below means drive-abci's value wins on collision. - const getBuildArgs = (configPath) => ( - config.has(configPath) ? (config.get(configPath) || {}) : {} - ); - const mergedBuildArgs = { - ...getBuildArgs('platform.dapi.rsDapi.docker.build.buildArgs'), - ...getBuildArgs('platform.drive.abci.docker.build.buildArgs'), - }; - for (const [key, value] of Object.entries(mergedBuildArgs)) { - envs[key] = value; - } - const configuredAccessLogPath = config.get('platform.dapi.rsDapi.logs.accessLogPath'); const hasConfiguredPath = typeof configuredAccessLogPath === 'string' && configuredAccessLogPath.trim() !== ''; diff --git a/packages/dashmate/templates/dynamic-compose.yml.dot b/packages/dashmate/templates/dynamic-compose.yml.dot index b0e723cfac..6b3e90302e 100644 --- a/packages/dashmate/templates/dynamic-compose.yml.dot +++ b/packages/dashmate/templates/dynamic-compose.yml.dot @@ -13,13 +13,23 @@ services: {{?}} {{ driveLogs = Object.entries(it.platform.drive.abci.logs).filter(([, settings]) => settings.destination !== 'stderr' && settings.destination !== 'stdout'); }} - {{? driveLogs.length > 0 }} + {{ driveBuildArgs = Object.entries((it.platform.drive.abci.docker.build && it.platform.drive.abci.docker.build.buildArgs) || {}); }} + {{? driveLogs.length > 0 || driveBuildArgs.length > 0 }} drive_abci: + {{? driveLogs.length > 0 }} volumes: {{~ driveLogs :logger }} {{ [name, settings] = logger; }} - {{=settings.destination}}:/var/log/dash/drive/{{=name}}/{{=settings.destination.split('/').reverse()[0]}} {{~}} + {{?}} + {{? driveBuildArgs.length > 0 }} + build: + args: + {{~ driveBuildArgs :pair }} + {{=pair[0]}}: "{{=pair[1]}}" + {{~}} + {{?}} {{?}} {{? it.platform.drive.tenderdash.log.path !== null }} @@ -29,6 +39,7 @@ services: {{?}} {{? it.platform.dapi && it.platform.dapi.rsDapi }} + {{ rsDapiBuildArgs = Object.entries((it.platform.dapi.rsDapi.docker.build && it.platform.dapi.rsDapi.docker.build.buildArgs) || {}); }} rs_dapi: expose: - 3009 @@ -40,6 +51,13 @@ services: ports: - {{=it.platform.dapi.rsDapi.metrics.host}}:{{=it.platform.dapi.rsDapi.metrics.port}}:{{=it.platform.dapi.rsDapi.metrics.port}} {{?}} + {{? rsDapiBuildArgs.length > 0 }} + build: + args: + {{~ rsDapiBuildArgs :pair }} + {{=pair[0]}}: "{{=pair[1]}}" + {{~}} + {{?}} {{?}} {{ gatewayLogs = it.platform.gateway.log.accessLogs.filter((l) => l.type === 'file'); }} diff --git a/packages/dashmate/test/unit/templates/dynamicComposeBuildArgs.spec.js b/packages/dashmate/test/unit/templates/dynamicComposeBuildArgs.spec.js new file mode 100644 index 0000000000..5434cfdb16 --- /dev/null +++ b/packages/dashmate/test/unit/templates/dynamicComposeBuildArgs.spec.js @@ -0,0 +1,65 @@ +import { expect } from 'chai'; +import fs from 'fs'; +import path from 'path'; +import { fileURLToPath } from 'url'; +import dot from 'dot'; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); +const TEMPLATE_PATH = path.resolve( + __dirname, + '../../../templates/dynamic-compose.yml.dot', +); + +function render(buildArgsByService) { + dot.templateSettings.strip = false; + const tpl = fs.readFileSync(TEMPLATE_PATH, 'utf8'); + const fn = dot.template(tpl); + return fn({ + core: { docker: { commandArgs: [] }, log: { filePath: null } }, + platform: { + drive: { + abci: { + logs: {}, + docker: { build: { buildArgs: buildArgsByService.drive || {} } }, + }, + tenderdash: { log: { path: null } }, + }, + dapi: { + rsDapi: { + metrics: { enabled: false }, + docker: { build: { buildArgs: buildArgsByService.rsDapi || {} } }, + }, + }, + gateway: { log: { accessLogs: [] } }, + }, + }); +} + +describe('dynamic-compose buildArgs rendering', () => { + it('emits drive_abci build.args from drive.abci.docker.build.buildArgs', () => { + const out = render({ + drive: { SDK_TEST_DATA: 'true', CARGO_BUILD_PROFILE: 'release' }, + }); + expect(out).to.match(/drive_abci:[\s\S]*build:[\s\S]*args:[\s\S]*SDK_TEST_DATA:\s*"true"/); + expect(out).to.match(/drive_abci:[\s\S]*CARGO_BUILD_PROFILE:\s*"release"/); + }); + + it('emits rs_dapi build.args from dapi.rsDapi.docker.build.buildArgs', () => { + const out = render({ + rsDapi: { CARGO_BUILD_PROFILE: 'release' }, + }); + expect(out).to.match(/rs_dapi:[\s\S]*build:[\s\S]*args:[\s\S]*CARGO_BUILD_PROFILE:\s*"release"/); + }); + + it('omits the build block entirely when buildArgs is empty', () => { + const out = render({}); + // The drive_abci service block is only emitted by this template when + // either driveLogs or driveBuildArgs has entries — empty buildArgs + + // empty logs ⇒ no drive_abci section at all. + expect(out).to.not.match(/drive_abci:\s*\n\s*build:/); + // rs_dapi is emitted regardless (it carries the expose stanza); but it + // must NOT carry a build section when buildArgs is empty. + expect(out).to.not.match(/rs_dapi:[\s\S]*build:\s*\n\s*args:/); + }); +}); diff --git a/scripts/setup_local_network.sh b/scripts/setup_local_network.sh index b506c18afe..6676f9d889 100755 --- a/scripts/setup_local_network.sh +++ b/scripts/setup_local_network.sh @@ -17,16 +17,8 @@ yarn run dashmate setup local --verbose \ # enable insight yarn dashmate config set core.insight.enabled true --config local_seed -# Bake SDK_TEST_DATA=true into the drive-abci docker build for each masternode -# so the genesis shielded-pool seeder + identity/contract fixtures run on every -# local devnet bring-up. Production / release builds explicitly do NOT set this. -# -# Note: the seeder runs at `docker build` time inside the Dockerfile's bake -# stage, so a release-mode binary is highly recommended when SDK_TEST_DATA is -# on (see `docs/shielded-seeder-performance.md`). Set it per-invocation via -# `CARGO_BUILD_PROFILE=release yarn start`, or pin per config with -# `yarn dashmate config set --config=local_N -# platform.drive.abci.docker.build.buildArgs.CARGO_BUILD_PROFILE release`. +# Enable SDK_TEST_DATA in drive-abci builds for each local masternode so the +# genesis shielded-pool seeder + identity/contract fixtures run at bring-up. for i in $(seq 1 ${MASTERNODES_COUNT}); do yarn dashmate config set --config=local_${i} \ platform.drive.abci.docker.build.buildArgs.SDK_TEST_DATA "true"