diff --git a/kb/wildlife-reid-mobile/.gitignore b/kb/wildlife-reid-mobile/.gitignore new file mode 100644 index 00000000..90faa3da --- /dev/null +++ b/kb/wildlife-reid-mobile/.gitignore @@ -0,0 +1,2 @@ +# Binary outputs are too large for git — transferred separately per HANDOFF doc +output/ diff --git a/kb/wildlife-reid-mobile/HANDOFF-2026-04-25.md b/kb/wildlife-reid-mobile/HANDOFF-2026-04-25.md new file mode 100644 index 00000000..e38ff000 --- /dev/null +++ b/kb/wildlife-reid-mobile/HANDOFF-2026-04-25.md @@ -0,0 +1,150 @@ +# Session Handoff — 2026-04-25 + +**From:** Claude (WSL session, `/mnt/c/off-grid-mobile`) +**To:** Claude (Mac session, same repo cloned locally) +**Purpose:** Continue work on the wildlife re-ID pivot of off-grid-mobile, picking up at Step (A) of the current plan — sideload synthetic pack onto iPhone 13 + Pixel 8 dev builds and validate the pipeline end-to-end. + +--- + +## Read this first + +If you are the new Mac-Claude session, your reading order: + +1. **This doc** (you're here) — current state, decisions, what to do next, gotchas. +2. **`kb/wildlife-reid-mobile/INDEX.md`** — durable knowledge base index. Source registry, wiki articles, outputs. +3. **`kb/wildlife-reid-mobile/outputs/plans/miewid-v41-integration-plan.md`** — the 5-stage integration plan. Stage 0 done, Stage 1.2 done, you're working on the bridge to Stage 2. +4. **`kb/wildlife-reid-mobile/outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md`** — pre-emptively identified issues you'll hit when you get to mobile-side rewiring (Stage 2). +5. **`CLAUDE.md`** at repo root — project-wide instructions (pre-commit, push workflow, CI loop). Was just realigned via PR #12 to reflect actual setup (Gemini-only, no Codecov/SonarCloud). + +After reading, **rebuild your memory** by writing the relevant facts back into your local `~/.claude/projects/.../memory/` per your auto-memory instructions. The WSL session's memory is at `/home/jason/.claude/projects/-mnt-c-off-grid-mobile/memory/` and won't follow. + +## Project mission (one sentence) + +Repurpose the React Native off-grid LLM app (`WildMeOrg/off-grid-mobile`) into an offline wildlife individual re-identification platform — capture photo → species detector (YOLO ONNX) → MiewID v4.1 embedding → cosine match against a Wildbook-exported pack of known individuals → human review → sync to Wildbook when online. + +## Current state — who, what, where + +| Topic | State | +|---|---| +| **User** | Jason (CXL / Wild Me Lab employee). MiewID redistribution sign-off granted 2026-04-23. | +| **Repo** | `WildMeOrg/off-grid-mobile`, default branch `main`. Long-lived integration branch: **`wildlife-reid`**. Open PRs target `wildlife-reid`, NOT `main`. | +| **Reference devices** | iPhone 13 (A15, 6-core NPU); Google Pixel 8 (Tensor G3, clean AOSP). | +| **Dogfood target** | Wild horses, State of Washington. | +| **CI** | GitHub Actions only (`typecheck`, `test`, `lint`, `android-build`). Triggered on push + PR against `main` and `wildlife-reid`. Workflows had to be manually enabled on the fork (one-time UI step Jason did 2026-04-24). | +| **Code review bots** | **Gemini Code Assist** installed on the fork — auto-reviews on PR open, retrigger with `/gemini review`. Codecov / SonarCloud are NOT installed (see realigned CLAUDE.md). | +| **Codex CLI** | `/usr/bin/codex`, model upgraded to **5.5** as of 2026-04-24. Use for scoped second-opinion reviews; not part of routine PR loop. File substantive findings to `kb/wildlife-reid-mobile/outputs/reports/`. | + +## What's done (chronological) + +**2026-04-22** +- Built the KB scaffold under `kb/wildlife-reid-mobile/`. Ingested upstream docs + ran codex review + Explore-agent codebase map. +- Identified 2 latent preprocessing bugs (native scale divide-not-multiply + MiewID TS wrapper passing scale=1.0). + +**2026-04-23** +- Resolved licensing for MiewID v4.1 redistribution (CXL employee sign-off). +- Confirmed local v4.1 checkpoint at `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` (n_classes=20,191). +- **PR #11** merged — fixed native scale math (Kotlin + Swift), TS wrapper, added cross-platform golden parity fixture (1×1 pure-red pixel through ImageNet norm + scale=1/255 → R≈2.249, G≈-2.036, B≈-1.804). Also patched `.husky/pre-commit` to skip iOS tests when xcodebuild absent (WSL friendliness). Created `wildlife-reid` integration branch, pushed CI workflow to trigger on it. +- Codex 5.5 Stage-2 risk review surfaced 2 blockers + 3 high + 3 medium pre-existing issues — filed at `kb/.../outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md`. Stage 2 plan rewritten in light of these. + +**2026-04-24** +- Horse-face detector trained via yolo-trainer skill: `/mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx` (10 MB YOLO11n, mAP50 0.9999). Pack-ready, no code changes needed. +- **MiewID v4.1 ONNX export done** via `tools/export_miewid_v41_onnx.py`. FP16 single-file 103.9 MB at `tools/output/miewid_v4_1_fp16.onnx`. Cosine vs PyTorch ref: 0.999994 (well above 0.999 threshold). Embedding dim 2152 confirmed. +- Gemini installed on fork; **PR #12** merged (CLAUDE.md realigned to reflect Gemini-only setup). + +**2026-04-25** +- **Pack bundler** at `tools/build_pack.py` with `synthetic` + `from-coco` (stub) modes. +- **Synthetic horse pack** at `tools/output/horse-synthetic-2026-04-25.zip` (9.2 MB, 5 individuals × 2 embeddings, real WHORSE detector bundled, validates against pack format spec, within-individual cos 0.995, cross 0.02). +- Session handoff to Mac (this doc). + +## Where we are in the plan (B → A → C) + +| Step | Status | Owner | +|---|---|---| +| (B) Pack-bundling tooling | ✅ done | WSL session | +| **(A) Vertical slice on iPhone 13 + Pixel 8** | ⏳ **NEXT** | **You (Mac session)** | +| (C) Stage 2 foundation trio | queued | Mac session afterwards | + +## Step (A) brief — what to do next + +**Goal:** Sideload `horse-synthetic-2026-04-25.zip` + `miewid_v4_1_fp16.onnx` onto an iPhone 13 dev build, validate the entire stack runs end-to-end, repeat on Pixel 8, confirm cross-platform embedding cosine ≥ 0.999. + +**Suggested PR (`feat/dev-pack-bundling`):** + +1. **Bundle assets in dev builds.** Drop `horse-synthetic-2026-04-25.zip` and `miewid_v4_1_fp16.onnx` into a new `assets/dev/` directory. Add to RN bundling config so they ship with debug builds only (release builds should NOT include these). +2. **Debug startup hook.** In `App.tsx` (or a debug-only helper), on first launch: + - Unzip the bundled pack into `DocumentDirectoryPath/embedding_packs/horse-synthetic-2026-04-25/` + - Copy MiewID ONNX into `DocumentDirectoryPath/models/miewid_v4_1.onnx` + - Call `wildlifeStore.setMiewidModelPath(...)` and `wildlifeStore.addPack(...)` so the existing pipeline picks them up + - Guard with `__DEV__` so this never runs in release +3. **Tiny debug screen** — "Run pipeline test" button. Loads a bundled test photo (or a colored square if no real photo handy), runs `wildlifePipeline.processPhoto`, displays: + - Detection bbox + species + confidence + - First few embedding values (sanity check that they're not garbage) + - Top-5 match candidates + - Inference latency per stage +4. **Run on iPhone 13** via Xcode. Note any iOS-specific issues. +5. **Run on Pixel 8** via Android Studio / `npx react-native run-android`. +6. **Cross-platform parity check** — load the same fixed test image on both, embed, compare cosine. Should be ≥ 0.999. +7. **PR against `wildlife-reid`.** Pre-commit gates will run JS + Swift + Kotlin checks. + +## Binary artifacts you'll need (NOT in git) + +These are too big or out-of-tree to commit. Jason will transfer them to the Mac via OneDrive / SCP / USB / whatever: + +| File | Size | Source path on WSL | Place on Mac at | +|---|---|---|---| +| `miewid_v4_1_fp16.onnx` | 103.9 MB | `kb/wildlife-reid-mobile/tools/output/miewid_v4_1_fp16.onnx` | `kb/wildlife-reid-mobile/tools/output/miewid_v4_1_fp16.onnx` | +| `horse-synthetic-2026-04-25.zip` | 9.2 MB | `kb/wildlife-reid-mobile/tools/output/horse-synthetic-2026-04-25.zip` | `kb/wildlife-reid-mobile/tools/output/horse-synthetic-2026-04-25.zip` | +| `horse_wild_face_yolo11n_416.onnx` | 10 MB | `/mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx` | wherever convenient — used to rebuild the synthetic pack if needed | +| `detector_config.json` | 567 B | `/mnt/c/claude-skills/output/whorse-face-detector/detector_config.json` | same as above | + +If you want to **regenerate the synthetic pack** on the Mac instead of transferring it: clone `wbia-plugin-miew-id` (not strictly needed for `build_pack.py`, only for re-exporting MiewID), `pip install pillow numpy onnxruntime`, then `python3 build_pack.py synthetic ...` per the documented command in `kb/.../raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md`. Detector + MiewID FP16 still need transfer either way. + +## Memory porting + +The WSL session's memory at `/home/jason/.claude/projects/-mnt-c-off-grid-mobile/memory/` has three durable entries that should be recreated on the Mac: + +1. `user_role.md` — Jason at CXL / Wild Me Lab; on-device wildlife re-ID mission. +2. `project_wildlife_reid.md` — KB at `kb/wildlife-reid-mobile/`; preprocessing fix landed; v4.1 export done; horse detector ready; all the dated facts. +3. `reference_skills_and_tools.md` — `/mnt/c/claude-skills/` skill library (assuming same path on Mac); codex 5.5 CLI; miewid-trainer MCP. + +Mac-Claude: write equivalents into your local `~/.claude/projects/.../memory/` based on this handoff doc + the KB INDEX. Don't try to copy verbatim from WSL — paths differ (the user's home dir, `/mnt/c/` mount point, etc.). + +## Known gotchas + +1. **CRLF/LF drift.** Repo files often show as "modified" in `git status` due to line-ending differences between the upstream repo and Windows clones. **Stage files individually by name; never `git add -A`.** When edits show suspiciously large diffs, normalize the file with `python3 -c "open(p,'wb').write(open(p,'rb').read().replace(b'\\r\\n', b'\\n'))"` before staging. Likely less of an issue on Mac than on WSL. +2. **Pre-commit hook + iOS tests.** The hook now skips iOS tests when `xcodebuild` is absent (WSL friendliness, PR #11). On Mac with Xcode installed, iOS tests **will** run on every Swift-touching commit — slower but correct. +3. **GitHub fork workflows.** Actions are enabled and CI fires on `main` + `wildlife-reid`. PRs targeting other branches won't fire CI unless added to `.github/workflows/ci.yml` `branches:` filters. +4. **Gemini false positives.** It pattern-matched the `kb/wildlife-reid-mobile/` directory name as "fork name mismatch" in PR #12. The directory is named for the **product domain**, not the fork. Reject similar suggestions on the comment thread with a brief explanation; don't blindly apply. +5. **Codex 5.5 use.** It's worth running for any non-trivial PR (it caught a real bug in the Stage-0 PR that I missed). Scope the prompt — give it specific files + specific questions. +6. **iOS bundle size.** MiewID FP16 is 103.9 MB. Bundling in debug builds is fine, but DO NOT bundle in release — the design ([[model-acquisition]]) requires download-on-demand for production. + +## Open decisions you may need to make + +- **Hosting for shipped MiewID ONNX** (Stage 1.5) — HF / CXL CDN / GitHub release asset. No decision yet. +- **iOS bundle ID + signing identity** — Off Grid's existing identity vs a new one for the wildlife fork. Not yet discussed. +- **Real Washington-horses COCO acquisition** — does Jason have horses.wildbook.org credentials for an export? Pending. +- **Renaming the GitHub repo** to reflect the wildlife pivot (e.g. `wildlife-reid-mobile`)? Optional, not blocking. + +## Tasks state at handoff + +``` +#15 [pending] Step (A) Vertical slice — sideload synthetic pack on iPhone 13 + Pixel 8 +``` + +All earlier tasks (preprocessing fixes, MiewID export, pack bundler, etc.) are completed and recorded in the KB. + +## After Step (A) + +When the vertical slice works on both devices and you've measured real latencies + cross-platform parity, the next phase is **Step (C) — Stage 2 foundation trio:** + +- 2.1 `MiewIDModelRecord` + state machine (replaces the bare `miewidModelPath: string | null`) +- 2.2 Pack validator (manifest schema + checksums + offset bounds) +- 2.3 JS download service (staging + SHA-256 + atomic move) + +These fix the codex-5.5-flagged blockers and replace the debug startup hook from Step (A) with proper production code. See `outputs/plans/miewid-v41-integration-plan.md` Stage 2 section for the full sub-stage breakdown. + +## Questions you'll likely ask Jason early + +- Where do you want the binary artifacts dropped on the Mac? +- Are iPhone 13 + Pixel 8 already provisioned for dev builds, or do we need to set up signing / `adb` first? +- Do you have a horse photo (or any real wildlife photo) we can use as a fixed test input for cross-platform parity? diff --git a/kb/wildlife-reid-mobile/INDEX.md b/kb/wildlife-reid-mobile/INDEX.md new file mode 100644 index 00000000..3d75b99c --- /dev/null +++ b/kb/wildlife-reid-mobile/INDEX.md @@ -0,0 +1,75 @@ +# Wildlife Re-ID Mobile Knowledge Base + +**Last compiled:** 2026-04-23 +**Articles:** 9 wiki + 1 output +**Sources:** 7 raw documents (+ 4 live docs via symlink) +**Coverage:** Good on architecture, pipeline, bugs, plan. V4.1 checkpoint now ingested. Gaps: device performance data, license resolution, real pack samples, ONNX hosting decision. + +## Source Registry + +| File | Type | Ingested | Summary | +|---|---|---|---| +| [raw/docs-live → docs/WILDLIFE_REID_FEASIBILITY.md](raw/docs-live/WILDLIFE_REID_FEASIBILITY.md) | design | yes | Feasibility assessment; Off Grid + ONNX Runtime recommendation; MiewID profile + storage budget | +| [raw/docs-live → docs/EMBEDDING_PACK_FORMAT.md](raw/docs-live/EMBEDDING_PACK_FORMAT.md) | spec | yes | Pack format v1.0 — manifest, embeddings.bin layout, detector config, checksums | +| [raw/docs-live → docs/plans/2026-02-25-wildlife-reid-design.md](raw/docs-live/plans/2026-02-25-wildlife-reid-design.md) | design | yes | Fork strategy (what stripped vs kept), data model, pipeline stages | +| [raw/docs-live → docs/plans/2026-02-25-wildlife-reid-implementation.md](raw/docs-live/plans/2026-02-25-wildlife-reid-implementation.md) | plan | partial | Task-by-task TDD implementation plan, executed ~through Phase 5 | +| [raw/docs-live → docs/ARCHITECTURE.md](raw/docs-live/ARCHITECTURE.md) | arch | skim | General Off Grid architecture patterns reused by the wildlife fork | +| [raw/codex/2026-04-22-codebase-review.md](raw/codex/2026-04-22-codebase-review.md) | review | yes | Codex second opinion: strengths, risks, preprocessing bugs, acquisition recommendation | +| [raw/exploration/2026-04-22-codebase-map.md](raw/exploration/2026-04-22-codebase-map.md) | map | yes | Thorough Explore-agent mapping of services, screens, stores, native modules, tests | +| [raw/models/2026-04-23-miewid-v41-checkpoint.md](raw/models/2026-04-23-miewid-v41-checkpoint.md) | model | yes | Local MiewID v4.1 checkpoint: file path, MD5, training config, n_classes=20,191 | +| [raw/models/2026-04-24-whorse-face-detector.md](raw/models/2026-04-24-whorse-face-detector.md) | model | yes | YOLO11n horse-face detector (10 MB, mAP50 0.9999) at /mnt/c/claude-skills/output/whorse-face-detector/ — pack-ready, no code changes needed | +| [raw/models/2026-04-24-miewid-v41-onnx-export.md](raw/models/2026-04-24-miewid-v41-onnx-export.md) | model | yes | MiewID v4.1 ONNX exports (FP32 205.7 MB, FP16 103.9 MB) at tools/output/. Parity vs PyTorch ref: FP32 cos_min 1.0000, FP16 cos_min 0.99999, embedding_dim=2152 confirmed. | +| [raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md](raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md) | tool | yes | `build_pack.py` two-mode pack assembler + `horse-synthetic-2026-04-25.zip` (9.2 MB, 5 individuals × 2 embeddings, validates against pack format spec). Bridge artifact between Stage 1 and Stage 2. | + +## Wiki Articles + +| Article | Category | Sources | Last Updated | +|---|---|---|---| +| [MiewID v4](wiki/entities/miewid-v4.md) | entity | 3 | 2026-04-22 | +| [Off Grid Mobile (Wildlife Re-ID Fork)](wiki/entities/off-grid-mobile-app.md) | entity | 3 | 2026-04-22 | +| [Embedding Pack](wiki/concepts/embedding-pack.md) | concept | 2 | 2026-04-22 | +| [On-Device Re-ID Pipeline](wiki/concepts/on-device-reid-pipeline.md) | concept | 2 | 2026-04-22 | +| [Image Preprocessing](wiki/methods/image-preprocessing.md) | method | 4 files + spec | 2026-04-22 | +| [YOLO Postprocessing](wiki/methods/yolo-postprocessing.md) | method | 2 | 2026-04-22 | +| [Model Acquisition](wiki/methods/model-acquisition.md) | method | 3 | 2026-04-22 | +| [Critical Bugs](wiki/bugs/critical-bugs.md) | bugs | source reads | 2026-04-22 | +| [State of Implementation](wiki/meta/state-of-implementation.md) | meta | 3 | 2026-04-22 | + +## Outputs + +| Output | Format | Date | +|---|---|---| +| [MiewID v4.1 On-Device Integration Plan](outputs/plans/miewid-v41-integration-plan.md) | plan | 2026-04-22 | + +## Coverage Gaps + +- [ ] Device-class latency benchmarks (iPhone Neural Engine, Android NNAPI) — populate after Stage 3.2 of plan +- [x] ~~MiewID weights **license** resolution~~ — resolved 2026-04-23 via CXL employee sign-off (Jason) +- [x] ~~Confirmed **v4.1** availability~~ — local checkpoint found at `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` (2026-04-23) +- [ ] Hosting decision for exported ONNX (HF / CXL CDN / GitHub release) +- [~] Real sample pack — **target: wild horses, State of Washington** (2026-04-23). + - [x] Detector model — YOLO11n trained on WBIA `horse_wild+face`, ready as ONNX (2026-04-24). See [[2026-04-24-whorse-face-detector]]. + - [x] MiewID v4.1 ONNX export — FP16 103.9 MB, parity ≥ 0.99999 (2026-04-24). See [[2026-04-24-miewid-v41-onnx-export]]. + - [ ] Washington-horses COCO export from Wildbook + matchability assessment + - [ ] Bundle `embeddings.bin` + `index.json` + reference photos + manifest into pack zip +- [ ] Wildbook sync API contract — endpoints, auth, Encounter upload shape +- [ ] Core ML / NNAPI EP compatibility testing for MiewID +- [ ] Gallery dilution curves per species at expected production scale +- [ ] Species-specific cosine thresholds (calibration data) + +## Recent Queries + +| Date | Query | Output | +|---|---|---| +| 2026-04-22 | "What's built, what's stubbed, and how to ship MiewID v4.1 on-device?" | [integration plan](outputs/plans/miewid-v41-integration-plan.md) + [state of implementation](wiki/meta/state-of-implementation.md) + [critical bugs](wiki/bugs/critical-bugs.md) | +| 2026-04-23 | "Fix the preprocessing scale bugs (Stage 0 of integration plan)" | PR [#11](https://github.com/WildMeOrg/off-grid-mobile/pull/11) — `fix/preprocessing-scale-math` → `wildlife-reid`. Native multiply-not-divide on both platforms + MiewID TS wrapper `1/255` + cross-platform golden parity fixture. 410/410 Jest, 11/11 Kotlin tests pass. | + +## Key Conclusions (TL;DR) + +1. **Pipeline is ~70-80% wired end-to-end.** Architecture is sound; most remaining work is glue, not foundation. +2. **MiewID v4.1 is not actually loaded anywhere.** `miewidModelPath` setter exists but no call site populates it — highest-leverage gap. +3. **Two real preprocessing bugs** will produce garbage embeddings even after MiewID wiring: + - Native `scale` divided instead of multiplied (both Kotlin & Swift) + - TS MiewID wrapper passes `scale = 1.0` (feeds 0-255 into ImageNet normalization) +4. **Recommendation:** ship MiewID as a separate FP16 download (~80-120 MB), not bundled. Keep detectors in packs. Avoid INT8 until rank-order parity is proven on real wildlife crops. +5. **Next steps:** fix bugs + golden parity test → MiewID ONNX export + HF publish → in-app acquisition service → E2E verification. See [integration plan](outputs/plans/miewid-v41-integration-plan.md). diff --git a/kb/wildlife-reid-mobile/outputs/plans/miewid-v41-integration-plan.md b/kb/wildlife-reid-mobile/outputs/plans/miewid-v41-integration-plan.md new file mode 100644 index 00000000..f02bb0cc --- /dev/null +++ b/kb/wildlife-reid-mobile/outputs/plans/miewid-v41-integration-plan.md @@ -0,0 +1,109 @@ +# MiewID v4.1 On-Device Integration Plan + +**Category:** outputs/plans +**Date:** 2026-04-22 +**Sources:** [[critical-bugs]], [[model-acquisition]], [[state-of-implementation]], [[off-grid-mobile-app]] +**Related:** [[miewid-v4]], [[embedding-pack]], [[image-preprocessing]] + +## Goal + +Make MiewID v4.1 individual re-ID work **end-to-end on device** in the Off Grid Mobile wildlife fork: user photographs an animal → detector finds it → MiewID v4.1 produces a valid 2152-dim embedding → cosine match against pack + local individuals surfaces top-5 candidates for review. + +## Sequencing + +### Stage 0 — Foundation Fixes (blockers) + +Must land before any MiewID work is meaningful, or we'll be tuning on garbage. + +- [x] **0.1** Fix native `imageToTensor` scale semantics — multiply, not divide. Kotlin + Swift. Update unit tests. **Done 2026-04-23 (PR #11).** +- [x] **0.2** Fix MiewID TS preprocessor to pass `scale = 1.0/255.0`. Update test. **Done 2026-04-23 (PR #11).** +- [x] **0.3** Cross-platform golden parity fixture (1×1 pure-red pixel through ImageNet norm + scale=1/255 → R≈2.249, G≈-2.036, B≈-1.804). Encoded identically in Kotlin and Swift tests. **Done 2026-04-23 (PR #11).** Follow-on: real Python ↔ device parity on a larger image once MiewID ONNX exists. +- [ ] **0.4** (Defensive) Branch YOLO postprocessing on `architecture` for `5 + C` v5/v7 support. Deferred — only matters if a pack ever ships a YOLOv5/v7 detector. + +### Stage 1 — MiewID v4.1 ONNX Export + +Done outside the app (in the miewid-trainer workspace or a throwaway repo). + +**Starting artifact (confirmed 2026-04-23):** +- PyTorch checkpoint: `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` + (206 MB FP32, md5 `71c0bdd9bf8cbdb1b9e967a2e96949bb`, n_classes=20,191) +- Config: `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.yaml` +- Architecture: EfficientNetV2-RW-M + GeM + sub-center ArcFace dynamic (m=0.328, s=49.33, k=2), `use_fc=false` +- See [[2026-04-23-miewid-v41-checkpoint]] for full details. + +Tasks: + +- ~~**1.1** Confirm with CXL: license for mobile redistribution.~~ **Done 2026-04-23**: Jason signed off as a CXL employee. We redistribute the ONNX derivative, not the raw `.bin`. +- [x] **1.2** ~~Write export script~~ **Done 2026-04-24.** Script at `kb/wildlife-reid-mobile/tools/export_miewid_v41_onnx.py`. Outputs in `tools/output/`: FP32 205.7 MB cos_min 1.0000; FP16 103.9 MB cos_min 0.999994. Single self-contained files (no external-data shards). Manifest with checksums + provenance. See [[2026-04-24-miewid-v41-onnx-export]]. *(Original task description retained below for reference.)* + - Instantiate the model class from `wbia-plugin-miew-id` using `miew_id.msv4_1_main.yaml` `model_params` + - `load_state_dict(…, strict=False)` to drop the ArcFace classification head + - Wrap in an `EmbeddingOnly` module that returns only the 2152-dim normalized feature vector + - `torch.onnx.export(opset_version=17, dynamic_axes={"input": {0:"batch"}, "embedding": {0:"batch"}})` + - Output: `miewid_v4_1_fp32.onnx` (~200 MB) +- [~] **1.3** Parity test: 100 random tensors → cos_min 1.0000 FP32 / 0.99999 FP16 (done as part of 1.2). **Still pending:** 50 diverse real wildlife crops (grab from an existing miewid-trainer test set). For each, run: + - PyTorch forward on the `.bin` checkpoint → reference embedding + - `onnxruntime.InferenceSession` on the ONNX → test embedding + - Assert cosine(ref, test) ≥ 0.9999 for all 50 +- [x] **1.4** ~~FP16 conversion~~ **Done 2026-04-24** via `onnxconverter_common.float16.convert_float_to_float16`. cos_min 0.999994 vs PyTorch ref (well above the 0.999 threshold). Per-sample l2 drift ≈ 0.23 on embeddings of magnitude O(2-3). **Still pending:** rank-order parity on a 100-pair cross-gallery retrieval test using real crops (defer to 1.3-real-crop). +- **1.5** Publish: upload `miewid_v4_1_fp16.onnx` (~100 MB) to either (a) a `conservationxlabs/miewid-msv4-1-onnx` HF repo, (b) a CXL-hosted CDN URL, or (c) a GitHub release asset on the off-grid-mobile repo. Record SHA-256 for app-side verification. + +### Stage 2 — In-App Acquisition (expanded based on [codex 5.5 Stage-2 risk review, 2026-04-24](../reports/2026-04-24-codex-5-5-stage-2-risk-review.md)) + +The original "add a download service + wire the path" framing was too thin. Before any download work is meaningful we need a model state machine, a versioned identity record, and a pack validator. These also unlock fixes for lifecycle races and partial-success observations that are currently latent bugs. + +- **2.1** **Model record + state machine.** Define `MiewIDModelRecord { path, name, version, sha256, sizeBytes, status }` with states `missing | downloading | ready | corrupt | incompatible`. Replace `wildlifeStore.miewidModelPath: string | null`. Reconcile on app startup — verify file existence + hash + optionally resume in-flight downloads. +- **2.2** **Pack validator** (blocker). Manifest schema, `formatVersion` support gate, required-files existence, SHA-256 vs `manifest.checksums`, `embeddings.bin` size === `embeddingCount * embeddingDim * 4`, per-individual offset/count bounds. Quarantine invalid packs rather than admit them to the store. +- **2.3** **JS download/cache service** (`src/services/modelDownloadService` or similar). Wraps existing native `DownloadManagerModule` with staging-path download, HTTP status check, expected length check, SHA-256 verify, atomic move into cache, retry/backoff, cancellation. Used by both MiewID and pack downloads. +- **2.4** **Pack grouping + compatibility.** Group by `{species, featureClass, detectorModelFile, embeddingModelVersion}` in capture flow. Run one detector per group. Reject or isolate incompatible embedding spaces. Currently two horse packs produce double-detections + mixed DBs. +- **2.5** **ONNX lifecycle hardening** in `onnxInferenceService`. Per-path load-promise dedup (so concurrent loads don't leak sessions). Release waits for active inference via refcount or read/write lock. Fixes: duplicate sessions, use-after-free during unload. +- **2.6** **Partial-success observations.** `wildlifePipeline.processPhoto` returns `{detections, errors}` with per-species/per-detection failure annotations. Capture saves partial observations with explicit failed statuses instead of losing all completed detections when one species fails. +- **2.7** **Model descriptors + I/O contract validation.** Declare expected input/output names, tensor ranks, dtype, embedding dim for detector and MiewID v4.1. Validate at session load time with actionable errors. Fail fast on model/config drift. +- **2.8** **Settings UX for model + pack management.** New routes: MiewID download/update/delete; broken-pack recovery. Also a first-run prompt to acquire MiewID when no pack has been loaded yet. + +**Suggested order:** 2.1 + 2.2 + 2.3 together (foundation trio), then 2.4 (capture flow), then 2.5–2.7 as parallel hardening, then 2.8 last. + +### Stage 3 — End-to-End Verification + +- **3.1** Maestro E2E: bundle a tiny test pack (2-3 synthetic individuals, 3 embeddings each) + the FP16 MiewID ONNX in a `fixtures/` dir — sideload on test devices. Run full capture flow; assert a self-sighting (same photo → top candidate = same individual with score > 0.95). +- **3.2** Measure device latency: detection ms, embedding ms, match ms, total. Reference devices: **iPhone 13** (A15 Bionic, 6-core NPU) on iOS; **Pixel 8** (Tensor G3, clean AOSP for predictable NNAPI) on Android. Budgets: detection <80ms, embedding <500ms cold / <300ms warm. Log results to `outputs/reports/` in the KB. +- **3.3** Cross-platform consistency: same test image on iOS and Android → embeddings cosine similarity ≥ 0.999. Catches native preprocessing drift. + +### Stage 4 — Review UX Polish + +Not strictly required for "re-ID works," but unlocks real field usage. + +- **4.1** Pack candidate name resolution in `MatchReviewScreen`: cache `pack.index.json` in memory on pack load; map `candidate.individualId` → name + ref photo URI. +- **4.2** Calibrated thresholds: per-species cosine thresholds (from test gallery stats) for "auto-approve" / "strong candidate" / "weak candidate" coloring. +- **4.3** Show top-5 with reference photos side-by-side with the capture crop. + +### Stage 5 — Scale + Quality (Follow-on) + +- Gallery dilution study per species (from miewid-trainer skill) to know how accuracy drops with gallery size. +- Move `embeddingMatchService` to sqlite-vec or FAISS when pack × gallery crosses ~5K individuals. +- Consider INT8 quantization after FP16 ships and proves stable — re-run rank-order parity against real wildlife test set. + +## Risks & Mitigations + +| Risk | Mitigation | +|---|---| +| ~~License ambiguity for MiewID weights~~ | Resolved 2026-04-23 (CXL employee sign-off). | +| FP16 drift on Android GPU delegate | Stage 3.3 cross-platform test; fall back to CPU EP | +| Pack schema drift (new v2 manifest) | `formatVersion` is already in the manifest; gate parsing on it | +| INT8 rank regressions | Don't ship INT8 until rank-order parity is proven on real crops | +| Accuracy degradation vs. Python reference | Golden parity test (Stage 0.3) + embedding parity test (Stage 3.3) | + +## Success Criteria + +- ✅ Stages 0 + 1 + 2 done, app can photograph an animal and show meaningful top-5 candidates from a real pack with real MiewID v4.1 embeddings. +- ✅ Self-sighting test: same photo re-captured returns the same individual at the top with cosine > 0.95. +- ✅ Cross-platform embedding cosine ≥ 0.999 on a fixed test image. +- ✅ First capture (cold) < 3 s on a reference flagship; subsequent captures < 1 s. +- ✅ No preprocessing tensor diverges from Python reference by > 1e-3 per element. + +## Open Items to Confirm with User + +1. ~~Is v4.1 published yet?~~ **Resolved 2026-04-23** — v4.1 checkpoint is local at `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin`. Will export it ourselves. +2. ~~License status with CXL~~ — **Resolved 2026-04-23** (Jason, CXL employee, granted sign-off). +3. ~~Target pack for dogfooding~~ — **wild horses in the State of Washington** (confirmed 2026-04-23). Likely sourced from `horses.wildbook.org`. **Caveat:** per miewid-trainer MODEL_DATA.md, `horse_wild_tunisian+face` gets 98.5% R1, but generic `horse_wild+face` scores 0% (likely dataset-too-small). Before shipping the pack, run a matchability assessment on a real Washington-horses COCO export using the miewid-trainer skill; if R1 < 40%, reserve extra budget for fine-tuning before declaring dogfood ready. +4. ~~Reference device(s) for latency + parity testing~~ — **iPhone 13 + Pixel 8** (confirmed 2026-04-23). +5. Preferred hosting for the published ONNX: HF repo / CXL CDN / GitHub release asset? diff --git a/kb/wildlife-reid-mobile/outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md b/kb/wildlife-reid-mobile/outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md new file mode 100644 index 00000000..2a5d8c34 --- /dev/null +++ b/kb/wildlife-reid-mobile/outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md @@ -0,0 +1,124 @@ +# Codex 5.5 — Stage-2-Risk Deep Review + +**Date:** 2026-04-24 +**Reviewer:** codex-cli 0.121.0, model GPT-5.5 (upgraded same day) +**Scope:** Pre-Stage-2 risk scan of code paths Stage 2 will touch — model lifecycle, pack manager, embedding DB, capture flow, wildlife pipeline, download service. +**Related:** [[miewid-v41-integration-plan]], [[state-of-implementation]], [[critical-bugs]] + +## Summary + +Two blockers, three highs, three mediums. None in the code I just fixed in PR #11 — all in pre-existing paths that Stage 2 (MiewID download/cache/version-gate) will amplify. Three are architecture-shaped (model lifecycle, pack grouping, download integrity) and need design decisions before implementation. + +## Findings (ranked by severity) + +### 1. [blocker] No authoritative MiewID version identity + +No way to gate `manifest.embeddingModel.version` against the installed MiewID. + +- `src/stores/wildlifeStore.ts:22` persists only `miewidModelPath` +- `setMiewidModelPath` at `:162` stores a bare string +- `useCaptureFlow.ts:90` gates only on non-null +- `EmbeddingPack` in `src/types/wildlife.ts:55` flattens only `embeddingModelVersion`, discarding model name + input config + +Conflicts with spec's required major/minor behavior (`docs/EMBEDDING_PACK_FORMAT.md:549`). + +**Fix:** persist a `MiewIDModelRecord { path, name, version, sha256, sizeBytes, status }`. Verify file existence + hash on hydration. Gate packs/capture by semver — block major mismatch, warn minor mismatch. + +### 2. [blocker] Pack integrity not validated before use + +- `packManager.loadManifest` and `loadPackIndex` only `JSON.parse` — no schema, `formatVersion`, file existence, checksum, byte-size, or offset validation +- `buildEmbeddingDatabase:43` reads `embeddings.bin` blindly +- `getEmbeddingsForIndividual:33` slices without bounds checks +- Corrupt/truncated pack → short vectors → `cosineSimilarity` reads `undefined` and returns `NaN` + +**Fix:** pack install/rehydrate validator checking: +- Manifest schema + supported `formatVersion` +- Required files present +- `RNFS.hash(..., 'sha256')` matches `manifest.checksums` +- `embeddings.bin` size === `embeddingCount * embeddingDim * 4` +- Every individual's offset/count in bounds + +Quarantine invalid packs rather than admitting them to `wildlifeStore`. + +### 3. [high] Capture config: one per pack, but DB merges all packs per species + +- `useCaptureFlow.ts:99` builds one `SpeciesConfig` per pack +- `buildEmbeddingDatabase:37` pulls every pack with that species + +→ two horse packs → horse detector runs twice, both runs match the merged DB. Worse, packs with same species but different feature class or different MiewID version get mixed into one embedding space. + +**Fix:** group by `{ species, featureClass, detectorModelFile, embeddingModelVersion }` and run one detector per group. Reject/isolate incompatible embedding spaces. + +### 4. [high] ONNX session lifecycle is race-prone + +- `loadModel:11` checks the map before awaiting `InferenceSession.create` — concurrent callers create duplicate sessions, leaking the first +- `unloadModel:131` can release a session while `runDetection:57` or `extractEmbedding:116` is in flight +- `useCaptureFlow.ts:148` has no concurrency guard against repeated camera/gallery actions + +**Fix:** per-path load promises + session states; serialize load/release; operation refcounts or read/write lock so release waits for active inference. + +### 5. [high] MiewID-missing = dead-end alert, not recoverable flow + +- `useCaptureFlow.ts:90` shows "MiewID model not loaded" and returns +- `App.tsx:85` hydrates stores and initializes packs but never reconciles restored model path with actual file, version, checksum, or active download +- `SettingsScreen.tsx:81` has no MiewID management route + +**Fix:** model state machine with `missing | downloading | ready | corrupt | incompatible`. Capture routes to/resumes the required download from the pack manifest. App startup reconciles/resumes any in-flight download. + +### 6. [medium] Pipeline discards partial successful work on error + +- `wildlifePipeline.processPhoto:24` accumulates detections across species +- Any later detector/MiewID/crop/match error rejects the whole call +- `useCaptureFlow.ts:137` then saves nothing + +→ one species succeeds, another fails → user loses all completed detections. + +**Fix:** either validate/load MiewID before any detector work, or return `{ detections, errors }` with per-species/per-detection failure annotations so capture can save partial observations with explicit failed statuses. + +### 7. [medium] ONNX I/O contracts are assumed, not checked + +- `runDetection:50` uses `session.inputNames[0]` and a guessed output name +- Dereferences `outputTensor.data` without existence check +- `extractEmbedding:119` takes first output; never validates embedding length vs pack/model expectations + +**Fix:** model descriptors for detector and MiewID v4.1 with expected input/output names, tensor ranks, dtype, embedding dim. Validate at load time with actionable errors. + +### 8. [medium] Existing download layer lacks integrity guarantees + +- No JS `backgroundDownloadService` under `src/services` — only native modules +- Android `DownloadManagerModule.kt:405`: treats `unknown` DownloadManager row + any non-empty file as completed +- iOS `DownloadManagerModule.swift:890`: saves completed downloads without checksum or HTTP status validation + +**Fix:** JS download/cache service that: +- Downloads to a staging path +- Validates HTTP status + expected length + SHA-256 +- Atomically moves into the model cache +- Records the verified model record +- Handles retry/backoff and cancellation in JS, not just via native progress events + +## Implications for the Stage-2 Integration Plan + +These findings reshape Stage 2 significantly. The existing plan in [[miewid-v41-integration-plan]] treats Stage 2 as "add a download service + wire the path." Reality: Stage 2 needs a model *state machine* (finding 5), a *version identity record* (finding 1), and a *pack validator* (finding 2) before any download work is meaningful — findings 3, 4, 7, 8 are correctness/robustness wins that should land alongside. + +### Revised Stage 2 sub-stages + +- **2.1 — Model record + state machine.** Define `MiewIDModelRecord`, states `missing|downloading|ready|corrupt|incompatible`, persist in `wildlifeStore`. Reconcile on startup. +- **2.2 — Pack validator.** Schema check, checksum verify, byte-size verify, offset-bounds verify. Quarantine invalid packs. Errors flow to new "broken packs" Settings section. +- **2.3 — JS download service.** Wrap native downloads with staging + HTTP status check + SHA-256 + atomic move + retry/backoff. Used by both MiewID and pack download. +- **2.4 — Pack grouping + compatibility check.** Group packs by `{species, featureClass, detectorModelFile, embeddingModelVersion}` for capture. Reject mixing incompatible embedding spaces. +- **2.5 — Lifecycle hardening.** Per-path load-promise dedup in `onnxInferenceService`; release waits for active inference via refcount. +- **2.6 — Partial-success observations.** `processPhoto` returns `{detections, errors}`; capture saves partial observations with per-detection failure flags. +- **2.7 — Model descriptors + I/O contract validation** at session load time. +- **2.8 — Settings UX** for MiewID management + broken-pack recovery. + +### Suggested order + +Land 2.1 + 2.2 + 2.3 as a trio (they're interdependent and form the foundation). Then 2.4 (capture flow), then 2.5-2.7 as parallel hardening. 2.8 last. + +## Note on Scope + +Codex explicitly skipped what was fixed in PR #11 (native scale multiply, TS wrapper `1/255`, `DEFAULT_DETECTOR_CONFIG` fallback, stub `MatchReviewScreen.tsx` removal). Focus here is genuinely pre-existing latent issues. + +## Source + +Full codex output archived at `/tmp/claude-1000/-mnt-c-off-grid-mobile/.../bhuoqrwdt.output` (transient; this file is the durable record). diff --git a/kb/wildlife-reid-mobile/raw/codex/2026-04-22-codebase-review.md b/kb/wildlife-reid-mobile/raw/codex/2026-04-22-codebase-review.md new file mode 100644 index 00000000..7ace174b --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/codex/2026-04-22-codebase-review.md @@ -0,0 +1,32 @@ +# Codex Review — Off Grid Mobile → Wildlife Re-ID + +**Date:** 2026-04-22 +**Reviewer:** codex-cli 0.121.0 (read-only mode) +**Scope:** Assessment of work done toward on-device MiewID v4.1 individual re-ID. + +## Top 3 Strengths + +- **Layering is solid.** Session management in `src/services/onnxInferenceService/index.ts:8`, orchestration in `src/services/wildlifePipeline/index.ts:11`, DB assembly in `src/services/embeddingDatabaseBuilder.ts:29`, and persisted offline state in `src/stores/wildlifeStore.ts:79`. The separation will survive real-model integration. +- **Symmetric cross-platform native image prep/crop.** Kotlin `android/app/src/main/java/ai/offgridmobile/imagetensor/ImageTensorModule.kt:21` and Swift `ios/ImageTensorModule.swift:12` mirror each other, removing a common RN bottleneck. +- **Disciplined commit progression.** docs/spec (`7704aed`) → core services (`97450e8`) → detector preprocessing (`79a14bf`) → pipeline (`68f5729`) → screens/store/tests (`28f0b57`, `c6a5a02`) → real native tensor + ONNX wiring (`5a67794`). + +## Top 3 Risks / Gaps + +1. **Licensing blocker.** Feasibility doc still says MiewID weights/code have no explicit open-source license. Must be resolved with Conservation X Labs before shipping. +2. **MiewID v4.1 is not actually wired as a product feature.** `miewidModelPath` is set-only in `src/stores/wildlifeStore.ts:56` / `:162`, checked in `src/screens/CaptureScreen/useCaptureFlow.ts:90`, but no call site ever populates it. Spec says MiewID is a separately downloaded shared model with version checks (`docs/EMBEDDING_PACK_FORMAT.md:151,549`) — none of that exists. +3. **Accuracy bugs in preprocessing/postprocessing.** + - Native code divides by `scale` in `android/app/src/main/java/ai/offgridmobile/imagetensor/ImageTensorModule.kt:62` and `ios/ImageTensorModule.swift:188`, but the pack spec defines `scale` as a multiplier (`1/255`) (`docs/EMBEDDING_PACK_FORMAT.md:166`). If a pack ships `scale: 0.00392...`, division yields `r * 255` — catastrophically wrong. + - Embedding preprocessing passes `scale=1.0` in `src/services/onnxInferenceService/preprocessing.ts:62`, meaning 0-255 pixel values get ImageNet-normalized with `mean=[0.485,…]` / `std=[0.229,…]` that expects 0-1. MiewID embeddings will be garbage. + - YOLO parser at `src/services/onnxInferenceService/postprocessing.ts:142` uses `4 + numClasses` rows. Matches YOLOv8/v11 (objectness-free) but not legacy YOLOv5 which is `5 + C`. Architecture-dependent — ensure detectors are v8+. + +## Highest-Leverage Improvements + +- **Build the real acquisition path first.** Pack import/unzip/checksum, MiewID download/cache, manifest version enforcement, startup restore. Today packs are only a directory abstraction (`src/services/packManager/index.ts:8`) and the UI has no install flow. +- **Fix tensor math and add golden parity tests against Python** for one detector and one MiewID export. Do not optimize before this. +- **Make review field-usable.** Resolve pack candidate names/photos (not raw IDs) in `src/screens/MatchReviewScreen/index.tsx:67`, add calibrated acceptance thresholds, replace GPS/sync stubs in `src/screens/CaptureScreen/useCaptureFlow.ts:63` and `src/screens/SyncScreen.tsx:55`. + +## Answers to Specific Questions + +**Is MiewID v4 actually wired in?** No. `extractEmbedding()` in `onnxInferenceService` is generic and would work if given a path, but no call site provides a real MiewID model. In current app UX, neither the detector nor MiewID is end-to-end without manually seeding pack/model paths. + +**Model acquisition recommendation.** Ship MiewID as a **separate versioned download**, not bundled in the app binary. Start **FP16**, target ~80-120 MB. Keep detector ONNX in each pack. Avoid INT8 until you prove rank-order parity on real wildlife crops; re-ID quality is more fragile than detection. diff --git a/kb/wildlife-reid-mobile/raw/docs-live b/kb/wildlife-reid-mobile/raw/docs-live new file mode 120000 index 00000000..48c4a0c0 --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/docs-live @@ -0,0 +1 @@ +../../../docs \ No newline at end of file diff --git a/kb/wildlife-reid-mobile/raw/exploration/2026-04-22-codebase-map.md b/kb/wildlife-reid-mobile/raw/exploration/2026-04-22-codebase-map.md new file mode 100644 index 00000000..ccac3bf9 --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/exploration/2026-04-22-codebase-map.md @@ -0,0 +1,120 @@ +# Codebase Map — Wildlife Re-ID Pipeline (Off Grid Mobile) + +**Date:** 2026-04-22 +**Source:** Explore agent thorough pass over `/mnt/c/off-grid-mobile` +**Purpose:** Full map of what's been built in service of on-device MiewID re-ID, filed as the primary raw source for the KB. + +## 1. Services Layer + +### onnxInferenceService (`src/services/onnxInferenceService/`) +Unified ONNX Runtime wrapper for both detector (YOLO) and embedding (MiewID) inference. + +**Public API:** +- `loadModel(modelPath, type)` — load detector or embedding ONNX +- `runDetection(imageUri, detectorModelPath, config) → DetectionOutput` +- `extractEmbedding(croppedImageUri, miewidModelPath, opts?) → EmbeddingOutput` +- `unloadModel(modelPath)`, `unloadAll()`, `isModelLoaded(modelPath)` + +**Data flow:** +1. Detection: imageUri → `preprocessImageForDetection()` → native `ImageTensorModule` (NCHW tensor) → ONNX `session.run()` → `parseYoloOutput()` (NMS, filter, box convert) → `DetectionResult[]` +2. Embedding: croppedImageUri → `preprocessImageForEmbedding()` (440×440, ImageNet norm) → ONNX → Float32Array[2152] + +**Details:** NCHW layout, supports YOLOv8/11 cxcywh + xyxy/xywh via `DetectorConfig.outputSpec`; MiewID defaults 440×440 ImageNet mean/std; IoU-based NMS; inference timing tracked. Status: real, tested. + +### Native ImageTensor Module +- Android: `android/app/src/main/java/ai/offgridmobile/imagetensor/ImageTensorModule.kt` (Kotlin) +- iOS: `ios/ImageTensorModule.swift` (Swift) +- API: `imageToTensor(uri, w, h, mean, std, scale, channelOrder) → number[]`, `cropImage(uri, x, y, w, h, outputPath) → string` +- Loads `content://`, `file://`, plain paths; bilinear resize; per-channel `(pixel/scale - mean[i]) / std[i]` (**note: divide, not multiply — see bugs**); NCHW pack; RGB↔BGR swap; JPEG crop save (95% q). + +### packManager (`src/services/packManager/`) +Loads embedding packs from disk. +- `initialize()` creates `DocumentDirectoryPath/embedding_packs/` +- `loadPackIndex(path) → PackIndividual[]` +- `loadManifest(path) → EmbeddingPackManifest` +- `getEmbeddingsForIndividual(allEmbeddings, individual, embeddingDim) → number[][]` +- `deletePack(dir)`, `getPacksDir()` +- Pack layout: manifest.json, config/detector.json, models/{detector}.onnx, embeddings/index.json, embeddings.bin (flat FP32 LE), reference_photos/{id}/ +- Offset math: `individual.embeddingOffset * embeddingDim * 4` bytes + +### wildlifePipeline (`src/services/wildlifePipeline/`) +Orchestrates detect → crop → embed → match. +- `processPhoto(params) → PipelineResult` +- Per-species: load detector → detect → for each detection: crop (native), embed, match → Detection with `matchResult.topCandidates` +- Saves observations to store; returns pipeline result + +### embeddingMatchService (`src/services/embeddingMatchService/`) +- `matchEmbedding(query, database, topN) → MatchCandidate[]` +- Cosine similarity brute-force; per-individual best score across their embeddings +- Sufficient for <5K individuals + +### embeddingDatabaseBuilder.ts +- `buildEmbeddingDatabase(species, packs, localIndividuals) → EmbeddingDatabaseEntry[]` +- Filters packs by species, loads `index.json` + `embeddings.bin` (base64 → Uint8Array → Float32Array), merges with local individuals. Broken packs logged but non-fatal. + +## 2. Screens (User Flow) + +- **WildlifeHomeScreen.tsx** — dashboard (quick capture, pack summary, recent observations, sync status) +- **CaptureScreen** + `useCaptureFlow.ts` — take/pick photo → build `SpeciesConfig[]` per loaded pack → call pipeline → save observation → navigate. **GPS stub returns null.** +- **DetectionResultsScreen** — photo + bounding box overlays; tap detection → MatchReview +- **MatchReviewScreen** — cropped crop + candidates (local shows userLabel/ref photo; pack shows raw ID — name resolution TODO). Actions: Approve → `addEmbeddingToLocalIndividual`; No Match → create `FIELD-XXX` LocalIndividual; Skip. +- **ObservationsScreen** — list + filters (All / Pending / Reviewed / Synced) +- **PacksScreen + PackDetailScreen** — lists loaded packs +- **SyncScreen.tsx** — stubbed queue UI + +## 3. Types & Stores + +### `src/types/wildlife.ts` +`EmbeddingPackManifest`, `EmbeddingPack`, `PackIndividual`, `LocalIndividual`, `Observation`, `Detection`, `MatchCandidate`, `DetectorConfig`, `SyncQueueItem` + +### `src/stores/wildlifeStore.ts` +State: packs, observations, localIndividuals, syncQueue, miewidModelPath, nextFieldId. +Actions: addPack, addObservation, updateDetection, addLocalIndividual, addEmbeddingToLocalIndividual, addToSyncQueue, updateSyncStatus, setMiewidModelPath, getNextFieldId. Zustand + AsyncStorage. + +## 4. Tests + +- Unit: `__tests__/unit/services/onnxInferenceService.test.ts` (NMS, box formats, multi-class), embeddingMatchService, packManager, embeddingDatabaseBuilder, wildlifeStore. +- Integration: `__tests__/integration/wildlife/pipelineFlow.test.ts` — first sighting → no match → local individual → re-sighting match → accumulate embeddings. + +## 5. Data Flow + +``` +capture photo → useCaptureFlow.processPhoto + → load detector config from pack manifest + → buildEmbeddingDatabase(species, packs, locals) + → wildlifePipeline.processPhoto + for each species: load detector → detect → for each det: crop → embed → match + → save Observation → navigate to DetectionResults +MatchReviewScreen: approve / no-match (new local FIELD-XXX) / skip +Sync: syncQueue → (stub; no HTTP client yet) +``` + +## 6. What's Built vs Stubbed + +| Layer | Status | +|---|---| +| ONNX inference | Real | +| Image preprocessing | Real (but see accuracy bugs) | +| Detection postprocessing / NMS | Real | +| Embedding extraction | Real (called, but model never installed) | +| Vector matching | Real (cosine brute-force) | +| Pack loading | Real | +| Observation storage | Real (Zustand + AsyncStorage) | +| Local individual management | Real | +| Match review UI | Real; pack-name resolution TODO | +| Detector model path | Loaded from manifest; no auto-download | +| **MiewID model path** | **Never populated** | +| Pack download | Missing | +| Geolocation | Stubbed (returns null) | +| Wildbook sync | Queue exists, no HTTP client | +| Model timeout | Missing | +| Download retry | Missing | + +## 7. Critical Gaps for MiewID v4.1 On-Device Re-ID End-to-End + +1. MiewID v4.1 model acquisition flow (download, cache, version enforcement) +2. Pack install UX (import .zip → unzip → checksum verify → register) +3. Preprocessing bugs: `scale` semantics (divide vs multiply), MiewID pipeline feeds 0-255 to ImageNet norm +4. Pack name/photo resolution in MatchReviewScreen +5. GPS + sync (out of scope for pure re-ID, but needed for field use) +6. MiewID license clarification with CXL diff --git a/kb/wildlife-reid-mobile/raw/models/2026-04-23-miewid-v41-checkpoint.md b/kb/wildlife-reid-mobile/raw/models/2026-04-23-miewid-v41-checkpoint.md new file mode 100644 index 00000000..b784105f --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/models/2026-04-23-miewid-v41-checkpoint.md @@ -0,0 +1,77 @@ +# MiewID v4.1 Checkpoint — Local Reference + +**Date ingested:** 2026-04-23 +**Source path:** `/mnt/c/claude-skills/models/reference/` + +## Files + +| File | Size | MD5 | Notes | +|---|---|---|---| +| `miew_id.msv4_1_main.bin` | 206,040,591 B (~196 MB) | `71c0bdd9bf8cbdb1b9e967a2e96949bb` | PyTorch state dict, FP32 | +| `miew_id.msv4_1_main.yaml` | 1,477 B | — | Training config | +| `detect.yolov11.msv3.pt` | 5.4 MB | (see `.md5` sidecar) | Companion YOLOv11 detector (msv3-era) | + +Same file size (206 MB) as v3 and v4 main — architecture is identical, only weights change. + +## Training Config (from .yaml) + +```yaml +model_params: + model_name: efficientnetv2_rw_m + loss_module: arcface_subcenter_dynamic + margin: 0.32841442327915477 + s: 49.32675426153405 + k: 2 + n_classes: 20191 # v4 main was 19911 → v4.1 adds 280 individuals + fc_dim: 2048 + dropout: 0 + ls_eps: 0 + theta_zero: 0.785 + use_fc: false # Output is backbone GeM features, NOT the 2048 FC + pretrained: true + +data: + image_size: [440, 440] + crop_bbox: true + +engine: + epochs: 30 + loss_module: arcface + seed: 42 + use_swa: false + +test: + batch_size: 8 + fliplr: false + fliplr_view: [] +``` + +Exp name: `EDA-msv4_1` · Project: `msv4-main` · Trained on CUDA device 6. + +## Compared to v4 main (`miew_id.msv4_v3_main.yaml`) + +| Field | v4 main | v4.1 | +|---|---|---| +| `n_classes` | 19,911 | **20,191** (+280) | +| `margin`, `s`, `k` | same | same | +| Architecture | `efficientnetv2_rw_m` | same | +| Image size | 440×440 | same | +| `use_fc` | false | false | + +Interpretation: v4.1 is an **additive retrain** on a superset of individuals — same architecture, same hyperparameters. ONNX export recipe will be identical to any v3/v4 main recipe, just pointed at this checkpoint. + +## Embedding Dimension + +The pack spec ([[embedding-pack]]) and feasibility doc ([[miewid-v4]]) state `embeddingDim: 2152`. With `use_fc: false`, the output comes from the backbone's GeM-pooled features. EfficientNetV2-RW-M has a natural feature dim around 2152 (per timm spec), which matches the pack format. + +## Implications for On-Device Integration + +- We have a **concrete v4.1 checkpoint to export**, no dependency on an external HF release. +- Export path (from [[model-acquisition]]): load `.bin` into the MiewID model class (`wbia-plugin-miew-id`) → `torch.onnx.export` at opset 17 with dynamic batch → parity test on CPU ORT → FP16 convert → parity test again. +- The companion `detect.yolov11.msv3.pt` (5.4 MB) can be exported to ONNX as the default detector for early testing — well under the 15 MB detector budget in the feasibility doc. + +## Open Items + +- License status for redistribution — the file is local to Jason's workspace; mobile distribution still requires CXL sign-off. +- Confirm exported ONNX preserves the normalization path (MiewID ingests 440×440 RGB, ImageNet mean/std, [0,1] range). +- Confirm the `.bin` loads cleanly from the `wbia-plugin-miew-id` repo class without a config mismatch (n_classes=20191). diff --git a/kb/wildlife-reid-mobile/raw/models/2026-04-24-miewid-v41-onnx-export.md b/kb/wildlife-reid-mobile/raw/models/2026-04-24-miewid-v41-onnx-export.md new file mode 100644 index 00000000..48bb0073 --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/models/2026-04-24-miewid-v41-onnx-export.md @@ -0,0 +1,95 @@ +# MiewID v4.1 ONNX Export — Stage 1.2 Deliverable + +**Date:** 2026-04-24 +**Tool:** `kb/wildlife-reid-mobile/tools/export_miewid_v41_onnx.py` +**Run command:** `python3 export_miewid_v41_onnx.py --fp16 --parity-trials 25` +**Source:** [[2026-04-23-miewid-v41-checkpoint]] + +## Outputs (in `tools/output/`) + +| File | Size | SHA-256 | Status | +|---|---|---|---| +| `miewid_v4_1_fp32.onnx` | 205.7 MB single file | `feb5ff43f548a12f3b92b6ab5117e80bfe1638216d8530617d7c3147ab8c7fd6` | reference (parity test target) | +| `miewid_v4_1_fp16.onnx` | **103.9 MB** single file | `3c1f1db5bda5d2434b01c63daa0bfa6a1f62b77ef89c40c335699eea2b04c0bd` | **mobile deployment artifact** | +| `miewid_v4_1_export_manifest.json` | 1.8 KB | — | provenance | + +Both ONNX files are self-contained (no `.onnx.data` external-data shards) thanks to the consolidation pass added 2026-04-24. + +## Source provenance + +- **PyTorch checkpoint:** `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` +- **Checkpoint SHA-256:** `032a41a0472697f1fc403b16ee1a20ef5cf01238b88cc4172ea0910d0b3808c9` (matches md5 noted in [[2026-04-23-miewid-v41-checkpoint]]) +- **Backbone:** `efficientnetv2_rw_m` +- **Loss module:** `arcface_subcenter_dynamic` +- **n_classes:** 20,191 (training-time head; stripped at export) +- **`use_fc`:** false (so output is backbone GeM-pooled features, not the 2048 FC dim) + +## Parity (vs PyTorch FP32 reference) + +100 random batches × 4 samples = 400 actually wait. 25 trials × batch 4 = 100 samples each. + +### FP32 ONNX vs PyTorch +- `cosine_min`: **0.9999998807907104** (well above the 0.9999 threshold) +- `cosine_mean`: 0.9999999958276748 +- `l2_max` per-sample: 1.5e-4 (effectively floating-point round-trip noise) +- `embedding_dim`: **2152** (matches pack spec exactly) +- All 100 samples pass + +Interpretation: the ONNX export is byte-for-byte equivalent to the PyTorch reference up to numerical noise. ArcFace head was correctly stripped — `missing=0, unexpected=0` on `load_state_dict`, meaning the canonical `wbia_miew_id.MiewIdNet` class is a perfect structural match for this checkpoint. + +### FP16 ONNX vs PyTorch +- `cosine_min`: **0.9999938011169434** (well above the looser 0.999 threshold) +- `cosine_mean`: 0.9999958050251007 +- `l2_max` per-sample: 0.23 (FP16 quantization noise — embedding magnitudes are O(2-3)) +- All 100 samples pass + +Interpretation: FP16 introduces ~5e-6 cosine drift, far below the precision floor that would affect re-ID rank ordering. Safe for production. + +## Sanity check vs the [[on-device-reid-pipeline]] spec + +| Pack spec field | Required value | Export observed | +|---|---|---| +| `embeddingModel.name` | `miewid-v4` (or v4.1) | `efficientnetv2_rw_m` backbone, MiewID v4.1 weights ✓ | +| `embeddingModel.version` | semver string | `4.1.0` (to be set in pack manifest) | +| `embeddingModel.inputSize` | `[440, 440]` | exported with `(1,3,440,440)` ✓ | +| `embeddingModel.normalize.mean` | `[0.485, 0.456, 0.406]` | ImageNet — model expects this ✓ | +| `embeddingModel.normalize.std` | `[0.229, 0.224, 0.225]` | ImageNet ✓ | +| `embeddingDim` | `2152` | confirmed via parity test ✓ | +| ONNX opset | ≥ 11 (ORT Mobile) | 17 ✓ | +| Input dtype | float32 | ✓ | +| Output | single tensor `(N, 2152)` | ✓ | + +End-to-end compatible with the existing `onnxInferenceService.extractEmbedding()` after the [[critical-bugs]] fix in PR #11. No code changes required to consume. + +## What this unlocks + +This is the **embedding** half of the Washington horses pack. With the [[2026-04-24-whorse-face-detector]] (detector half) already in hand, the remaining pack ingredients are: + +| Component | Status | +|---|---| +| Detector ONNX + config | ✅ done 2026-04-24 | +| **MiewID v4.1 ONNX (FP16)** | ✅ **done 2026-04-24 (this artifact)** | +| `embeddings.bin` (per-individual MiewID vectors) | ❌ pending Wildbook horse-face COCO export + batch inference | +| `index.json` (individual metadata + offsets) | ❌ same | +| `reference_photos/{id}/` | ❌ same | +| `manifest.json` | ❌ assembled last with checksums | + +## Followups + +- **Hosting decision** (Stage 1.5) — where does the FP16 ONNX live for app-side download? HF / CXL CDN / GitHub release. Untouched. +- **Cross-platform on-device parity** — Stage 3 in the integration plan: load `miewid_v4_1_fp16.onnx` on iPhone 13 + Pixel 8, embed a fixed test crop, assert cross-platform cosine ≥ 0.999. +- **Real-crop matchability assessment** — use miewid-trainer's `benchmark_model` on a Washington-horses COCO before bundling. If R1 < 40% on horse_wild+face, consider fine-tune before shipping the pack. + +## Reproduce + +```bash +cd /mnt/c/off-grid-mobile/kb/wildlife-reid-mobile/tools +python3 export_miewid_v41_onnx.py --fp16 --parity-trials 25 +``` + +Runtime observed: ~2 minutes total on RTX 5080 Laptop (mostly FP16 conversion + parity). + +## Notes + +- **Dependency that wasn't in the original requirements list:** `onnxscript` (for PyTorch ≥ 2.10's dynamo-backed ONNX export path). Added during this run via `pip install --user --break-system-packages onnxscript`. Documented in tools/README.md. +- The `efficientnetv2_rw_m` backbone is **not** a public timm model name today — wbia_miew_id includes a registered alias. The script's `wbia_miew_id` canonical path uses this; the timm-fallback would need extra wiring. Stick with the canonical class. diff --git a/kb/wildlife-reid-mobile/raw/models/2026-04-24-whorse-face-detector.md b/kb/wildlife-reid-mobile/raw/models/2026-04-24-whorse-face-detector.md new file mode 100644 index 00000000..30b5c9bc --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/models/2026-04-24-whorse-face-detector.md @@ -0,0 +1,98 @@ +# WHORSE Face Detector — Pack-Ready Deliverable + +**Date ingested:** 2026-04-24 +**Source path:** `/mnt/c/claude-skills/output/whorse-face-detector/` +**Built with:** yolo-trainer skill (1× RTX 5080 Laptop, 50 epochs, batch 64) +**Run provenance:** `/mnt/c/claude-skills/runs/runs/run_20260424_093150/run_manifest.json` + +## Files + +| File | Size | Purpose | +|---|---|---| +| `horse_wild_face_yolo11n_416.onnx` | 10 MB | YOLO11n single-class face detector, opset 12, simplified | +| `detector_config.json` | 567 B | DetectorConfig — drops into pack `config/detector.json` | +| `README.md` | 1.6 KB | Training + eval summary | + +## Spec & Compatibility + +The `detector_config.json` is byte-compatible with `src/types/wildlife.ts → DetectorConfig`: + +```json +{ + "modelFile": "horse_wild_face_yolo11n_416.onnx", + "architecture": "yolo11n", + "inputSize": [416, 416], + "inputChannels": 3, + "channelOrder": "RGB", + "normalize": { "mean": [0,0,0], "std": [1,1,1], "scale": 0.00392156862745098 }, + "confidenceThreshold": 0.25, + "nmsThreshold": 0.45, + "maxDetections": 100, + "outputFormat": "yolov11-detect", + "classLabels": ["horse_wild+face"], + "outputSpec": { + "boxFormat": "cxcywh", + "coordinateType": "absolute", + "outputTensorName": "output0", + "layout": "1x(4+C)xN" + } +} +``` + +End-to-end check vs the app's runtime constraints (see [[yolo-postprocessing]]): + +| Constraint | This model | +|---|---| +| ONNX opset ≥ 11 (ORT Mobile) | opset 12 ✓ | +| Single output tensor `[1, 4+C, N]` | `[1, 5, 3549]` (1 class → 4+1=5; 3549 = 52² + 26² + 13² for 416×416 multi-scale heads) ✓ | +| No objectness channel (anchor-free) | YOLO11 anchor-free ✓ | +| Box format ∈ {cxcywh, xyxy, xywh} | cxcywh ✓ | +| Channel order RGB or BGR | RGB ✓ | +| `scale` is a multiplier (per `EMBEDDING_PACK_FORMAT.md`) | 1/255 = 0.00392... ✓ — matches the [[critical-bugs]] fix landed in PR #11 | +| Output tensor name | `output0` (hardcoded fallback in `onnxInferenceService.runDetection`) ✓ | +| Mobile-friendly size | 10 MB (well under feasibility-doc 15 MB detector budget) ✓ | + +**No code changes required to consume this model in `parseYoloOutput`.** + +## Eval (345-image WBIA holdout) + +| Metric | Value | +|---|---| +| mAP@50 | 0.9999 | +| mAP@50-95 | 0.9576 | +| Optimal F1 | 0.997 @ conf 0.2 | + +## Caveats (flagged by trainer) + +1. **Metrics are near-ceiling because the task is easy.** 1725 training images, mostly one large face per close-cropped photo. Real-world mobile-camera photos (small, multi-subject, harsh lighting) will be the honest test. **Need a small field-photo eval set before broad deployment.** +2. **Possible viewpoint blind spots.** WBIA images are mostly frontal close-ups; expect under-firing on side profiles or distant subjects. **Remedy is more diverse training data, not more epochs.** + +Both caveats are accuracy/recall risks for the dogfood pack — they don't block integration. + +## What this unlocks + +| Pack component | Status | +|---|---| +| Detector model + config | ✅ this deliverable | +| MiewID embedding model | ❌ Stage 1 of [[miewid-v41-integration-plan]] (export from `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin`) | +| `embeddings.bin` (per-individual MiewID vectors) | ❌ pending Wildbook horse-face COCO export + MiewID inference | +| `index.json` (individual metadata) | ❌ same | +| `reference_photos/{id}/` | ❌ pulled from Wildbook export | +| `manifest.json` | ❌ assembled last, references all of the above with checksums | + +## Recommended next step + +The **detector half** of the Washington horses pack is now done. Sequence to complete the pack: + +1. Export MiewID v4.1 → ONNX (Stage 1.2 in [[miewid-v41-integration-plan]]) +2. Pull a Washington-horses COCO from `horses.wildbook.org` filtered to `feature_class = horse_wild+face` +3. Use the miewid-trainer MCP `benchmark_model` to run a zero-shot matchability check against MiewID v4.1 on those crops +4. If matchability ≥ marginal tier, batch-infer MiewID v4.1 over the COCO crops to populate `embeddings.bin` + `index.json` +5. Bundle as a `.zip` per `EMBEDDING_PACK_FORMAT.md` + +## References + +- `/mnt/c/claude-skills/output/whorse-face-detector/README.md` +- `/mnt/c/claude-skills/runs/runs/run_20260424_093150/run_manifest.json` +- yolo-trainer skill at `/mnt/c/claude-skills/.claude/skills/yolo-trainer/` +- [[2026-04-23-miewid-v41-checkpoint]] diff --git a/kb/wildlife-reid-mobile/raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md b/kb/wildlife-reid-mobile/raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md new file mode 100644 index 00000000..c309989b --- /dev/null +++ b/kb/wildlife-reid-mobile/raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md @@ -0,0 +1,84 @@ +# Pack Bundler + Synthetic Horse Pack + +**Date:** 2026-04-25 +**Tool:** `kb/wildlife-reid-mobile/tools/build_pack.py` +**Output:** `tools/output/horse-synthetic-2026-04-25.zip` (9.2 MB) + +## Tool: `build_pack.py` + +Two-mode CLI for assembling spec-compliant `.zip` packs per `docs/EMBEDDING_PACK_FORMAT.md`. + +### `synthetic` mode (working today) + +Random unit-norm 2152-d vectors with controlled per-individual noise (σ=0.0015 → within-individual cos ≈ 0.995, cross-individual cos ≈ ±0.02). Reference photos are HSV-spread colored squares so each individual is visually distinct in review UI. Real detector ONNX is bundled (the WHORSE one). Useful for: +- Mobile pipeline plumbing tests (file loading, byte layout, manifest parsing) +- Match-review UI dev (real-looking ranked candidates) +- Cross-platform parity tests once on-device + +### `from-coco` mode (skeleton) + +Stub for now — designed to ingest a Wildbook horses COCO export, batch-infer MiewID v4.1 over each annotation's bbox crop, and produce a real pack. Implementation deferred to when we have the Washington-horses COCO in hand. + +## Pack: `horse-synthetic-2026-04-25.zip` + +**Layout** (validated against `docs/EMBEDDING_PACK_FORMAT.md`): + +``` +horse-synthetic-2026-04-25.zip (9.2 MB) +├── manifest.json +├── config/detector.json +├── models/horse_wild_face_yolo11n_416.onnx (10.5 MB) +├── embeddings/ +│ ├── index.json (5 individuals × 2 embeddings each) +│ └── embeddings.bin (10 × 2152 × 4 = 86 KB) +└── reference_photos/SYN-{0001..0005}/ref_{01..03}.jpg (15 colored squares) +``` + +**Manifest highlights:** +- `formatVersion: "1.0"` ✓ +- `species: "horse"`, `featureClass: "horse_wild+face"` ✓ +- `embeddingDim: 2152` ✓ +- `embeddingModel.{name, version, inputSize, normalize}` populated for v4.1 ✓ +- `detectorModel.{filename, configFile}` ✓ +- SHA-256 checksums for `embeddings.bin` and the detector ✓ + +**Self-validation passes:** +- All required files present +- `embeddings.bin` size === `embeddingCount × embeddingDim × 4` (87,040 B) +- Per-individual offsets cumulate cleanly (0, 2, 4, 6, 8) +- Reference photos in `index.json` resolve to real files + +**Consumer-side parity** (Python sim of mobile load): +- Within-individual cosine: 0.9950–0.9953 (target ~0.995) ✓ +- Cross-individual cosine: ±0.007 to ±0.026 (target ~0) ✓ + +## What this unlocks + +This is the **bridge artifact** between Stage 1 and Stage 2: + +- For **Stage 2 development** — an end-to-end pack that satisfies the spec, so we can build/test mobile-side validators and loaders against something concrete instead of mocks. +- For **vertical-slice on-device test (step A)** — sideload this pack + the FP16 MiewID ONNX onto iPhone 13 / Pixel 8 and validate that the existing pipeline (post-PR-#11 fix) loads everything and produces sensible embeddings + match candidates against real test photos. +- For **cross-platform parity** — feeding a fixed test photo through both devices' pipelines should yield embeddings cosine ≥ 0.999 against each other and against a Python reference. + +## Reproduce + +```bash +cd /mnt/c/off-grid-mobile/kb/wildlife-reid-mobile/tools +python3 build_pack.py synthetic \ + --species horse \ + --feature-class horse_wild+face \ + --display-name "Synthetic Horses (Dev)" \ + --detector /mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx \ + --detector-config /mnt/c/claude-skills/output/whorse-face-detector/detector_config.json \ + --num-individuals 5 \ + --photos-per-individual 3 \ + --embeddings-per-individual 2 \ + --output-dir output/ +``` + +Runtime ~3 seconds. Output goes to `output/horse-synthetic-2026-04-25/` (kept) and `output/horse-synthetic-2026-04-25.zip`. + +## Followups + +- **Implement `from-coco` mode** when we have a Washington-horses COCO. Needs MiewID inference loop + crop preprocessing matching the mobile pipeline. ~half day. +- **Add a `validate-zip` mode** to `build_pack.py` that takes a `.zip` and runs the same self-validation a Stage-2 pack-validator would. Useful for CI gates on real pack exports. diff --git a/kb/wildlife-reid-mobile/tools/README.md b/kb/wildlife-reid-mobile/tools/README.md new file mode 100644 index 00000000..a4fe43a3 --- /dev/null +++ b/kb/wildlife-reid-mobile/tools/README.md @@ -0,0 +1,67 @@ +# Tools + +Reproducible scripts that produce artifacts referenced by the wiki. + +## `export_miewid_v41_onnx.py` + +Exports the MiewID v4.1 PyTorch checkpoint at +`/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` to ONNX +(FP32 + optional FP16), runs cosine-similarity parity tests against the +PyTorch reference, and writes a JSON manifest with provenance. + +**Quick start (in your existing miewid Python env):** + +```bash +cd /mnt/c/off-grid-mobile/kb/wildlife-reid-mobile/tools + +# Standard production export — FP32 + FP16 + 50-trial parity +python export_miewid_v41_onnx.py --fp16 + +# Minimal sanity-check export — FP32 only, fewer parity trials +python export_miewid_v41_onnx.py --parity-trials 10 + +# Verify an already-exported ONNX without re-exporting +python export_miewid_v41_onnx.py --validate-only --onnx output/miewid_v4_1_fp32.onnx +``` + +**Inputs:** +- `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` (overridable via `--checkpoint`) +- `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.yaml` (overridable via `--config`) + +**Outputs (default `--output-dir output/`):** +- `miewid_v4_1_fp32.onnx` — full-precision, ~200 MB +- `miewid_v4_1_fp16.onnx` — half-precision, ~100 MB (only with `--fp16`) +- `miewid_v4_1_export_manifest.json` — checkpoint hash, model config, opset, parity stats, output hashes + +**Parity acceptance:** +- FP32 vs PyTorch reference: per-sample cosine similarity ≥ 0.9999 across `--parity-trials` × 4 random batches. Fails the script if any sample drops below. +- FP16 vs PyTorch reference: ≥ 0.999 (looser). Logged as warning if it fails — does not abort. + +**Requirements:** +``` +pip install torch onnx onnxruntime onnxconverter-common onnxscript timm pyyaml numpy +# Strongly preferred — gives canonical model class: +pip install -e /path/to/wbia-plugin-miew-id +``` + +`onnxscript` is needed for PyTorch ≥ 2.10's dynamo-backed `torch.onnx.export` path. +On a system with externally-managed Python (Debian/Ubuntu PEP 668), add +`--user --break-system-packages` to the pip install command. + +If `wbia_miew_id` isn't importable, the script falls back to a minimal +`timm` + GeM reconstruction. The fallback may not be byte-identical to the +training-time forward — the parity check will catch this. + +**What this script does NOT do:** +- Upload to HuggingFace / CDN — that's a separate publishing step +- Embed test wildlife crops — use the miewid-trainer skill's + `benchmark_model` for that +- Bundle into a pack `.zip` — that's the next step after both detector and + MiewID ONNX are in hand + +## After running + +1. Move the verified ONNX into a stable location for pack bundling +2. Record the SHA-256 from the manifest into the pack `manifest.json` under + `embeddingModel.checksums` +3. Continue with [miewid-v4.1 integration plan Stage 1.5](../outputs/plans/miewid-v41-integration-plan.md) (publish step) diff --git a/kb/wildlife-reid-mobile/tools/build_pack.py b/kb/wildlife-reid-mobile/tools/build_pack.py new file mode 100644 index 00000000..1f7d4406 --- /dev/null +++ b/kb/wildlife-reid-mobile/tools/build_pack.py @@ -0,0 +1,547 @@ +#!/usr/bin/env python3 +""" +Build a wildlife re-ID embedding pack conforming to +docs/EMBEDDING_PACK_FORMAT.md. + +Two modes: + + synthetic — Random 2152-d unit-norm vectors + colored-square reference + photos + made-up individual IDs. For development / on-device + plumbing tests; the detector still works against real photos + but the embeddings won't actually match anything sensible. + + from-coco — Real pack assembled from a COCO export (e.g. horses.wildbook.org). + Batch-infers MiewID over each annotation's bbox crop to populate + embeddings.bin; copies reference photos from the source images. + +Pack layout produced (per EMBEDDING_PACK_FORMAT.md §File Structure): + + {species}-{context}-{YYYY-MM-DD}.zip + ├── manifest.json # EmbeddingPackManifest + ├── models/ + │ └── {detector-filename}.onnx + ├── config/ + │ └── detector.json # DetectorConfig + ├── embeddings/ + │ ├── index.json # PackIndividual[] + │ └── embeddings.bin # Flat float32 LE, NCHW(?) — flat per spec + └── reference_photos/{individual-id}/ref_NN.jpg + +Important: MiewID is **NOT** bundled in the pack (per spec) — it's a shared +model the app downloads separately. We reference its version + repo in the +manifest so the app can version-gate. + +Usage examples: + + # Synthetic mini-pack for on-device dev/testing + python build_pack.py synthetic \\ + --species horse \\ + --feature-class horse_wild+face \\ + --display-name "Synthetic Horses (Dev)" \\ + --detector /mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx \\ + --detector-config /mnt/c/claude-skills/output/whorse-face-detector/detector_config.json \\ + --miewid-version 4.1.0 \\ + --num-individuals 5 \\ + --photos-per-individual 3 \\ + --embeddings-per-individual 2 \\ + --output-dir output/ + + # Real pack from a COCO + MiewID inference + python build_pack.py from-coco \\ + --species horse \\ + --feature-class horse_wild+face \\ + --display-name "Washington Wild Horses 2026-04" \\ + --wildbook-url https://horses.wildbook.org \\ + --coco /path/to/washington_horses_coco.json \\ + --images-dir /path/to/images \\ + --detector /mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx \\ + --detector-config /mnt/c/claude-skills/output/whorse-face-detector/detector_config.json \\ + --miewid /mnt/c/off-grid-mobile/kb/wildlife-reid-mobile/tools/output/miewid_v4_1_fp16.onnx \\ + --miewid-version 4.1.0 \\ + --output-dir output/ + +Requirements: + pip install pillow numpy onnx + # Plus, for `from-coco` mode: + pip install onnxruntime +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import hashlib +import io +import json +import logging +import shutil +import struct +import sys +import zipfile +from pathlib import Path +from typing import Optional + +import numpy as np + +logger = logging.getLogger("build_pack") + +PACK_FORMAT_VERSION = "1.0" +DEFAULT_EMBEDDING_DIM = 2152 + +# ImageNet normalize defaults — must match what the mobile app expects. +DEFAULT_IMAGENET_MEAN = [0.485, 0.456, 0.406] +DEFAULT_IMAGENET_STD = [0.229, 0.224, 0.225] +DEFAULT_MIEWID_INPUT = [440, 440] + + +# ---------------------------------------------------------------------------- +# Helpers +# ---------------------------------------------------------------------------- + +def sha256_of_file(path: Path, chunk: int = 1 << 20) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for block in iter(lambda: f.read(chunk), b""): + h.update(block) + return h.hexdigest() + + +def write_jpeg_square(path: Path, color: tuple[int, int, int], size: int = 256): + """Write a solid-color JPEG. Used as a synthetic reference photo.""" + from PIL import Image + img = Image.new("RGB", (size, size), color) + path.parent.mkdir(parents=True, exist_ok=True) + img.save(str(path), format="JPEG", quality=85) + + +# ---------------------------------------------------------------------------- +# Pack data model (matches src/types/wildlife.ts) +# ---------------------------------------------------------------------------- + +def make_individual_record( + individual_id: str, + name: Optional[str], + embedding_offset: int, + embedding_count: int, + reference_photos: list[str], + sex: Optional[str] = None, + life_stage: Optional[str] = None, + encounter_count: Optional[int] = None, +) -> dict: + return { + "id": individual_id, + "name": name, + "alternateId": None, + "sex": sex, + "lifeStage": life_stage, + "firstSeen": None, + "lastSeen": None, + "encounterCount": encounter_count if encounter_count is not None else embedding_count, + "embeddingCount": embedding_count, + "embeddingOffset": embedding_offset, + "referencePhotos": reference_photos, + "notes": None, + } + + +def make_manifest( + *, + species: str, + feature_class: str, + display_name: str, + description: Optional[str], + wildbook_url: str, + individual_count: int, + embedding_count: int, + embedding_dim: int, + miewid_name: str, + miewid_version: str, + miewid_hf_repo: Optional[str], + miewid_input_size: list, + miewid_normalize: dict, + detector_filename: str, + detector_config_relpath: str, + checksums: dict, + exported_by: Optional[str] = None, + search_query: Optional[str] = None, +) -> dict: + return { + "formatVersion": PACK_FORMAT_VERSION, + "species": species, + "featureClass": feature_class, + "displayName": display_name, + "description": description, + "wildbookInstanceUrl": wildbook_url, + "exportDate": dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"), + "exportedBy": exported_by, + "searchQuery": search_query, + "individualCount": individual_count, + "embeddingCount": embedding_count, + "embeddingDim": embedding_dim, + "embeddingModel": { + "name": miewid_name, + "version": miewid_version, + "huggingFaceRepo": miewid_hf_repo, + "inputSize": miewid_input_size, + "normalize": miewid_normalize, + }, + "detectorModel": { + "filename": detector_filename, + "configFile": detector_config_relpath, + }, + "checksums": checksums, + } + + +# ---------------------------------------------------------------------------- +# Pack assembly +# ---------------------------------------------------------------------------- + +def write_embeddings_bin(path: Path, embeddings: np.ndarray) -> None: + """Flat float32 little-endian NCHW(N×D) layout per spec.""" + assert embeddings.dtype == np.float32 + assert embeddings.ndim == 2 + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("wb") as f: + # numpy tofile uses native byte order on most platforms; force little-endian. + f.write(embeddings.astype(" list[str]: + """Self-check before zipping. Returns list of validation errors (empty = OK).""" + errors: list[str] = [] + manifest_path = pack_dir / "manifest.json" + if not manifest_path.exists(): + errors.append("manifest.json missing") + return errors + + manifest = json.loads(manifest_path.read_text()) + + # Required files + required = [ + "manifest.json", + f"models/{manifest['detectorModel']['filename']}", + manifest["detectorModel"]["configFile"], + "embeddings/index.json", + "embeddings/embeddings.bin", + ] + for rel in required: + if not (pack_dir / rel).exists(): + errors.append(f"missing required file: {rel}") + + # embeddings.bin size matches embeddingCount * embeddingDim * 4 + bin_path = pack_dir / "embeddings" / "embeddings.bin" + if bin_path.exists(): + expected = manifest["embeddingCount"] * manifest["embeddingDim"] * 4 + actual = bin_path.stat().st_size + if expected != actual: + errors.append(f"embeddings.bin size {actual} != expected {expected} " + f"({manifest['embeddingCount']} × {manifest['embeddingDim']} × 4)") + + # index.json offsets in bounds + index_path = pack_dir / "embeddings" / "index.json" + if index_path.exists(): + index = json.loads(index_path.read_text()) + if len(index) != manifest["individualCount"]: + errors.append(f"index has {len(index)} entries, manifest says {manifest['individualCount']}") + total_emb = 0 + for i, ind in enumerate(index): + if ind["embeddingOffset"] != total_emb: + errors.append(f"individual[{i}] {ind['id']}: offset {ind['embeddingOffset']} " + f"!= cumulative {total_emb}") + total_emb += ind["embeddingCount"] + if total_emb != manifest["embeddingCount"]: + errors.append(f"sum of individual embeddingCount ({total_emb}) " + f"!= manifest embeddingCount ({manifest['embeddingCount']})") + + # Reference photos in index point to files that exist + if index_path.exists(): + for ind in json.loads(index_path.read_text()): + for photo_name in ind["referencePhotos"]: + p = pack_dir / "reference_photos" / ind["id"] / photo_name + if not p.exists(): + errors.append(f"reference photo missing: {p}") + + return errors + + +# ---------------------------------------------------------------------------- +# Synthetic mode +# ---------------------------------------------------------------------------- + +def build_synthetic(args): + output_dir = Path(args.output_dir) + pack_id = f"{args.species}-{args.synthetic_label}-{dt.date.today().isoformat()}" + pack_dir = output_dir / pack_id + if pack_dir.exists(): + logger.info("Removing existing %s", pack_dir) + shutil.rmtree(pack_dir) + + rng = np.random.default_rng(args.seed) + + # Make individuals + per-individual unit-norm embeddings + colored-square photos + individuals = [] + all_embeddings: list[np.ndarray] = [] + reference_photos: dict[str, list[Path]] = {} + photo_staging = output_dir / f"_synthetic_photos_{pack_id}" + if photo_staging.exists(): + shutil.rmtree(photo_staging) + photo_staging.mkdir(parents=True) + + offset = 0 + for i in range(args.num_individuals): + individual_id = f"SYN-{i+1:04d}" + # Center direction per individual; per-embedding noise + center = rng.standard_normal(args.embedding_dim).astype(np.float32) + center /= (np.linalg.norm(center) + 1e-8) + # Per-dim noise scale: cos(center+n, center+n') ≈ 1 - σ²·d when both + # are L2-normalized. For d=2152 and target cos ≈ 0.995, σ ≈ 0.0015. + # Keep noise small relative to the curse-of-dimensionality effect. + sigma = 0.0015 + embeddings = [] + for _ in range(args.embeddings_per_individual): + v = center + sigma * rng.standard_normal(args.embedding_dim).astype(np.float32) + v /= (np.linalg.norm(v) + 1e-8) + embeddings.append(v) + emb_arr = np.stack(embeddings, axis=0) + all_embeddings.append(emb_arr) + + # Reference photos: one solid-color JPEG per photo + photo_paths = [] + # Hue spread across individuals so each gets a distinct color + hue = (i / max(args.num_individuals, 1)) * 360 + for j in range(args.photos_per_individual): + # Vary brightness slightly per photo + brightness = 0.7 + 0.2 * (j / max(args.photos_per_individual, 1)) + color = _hsv_to_rgb(hue, 0.7, brightness) + p = photo_staging / individual_id / f"ref_{j+1:02d}.jpg" + write_jpeg_square(p, color) + photo_paths.append(p) + + individuals.append(make_individual_record( + individual_id=individual_id, + name=f"Synthetic {i+1}", + embedding_offset=offset, + embedding_count=args.embeddings_per_individual, + reference_photos=[p.name for p in photo_paths], + sex=("male" if i % 2 == 0 else "female"), + life_stage="adult", + )) + reference_photos[individual_id] = photo_paths + offset += args.embeddings_per_individual + + embeddings_concat = np.concatenate(all_embeddings, axis=0).astype(np.float32) + + # Detector + config (real, not synthetic) + detector_path = Path(args.detector) + detector_config = json.loads(Path(args.detector_config).read_text()) + + # Embedding model normalize info — synthesized but spec-shaped + miewid_normalize = { + "mean": DEFAULT_IMAGENET_MEAN, + "std": DEFAULT_IMAGENET_STD, + } + + manifest_kwargs = dict( + species=args.species, + feature_class=args.feature_class, + display_name=args.display_name, + description=f"Synthetic development pack — {args.num_individuals} individuals × " + f"{args.embeddings_per_individual} embeddings (random unit vectors). " + "Detector is real; embeddings are NOT — do not use for actual matching.", + wildbook_url=args.wildbook_url, + miewid_name=args.miewid_name, + miewid_version=args.miewid_version, + miewid_hf_repo=args.miewid_hf_repo, + miewid_input_size=DEFAULT_MIEWID_INPUT, + miewid_normalize=miewid_normalize, + exported_by="build_pack.py (synthetic)", + search_query=None, + ) + + manifest = assemble_pack_dir( + pack_dir=pack_dir, + detector_path=detector_path, + detector_config=detector_config, + embeddings=embeddings_concat, + individuals=individuals, + reference_photos=reference_photos, + manifest_partial_kwargs=manifest_kwargs, + ) + + errors = validate_pack(pack_dir) + if errors: + logger.error("Validation errors:") + for e in errors: + logger.error(" - %s", e) + return 4 + logger.info("Pack validated OK.") + + # Zip + zip_path = output_dir / f"{pack_id}.zip" + if zip_path.exists(): + zip_path.unlink() + zip_pack(pack_dir, zip_path) + logger.info("Wrote %s (%.1f MB)", zip_path, zip_path.stat().st_size / 1e6) + + # Cleanup staging + if photo_staging.exists(): + shutil.rmtree(photo_staging) + + print() + print("=" * 72) + print(f"Synthetic pack ready: {zip_path}") + print(f" Individuals: {manifest['individualCount']}") + print(f" Embeddings: {manifest['embeddingCount']} × {manifest['embeddingDim']}") + print(f" Detector: {detector_path.name} ({detector_path.stat().st_size / 1e6:.1f} MB)") + print(f" Pack dir (kept): {pack_dir}") + print("=" * 72) + return 0 + + +def _hsv_to_rgb(h, s, v) -> tuple[int, int, int]: + """Tiny self-contained HSV→RGB so we don't need colorsys with nice clamping.""" + import colorsys + r, g, b = colorsys.hsv_to_rgb(h / 360.0, s, v) + return int(r * 255), int(g * 255), int(b * 255) + + +# ---------------------------------------------------------------------------- +# from-coco mode (skeleton — full impl in next iteration) +# ---------------------------------------------------------------------------- + +def build_from_coco(args): + """Real-data pack from a COCO export. Stub for now — needs Wildbook COCO + crop pipeline.""" + raise NotImplementedError( + "from-coco mode is the next iteration. The script structure supports it; " + "we still need: (1) a COCO JSON spec'd to horses.wildbook.org export shape, " + "(2) a crop+preprocess loop matching mobile's preprocessing, " + "(3) MiewID ONNX inference loop. Do this once we have a real Washington-horses COCO." + ) + + +# ---------------------------------------------------------------------------- +# CLI +# ---------------------------------------------------------------------------- + +def add_common_args(p: argparse.ArgumentParser): + p.add_argument("--species", required=True, help="e.g. horse") + p.add_argument("--feature-class", required=True, help="e.g. horse_wild+face") + p.add_argument("--display-name", required=True) + p.add_argument("--description", default=None) + p.add_argument("--wildbook-url", default="https://horses.wildbook.org") + p.add_argument("--detector", required=True, help="Path to detector .onnx") + p.add_argument("--detector-config", required=True, help="Path to detector.json") + p.add_argument("--miewid-name", default="miewid-v4") + p.add_argument("--miewid-version", default="4.1.0") + p.add_argument("--miewid-hf-repo", default="conservationxlabs/miewid-msv4") + p.add_argument("--embedding-dim", type=int, default=DEFAULT_EMBEDDING_DIM) + p.add_argument("--output-dir", default="output") + p.add_argument("--log-level", default="INFO") + + +def parse_args(argv=None): + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = p.add_subparsers(dest="mode", required=True) + + syn = sub.add_parser("synthetic", help="Build a synthetic dev pack with random embeddings") + add_common_args(syn) + syn.add_argument("--synthetic-label", default="synthetic", + help="Tag in the pack id, e.g. 'synthetic' or 'dev'") + syn.add_argument("--num-individuals", type=int, default=5) + syn.add_argument("--embeddings-per-individual", type=int, default=2) + syn.add_argument("--photos-per-individual", type=int, default=2) + syn.add_argument("--seed", type=int, default=42) + + coco = sub.add_parser("from-coco", help="Build a real pack from a COCO export") + add_common_args(coco) + coco.add_argument("--coco", required=True) + coco.add_argument("--images-dir", required=True) + coco.add_argument("--miewid", required=True, help="Path to MiewID ONNX (FP16 recommended)") + coco.add_argument("--max-photos-per-individual", type=int, default=3) + + return p.parse_args(argv) + + +def main(argv=None) -> int: + args = parse_args(argv) + logging.basicConfig( + level=getattr(logging, args.log_level.upper()), + format="%(asctime)s [%(levelname)s] %(message)s", + ) + + if args.mode == "synthetic": + return build_synthetic(args) + elif args.mode == "from-coco": + return build_from_coco(args) + else: + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/kb/wildlife-reid-mobile/tools/export_miewid_v41_onnx.py b/kb/wildlife-reid-mobile/tools/export_miewid_v41_onnx.py new file mode 100644 index 00000000..adce32e5 --- /dev/null +++ b/kb/wildlife-reid-mobile/tools/export_miewid_v41_onnx.py @@ -0,0 +1,590 @@ +#!/usr/bin/env python3 +""" +Export MiewID v4.1 PyTorch checkpoint to ONNX for on-device inference. + +Source artifacts: + /mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin (~206 MB FP32) + /mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.yaml (training config) + +Outputs (default --output-dir output/): + miewid_v4_1_fp32.onnx (~200 MB) + miewid_v4_1_fp16.onnx (~100 MB, only with --fp16) + miewid_v4_1_export_manifest.json (provenance + parity stats) + +Usage examples: + # Full export with FP16 conversion + parity tests + python export_miewid_v41_onnx.py --fp16 + + # Custom paths + python export_miewid_v41_onnx.py \\ + --checkpoint /path/to/miew_id.msv4_1_main.bin \\ + --config /path/to/miew_id.msv4_1_main.yaml \\ + --output-dir ./output \\ + --fp16 --parity-trials 100 + + # Skip export, only verify an already-exported ONNX file + python export_miewid_v41_onnx.py --validate-only \\ + --onnx output/miewid_v4_1_fp32.onnx + +Requirements: + pip install torch onnx onnxruntime onnxconverter-common timm pyyaml numpy + + Strongly preferred: wbia-plugin-miew-id installed in editable mode so the + canonical MiewIdNet class is used. The script falls back to a minimal + timm + GeM reconstruction if wbia_miew_id can't be imported, but the + canonical class guarantees byte-identical inference to the trainer. + +What this script does: + + 1. Load the YAML config + .bin checkpoint. + 2. Instantiate a MiewIdNet (or timm-fallback equivalent), then strip the + ArcFace classification head (we only need the 2152-dim embedding). + 3. Wrap in an EmbeddingOnly nn.Module that returns the L2-normalized feature. + 4. Export to ONNX (opset 17, dynamic batch dimension). + 5. Parity check: run N random batches through both PyTorch (FP32) and + ONNX Runtime (CPU EP), compute per-sample cosine similarity, and assert + all >= 0.9999. Reports min/mean/max + L2-distance stats. + 6. (Optional) Convert to FP16 via onnxconverter_common, re-run parity. + 7. SHA-256 every output. Write a JSON manifest with full provenance. + +Why "embedding only": + MiewID's training-time forward returns logits over n_classes (20,191 for + v4.1). For inference we only need the embedding before the ArcFace head; + exporting only the embedding path drops ~80 MB of unnecessary classification + weights and avoids exporting the head's specialized ops. +""" + +from __future__ import annotations + +import argparse +import dataclasses +import hashlib +import json +import logging +import sys +from pathlib import Path +from typing import Optional + +import numpy as np + +logger = logging.getLogger("export_miewid_v41") + + +# ---------------------------------------------------------------------------- +# Defaults derived from miew_id.msv4_1_main.yaml +# ---------------------------------------------------------------------------- + +DEFAULT_CHECKPOINT = "/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin" +DEFAULT_CONFIG = "/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.yaml" +DEFAULT_OUTPUT_DIR = "output" + +EXPECTED_EMBEDDING_DIM = 2152 # EfficientNetV2-RW-M GeM-pooled feature dim per pack spec +DEFAULT_INPUT_SIZE = (440, 440) + + +# ---------------------------------------------------------------------------- +# Helpers +# ---------------------------------------------------------------------------- + +def sha256_of_file(path: Path, chunk: int = 1 << 20) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for block in iter(lambda: f.read(chunk), b""): + h.update(block) + return h.hexdigest() + + +def load_yaml_config(path: Path) -> dict: + import yaml + with path.open("r") as f: + return yaml.safe_load(f) + + +# ---------------------------------------------------------------------------- +# Model loading +# ---------------------------------------------------------------------------- + +def build_model_canonical(model_params: dict): + """Use the canonical wbia-plugin-miew-id model class. Preferred path.""" + try: + # The wbia-plugin-miew-id repo ships a Python package usually called + # `wbia_miew_id`. The classification module is typically `MiewIdNet` + # under `wbia_miew_id.models`. Adjust if your install differs. + from wbia_miew_id.models import MiewIdNet + except ImportError as e: + raise ImportError( + "wbia_miew_id not installed. Install it from " + "https://github.com/WildMeOrg/wbia-plugin-miew-id " + "or rerun with the timm fallback (set MIEWID_FALLBACK_TIMM=1)." + ) from e + + model = MiewIdNet( + model_name=model_params["model_name"], + use_fc=model_params.get("use_fc", False), + fc_dim=model_params.get("fc_dim", 2048), + dropout=model_params.get("dropout", 0.0), + loss_module=model_params["loss_module"], + s=model_params["s"], + margin=model_params["margin"], + ls_eps=model_params.get("ls_eps", 0.0), + theta_zero=model_params.get("theta_zero", 0.785), + pretrained=False, # we'll load weights ourselves + n_classes=model_params["n_classes"], + k=model_params.get("k", 1), + ) + return model + + +def build_model_timm_fallback(model_params: dict): + """Minimal timm + GeM pooling reconstruction. + + Used only if wbia_miew_id can't be imported. This may NOT be byte-identical + to the canonical model — verify with the parity test before trusting. + """ + import timm + import torch + import torch.nn as nn + import torch.nn.functional as F + + backbone = timm.create_model( + model_params["model_name"], + pretrained=False, + num_classes=0, # drop classification head + global_pool="", # we'll add our own GeM + ) + + class GeM(nn.Module): + def __init__(self, p: float = 3.0, eps: float = 1e-6): + super().__init__() + self.p = nn.Parameter(torch.ones(1) * p) + self.eps = eps + + def forward(self, x): # (N, C, H, W) + return F.avg_pool2d( + x.clamp(min=self.eps).pow(self.p), + kernel_size=(x.size(-2), x.size(-1)), + ).pow(1.0 / self.p).flatten(1) + + class MiewIdMinimal(nn.Module): + def __init__(self, backbone, embedding_dim: int): + super().__init__() + self.backbone = backbone + self.global_pool = GeM() + self.bn = nn.BatchNorm1d(embedding_dim) + + def forward(self, x): + features = self.backbone.forward_features(x) + pooled = self.global_pool(features) + return self.bn(pooled) + + # match canonical class API for EmbeddingOnly to call + def extract_features(self, x): + return self.forward(x) + + # Probe feature dim from a dummy forward + backbone.eval() + with torch.no_grad(): + feats = backbone.forward_features(torch.zeros(1, 3, *DEFAULT_INPUT_SIZE)) + embedding_dim = feats.shape[1] + return MiewIdMinimal(backbone, embedding_dim) + + +def build_model(model_params: dict): + import os + if os.environ.get("MIEWID_FALLBACK_TIMM") == "1": + logger.warning("MIEWID_FALLBACK_TIMM=1 — using timm fallback reconstruction.") + return build_model_timm_fallback(model_params) + try: + return build_model_canonical(model_params) + except ImportError as e: + logger.warning("Canonical wbia_miew_id import failed (%s); using timm fallback.", e) + return build_model_timm_fallback(model_params) + + +def load_checkpoint(model, checkpoint_path: Path): + import torch + state = torch.load(checkpoint_path, map_location="cpu") + # Some checkpoints are dicts with 'model_state_dict'; handle both shapes. + if isinstance(state, dict) and "model_state_dict" in state: + state = state["model_state_dict"] + elif isinstance(state, dict) and "state_dict" in state: + state = state["state_dict"] + + incompatible = model.load_state_dict(state, strict=False) + missing = list(incompatible.missing_keys) + unexpected = list(incompatible.unexpected_keys) + + # Expect ArcFace head keys to be 'unexpected' since we strip the head later. + head_keylike = ("final.", "arcface", "loss_module", "head.") + benign_unexpected = [k for k in unexpected if any(t in k.lower() for t in head_keylike)] + surprising_unexpected = [k for k in unexpected if k not in benign_unexpected] + + logger.info("Loaded checkpoint: missing=%d, unexpected=%d (%d benign head keys)", + len(missing), len(unexpected), len(benign_unexpected)) + if missing: + logger.warning("Missing keys (first 5): %s", missing[:5]) + if surprising_unexpected: + logger.warning("Surprising unexpected keys (first 5): %s", surprising_unexpected[:5]) + + return {"missing": missing, "unexpected": unexpected, + "benign_unexpected": benign_unexpected, + "surprising_unexpected": surprising_unexpected} + + +# ---------------------------------------------------------------------------- +# Embedding-only wrapper +# ---------------------------------------------------------------------------- + +def make_embedding_only(model): + """Wrap the model so forward() returns ONLY the embedding, no logits.""" + import torch + import torch.nn as nn + + class EmbeddingOnly(nn.Module): + def __init__(self, m): + super().__init__() + self.m = m + + def forward(self, x): + # Try canonical extract_features first, then various fallbacks. + for method in ("extract_features", "embedding", "forward_features"): + fn = getattr(self.m, method, None) + if callable(fn): + return fn(x) + # Last resort: assume forward(x) returns (embedding, logits) tuple + out = self.m(x) + if isinstance(out, (tuple, list)) and len(out) >= 1: + return out[0] + return out + + wrapper = EmbeddingOnly(model) + wrapper.eval() + return wrapper + + +# ---------------------------------------------------------------------------- +# ONNX export +# ---------------------------------------------------------------------------- + +def export_onnx(wrapper, output_path: Path, opset: int = 17, + input_size=DEFAULT_INPUT_SIZE) -> dict: + """Export to ONNX, then consolidate any external-data weights into a single file. + + PyTorch's dynamo exporter (default in torch>=2.10) can split large models into + `model.onnx` (graph) + `model.onnx.data` (weights). Mobile pack format expects + a single self-contained `.onnx`; we re-save with `save_as_external_data=False` + after export so downstream consumers don't have to think about it. + """ + import onnx + import torch + h, w = input_size + dummy = torch.randn(1, 3, h, w) + + output_path.parent.mkdir(parents=True, exist_ok=True) + torch.onnx.export( + wrapper, + dummy, + str(output_path), + input_names=["input"], + output_names=["embedding"], + opset_version=opset, + dynamic_axes={ + "input": {0: "batch"}, + "embedding": {0: "batch"}, + }, + do_constant_folding=True, + ) + + # Consolidate external data (if any) back into the .onnx file + data_path = output_path.with_suffix(output_path.suffix + ".data") + if data_path.exists(): + logger.info("Consolidating external weights from %s into %s", data_path, output_path) + model = onnx.load(str(output_path)) + # save_as_external_data=False forces all initializer tensors back into the model proto + onnx.save_model(model, str(output_path), save_as_external_data=False) + data_path.unlink() + + return {"path": str(output_path), "opset": opset, "input_size": [h, w]} + + +# ---------------------------------------------------------------------------- +# Parity check +# ---------------------------------------------------------------------------- + +@dataclasses.dataclass +class ParityResult: + trials: int + cosine_min: float + cosine_mean: float + cosine_max: float + l2_max: float + embedding_dim: int + embedding_dim_matches_expected: bool + all_pass: bool + threshold: float + + +def _cosine(a: np.ndarray, b: np.ndarray, eps: float = 1e-12) -> np.ndarray: + """Per-row cosine similarity. a,b shape: (N, D).""" + an = a / (np.linalg.norm(a, axis=1, keepdims=True) + eps) + bn = b / (np.linalg.norm(b, axis=1, keepdims=True) + eps) + return (an * bn).sum(axis=1) + + +def parity_check(wrapper, onnx_path: Path, + trials: int = 50, + batch: int = 4, + input_size=DEFAULT_INPUT_SIZE, + cosine_threshold: float = 0.9999, + seed: int = 42) -> ParityResult: + import torch + import onnxruntime as ort + + rng = np.random.default_rng(seed) + h, w = input_size + + sess = ort.InferenceSession(str(onnx_path), providers=["CPUExecutionProvider"]) + input_name = sess.get_inputs()[0].name + output_name = sess.get_outputs()[0].name + + cos_all: list[float] = [] + l2_all: list[float] = [] + emb_dim: Optional[int] = None + + wrapper.eval() + with torch.no_grad(): + for _ in range(trials): + x_np = rng.standard_normal((batch, 3, h, w), dtype=np.float32) + x_torch = torch.from_numpy(x_np) + + torch_emb = wrapper(x_torch).cpu().numpy() + onnx_emb = sess.run([output_name], {input_name: x_np})[0] + + if emb_dim is None: + emb_dim = torch_emb.shape[-1] + + assert torch_emb.shape == onnx_emb.shape, ( + f"Shape mismatch: torch {torch_emb.shape} vs onnx {onnx_emb.shape}") + + cos = _cosine(torch_emb, onnx_emb) + l2 = np.linalg.norm(torch_emb - onnx_emb, axis=1) + cos_all.extend(cos.tolist()) + l2_all.extend(l2.tolist()) + + return ParityResult( + trials=trials * batch, + cosine_min=float(min(cos_all)), + cosine_mean=float(sum(cos_all) / len(cos_all)), + cosine_max=float(max(cos_all)), + l2_max=float(max(l2_all)), + embedding_dim=emb_dim or -1, + embedding_dim_matches_expected=(emb_dim == EXPECTED_EMBEDDING_DIM), + all_pass=min(cos_all) >= cosine_threshold, + threshold=cosine_threshold, + ) + + +# ---------------------------------------------------------------------------- +# FP16 conversion +# ---------------------------------------------------------------------------- + +def convert_fp16(fp32_path: Path, fp16_path: Path) -> None: + import onnx + from onnxconverter_common import float16 + model = onnx.load(str(fp32_path)) + fp16_model = float16.convert_float_to_float16(model, keep_io_types=True) + onnx.save(fp16_model, str(fp16_path)) + + +def parity_check_fp16(wrapper, fp16_path: Path, + trials: int = 25, + batch: int = 4, + input_size=DEFAULT_INPUT_SIZE, + cosine_threshold: float = 0.999, + seed: int = 43) -> ParityResult: + """FP16 has slightly looser parity (1e-3 vs 1e-4).""" + return parity_check( + wrapper, fp16_path, + trials=trials, batch=batch, input_size=input_size, + cosine_threshold=cosine_threshold, seed=seed, + ) + + +# ---------------------------------------------------------------------------- +# Main +# ---------------------------------------------------------------------------- + +def parse_args(argv=None): + p = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--checkpoint", default=DEFAULT_CHECKPOINT, + help="Path to miew_id.msv4_1_main.bin") + p.add_argument("--config", default=DEFAULT_CONFIG, + help="Path to miew_id.msv4_1_main.yaml") + p.add_argument("--output-dir", default=DEFAULT_OUTPUT_DIR, + help="Where to write ONNX + manifest") + p.add_argument("--opset", type=int, default=17, + help="ONNX opset version (>= 11 required for ORT Mobile)") + p.add_argument("--input-size", type=int, nargs=2, default=list(DEFAULT_INPUT_SIZE), + metavar=("H", "W")) + p.add_argument("--fp16", action="store_true", + help="Also produce FP16 ONNX (recommended for mobile)") + p.add_argument("--parity-trials", type=int, default=50, + help="Number of random batches for FP32 parity check") + p.add_argument("--validate-only", action="store_true", + help="Skip export; run parity check on an existing ONNX") + p.add_argument("--onnx", default=None, + help="(--validate-only) Path to existing ONNX to verify") + p.add_argument("--log-level", default="INFO") + return p.parse_args(argv) + + +def main(argv=None) -> int: + args = parse_args(argv) + logging.basicConfig( + level=getattr(logging, args.log_level.upper()), + format="%(asctime)s [%(levelname)s] %(message)s", + ) + + checkpoint = Path(args.checkpoint) + config_path = Path(args.config) + output_dir = Path(args.output_dir) + + if not checkpoint.is_file(): + logger.error("Checkpoint not found: %s", checkpoint) + return 2 + if not config_path.is_file(): + logger.error("Config not found: %s", config_path) + return 2 + + cfg = load_yaml_config(config_path) + model_params = cfg["model_params"] + logger.info("Loaded config: model=%s n_classes=%d use_fc=%s", + model_params["model_name"], + model_params["n_classes"], + model_params.get("use_fc", False)) + + logger.info("Building model...") + model = build_model(model_params) + + logger.info("Loading checkpoint %s", checkpoint) + load_report = load_checkpoint(model, checkpoint) + + wrapper = make_embedding_only(model) + + # Validate-only short-circuit + if args.validate_only: + if not args.onnx: + logger.error("--validate-only requires --onnx PATH") + return 2 + onnx_path = Path(args.onnx) + logger.info("Running parity check on %s", onnx_path) + result = parity_check(wrapper, onnx_path, + trials=args.parity_trials, + input_size=tuple(args.input_size)) + print(json.dumps(dataclasses.asdict(result), indent=2)) + return 0 if result.all_pass else 1 + + # Export + fp32_path = output_dir / "miewid_v4_1_fp32.onnx" + logger.info("Exporting FP32 ONNX → %s (opset=%d, input=%s)", + fp32_path, args.opset, args.input_size) + export_info = export_onnx(wrapper, fp32_path, + opset=args.opset, + input_size=tuple(args.input_size)) + + logger.info("Running FP32 parity check (%d trials)...", args.parity_trials) + fp32_parity = parity_check(wrapper, fp32_path, + trials=args.parity_trials, + input_size=tuple(args.input_size)) + logger.info("FP32 parity: cos_min=%.6f cos_mean=%.6f l2_max=%.4e dim=%d pass=%s", + fp32_parity.cosine_min, fp32_parity.cosine_mean, + fp32_parity.l2_max, fp32_parity.embedding_dim, + fp32_parity.all_pass) + if not fp32_parity.all_pass: + logger.error("FP32 parity FAILED (cosine_min=%.6f < threshold=%.4f). " + "Check that the model class matches the checkpoint and that " + "EmbeddingOnly is calling the right inference path.", + fp32_parity.cosine_min, fp32_parity.threshold) + return 3 + if not fp32_parity.embedding_dim_matches_expected: + logger.warning("Embedding dim is %d, expected %d (per pack spec). " + "If you change embedding_dim, update EmbeddingPackManifest.", + fp32_parity.embedding_dim, EXPECTED_EMBEDDING_DIM) + + # Optional FP16 + fp16_path: Optional[Path] = None + fp16_parity: Optional[ParityResult] = None + if args.fp16: + fp16_path = output_dir / "miewid_v4_1_fp16.onnx" + logger.info("Converting to FP16 → %s", fp16_path) + convert_fp16(fp32_path, fp16_path) + logger.info("Running FP16 parity check (looser threshold 0.999)...") + fp16_parity = parity_check_fp16(wrapper, fp16_path, + input_size=tuple(args.input_size)) + logger.info("FP16 parity: cos_min=%.6f cos_mean=%.6f l2_max=%.4e pass=%s", + fp16_parity.cosine_min, fp16_parity.cosine_mean, + fp16_parity.l2_max, fp16_parity.all_pass) + if not fp16_parity.all_pass: + logger.warning("FP16 parity below threshold — embeddings may diverge " + "from PyTorch reference. Consider sticking with FP32.") + + # Manifest + manifest = { + "schema_version": "1", + "exported_at": __import__("datetime").datetime.utcnow().isoformat() + "Z", + "source": { + "checkpoint": str(checkpoint), + "checkpoint_sha256": sha256_of_file(checkpoint), + "config": str(config_path), + "model_name": model_params["model_name"], + "n_classes": model_params["n_classes"], + "loss_module": model_params["loss_module"], + "use_fc": model_params.get("use_fc", False), + }, + "load_report": { + "missing_keys": len(load_report["missing"]), + "unexpected_keys": len(load_report["unexpected"]), + "benign_head_keys": len(load_report["benign_unexpected"]), + "surprising_unexpected_keys": load_report["surprising_unexpected"][:10], + }, + "export": { + "opset": args.opset, + "input_size": list(args.input_size), + "fp32": { + "path": str(fp32_path), + "sha256": sha256_of_file(fp32_path), + "size_bytes": fp32_path.stat().st_size, + "parity": dataclasses.asdict(fp32_parity), + }, + }, + } + if fp16_path is not None: + manifest["export"]["fp16"] = { + "path": str(fp16_path), + "sha256": sha256_of_file(fp16_path), + "size_bytes": fp16_path.stat().st_size, + "parity": dataclasses.asdict(fp16_parity), + } + + manifest_path = output_dir / "miewid_v4_1_export_manifest.json" + manifest_path.write_text(json.dumps(manifest, indent=2)) + logger.info("Wrote manifest: %s", manifest_path) + + # Summary + print() + print("=" * 72) + print("Export complete.") + print(f" FP32: {fp32_path} ({fp32_path.stat().st_size / 1e6:.1f} MB)") + if fp16_path is not None: + print(f" FP16: {fp16_path} ({fp16_path.stat().st_size / 1e6:.1f} MB)") + print(f" Manifest: {manifest_path}") + print(f" FP32 parity: cosine_min={fp32_parity.cosine_min:.6f} (>= {fp32_parity.threshold})") + if fp16_parity is not None: + print(f" FP16 parity: cosine_min={fp16_parity.cosine_min:.6f} (>= {fp16_parity.threshold})") + print("=" * 72) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/kb/wildlife-reid-mobile/wiki/bugs/critical-bugs.md b/kb/wildlife-reid-mobile/wiki/bugs/critical-bugs.md new file mode 100644 index 00000000..35fb89e5 --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/bugs/critical-bugs.md @@ -0,0 +1,92 @@ +# Critical Bugs (Pre-MiewID Integration) + +**Category:** bugs +**Sources:** [[codex-review-2026-04-22]], direct read of `src/services/onnxInferenceService/preprocessing.ts`, native `ImageTensorModule` on both platforms +**Related:** [[image-preprocessing]], [[on-device-reid-pipeline]], [[miewid-v41-integration-plan]] +**Last updated:** 2026-04-22 + +## Summary + +Two real preprocessing bugs will produce garbage embeddings and skewed detections even after MiewID is wired. They must be fixed first and covered by a golden parity test. Both were independently confirmed via file reads. + +## Bug 1 — Native `scale` is divided instead of multiplied + +**Files:** +- `android/app/src/main/java/ai/offgridmobile/imagetensor/ImageTensorModule.kt:62` +- `ios/ImageTensorModule.swift:188` + +**Current code (Android):** +```kotlin +output[rIdx * h * w + i] = (r / scale - mean[0]) / std[0] +``` + +**Spec (`docs/EMBEDDING_PACK_FORMAT.md:166`):** +> `normalize.scale` … `1/255 = 0.00392156862` converts uint8 [0-255] to float [0-1]. Set to `1.0` if the model expects [0-255] input. + +With a pack that ships `scale = 0.00392…` (correct per spec), the native code computes `r / 0.00392 = r * 255`, pushing values to [0, 65025] — completely out of range for any normalization. + +**Fix:** +```kotlin +output[rIdx * h * w + i] = (r * scale - mean[0]) / std[0] +``` +(Mirror in Swift.) + +## Bug 2 — MiewID TS wrapper passes `scale = 1.0` + +**File:** `src/services/onnxInferenceService/preprocessing.ts:62-70` + +```ts +const rawArray = await ImageTensorModule.imageToTensor( + imageUri, width, height, + norm.mean, norm.std, + 1.0, // ← should be 1.0/255.0 + 'RGB', +); +``` + +Combined with the native bug, result is `r / 1.0 = r` (0-255), then `(r - 0.485) / 0.229` which yields values in [~-2.1, ~+1110]. MiewID expects inputs in roughly [-2.1, +2.7]. Embeddings are garbage → matching is meaningless. + +**Fix after Bug 1 is resolved:** +```ts +1.0 / 255.0, +``` + +## Bug 3 (Architectural) — MiewID model path never populated + +**Files:** +- `src/stores/wildlifeStore.ts:56,162` — `setMiewidModelPath` exists +- `src/screens/CaptureScreen/useCaptureFlow.ts:90` — early-returns when null +- **No call site ever populates it.** + +The spec says MiewID is a separately downloaded shared model (`docs/EMBEDDING_PACK_FORMAT.md:151,549`). Download flow doesn't exist. See [[model-acquisition]]. + +## Bug 4 (Watch) — YOLO parser assumes v8/v11 layout + +**File:** `src/services/onnxInferenceService/postprocessing.ts:143` +```ts +const numRows = 4 + numClasses; +``` + +Correct for YOLOv8/11 (no objectness). Wrong for legacy YOLOv5/v7. Low risk **if** we enforce v8+ detectors in packs. See [[yolo-postprocessing]] for branching strategy. + +## Suggested Test Additions + +1. **Golden parity test** (new): fix a test JPEG → load via native `imageToTensor(scale=1/255, mean=ImageNet, std=ImageNet)` → compare to a NumPy-generated reference tensor. ±1e-4 per element. +2. **Schema-driven test** (new): fabricate a `DetectorConfig` with `scale=1/255` and assert a solid-red 255/0/0 image lands at roughly `R=[(1-0.485)/0.229, ...]` after preprocessing. +3. **Update existing** `__tests__/unit/services/onnxInferenceService.test.ts` to expect the corrected semantics. + +## Remediation Order (Recommended) + +1. Land native fix (multiply instead of divide) + Swift mirror (one PR). +2. Land TS MiewID fix (`1/255`) + test updates (one PR). +3. Add golden parity test (one PR, can be same as #2). +4. Branch YOLO parser on architecture (defensive; separate PR). +5. Proceed to MiewID acquisition + wiring ([[miewid-v41-integration-plan]]). + +## Status + +- [x] **Fix Bug 1 (native `*` vs `/`)** — done 2026-04-23 on branch `fix/preprocessing-scale-math`. Kotlin verified green (11/11 tests); Swift mirrored, needs macOS/CI to verify. +- [x] **Fix Bug 2 (TS scale)** — done 2026-04-23. 410/410 Jest tests pass. +- [x] **Golden parity test** — cross-platform fixture added: 1×1 pure-red pixel through ImageNet norm + scale=1/255 → `R≈2.2489, G≈-2.0357, B≈-1.8044`. Encoded identically in Kotlin (`bitmapToNchw MiewID parity fixture — solid red 255 with ImageNet norm`) and Swift (`testExtractNchwMiewIDParityFixtureSolidRed`). +- [ ] YOLO architecture branching (deferred — only matters if a pack ever ships a YOLOv5/v7 detector) +- [ ] MiewID path populated end-to-end (Stage 2 of [[miewid-v41-integration-plan]]) diff --git a/kb/wildlife-reid-mobile/wiki/concepts/embedding-pack.md b/kb/wildlife-reid-mobile/wiki/concepts/embedding-pack.md new file mode 100644 index 00000000..24840dce --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/concepts/embedding-pack.md @@ -0,0 +1,64 @@ +# Embedding Pack + +**Category:** concept +**Sources:** [[docs-live/EMBEDDING_PACK_FORMAT.md]], [[docs-live/plans/2026-02-25-wildlife-reid-design.md]] +**Related:** [[miewid-v4]], [[on-device-reid-pipeline]], [[off-grid-mobile-app]] +**Last updated:** 2026-04-22 + +## Summary + +An **embedding pack** is a self-contained zip bundling a species-specific detector, pre-computed MiewID embeddings for all known individuals, reference photos for review, and metadata. Exported from a Wildbook instance via an Encounter Search export; consumed by the mobile app to enable offline individual re-identification for a single (species, feature class). + +## Layout + +``` +{species}-{context}-{YYYY-MM}.zip +├── manifest.json +├── models/{detector-filename}.onnx +├── embeddings/ +│ ├── index.json +│ └── embeddings.bin # flat float32, little-endian, NCHW layout +├── reference_photos/{individual-id}/ref_NN.jpg +└── config/detector.json +``` + +## manifest.json (key fields) + +- `formatVersion` (currently "1.0") +- `species`, `featureClass` (e.g., `horse+face`), `displayName` +- `wildbookInstanceUrl` — sync target +- `individualCount`, `embeddingCount`, `embeddingDim: 2152` (MiewID v4) +- `embeddingModel` — `name`, `version` (e.g., `4.0.0`), `huggingFaceRepo`, `inputSize: [440, 440]`, `normalize.{mean,std}` +- `detectorModel.filename`, `detectorModel.configFile` +- `checksums` — SHA-256 per critical file + +## config/detector.json (key fields) + +- `architecture` ("yolo11", "yolov8", "efficientdet", …) +- `inputSize`, `inputChannels: 3`, `channelOrder` ("RGB"/"BGR") +- `normalize.{mean, std, scale}` — where `scale` is intended as a **multiplier** (`1/255 = 0.00392156862`) applied **before** mean/std +- `confidenceThreshold`, `nmsThreshold`, `maxDetections` +- `outputFormat`, `classLabels` +- `outputSpec.{boxFormat: xyxy|xywh|cxcywh, coordinateType: normalized|absolute, layout}` + +## embeddings.bin + +- Flat float32, little-endian +- Individual *i*'s embeddings start at byte offset `individual.embeddingOffset * embeddingDim * 4` +- One individual may have multiple embeddings (multi-encounter) — count via `embeddingCount` in `index.json` + +## Important Design Decisions + +- **MiewID is NOT in the pack.** It's downloaded separately, since the same model works across all species. Packs reference the MiewID version so the app can warn on mismatch. +- **Detector IS in the pack.** Species-specific (e.g., different YOLO per feature class). +- **Reference photos are JPEG, ≤ 512×512**, ordered best-first. +- **Packs are content-addressed via SHA-256 checksums.** + +## Open Questions / Contradictions + +- **`normalize.scale` semantics.** The spec intends it as a **multiplier** (value `1/255`), but the native Kotlin (`android/.../ImageTensorModule.kt:62`) and Swift (`ios/ImageTensorModule.swift:188`) implementations **divide** by `scale`. With a pack that ships `scale: 0.00392…`, division yields `pixel × 255` — catastrophically wrong. See [[critical-bugs]]. Resolution: either (a) change native code to multiply, or (b) amend spec to clarify. Preferred: change native code + add golden parity test. + +## References + +- `docs/EMBEDDING_PACK_FORMAT.md` +- `docs/plans/2026-02-25-wildlife-reid-design.md` §3 Data Model diff --git a/kb/wildlife-reid-mobile/wiki/concepts/on-device-reid-pipeline.md b/kb/wildlife-reid-mobile/wiki/concepts/on-device-reid-pipeline.md new file mode 100644 index 00000000..9a394645 --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/concepts/on-device-reid-pipeline.md @@ -0,0 +1,79 @@ +# On-Device Re-ID Pipeline + +**Category:** concept +**Sources:** [[docs-live/plans/2026-02-25-wildlife-reid-design.md]], [[codebase-map]] +**Related:** [[off-grid-mobile-app]], [[miewid-v4]], [[embedding-pack]], [[image-preprocessing]], [[yolo-postprocessing]] +**Last updated:** 2026-04-22 + +## Summary + +Offline detect-then-embed-then-match pipeline running entirely on-device via ONNX Runtime. User takes a photo, the app runs each loaded species' detector, crops each detection, extracts a 2152-dim MiewID embedding, ranks against an in-memory database (pack + local individuals), and shows top-5 candidates for human review. Reviewed results are queued for later Wildbook sync. + +## Stages + +``` +1. CAPTURE + photoUri saved, GPS (stub) + timestamp recorded + +2. DETECT (per loaded species detector) + - Preprocess image → NCHW Float32 tensor (native ImageTensorModule) + - ONNX session.run() + - Parse output (YOLOv8/11 layout: 4+numClasses rows × N detections) + - Confidence filter → NMS → slice to maxDetections + +3. CROP & EMBED (per detection) + - Native cropImage() → JPEG on disk + - Preprocess 440×440 ImageNet norm + - MiewID ONNX session.run() → Float32Array[2152] + +4. MATCH (per detection) + - cosineSimilarity vs. database + · pack individuals (multi-embedding per individual; take best score) + · local individuals (same) + - Sort desc, top-5 → MatchCandidate[] + +5. REVIEW (human, async) + - Approve → attach embedding to approved individual (accumulates) + - No-Match → create FIELD-XXX LocalIndividual with this embedding + - Skip → leave pending + +6. SYNC (when online, not implemented) + - syncQueue → Wildbook REST API + - Detection → Encounter; LocalIndividual → MarkedIndividual +``` + +## Key Code Paths + +| Stage | File | +|---|---| +| Orchestration | `src/services/wildlifePipeline/index.ts` | +| Detection inference | `src/services/onnxInferenceService/index.ts` → `runDetection` | +| Detection preproc | `src/services/onnxInferenceService/preprocessing.ts` | +| Detection postproc / NMS | `src/services/onnxInferenceService/postprocessing.ts` | +| Crop | Native `ImageTensorModule.cropImage` | +| Embedding inference | `src/services/onnxInferenceService/index.ts` → `extractEmbedding` | +| Embedding preproc | `src/services/onnxInferenceService/preprocessing.ts` | +| Matching | `src/services/embeddingMatchService/index.ts` | +| DB assembly | `src/services/embeddingDatabaseBuilder.ts` | +| User flow | `src/screens/CaptureScreen/useCaptureFlow.ts` | +| Review | `src/screens/MatchReviewScreen/index.tsx` | + +## Performance Budget (flagship phone, from feasibility) + +| Stage | Target | +|---|---| +| Detection (YOLO nano/small, quantized) | 20-80ms | +| MiewID embedding (FP16) | 100-500ms | +| Vector search (10K individuals) | < 50ms | + +## Scale Notes + +- Brute-force cosine sim is fine for < ~5K individuals. For larger galleries, migrate to sqlite-vec / ObjectBox / FAISS-CPU. +- Storage: ~550 MB total for a realistic deployment (detector 15 MB + MiewID 100 MB FP16 + 10K individuals × 5 embeddings × 2152 × 4 B ≈ 430 MB). + +## Known Issues + +- Preprocessing scale math — see [[critical-bugs]] +- MiewID model is never loaded in production flow — see [[state-of-implementation]] +- Pack candidate names/photos not resolved in review UI +- GPS returns null; sync is a stub diff --git a/kb/wildlife-reid-mobile/wiki/entities/miewid-v4.md b/kb/wildlife-reid-mobile/wiki/entities/miewid-v4.md new file mode 100644 index 00000000..91086329 --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/entities/miewid-v4.md @@ -0,0 +1,77 @@ +# MiewID v4 + +**Category:** entity +**Sources:** [[docs-live/WILDLIFE_REID_FEASIBILITY.md]], [[docs-live/EMBEDDING_PACK_FORMAT.md]], miewid-trainer skill (MODEL_DATA.md reference) +**Related:** [[embedding-pack]], [[on-device-reid-pipeline]], [[image-preprocessing]], [[miewid-v41-integration-plan]] +**Last updated:** 2026-04-22 + +## Summary + +MiewID is Wild Me / Conservation X Labs' multispecies animal re-identification model. **v4** (Jan 2026) covers ~90 species and ~110 feature classes with ArcFace subcenter dynamic-margin training on ~20K individuals. Used to produce 2152-dim L2-normalizable embedding vectors from 440×440 animal crops; cosine similarity against a gallery of known individuals drives re-ID. + +## Architecture & Profile + +| Property | Value | +|---|---| +| Backbone | EfficientNetV2-RW-M (timm) | +| Parameters | 51.1M | +| GMACs | 24.38 | +| Input | 440×440 RGB, ImageNet-normalized (mean `[0.485, 0.456, 0.406]`, std `[0.229, 0.224, 0.225]`) | +| Pooling | GeM (Generalized Mean Pooling), learnable p=3 | +| Output | 2,152-dim embedding (batch-normalized) | +| Loss | Sub-center ArcFace with dynamic margins (s=49.3, m=0.33, k=2) | +| Framework | PyTorch (timm + HuggingFace transformers) | +| Distribution | Safetensors on HuggingFace | +| Size estimates | ~200 MB FP32, ~100 MB FP16, ~50 MB INT8 | +| Accuracy (v4) | 78% top-1, 87% top-5, 89% top-10 on held-out test data | + +## Versions + +| Version | Source | Species / n_classes | Notes | +|---|---|---|---| +| v2 | `conservationxlabs/miewid-msv2` | 54 species | | +| v3 | `conservationxlabs/miewid-msv3` | 64 species | Common zero-shot baseline | +| v4 (main) | `conservationxlabs/miewid-msv4`; local `/mnt/c/claude-skills/models/reference/miew_id.msv4_v3_main.bin` | 19,911 classes | Current production; includes deer | +| **v4.1 (main)** | **Local: `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin`** (md5 `71c0bdd9bf8cbdb1b9e967a2e96949bb`, 206,040,591 B) + `.yaml` | **20,191 classes** (+280 over v4 main) | **Target for this project.** Same architecture / hyperparameters as v4 main — additive retrain. See [[2026-04-23-miewid-v41-checkpoint]]. | + +## Relation to this project + +The Off Grid Mobile fork targets MiewID **v4.1** on-device. The embedding pack format (`embeddingModel.version`) is explicitly versioned so the app can version-gate packs. `embeddingDim: 2152` is baked into the pack spec ([[embedding-pack]]). + +### Current wiring status + +MiewID is **not yet operationally wired**: +- `extractEmbedding()` in `src/services/onnxInferenceService/index.ts` is generic and would call ONNX Runtime if given a model path, but +- `wildlifeStore.miewidModelPath` is never populated by any call site +- The spec-expected flow (download MiewID separately from pack, version-gate against `embeddingModel.version`) is not implemented +- See [[state-of-implementation]] and [[critical-bugs]]. + +## ONNX Export Path + +1. `torch.onnx.export()` from PyTorch checkpoint (opset ≥ 11 for ONNX Runtime Mobile) +2. Verify numerical parity against Python reference on a fixed batch of crops (rank-order parity > cosine-closeness) +3. FP16 for mobile first (~100 MB). INT8 only after rank-order parity is proven on real wildlife crops. +4. Distribution: separate download, not bundled in the app binary — the same MiewID works across all species packs, and the app should warn when pack `embeddingModel.version` doesn't match loaded MiewID. + +## Literature Context (from miewid-trainer skill) + +- **EfficientNetV2-M backbone** outperforms SwinV2-Base by ~4.5% on multispecies Re-ID (fine-grained CNNs still competitive). +- **Sub-center ArcFace with dynamic margins** differentiates MiewID from MegaDescriptor (which uses standard ArcFace m=0.5, s=64). +- GeM pooling validated as standard. +- 440×440 square input is optimal for fine-grained Re-ID. +- Multi-species training provides ~12.5% avg improvement over single-species. + +## Open Questions + +- ~~**License** — MiewID weights have no explicit open-source license~~. **Resolved 2026-04-23**: Jason (CXL employee) signed off on mobile redistribution of the v4.1 ONNX export. The PyTorch `.bin` is not distributed directly; we ship a derivative ONNX. +- **v4.1 public availability** — checkpoint exists locally (see above); not confirmed published to a public HF repo. For app distribution we need either (a) a public HF mirror or (b) a CXL-hosted CDN URL for the ONNX export. +- **INT8 quantization stability** — untested for re-ID; start FP16. +- **Core ML / NNAPI delegate parity** — if using EP-accelerated inference on iOS Neural Engine / Android NNAPI, verify embeddings are bitwise close to CPU/ORT reference. + +## References + +- `docs/WILDLIFE_REID_FEASIBILITY.md` §MiewID Technical Profile +- `docs/EMBEDDING_PACK_FORMAT.md` §manifest.json (embeddingModel block) +- miewid-trainer skill MODEL_DATA.md (Wild Me reference checkpoints) +- arxiv 2412.05602 (MiewID v2 paper) +- Beyan et al. 2026 Information Fusion 133 — animal Re-ID survey diff --git a/kb/wildlife-reid-mobile/wiki/entities/off-grid-mobile-app.md b/kb/wildlife-reid-mobile/wiki/entities/off-grid-mobile-app.md new file mode 100644 index 00000000..84df8d60 --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/entities/off-grid-mobile-app.md @@ -0,0 +1,74 @@ +# Off Grid Mobile (Wildlife Re-ID Fork) + +**Category:** entity +**Sources:** [[docs-live/WILDLIFE_REID_FEASIBILITY.md]], [[docs-live/plans/2026-02-25-wildlife-reid-design.md]], [[docs-live/plans/2026-02-25-wildlife-reid-implementation.md]], [[codebase-map]] +**Related:** [[miewid-v4]], [[embedding-pack]], [[on-device-reid-pipeline]], [[state-of-implementation]] +**Last updated:** 2026-04-22 + +## Summary + +`/mnt/c/off-grid-mobile` is a React Native 0.83 / TypeScript app originally built as an on-device LLM / image-gen / voice tool. It is being repurposed as an offline wildlife individual re-identification platform: capture photo → species detector (YOLO) → crop → MiewID embedding → cosine match against embedding pack + local individuals → human review → queue for Wildbook sync. Chat / LLM / image-gen / voice / tools modules were stripped in commit `a63d662`. + +## Stack + +| Layer | Choice | +|---|---| +| UI | React Native 0.83, React 19.2, Zustand 5, React Navigation 7 | +| ML runtime | `onnxruntime-react-native` | +| Native image ops | Custom Kotlin `ImageTensorModule` + Swift `ImageTensorModule` | +| Persistence | Zustand → AsyncStorage; files on RNFS DocumentDirectory | +| Testing | Jest, RNTL, Maestro (E2E) | +| CI | Gemini, Codecov, SonarCloud | + +## Architecture (wildlife pipeline) + +See [[on-device-reid-pipeline]] for the full flow. Key service modules: + +| Service | Purpose | Path | +|---|---|---| +| `onnxInferenceService` | Detector + embedding ONNX sessions | `src/services/onnxInferenceService/` | +| `wildlifePipeline` | Orchestrate detect → crop → embed → match | `src/services/wildlifePipeline/` | +| `packManager` | Load manifest, index, embeddings.bin from disk | `src/services/packManager/` | +| `embeddingMatchService` | Cosine similarity brute-force top-N | `src/services/embeddingMatchService/` | +| `embeddingDatabaseBuilder` | Merge pack + local individuals for a species | `src/services/embeddingDatabaseBuilder.ts` | +| `wildlifeStore` | Zustand state for packs/observations/locals/sync | `src/stores/wildlifeStore.ts` | + +Native modules (both platforms): + +- `imageToTensor(uri, w, h, mean, std, scale, channelOrder)` — load → resize → normalize → NCHW Float array +- `cropImage(uri, x, y, w, h, outputPath)` — JPEG crop save (95% q) + +## Screens (user flow) + +WildlifeHome → Capture (take/pick photo) → pipeline runs → DetectionResults (bboxes) → MatchReview (approve / new-individual / skip) → Observations → Sync (stub). + +## What Reused From Off Grid Infrastructure + +Background download service, model manager patterns, HuggingFace browser, Zustand + AsyncStorage patterns, theme system, auth (passphrase), navigation, Card/AppSheet/Button/Animated* components. + +## What Was Stripped + +LLM chat, image generation, voice/Whisper, tool calling, intent classifier, chat UI. See commit `a63d662`. + +## Status Snapshot (2026-04-22) + +Pipeline is ~70-80% wired end-to-end in code. The **critical missing pieces** are: +- MiewID v4.1 acquisition + wiring ([[miewid-v4]]) +- Pack install UX / unzip / checksum +- Preprocessing scale math bug ([[critical-bugs]]) +- Pack candidate name/photo resolution in review UI +- GPS + Wildbook sync HTTP client + +See [[state-of-implementation]] for the full gap matrix. + +## Quality Gates + +Pre-commit (Husky, file-scoped): ESLint + tsc + Jest for TS/JS; SwiftLint + `npm run test:ios` for Swift; Kotlin compile + lint + `npm run test:android` for Kotlin. +PR review loop: Gemini bot, Codecov (80% thresholds), SonarCloud. + +## References + +- `CLAUDE.md` — pre-commit gates, push/PR workflow, CI review loop +- `docs/plans/2026-02-25-wildlife-reid-design.md` +- `docs/plans/2026-02-25-wildlife-reid-implementation.md` +- `docs/WILDLIFE_REID_FEASIBILITY.md` diff --git a/kb/wildlife-reid-mobile/wiki/meta/state-of-implementation.md b/kb/wildlife-reid-mobile/wiki/meta/state-of-implementation.md new file mode 100644 index 00000000..37a0f7ed --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/meta/state-of-implementation.md @@ -0,0 +1,55 @@ +# State of Implementation (2026-04-22) + +**Category:** meta +**Sources:** [[codebase-map]], [[codex-review-2026-04-22]], git log, direct source reads +**Related:** [[off-grid-mobile-app]], [[on-device-reid-pipeline]], [[critical-bugs]], [[miewid-v41-integration-plan]] +**Last updated:** 2026-04-22 + +## Snapshot + +Pipeline is **~70-80% wired end-to-end in code**. Architecture is sound; most shortcomings are *unimplemented glue* rather than *wrong foundations*. The biggest single blocker is that MiewID is never actually loaded, followed by two preprocessing bugs that would silently degrade accuracy even once it is. + +## Matrix + +| Layer | Status | Notes | +|---|---|---| +| ONNX inference (ORT RN) | ✅ Real | Detector + generic embedding API | +| Native image preprocessing | ⚠️ Real but buggy | See [[critical-bugs]] Bug 1 + 2 | +| Detection postprocessing / NMS | ✅ Real | YOLOv8/11 only — see [[yolo-postprocessing]] | +| Embedding extraction | ⚠️ Code exists | Model never installed in practice | +| Vector matching | ✅ Real | Cosine brute-force, OK for < 5K individuals | +| Pack loading | ✅ Real | Manifest + index + embeddings.bin | +| Observation storage | ✅ Real | Zustand + AsyncStorage | +| Local individual management | ✅ Real | Accumulates embeddings on re-sighting | +| Match review UI | ⚠️ Partial | Pack IDs shown as raw strings — name/photo TODO | +| Detector model path | ⚠️ | Loaded from pack manifest; no auto-download | +| **MiewID model path** | ❌ | **Never populated**; setter exists but no caller | +| Pack install UX (.zip import) | ❌ | `packManager.initialize()` just mkdirs | +| Pack version check | ❌ | `embeddingModel.version` unenforced | +| Checksum verification | ❌ | `manifest.checksums` ignored | +| Geolocation | ❌ Stub | `useCaptureFlow` returns null | +| Wildbook sync (HTTP) | ❌ Stub | Queue exists, no client | +| Model download retry / timeout | ❌ | No exponential backoff, no hang guard | +| Golden parity test (Python ↔ device) | ❌ | Needed before trusting accuracy | + +## Where Work Should Go Next + +Ordered by leverage for the stated goal (on-app re-ID with MiewID v4.1): + +1. **Fix preprocessing bugs + add parity test.** Foundation for everything else. See [[critical-bugs]]. +2. **MiewID v4.1 acquisition flow.** Export → upload → app-side download/verify/cache/version-gate. See [[model-acquisition]], [[miewid-v41-integration-plan]]. +3. **Pack install UX.** .zip → unzip → checksum → register. +4. **Resolve pack candidate names/photos in review UI.** Field-usability unlock. +5. **Sync client** (can trail re-ID goal, but needed for real-world usage). + +## Non-Gaps (Already Strong) + +- Service layering, separation of concerns +- Native Kotlin/Swift parity for image ops (once scale fix lands) +- Test coverage on unit + integration layers +- Typed state model (`src/types/wildlife.ts`) +- Commit discipline (see git log since `7704aed`) + +## Licensing / External Blocker + +MiewID weights have no explicit OSS license (`docs/WILDLIFE_REID_FEASIBILITY.md:27,109`). Resolve with Conservation X Labs before distribution. diff --git a/kb/wildlife-reid-mobile/wiki/methods/image-preprocessing.md b/kb/wildlife-reid-mobile/wiki/methods/image-preprocessing.md new file mode 100644 index 00000000..3e9fa80e --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/methods/image-preprocessing.md @@ -0,0 +1,98 @@ +# Image Preprocessing (Detector + MiewID) + +**Category:** method +**Sources:** [[docs-live/EMBEDDING_PACK_FORMAT.md]], `src/services/onnxInferenceService/preprocessing.ts`, `android/app/src/main/java/ai/offgridmobile/imagetensor/ImageTensorModule.kt`, `ios/ImageTensorModule.swift` +**Related:** [[embedding-pack]], [[on-device-reid-pipeline]], [[critical-bugs]] +**Last updated:** 2026-04-22 + +## Summary + +All image→tensor work is done in a native module (`ImageTensorModule`) on both platforms for perf. TypeScript wrappers in `preprocessing.ts` exist for detection (per-pack `DetectorConfig`) and for MiewID embedding (hardcoded 440×440 + ImageNet norm). Two real accuracy bugs exist in the current implementation — see [[critical-bugs]]. + +## Intended Formula + +Per the pack spec (`docs/EMBEDDING_PACK_FORMAT.md:166`): + +> `normalize.scale` is the pixel value scaling factor applied BEFORE mean/std normalization. `1/255 = 0.00392156862` converts uint8 [0-255] to float [0-1]. Set to `1.0` if the model expects [0-255] input. + +So the correct formula is: + +``` +normalized[c] = (pixel[c] * scale - mean[c]) / std[c] +``` + +For MiewID (ImageNet): `scale = 1/255`, `mean = [0.485, 0.456, 0.406]`, `std = [0.229, 0.224, 0.225]`. +For typical YOLO: `scale = 1/255`, `mean = [0, 0, 0]`, `std = [1, 1, 1]` (→ `pixel/255`). + +## Actual Implementation (Bug) + +### Native (both Android and iOS) + +Android `ImageTensorModule.kt:62`: +```kotlin +output[rIdx * h * w + i] = (r / scale - mean[0]) / std[0] +``` + +iOS `ImageTensorModule.swift:188` mirrors this. **Divides by scale** instead of multiplying. With `scale = 1/255`, this yields `r × 255`, which is wildly out of range — effectively garbage tensors. + +### TypeScript MiewID wrapper + +`src/services/onnxInferenceService/preprocessing.ts:62-70`: +```ts +const rawArray = await ImageTensorModule.imageToTensor( + imageUri, width, height, + norm.mean, norm.std, + 1.0, // ← scale hardcoded to 1.0 + 'RGB', +); +``` +Comment says "[0, 255] divided by 1.0". Given the native bug, this produces raw 0-255 values minus ImageNet mean ≈ (0.485) — still completely wrong (pixel values 0-254.5 instead of ≈ -2 to +2). + +Even if the spec interpretation were "divide" and that were correct here by coincidence, the TypeScript layer doesn't match the detection layer, and no pack with `scale: 1/255` would work as intended. + +## Fix Strategy + +1. **Change native math to multiply** by scale, matching the spec: + ```kotlin + output[rIdx * h * w + i] = (r * scale - mean[0]) / std[0] + ``` +2. **Update MiewID TS wrapper** to pass `scale = 1.0/255.0`: + ```ts + await ImageTensorModule.imageToTensor(..., 1.0 / 255.0, 'RGB'); + ``` +3. **Update detector preprocessing** — pack's `config/detector.json` should always set `scale` to `1/255` for YOLO, which is what the spec already says. No change needed there after fix #1. +4. **Add golden parity test** that compares on-device tensor output vs a known Python / NumPy reference for a fixed test image. +5. **Update existing unit tests** to cover the corrected semantics. + +## NCHW Packing + +Output is laid out `[channel][h][w]`: +``` +output[channel * H * W + row * W + col] +``` +Channel indices swap for BGR: `rIdx=2, gIdx=1, bIdx=0`. Confirmed consistent between Kotlin and Swift. + +## Resize + +Bilinear resize to model input size before channel splitting. Both platforms use platform-native bilinear. + +## Crop (Pre-Embedding) + +`ImageTensorModule.cropImage(uri, x, y, w, h, outputPath)` — coords may be normalized or absolute (check per-call site). Saves JPEG at 95% quality. Called by `wildlifePipeline.processPhoto` between detection and embedding. + +## Test Coverage + +Exists: +- `__tests__/unit/services/onnxInferenceService.test.ts` — mocks native module; tests TS wrappers +- No golden parity test against Python reference + +Needed: +- Parity test with a fixed image + known expected NCHW float array +- iOS + Android crossvalidation (same inputs → same outputs) + +## References + +- `docs/EMBEDDING_PACK_FORMAT.md` §config/detector.json → normalize +- `src/services/onnxInferenceService/preprocessing.ts:62` +- `android/app/src/main/java/ai/offgridmobile/imagetensor/ImageTensorModule.kt:62` +- `ios/ImageTensorModule.swift:188` diff --git a/kb/wildlife-reid-mobile/wiki/methods/model-acquisition.md b/kb/wildlife-reid-mobile/wiki/methods/model-acquisition.md new file mode 100644 index 00000000..bfcd1244 --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/methods/model-acquisition.md @@ -0,0 +1,106 @@ +# Model Acquisition (MiewID + Detectors) + +**Category:** method +**Sources:** [[docs-live/plans/2026-02-25-wildlife-reid-design.md]], [[docs-live/EMBEDDING_PACK_FORMAT.md]], [[codex-review-2026-04-22]] +**Related:** [[miewid-v4]], [[embedding-pack]], [[miewid-v41-integration-plan]] +**Last updated:** 2026-04-22 + +## Summary + +MiewID is **one shared model** used across all species — downloaded separately from packs and reused. Detectors are **species-specific** and ship inside each pack. The app must enforce `embeddingModel.version` compatibility between the loaded MiewID and each pack's manifest. Neither side of this acquisition flow is implemented yet. + +## MiewID Export → ONNX + +1. **Start from PyTorch checkpoint.** Use the local v4.1 reference file we already have: + - `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` (md5 `71c0bdd9bf8cbdb1b9e967a2e96949bb`, 206 MB FP32) + - Companion config: `miew_id.msv4_1_main.yaml` (see [[2026-04-23-miewid-v41-checkpoint]]) + - Architecture: `efficientnetv2_rw_m`, `arcface_subcenter_dynamic`, `n_classes=20191`, `use_fc=false`, `image_size=[440,440]` +2. **Export:** load via the `wbia-plugin-miew-id` model class (the checkpoint is a raw state dict, not an HF transformers artifact). Strip the ArcFace head — we only need the backbone + GeM → L2 norm path since the app only uses embeddings, not classification logits. + ```python + import torch + from wbia_miew_id.models import MiewIdNet # or equivalent class in the repo + import yaml + + cfg = yaml.safe_load(open(".../miew_id.msv4_1_main.yaml"))["model_params"] + model = MiewIdNet( + model_name=cfg["model_name"], # efficientnetv2_rw_m + use_fc=cfg["use_fc"], # False + fc_dim=cfg["fc_dim"], + dropout=cfg["dropout"], + loss_module=cfg["loss_module"], + n_classes=cfg["n_classes"], + # ArcFace head parameters… + ) + state = torch.load(".../miew_id.msv4_1_main.bin", map_location="cpu") + model.load_state_dict(state, strict=False) # tolerate ArcFace head keys + model.eval() + + # Export only backbone+pool, NOT the classification logits. + class EmbeddingOnly(torch.nn.Module): + def __init__(self, m): super().__init__(); self.m = m + def forward(self, x): return self.m.extract_features(x) # backbone+GeM→L2 + + export_model = EmbeddingOnly(model) + dummy = torch.randn(1, 3, 440, 440) + torch.onnx.export( + export_model, dummy, "miewid_v4_1.onnx", + opset_version=17, + input_names=["input"], + output_names=["embedding"], + dynamic_axes={"input": {0: "batch"}, "embedding": {0: "batch"}}, + ) + ``` + (Final method name depends on the exact `wbia-plugin-miew-id` API — may be `extract_features`, `embedding`, or similar. See the miewid-trainer skill for the authoritative class signature.) +3. **Verify numerical parity.** On a fixed batch of ~50 crops, compare ONNX CPU vs PyTorch float32. Rank-order parity (cosine-sim ordering of gallery matches) is the relevant metric, not raw cosine closeness. +4. **Quantize:** Start FP16 (`onnxconverter_common.float16.convert_float_to_float16`), re-run parity. INT8 only after confirming rank parity on real wildlife crops. +5. **Publish:** Upload to HF (`conservationxlabs/miewid-msv4-onnx` or equivalent) with clear version tag. + +## Distribution Strategy (Recommended) + +| Asset | Distribution | Rationale | +|---|---|---| +| MiewID ONNX FP16 | **Separate download** (HuggingFace), ~80-120 MB | Shared across all species packs; app binary stays lean; can version-bump independently | +| Detector ONNX | **Bundled in pack** (.zip) | Species-specific; each pack's correct detector travels with its embeddings | +| Reference photos | **Bundled in pack** | Needed for review UI | + +Avoid bundling MiewID in the app binary — it blows up install size and makes version updates require app store rollouts. + +## App-Side Acquisition Flow (Needed) + +Currently missing. Target flow: + +1. **First run / model missing:** + - Surface "Download MiewID v4.1 (112 MB)" prompt + - Download via existing `backgroundDownloadService` (reused from Off Grid) + - Verify SHA-256 checksum + - Store in `DocumentDirectoryPath/models/miewid_v4_1.onnx` + - Set `wildlifeStore.miewidModelPath` +2. **Version check on pack load:** + - Compare `pack.manifest.embeddingModel.version` to loaded MiewID version + - Block or warn on mismatch; offer MiewID update +3. **Restore on startup:** + - On `App.initializeApp`, scan `models/` directory and rehydrate `miewidModelPath` + +## Pack Install Flow (Needed) + +Currently `packManager.initialize()` just creates the directory; no import UX exists. + +Target flow: +1. **Source .zip** via Files picker / deep-link / server download +2. **Unzip** to `packs/{pack-id}/` +3. **Validate:** parse `manifest.json`, check `formatVersion`, verify all listed `checksums`, spot-check `embeddings.bin` size vs `embeddingCount × embeddingDim × 4` +4. **Register** in `wildlifeStore.packs` +5. **Detector warm-load** (optional) so first capture isn't slow +6. **Delete flow** for storage reclamation + +## Open Questions + +- ~~**License** — MiewID weights license is unresolved~~. Signed off by Jason (CXL) on 2026-04-23 for the v4.1 ONNX export. +- **Hosting** — Where does the ONNX export live for app download? Options: (a) public HF mirror, (b) CXL CDN, (c) GitHub Release asset. +- **Delta updates** — If a pack is re-exported with +10 individuals, can we fetch only the delta? + +## References + +- `docs/plans/2026-02-25-wildlife-reid-design.md` §4 ML Inference Pipeline, §6 Distribution +- `docs/EMBEDDING_PACK_FORMAT.md` +- [[codex-review-2026-04-22]] diff --git a/kb/wildlife-reid-mobile/wiki/methods/yolo-postprocessing.md b/kb/wildlife-reid-mobile/wiki/methods/yolo-postprocessing.md new file mode 100644 index 00000000..cf73f8af --- /dev/null +++ b/kb/wildlife-reid-mobile/wiki/methods/yolo-postprocessing.md @@ -0,0 +1,71 @@ +# YOLO Postprocessing (Parse + NMS) + +**Category:** method +**Sources:** `src/services/onnxInferenceService/postprocessing.ts`, [[docs-live/EMBEDDING_PACK_FORMAT.md]] +**Related:** [[on-device-reid-pipeline]], [[embedding-pack]] +**Last updated:** 2026-04-22 + +## Summary + +`parseYoloOutput()` interprets a row-major Float32 tensor from a YOLO ONNX model, applies confidence threshold, converts boxes to normalized xywh, sorts, runs IoU-based NMS, and slices to `maxDetections`. Assumes YOLOv8/YOLO11 output layout `[1, (4 + numClasses), N]` — **no objectness channel**. + +## Output Layout Assumption + +Current implementation at `postprocessing.ts:143`: + +```ts +const numRows = 4 + numClasses; +``` + +This is correct for **YOLOv8 and YOLO11** (they fold objectness into class scores). Legacy **YOLOv5** outputs `[1, (5 + numClasses), N]` with objectness at row 4. If a pack ever ships a v5 detector, this code will silently misread the tensor. + +**Mitigation:** The pack's `config/detector.json` has an `architecture` field. The parser should branch on it (or on the observed tensor dimension) and handle `5 + C` for `yolov5` / `yolov7`. + +## Box Format Support + +Driven by `config.outputSpec.boxFormat`: + +| Format | Layout | Conversion | +|---|---|---| +| `cxcywh` | center x/y + w/h | `x = cx - w/2`, `y = cy - h/2` (YOLOv8/v11 default) | +| `xyxy` | top-left + bottom-right | `w = x2-x1`, `h = y2-y1` | +| `xywh` | top-left + w/h | pass-through | + +And `config.outputSpec.coordinateType`: `"normalized"` (pass through) vs `"absolute"` (divide by `inputSize`). + +## Confidence & Class Selection + +For each detection: +1. Read 4 box values (row 0..3, col i) +2. Read per-class scores (rows 4..4+numClasses-1, col i) +3. Take `argmax` across classes — pick best class + confidence +4. Skip if confidence < `config.confidenceThreshold` + +## NMS + +Standard greedy IoU-based: +1. Sort candidates by confidence desc +2. Repeatedly pick highest, drop any remaining with `IoU > nmsThreshold` against it +3. Return survivors + +Trimmed to `config.maxDetections`. + +## Test Coverage + +`__tests__/unit/services/onnxInferenceService.test.ts` covers: +- Multiple box formats (cxcywh, xyxy, xywh) +- Absolute vs normalized coords +- Multi-class scoring +- NMS suppression of overlapping boxes +- Confidence threshold filtering + +## Improvement Opportunities + +- **Objectness support** for YOLOv5/v7 detectors (as noted above). +- **Class-aware NMS** (different IoU per class) for packs that need it. +- **Batch inference** if a detector supports multi-crop input — not needed for single-photo capture flow. + +## References + +- `src/services/onnxInferenceService/postprocessing.ts` +- Ultralytics YOLOv8 output layout docs