Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions kb/wildlife-reid-mobile/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Binary outputs are too large for git — transferred separately per HANDOFF doc
output/
150 changes: 150 additions & 0 deletions kb/wildlife-reid-mobile/HANDOFF-2026-04-25.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Session Handoff — 2026-04-25

**From:** Claude (WSL session, `/mnt/c/off-grid-mobile`)
**To:** Claude (Mac session, same repo cloned locally)
**Purpose:** Continue work on the wildlife re-ID pivot of off-grid-mobile, picking up at Step (A) of the current plan — sideload synthetic pack onto iPhone 13 + Pixel 8 dev builds and validate the pipeline end-to-end.

---

## Read this first

If you are the new Mac-Claude session, your reading order:

1. **This doc** (you're here) — current state, decisions, what to do next, gotchas.
2. **`kb/wildlife-reid-mobile/INDEX.md`** — durable knowledge base index. Source registry, wiki articles, outputs.
3. **`kb/wildlife-reid-mobile/outputs/plans/miewid-v41-integration-plan.md`** — the 5-stage integration plan. Stage 0 done, Stage 1.2 done, you're working on the bridge to Stage 2.
4. **`kb/wildlife-reid-mobile/outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md`** — pre-emptively identified issues you'll hit when you get to mobile-side rewiring (Stage 2).
5. **`CLAUDE.md`** at repo root — project-wide instructions (pre-commit, push workflow, CI loop). Was just realigned via PR #12 to reflect actual setup (Gemini-only, no Codecov/SonarCloud).

After reading, **rebuild your memory** by writing the relevant facts back into your local `~/.claude/projects/.../memory/` per your auto-memory instructions. The WSL session's memory is at `/home/jason/.claude/projects/-mnt-c-off-grid-mobile/memory/` and won't follow.

## Project mission (one sentence)

Repurpose the React Native off-grid LLM app (`WildMeOrg/off-grid-mobile`) into an offline wildlife individual re-identification platform — capture photo → species detector (YOLO ONNX) → MiewID v4.1 embedding → cosine match against a Wildbook-exported pack of known individuals → human review → sync to Wildbook when online.

## Current state — who, what, where

| Topic | State |
|---|---|
| **User** | Jason (CXL / Wild Me Lab employee). MiewID redistribution sign-off granted 2026-04-23. |
| **Repo** | `WildMeOrg/off-grid-mobile`, default branch `main`. Long-lived integration branch: **`wildlife-reid`**. Open PRs target `wildlife-reid`, NOT `main`. |
| **Reference devices** | iPhone 13 (A15, 6-core NPU); Google Pixel 8 (Tensor G3, clean AOSP). |
| **Dogfood target** | Wild horses, State of Washington. |
| **CI** | GitHub Actions only (`typecheck`, `test`, `lint`, `android-build`). Triggered on push + PR against `main` and `wildlife-reid`. Workflows had to be manually enabled on the fork (one-time UI step Jason did 2026-04-24). |
| **Code review bots** | **Gemini Code Assist** installed on the fork — auto-reviews on PR open, retrigger with `/gemini review`. Codecov / SonarCloud are NOT installed (see realigned CLAUDE.md). |
| **Codex CLI** | `/usr/bin/codex`, model upgraded to **5.5** as of 2026-04-24. Use for scoped second-opinion reviews; not part of routine PR loop. File substantive findings to `kb/wildlife-reid-mobile/outputs/reports/`. |

## What's done (chronological)

**2026-04-22**
- Built the KB scaffold under `kb/wildlife-reid-mobile/`. Ingested upstream docs + ran codex review + Explore-agent codebase map.
- Identified 2 latent preprocessing bugs (native scale divide-not-multiply + MiewID TS wrapper passing scale=1.0).

**2026-04-23**
- Resolved licensing for MiewID v4.1 redistribution (CXL employee sign-off).
- Confirmed local v4.1 checkpoint at `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` (n_classes=20,191).
- **PR #11** merged — fixed native scale math (Kotlin + Swift), TS wrapper, added cross-platform golden parity fixture (1×1 pure-red pixel through ImageNet norm + scale=1/255 → R≈2.249, G≈-2.036, B≈-1.804). Also patched `.husky/pre-commit` to skip iOS tests when xcodebuild absent (WSL friendliness). Created `wildlife-reid` integration branch, pushed CI workflow to trigger on it.
- Codex 5.5 Stage-2 risk review surfaced 2 blockers + 3 high + 3 medium pre-existing issues — filed at `kb/.../outputs/reports/2026-04-24-codex-5-5-stage-2-risk-review.md`. Stage 2 plan rewritten in light of these.

**2026-04-24**
- Horse-face detector trained via yolo-trainer skill: `/mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx` (10 MB YOLO11n, mAP50 0.9999). Pack-ready, no code changes needed.
- **MiewID v4.1 ONNX export done** via `tools/export_miewid_v41_onnx.py`. FP16 single-file 103.9 MB at `tools/output/miewid_v4_1_fp16.onnx`. Cosine vs PyTorch ref: 0.999994 (well above 0.999 threshold). Embedding dim 2152 confirmed.
- Gemini installed on fork; **PR #12** merged (CLAUDE.md realigned to reflect Gemini-only setup).

**2026-04-25**
- **Pack bundler** at `tools/build_pack.py` with `synthetic` + `from-coco` (stub) modes.
- **Synthetic horse pack** at `tools/output/horse-synthetic-2026-04-25.zip` (9.2 MB, 5 individuals × 2 embeddings, real WHORSE detector bundled, validates against pack format spec, within-individual cos 0.995, cross 0.02).
- Session handoff to Mac (this doc).

## Where we are in the plan (B → A → C)

| Step | Status | Owner |
|---|---|---|
| (B) Pack-bundling tooling | ✅ done | WSL session |
| **(A) Vertical slice on iPhone 13 + Pixel 8** | ⏳ **NEXT** | **You (Mac session)** |
| (C) Stage 2 foundation trio | queued | Mac session afterwards |

## Step (A) brief — what to do next

**Goal:** Sideload `horse-synthetic-2026-04-25.zip` + `miewid_v4_1_fp16.onnx` onto an iPhone 13 dev build, validate the entire stack runs end-to-end, repeat on Pixel 8, confirm cross-platform embedding cosine ≥ 0.999.

**Suggested PR (`feat/dev-pack-bundling`):**

1. **Bundle assets in dev builds.** Drop `horse-synthetic-2026-04-25.zip` and `miewid_v4_1_fp16.onnx` into a new `assets/dev/` directory. Add to RN bundling config so they ship with debug builds only (release builds should NOT include these).
2. **Debug startup hook.** In `App.tsx` (or a debug-only helper), on first launch:
- Unzip the bundled pack into `DocumentDirectoryPath/embedding_packs/horse-synthetic-2026-04-25/`
- Copy MiewID ONNX into `DocumentDirectoryPath/models/miewid_v4_1.onnx`
- Call `wildlifeStore.setMiewidModelPath(...)` and `wildlifeStore.addPack(...)` so the existing pipeline picks them up
- Guard with `__DEV__` so this never runs in release
3. **Tiny debug screen** — "Run pipeline test" button. Loads a bundled test photo (or a colored square if no real photo handy), runs `wildlifePipeline.processPhoto`, displays:
- Detection bbox + species + confidence
- First few embedding values (sanity check that they're not garbage)
- Top-5 match candidates
- Inference latency per stage
4. **Run on iPhone 13** via Xcode. Note any iOS-specific issues.
5. **Run on Pixel 8** via Android Studio / `npx react-native run-android`.
6. **Cross-platform parity check** — load the same fixed test image on both, embed, compare cosine. Should be ≥ 0.999.
7. **PR against `wildlife-reid`.** Pre-commit gates will run JS + Swift + Kotlin checks.

## Binary artifacts you'll need (NOT in git)

These are too big or out-of-tree to commit. Jason will transfer them to the Mac via OneDrive / SCP / USB / whatever:

| File | Size | Source path on WSL | Place on Mac at |
|---|---|---|---|
| `miewid_v4_1_fp16.onnx` | 103.9 MB | `kb/wildlife-reid-mobile/tools/output/miewid_v4_1_fp16.onnx` | `kb/wildlife-reid-mobile/tools/output/miewid_v4_1_fp16.onnx` |
| `horse-synthetic-2026-04-25.zip` | 9.2 MB | `kb/wildlife-reid-mobile/tools/output/horse-synthetic-2026-04-25.zip` | `kb/wildlife-reid-mobile/tools/output/horse-synthetic-2026-04-25.zip` |
| `horse_wild_face_yolo11n_416.onnx` | 10 MB | `/mnt/c/claude-skills/output/whorse-face-detector/horse_wild_face_yolo11n_416.onnx` | wherever convenient — used to rebuild the synthetic pack if needed |
| `detector_config.json` | 567 B | `/mnt/c/claude-skills/output/whorse-face-detector/detector_config.json` | same as above |

If you want to **regenerate the synthetic pack** on the Mac instead of transferring it: clone `wbia-plugin-miew-id` (not strictly needed for `build_pack.py`, only for re-exporting MiewID), `pip install pillow numpy onnxruntime`, then `python3 build_pack.py synthetic ...` per the documented command in `kb/.../raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md`. Detector + MiewID FP16 still need transfer either way.

## Memory porting

The WSL session's memory at `/home/jason/.claude/projects/-mnt-c-off-grid-mobile/memory/` has three durable entries that should be recreated on the Mac:

1. `user_role.md` — Jason at CXL / Wild Me Lab; on-device wildlife re-ID mission.
2. `project_wildlife_reid.md` — KB at `kb/wildlife-reid-mobile/`; preprocessing fix landed; v4.1 export done; horse detector ready; all the dated facts.
3. `reference_skills_and_tools.md` — `/mnt/c/claude-skills/` skill library (assuming same path on Mac); codex 5.5 CLI; miewid-trainer MCP.

Mac-Claude: write equivalents into your local `~/.claude/projects/.../memory/` based on this handoff doc + the KB INDEX. Don't try to copy verbatim from WSL — paths differ (the user's home dir, `/mnt/c/` mount point, etc.).

## Known gotchas

1. **CRLF/LF drift.** Repo files often show as "modified" in `git status` due to line-ending differences between the upstream repo and Windows clones. **Stage files individually by name; never `git add -A`.** When edits show suspiciously large diffs, normalize the file with `python3 -c "open(p,'wb').write(open(p,'rb').read().replace(b'\\r\\n', b'\\n'))"` before staging. Likely less of an issue on Mac than on WSL.
2. **Pre-commit hook + iOS tests.** The hook now skips iOS tests when `xcodebuild` is absent (WSL friendliness, PR #11). On Mac with Xcode installed, iOS tests **will** run on every Swift-touching commit — slower but correct.
3. **GitHub fork workflows.** Actions are enabled and CI fires on `main` + `wildlife-reid`. PRs targeting other branches won't fire CI unless added to `.github/workflows/ci.yml` `branches:` filters.
4. **Gemini false positives.** It pattern-matched the `kb/wildlife-reid-mobile/` directory name as "fork name mismatch" in PR #12. The directory is named for the **product domain**, not the fork. Reject similar suggestions on the comment thread with a brief explanation; don't blindly apply.
5. **Codex 5.5 use.** It's worth running for any non-trivial PR (it caught a real bug in the Stage-0 PR that I missed). Scope the prompt — give it specific files + specific questions.
6. **iOS bundle size.** MiewID FP16 is 103.9 MB. Bundling in debug builds is fine, but DO NOT bundle in release — the design ([[model-acquisition]]) requires download-on-demand for production.

## Open decisions you may need to make

- **Hosting for shipped MiewID ONNX** (Stage 1.5) — HF / CXL CDN / GitHub release asset. No decision yet.
- **iOS bundle ID + signing identity** — Off Grid's existing identity vs a new one for the wildlife fork. Not yet discussed.
- **Real Washington-horses COCO acquisition** — does Jason have horses.wildbook.org credentials for an export? Pending.
- **Renaming the GitHub repo** to reflect the wildlife pivot (e.g. `wildlife-reid-mobile`)? Optional, not blocking.

## Tasks state at handoff

```
#15 [pending] Step (A) Vertical slice — sideload synthetic pack on iPhone 13 + Pixel 8
```

All earlier tasks (preprocessing fixes, MiewID export, pack bundler, etc.) are completed and recorded in the KB.

## After Step (A)

When the vertical slice works on both devices and you've measured real latencies + cross-platform parity, the next phase is **Step (C) — Stage 2 foundation trio:**

- 2.1 `MiewIDModelRecord` + state machine (replaces the bare `miewidModelPath: string | null`)
- 2.2 Pack validator (manifest schema + checksums + offset bounds)
- 2.3 JS download service (staging + SHA-256 + atomic move)

These fix the codex-5.5-flagged blockers and replace the debug startup hook from Step (A) with proper production code. See `outputs/plans/miewid-v41-integration-plan.md` Stage 2 section for the full sub-stage breakdown.

## Questions you'll likely ask Jason early

- Where do you want the binary artifacts dropped on the Mac?
- Are iPhone 13 + Pixel 8 already provisioned for dev builds, or do we need to set up signing / `adb` first?
- Do you have a horse photo (or any real wildlife photo) we can use as a fixed test input for cross-platform parity?
75 changes: 75 additions & 0 deletions kb/wildlife-reid-mobile/INDEX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Wildlife Re-ID Mobile Knowledge Base

**Last compiled:** 2026-04-23
**Articles:** 9 wiki + 1 output
**Sources:** 7 raw documents (+ 4 live docs via symlink)
**Coverage:** Good on architecture, pipeline, bugs, plan. V4.1 checkpoint now ingested. Gaps: device performance data, license resolution, real pack samples, ONNX hosting decision.

## Source Registry

| File | Type | Ingested | Summary |
|---|---|---|---|
| [raw/docs-live → docs/WILDLIFE_REID_FEASIBILITY.md](raw/docs-live/WILDLIFE_REID_FEASIBILITY.md) | design | yes | Feasibility assessment; Off Grid + ONNX Runtime recommendation; MiewID profile + storage budget |
| [raw/docs-live → docs/EMBEDDING_PACK_FORMAT.md](raw/docs-live/EMBEDDING_PACK_FORMAT.md) | spec | yes | Pack format v1.0 — manifest, embeddings.bin layout, detector config, checksums |
| [raw/docs-live → docs/plans/2026-02-25-wildlife-reid-design.md](raw/docs-live/plans/2026-02-25-wildlife-reid-design.md) | design | yes | Fork strategy (what stripped vs kept), data model, pipeline stages |
| [raw/docs-live → docs/plans/2026-02-25-wildlife-reid-implementation.md](raw/docs-live/plans/2026-02-25-wildlife-reid-implementation.md) | plan | partial | Task-by-task TDD implementation plan, executed ~through Phase 5 |
| [raw/docs-live → docs/ARCHITECTURE.md](raw/docs-live/ARCHITECTURE.md) | arch | skim | General Off Grid architecture patterns reused by the wildlife fork |
| [raw/codex/2026-04-22-codebase-review.md](raw/codex/2026-04-22-codebase-review.md) | review | yes | Codex second opinion: strengths, risks, preprocessing bugs, acquisition recommendation |
| [raw/exploration/2026-04-22-codebase-map.md](raw/exploration/2026-04-22-codebase-map.md) | map | yes | Thorough Explore-agent mapping of services, screens, stores, native modules, tests |
| [raw/models/2026-04-23-miewid-v41-checkpoint.md](raw/models/2026-04-23-miewid-v41-checkpoint.md) | model | yes | Local MiewID v4.1 checkpoint: file path, MD5, training config, n_classes=20,191 |
| [raw/models/2026-04-24-whorse-face-detector.md](raw/models/2026-04-24-whorse-face-detector.md) | model | yes | YOLO11n horse-face detector (10 MB, mAP50 0.9999) at /mnt/c/claude-skills/output/whorse-face-detector/ — pack-ready, no code changes needed |
| [raw/models/2026-04-24-miewid-v41-onnx-export.md](raw/models/2026-04-24-miewid-v41-onnx-export.md) | model | yes | MiewID v4.1 ONNX exports (FP32 205.7 MB, FP16 103.9 MB) at tools/output/. Parity vs PyTorch ref: FP32 cos_min 1.0000, FP16 cos_min 0.99999, embedding_dim=2152 confirmed. |
| [raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md](raw/models/2026-04-25-pack-bundler-and-synthetic-pack.md) | tool | yes | `build_pack.py` two-mode pack assembler + `horse-synthetic-2026-04-25.zip` (9.2 MB, 5 individuals × 2 embeddings, validates against pack format spec). Bridge artifact between Stage 1 and Stage 2. |

## Wiki Articles

| Article | Category | Sources | Last Updated |
|---|---|---|---|
| [MiewID v4](wiki/entities/miewid-v4.md) | entity | 3 | 2026-04-22 |
| [Off Grid Mobile (Wildlife Re-ID Fork)](wiki/entities/off-grid-mobile-app.md) | entity | 3 | 2026-04-22 |
| [Embedding Pack](wiki/concepts/embedding-pack.md) | concept | 2 | 2026-04-22 |
| [On-Device Re-ID Pipeline](wiki/concepts/on-device-reid-pipeline.md) | concept | 2 | 2026-04-22 |
| [Image Preprocessing](wiki/methods/image-preprocessing.md) | method | 4 files + spec | 2026-04-22 |
| [YOLO Postprocessing](wiki/methods/yolo-postprocessing.md) | method | 2 | 2026-04-22 |
| [Model Acquisition](wiki/methods/model-acquisition.md) | method | 3 | 2026-04-22 |
| [Critical Bugs](wiki/bugs/critical-bugs.md) | bugs | source reads | 2026-04-22 |
| [State of Implementation](wiki/meta/state-of-implementation.md) | meta | 3 | 2026-04-22 |

## Outputs

| Output | Format | Date |
|---|---|---|
| [MiewID v4.1 On-Device Integration Plan](outputs/plans/miewid-v41-integration-plan.md) | plan | 2026-04-22 |

## Coverage Gaps

- [ ] Device-class latency benchmarks (iPhone Neural Engine, Android NNAPI) — populate after Stage 3.2 of plan
- [x] ~~MiewID weights **license** resolution~~ — resolved 2026-04-23 via CXL employee sign-off (Jason)
- [x] ~~Confirmed **v4.1** availability~~ — local checkpoint found at `/mnt/c/claude-skills/models/reference/miew_id.msv4_1_main.bin` (2026-04-23)
- [ ] Hosting decision for exported ONNX (HF / CXL CDN / GitHub release)
- [~] Real sample pack — **target: wild horses, State of Washington** (2026-04-23).
- [x] Detector model — YOLO11n trained on WBIA `horse_wild+face`, ready as ONNX (2026-04-24). See [[2026-04-24-whorse-face-detector]].
- [x] MiewID v4.1 ONNX export — FP16 103.9 MB, parity ≥ 0.99999 (2026-04-24). See [[2026-04-24-miewid-v41-onnx-export]].
- [ ] Washington-horses COCO export from Wildbook + matchability assessment
- [ ] Bundle `embeddings.bin` + `index.json` + reference photos + manifest into pack zip
- [ ] Wildbook sync API contract — endpoints, auth, Encounter upload shape
- [ ] Core ML / NNAPI EP compatibility testing for MiewID
- [ ] Gallery dilution curves per species at expected production scale
- [ ] Species-specific cosine thresholds (calibration data)

## Recent Queries

| Date | Query | Output |
|---|---|---|
| 2026-04-22 | "What's built, what's stubbed, and how to ship MiewID v4.1 on-device?" | [integration plan](outputs/plans/miewid-v41-integration-plan.md) + [state of implementation](wiki/meta/state-of-implementation.md) + [critical bugs](wiki/bugs/critical-bugs.md) |
| 2026-04-23 | "Fix the preprocessing scale bugs (Stage 0 of integration plan)" | PR [#11](https://github.com/WildMeOrg/off-grid-mobile/pull/11) — `fix/preprocessing-scale-math` → `wildlife-reid`. Native multiply-not-divide on both platforms + MiewID TS wrapper `1/255` + cross-platform golden parity fixture. 410/410 Jest, 11/11 Kotlin tests pass. |

## Key Conclusions (TL;DR)

1. **Pipeline is ~70-80% wired end-to-end.** Architecture is sound; most remaining work is glue, not foundation.
2. **MiewID v4.1 is not actually loaded anywhere.** `miewidModelPath` setter exists but no call site populates it — highest-leverage gap.
3. **Two real preprocessing bugs** will produce garbage embeddings even after MiewID wiring:
- Native `scale` divided instead of multiplied (both Kotlin & Swift)
- TS MiewID wrapper passes `scale = 1.0` (feeds 0-255 into ImageNet normalization)
4. **Recommendation:** ship MiewID as a separate FP16 download (~80-120 MB), not bundled. Keep detectors in packs. Avoid INT8 until rank-order parity is proven on real wildlife crops.
5. **Next steps:** fix bugs + golden parity test → MiewID ONNX export + HF publish → in-app acquisition service → E2E verification. See [integration plan](outputs/plans/miewid-v41-integration-plan.md).
Loading
Loading