MyKindlebox/memory-bank/FEATURE_OCR_Pedigree_Scan.md

# MyKindlebox — OCR Pedigree Scanning Feature

> **Superseded 2026-10-28 — AI vision replaces Tesseract.** Position-based OCR could not handle the real variety of pedigrees (horizontal vs vertical, color vs B&W, handwritten vs printed, every breeder's own format). Per the user's decisions, scanning now sends the photo from the server to the **Claude API** (`claude-opus-5-5`), which returns the subject + all 14 ancestors as schema-validated JSON with per-field "uncertain" flags. Tesseract, the vendored `/tesseract/` assets, the `lib/ocr*` modules, the zone-picker plan, and the "no LLM / fully client-side / offline" constraint are **dropped**. The photo is **not stored**. Scanning needs internet and `ANTHROPIC_API_KEY` on the server; without it the button is hidden. Still in force from this doc: the Add Rabbit Step 1 integration point, Q2/Q3/Q6 (sire/dam herd matching: exact ear number, else unique exact name; a matched branch locks to the herd record; everything else lands as editable manual nodes), and Q4 normalization (decimal lbs, YYYY-MM-DD). See `architecture.md` → `lib/pedigreeScan.ts`, `routes/pedigreeScan.ts`, `PedigreeScan.jsx`. The rest of this document is historical.

**Feature:** OCR-assisted pedigree entry with manual zone correction fallback  
**Stack:** Node.js / React / SQLite  
**Packages:** `tesseract.js`, `jimp`, `react-image-crop`

---

## Integration Point

OCR scanning integrates into the **existing Add Rabbit wizard** rather than a separate route. No `/herd/scan` page is needed.

- Step 1 of the wizard gets a **"Scan Pedigree"** button alongside the existing manual entry option
- Scanning pre-fills the wizard's `PedigreeTree` + `details` state
- After OCR completes, the user is dropped into the normal wizard flow at Step 1 with fields populated
- The zone picker panel renders within the wizard context — no separate route or modal
- The existing wizard review/save path handles confirmation; no duplicate save flow is needed

## Overview

Allow users to photograph or upload a printed rabbit pedigree and have the app auto-populate the pedigree entry form. The system attempts automatic extraction first; if any field is wrong or missing, the user can click that field and manually draw a zone on the image to re-run OCR on just that region.

This feature does **not** use any LLM API calls. All processing is client-side.

---

## Architectural Decisions

### Q1 — Integration point
Resolved: option (a). OCR integrates into the existing Add Rabbit wizard via a "Scan Pedigree" button on Step 1. No separate `/herd/scan` route. See Integration Point section above.

### Q2 — How extracted ancestors land in PedigreeTree
- **Gen1 (sire/dam):** attempt herd match first using `rabbitName` + `earNumber` as the lookup key against existing herd rabbits via the existing autocomplete/search. On confident match → insert as a linked node. On no match → fall back to `serializeManualAncestor`. No user prompt to resolve — auto-link or auto-manual, silently.
- **Gen2/gen3:** always insert as manual JSON nodes. No FK columns exist below `sire_id`/`dam_id`. This is the intended target and no special handling is needed.

### Q3 — Subject populates both wizard steps simultaneously
Yes. On OCR complete, both state slices are pre-filled at once:
- `PedigreeTree` state ← sire, dam, gen2/gen3 ancestors
- `details` state ← subject `farmName` → `farm`, `rabbitName` → `name`, `earNumber` → `ear_number`, `dob` → `dob`, `weight` → `weight_lbs`

User reviews Step 1 (tree), advances normally, reviews Step 2 (details) which is already populated, confirms and saves. No special wizard behavior beyond pre-populating both slices.

### Q4 — Weight and DOB normalization
- **Weight:** normalize to decimal lbs at extraction time. Examples: `"8 lbs 4 oz"` → `8.25`, `"8.5 lbs"` → `8.5`, `"142 oz"` → `8.875`. The app's lb+oz vs decimal display setting handles rendering; storage is always a decimal float.
- **DOB:** convert to `YYYY-MM-DD` at extraction time. Common input formats: `MM/DD/YYYY`, `MM-DD-YY`, `M/D/YY`. If parsing fails or no DOB is found, use `"Unknown"` — the app's existing sentinel convention for manual ancestors.

### Q5 — Zone picker behavior for ancestor slots
Zone picker result opens/refills the slot's **manual-entry editor** — it does not write silently to the tree. Ancestor slots are mode-bearing buttons; the user must see the populated values in the editor and explicitly confirm. Silent write would bypass the existing UX contract for those slots.

### Q6 — Locking & matching behavior
The pedigree tree's chain-authority locking (a gen1 node linked to a herd rabbit causes that branch's grandparents to lock with a 🔒 and derive from the linked rabbit's own record) interacts with OCR population. Resolved model:

1. **OCR populates everything as editable manual nodes.** Subject → `details`; sire/dam/grandparents → manual ancestor JSON nodes (`mode: 'manual'`). Nothing locks during Step 1 — every scanned node is reviewable/editable exactly like hand entry, right up until the user clicks Next.
2. **Gen1 matches resolve into links at the Next boundary, not live.** When the user advances from Step 1, each gen1 node (sire/dam) that *confidently* matches an existing herd rabbit is converted to a linked node (`mode: 'linked'`, setting `sire_id`/`dam_id`). That branch's grandparents then **lock and defer to the stored record** — the OCR-extracted grandparents on that branch are intentionally **discarded** in favor of the linked rabbit's own pedigree (the herd record is authoritative).
3. **Unmatched ancestors stay manual.** A gen1 with no confident match remains a manual node, fully editable, and its scanned grandparents persist as manual `_name_text` JSON. Identical to hand entry.

**Mechanical caveat — only gen1 can link/lock.** The schema has FK columns only for the subject's sire/dam (`sire_id`/`dam_id`); there is no column to tie a grandparent (gen2/gen3) directly to a herd rabbit. Therefore:
- A gen2/gen3 node can never produce a link or a lock *on its own*, even if its scanned name happens to exist in the herd.
- A grandparent only ends up locked/stored-derived **because its gen1 parent linked** (chain-authority cascade). If gen1 stays manual, its grandparents stay manual text regardless.
- So "does this scanned ancestor match a herd rabbit?" is only ever asked at gen1; the gen2/gen3 case folds entirely into "did its gen1 parent match."

"Confident match" definition (gen1): exact `ear_number` match → link; else exactly one exact-`name` match → link; otherwise → manual node. Ear number takes precedence as the primary key (unique-ish and OCR-reliable); the existing `GET /api/rabbits/search` substring search is too loose to auto-link on directly.

### Q7 — Tesseract assets are self-hosted, not CDN-fetched
By default `tesseract.js` downloads its WASM core, worker script, and the `eng.traineddata` language file (~10–15 MB) from a CDN (jsDelivr/unpkg) on first use. This app is **self-hosted on a single PC** and is built to work offline (service worker, offline banner, PWA), so all OCR assets must be **bundled and served locally** — no third-party CDN calls at runtime.

- Vendor the WASM core, worker, and `eng.traineddata` into the client (e.g. `client/public/tesseract/`) and point `tesseract.js` at them via its `workerPath` / `corePath` / `langPath` options.
- The service worker (`public/sw.js`) should treat these as cacheable app-shell assets so the scanner works fully offline after first load. (Note: the `.traineddata` is several MB — confirm the SW cache strategy handles it; it's static and versioned, so cache-first is appropriate, unlike the network-first app shell.)
- Accept the resulting client bundle/footprint increase as a deliberate trade for offline, dependency-self-contained operation — consistent with the project's self-hosted ethos.

---

## Supported Input

- Printed pedigrees of any layout (ARBA-standard and non-standard)
- 3-generation family trees (subject + 2 parents + 4 grandparents)
- Photo captures from device camera or file upload
- Scanned flat documents or photographed paper (angled/shadowed)

---

## Pipeline

```
[1] Image Input
      ↓
[2] Client-side Preprocessing
      ↓
[3] Tesseract.js Full-Image OCR (with bounding boxes)
      ↓
[4] Layout Analysis → Tree Position Mapping
      ↓
[5] Form Population (nulls highlighted)
      ↓
[6] User Review
      ├── Happy → Save
      └── Field wrong → Zone Picker → Re-OCR → Field Update → Save
```

---

## Step-by-Step Implementation

### Step 1 — Image Input

Use a standard file input with camera capture support:

```jsx
<input
  type="file"
  accept="image/*"
  capture="environment"
  onChange={handleImageUpload}
/>
```

Both file picker and camera capture work via this single input on mobile and desktop.

---

### Step 2 — Client-Side Preprocessing

Before passing to Tesseract, normalize the image using canvas or `jimp`:

- **Resize:** Clamp max width to 1600px. Larger images don't improve OCR and slow processing.
- **Grayscale:** Convert to single-channel.
- **Binarization (threshold):** Apply adaptive threshold to convert to black-and-white. This is the single most impactful step for photos of paper.
- **Deskew:** Optional. Detect dominant angle via Hough transform and rotate to correct. Skip if too complex for v1 — binarization recovers most of the value.

Recommended: use the canvas API for resize and grayscale, then apply a simple threshold pass. `jimp` can handle all of this if preferred.

---

### Step 3 — Tesseract.js OCR

Run Tesseract on the preprocessed image with bounding box output enabled:

```js
import Tesseract from 'tesseract.js';

const { data } = await Tesseract.recognize(preprocessedImage, 'eng', {
  logger: m => console.log(m),
});

// data.words → array of { text, bbox: { x0, y0, x1, y1 }, confidence }
// data.lines → array of { text, bbox, words[] }
```

Use `data.words` with bounding boxes as the primary input to layout analysis. Discard words with confidence below ~60.

---

### Step 4 — Layout Analysis

Pedigrees are structured as a left-to-right generational tree. Map word bounding boxes to tree positions using x/y clustering.

#### Column detection

Divide the image width into 3 bands (or 4 if great-grandparents are present):

| Band | Generation |
|------|------------|
| Leftmost ~25% | Subject |
| 25–50% | Parents (sire top, dam bottom) |
| 50–75% | Grandparents (sire_sire, sire_dam, dam_sire, dam_dam) |
| 75–100% | Great-grandparents (optional) |

Band boundaries can be tuned — start with even splits of image width.

#### Sire / dam split within each column

For each column, sort all words by `y0`. Find the largest vertical gap — this is the dividing line between the sire half and dam half. Everything above the gap → sire lineage, below → dam lineage.

#### Text aggregation

Within each detected cell region, sort words by `y0` then `x0` and join into lines. The first line is typically the rabbit's name, subsequent lines are metadata (tattoo, DOB, breed, variety, weight).

#### Field parsing

Only five fields are extracted per cell:

- **Name:** first non-label line in the cell. Split on possessive pattern to extract farm name and rabbit name:
  ```js
  function parsePedigreeName(fullName) {
    // Handles both "Cody's Black Betty" and "Peachy Buns' Hoppin-pop"
    const match = fullName.match(/^(.+?'s?)\s+(.+)$/);
    if (match) return { farmName: match[1], rabbitName: match[2] };
    return { farmName: null, rabbitName: fullName };
  }
  ```
- **Ear number:** matches `[A-Z0-9]{2,8}` after label "Tattoo:", "Ear #:", "Ear No:", or similar
- **DOB:** matches date patterns (MM/DD/YYYY, MM-DD-YY, etc.)
- **Weight:** matches `\d+(\.\d+)?\s*(lbs?|oz)?`

Breed, variety, sex, and all other fields are not extracted. The user fills them in manually after scanning.

Unmatched or low-confidence fields → `null`.

#### Output shape

```js
{
  subject:   { farmName, rabbitName, earNumber, dob, weight },
  sire:      { farmName, rabbitName, earNumber, dob, weight },
  dam:       { farmName, rabbitName, earNumber, dob, weight },
  sire_sire: { farmName, rabbitName, earNumber, dob, weight },
  sire_dam:  { farmName, rabbitName, earNumber, dob, weight },
  dam_sire:  { farmName, rabbitName, earNumber, dob, weight },
  dam_dam:   { farmName, rabbitName, earNumber, dob, weight },
}
// breed, variety, sex — user fills in manually after scanning
```

---

### Step 5 — Form Population

Populate the existing pedigree entry form with extracted values.

- **Filled fields:** render normally
- **Null fields:** highlight with yellow background and a small "?" indicator
- Show a banner: *"OCR complete — review highlighted fields before saving."*

Do not auto-save. The user must explicitly confirm.

---

### Step 6 — User Review & Zone Picker

#### Normal flow

User scans the form, corrects any highlighted or wrong fields by typing, and saves.

#### Zone picker flow (triggered per field)

When the user clicks a field they want to correct using the image:

1. A panel slides open showing the original (un-preprocessed) image
2. The bounding region currently assigned to that field is highlighted with a colored overlay
3. User drags to draw a new rectangle over the correct area on the image
4. On release, `react-image-crop` returns the crop coordinates
5. Crop is extracted from the preprocessed image, passed back to Tesseract
6. Tesseract result populates the field
7. User can accept or discard the re-OCR result

The zone picker panel should sit alongside or below the form, not in a modal, so the user can see both at once.

#### Zone picker trigger

Add a small camera/scan icon button next to each form field. Clicking the field label or the icon opens the zone picker for that field. Do not require the user to enter a separate "correction mode."

---

## Component Structure

```
PedigreeScanFlow/
  ├── ImageCapture          — file input + camera, triggers preprocessing
  ├── OCRProcessor          — runs Tesseract, exposes result + status
  ├── LayoutAnalyzer        — maps bounding boxes to tree positions
  ├── PedigreeFormWithOCR   — existing pedigree form, extended with:
  │     ├── OCR value injection
  │     ├── Null field highlighting
  │     └── Zone picker trigger per field
  └── ZonePicker            — image display + react-image-crop + re-OCR
```

---

## State Shape

```js
{
  rawImage: Blob,               // original upload
  processedImage: ImageData,    // after preprocessing
  ocrResult: {                  // output of layout analysis
    subject, sire, dam,
    sire_sire, sire_dam,
    dam_sire, dam_dam
  },
  ocrStatus: 'idle' | 'preprocessing' | 'scanning' | 'done' | 'error',
  activeZoneField: string|null, // which field has zone picker open
}
```

---

## Packages

| Package | Purpose |
|---------|---------|
| `tesseract.js` | Client-side OCR |
| `jimp` | Image preprocessing (grayscale, threshold, resize) |
| `react-image-crop` | Zone selection UI for manual correction |

All processing is client-side. No server calls, no API tokens.

---

## Error Handling

- If Tesseract fails entirely → show error, fall back to manual entry
- If all fields come back null → warn the user the image may be too blurry or skewed, suggest retaking
- Individual null fields → highlighted in form, zone picker available

---

## Out of Scope (v1)

- Deskew / perspective correction (nice to have, not required)
- Great-grandparent (4th generation) extraction
- Handwriting support (degrades gracefully to nulls, zone picker handles correction)
- Multi-page pedigrees