# MyKindlebox — OCR Pedigree Scanning: Implementation Plan > **Superseded 2026-10-28.** This Tesseract plan was abandoned after Phase 3 in favor of server-side Claude vision extraction — see the note at the top of `FEATURE_OCR_Pedigree_Scan.md`. Kept for history only. **Feature:** OCR-assisted pedigree entry with manual zone correction fallback **Reference:** `FEATURE_OCR_Pedigree_Scan.md`, `architecture.md` **Scope:** Base feature only — image capture, OCR pipeline, form pre-fill, zone picker correction. --- ## Overview This plan wires a "Scan Pedigree" button into the existing Add Rabbit wizard (Step 1). Tapping it lets the user photograph or upload a printed pedigree; the app extracts up to five fields per ancestor cell and pre-fills the wizard. The user reviews, corrects with a zone picker if needed, then proceeds through the normal wizard flow. All OCR processing is client-side. No server changes are required for the OCR feature itself (Phase 5 adds one new API call for herd matching). --- ## Architectural notes **PedigreeTree injection:** `PedigreeTree` reads `initialState` once at mount and owns its state internally. OCR results must be ready before `PedigreeTree` mounts. The pattern mirrors the existing "Add Kit" flow: block `PedigreeTree` render in `AddRabbit.jsx` until the OCR `initialState` object is fully built, then mount. If the user rescans, bump `PedigreeTree`'s `key` prop to remount with a fresh `initialState`. Do not refactor `PedigreeTree` to accept controlled post-mount updates. **Chain-authority cascade:** `PedigreeTree` cascades live — the moment a gen1 node is `mode: 'linked'`, `fetchCascade` fires and its grandparent branch locks immediately. There is no deferred-link model. If a gen1 node matches a herd rabbit, seed it as `mode: 'linked'` in `initialState`; the OCR-extracted grandparents on that branch are discarded at mount and replaced by the linked rabbit's stored pedigree. This is correct behavior — the herd record is authoritative. **Herd matching endpoint:** Use `GET /api/rabbits/quick-search?fields=ear_number&q=` and filter client-side to an exact `ear_number` match. If no exact ear number match, fall back to `quick-search?fields=name&q=` with a single exact name match. Do not use `/search` — it is name-only substring and will not match ear numbers. **Field remapping:** OCR emits `{ farmName, rabbitName, earNumber, weight, dob }`. `serializeManualAncestor` expects `{ name, farm, ear_number, weight_lbs, dob }`. Always remap explicitly: `rabbitName → name`, `farmName → farm`, `earNumber → ear_number`, `weight → weight_lbs`. Never pass OCR output to `serializeManualAncestor` directly. **Zone picker image:** The zone picker displays the preprocessed image blob (not the original). All stored bounding box coordinates are in preprocessed (≤1600px) coordinate space and only align correctly against the preprocessed image. Discard the original file after preprocessing completes — only the preprocessed blob is retained in state. **Null farmName and the Save guard:** The manual-entry editor requires farm + name + ear number before enabling Save. An ancestor whose OCR-extracted `farmName` is null will pre-fill with an empty farm field and the Save button will be disabled until the user fills it in. The yellow highlight on the farm field signals this. This is intentional — do not weaken the existing Save guard. --- ## Phase 1 — Install and vendor OCR dependencies ### Step 1.1 — Install packages Install the two required packages into the client: - `tesseract.js` - `react-image-crop` Do not install `jimp` — all image preprocessing is handled via the canvas API (see Phase 2). **Test:** Run the client dev server. Confirm no import errors in the browser console. Confirm both packages appear in `client/package.json` dependencies. --- ### Step 1.2 — Vendor Tesseract assets locally Copy the following Tesseract runtime assets into `client/public/tesseract/`: - The Tesseract WASM core file - The Tesseract worker script - The English language training data file (`eng.traineddata`) These files must be served from the app itself — no CDN calls at runtime. The app is self-hosted and must work offline. Configure `tesseract.js` (wherever it is initialized) to point `workerPath`, `corePath`, and `langPath` at `./tesseract/` relative to the app's public root. **Test:** Disconnect the dev machine from the internet. Load the app. Trigger a Tesseract recognize call on any test image (a photo of text). Confirm it completes successfully without any network requests to jsDelivr, unpkg, or any external CDN. Check the Network tab in DevTools to verify. --- ### Step 1.3 — Add Tesseract assets to service worker cache In `client/public/sw.js`, add the `tesseract/` directory assets to the app-shell cache list alongside other static assets. Use cache-first strategy for these files — they are static, versioned, and large. **Test:** Load the app once online. Open DevTools → Application → Cache Storage. Confirm the Tesseract WASM, worker, and `eng.traineddata` files are present in the cache. Disconnect from the internet and reload. Confirm the scanner still initializes without errors. --- ## Phase 2 — Build the preprocessing utility ### Step 2.1 — Create `client/src/lib/ocrPreprocess.js` Create a new utility module using only the browser canvas API (no third-party image library). It exports a single async function `preprocessImage(file)` that accepts a `File` or `Blob` and returns a processed image blob ready for Tesseract. The function must perform these steps in order: 1. Decode the file into an `HTMLImageElement` via `URL.createObjectURL` 2. Draw it onto a canvas, scaling down proportionally if width exceeds 1600px 3. Read pixel data via `getImageData` and convert each pixel to grayscale (standard luminance formula: `0.299R + 0.587G + 0.114B`) 4. Apply a binarization pass: convert each grayscale pixel to black (0) or white (255) based on a luminance threshold (start at 128, tune if needed) 5. Write the result back to the canvas and return it as a PNG `Blob` via `canvas.toBlob` **Test:** Import the function in a temporary test component. Pass it a phone photo of printed text. Render the output blob in an `` tag. Confirm the result is black-and-white, high-contrast (text clearly black on white background), and no wider than 1600px. --- ### Step 2.2 — Validate preprocessing on a real pedigree photo Using a real printed pedigree photographed at a slight angle or in non-ideal lighting, run it through `preprocessImage`. Confirm the output is readable. **Test:** Visually inspect the output image. Text must be legible and clearly distinct from the background. If binarization produces large black blobs obscuring text, lower the threshold value. If text is faint or missing, raise it. --- ## Phase 3 — Build the OCR and layout analysis utilities ### Step 3.1 — Create `client/src/lib/ocrRun.js` Create a utility module that exports `runOCR(imageBlob)`. It accepts a preprocessed image blob and returns the raw Tesseract result object containing `words` (with bounding boxes and confidence scores) and `lines`. Initialize Tesseract using the local vendor paths configured in Step 1.2. Before returning, filter out any word objects with a confidence score below 60. **Test:** Pass a preprocessed pedigree image into `runOCR`. Log the returned `words` array. Confirm word objects contain `text`, `bbox` (with `x0`, `y0`, `x1`, `y1`), and `confidence`. Confirm low-confidence noise words are absent from the result. --- ### Step 3.2 — Create `client/src/lib/ocrLayout.js` — column detection Create a utility module for layout analysis. First export: `detectColumns(words, imageWidth)`. This function divides the image width into 3 equal bands and assigns each word to a column (0 = subject, 1 = parents, 2 = grandparents) based on its `x0` position. Returns an array of three word arrays, one per column. **Test:** Run `detectColumns` on a word list from a real pedigree OCR result. Log the three column arrays. Confirm the subject column contains the leftmost text, and the grandparent column contains the rightmost text. --- ### Step 3.3 — Add sire/dam split to `ocrLayout.js` Add export: `splitSireDam(columnWords)`. For a given column's word array, sort by `y0`. Find the largest vertical gap between adjacent words. Return `{ sire: wordsAboveGap, dam: wordsBelowGap }`. **Test:** Run `splitSireDam` on the parents column from a real pedigree OCR result. Confirm the top half contains the sire entry and the bottom half contains the dam entry. The split point should fall in the blank space between the two ancestor cells. --- ### Step 3.4 — Add cell text aggregation to `ocrLayout.js` Add export: `aggregateCellText(words)`. For a given array of words belonging to one cell, sort by `y0` then `x0`. Group words whose `y0` values are within ~10px of each other onto the same line. Join lines with newline characters. Returns a plain multiline string representing everything OCR'd in that cell. **Test:** Run `aggregateCellText` on the sire word array from a real pedigree. Confirm the output reads as natural lines of text matching what is printed in that cell on the pedigree. --- ### Step 3.5 — Create `client/src/lib/ocrParse.js` — field extraction Create a utility module that exports `parseCellFields(cellText)`. It accepts a multiline string (output of `aggregateCellText`) and returns: ``` { farmName: string | null, rabbitName: string | null, earNumber: string | null, dob: string | null, // YYYY-MM-DD or null weight: number | null, // decimal lbs or null } ``` Rules per field: - **Name (farmName + rabbitName):** Take the first non-label line of the cell. Apply the possessive split: match `^(.+?'s?)\s+(.+)$`. If matched, `farmName` = first capture group, `rabbitName` = second. If no possessive pattern is found, set `farmName = null` and `rabbitName` = the full first line. - **Ear number:** Scan all lines for a label (`Tattoo:`, `Ear #:`, `Ear No:`, or similar) followed by a value matching `[A-Z0-9]{2,8}`. Also accept a bare `[A-Z0-9]{2,8}` token on its own line when no label is present. - **DOB:** Scan all lines for date patterns: `MM/DD/YYYY`, `MM-DD-YYYY`, `MM/DD/YY`, `M/D/YY`. Convert any match to `YYYY-MM-DD`. Return `null` if no date is found or parsing fails. Do not use `"Unknown"` here — that sentinel is only written at save time. - **Weight:** Scan all lines for weight patterns and normalize to a decimal float in lbs: `"8 lbs 4 oz"` → `8.25`, `"8.5 lbs"` → `8.5`, `"8.5"` with a trailing `lbs`/`lb` → `8.5`. Return `null` if no weight is found. **Test:** Write at least three test cases using real cell text strings extracted from pedigree OCR results: 1. A cell with all five fields present and labelled 2. A cell with only name and ear number (DOB and weight absent) 3. A cell where the name has no possessive (rabbit name only, no farm) Confirm outputs match expected values for each case. --- ### Step 3.6 — Create `client/src/lib/ocrPipeline.js` — full pipeline orchestrator Create a utility that exports `extractPedigree(imageBlob)`. It: 1. Calls `runOCR(imageBlob)` to get the word list 2. Calls `detectColumns` to split words into subject / parents / grandparents columns 3. Subject column: calls `aggregateCellText` on all words, then `parseCellFields` 4. Parents column: calls `splitSireDam`, then `aggregateCellText` + `parseCellFields` on each half 5. Grandparents column: divides into four quadrants at the horizontal and vertical midpoints of the column (top-left = sire_sire, bottom-left = sire_dam, top-right = dam_sire, bottom-right = dam_dam), then `aggregateCellText` + `parseCellFields` on each 6. Returns the full output shape: ``` { subject: { farmName, rabbitName, earNumber, dob, weight }, sire: { farmName, rabbitName, earNumber, dob, weight }, dam: { farmName, rabbitName, earNumber, dob, weight }, sire_sire: { farmName, rabbitName, earNumber, dob, weight }, sire_dam: { farmName, rabbitName, earNumber, dob, weight }, dam_sire: { farmName, rabbitName, earNumber, dob, weight }, dam_dam: { farmName, rabbitName, earNumber, dob, weight }, } ``` **Test:** Run `extractPedigree` on a preprocessed real pedigree image. Log the output. Confirm all seven keys are present. Confirm at minimum the subject name and sire name are correctly extracted. Null values on sparse fields are acceptable — confirm no crashes or thrown errors. --- ## Phase 4 — Build the OCR UI components ### Step 4.1 — Create `ImageCapture` component Create `client/src/components/ocr/ImageCapture.jsx`. Renders a styled button labelled "Scan Pedigree". When clicked, it triggers a hidden file input that accepts images and requests the rear camera on mobile (`accept="image/*" capture="environment"`). On file selection, calls an `onImage(file)` prop with the selected `File` object. Does not perform any processing — the caller handles that. **Test:** Mount `ImageCapture` in isolation. Click the button. Confirm the file picker opens. Select an image file. Confirm `onImage` is called with the correct `File` object. In Chrome's mobile device emulator, confirm the camera capture option appears. --- ### Step 4.2 — Create `OCRProcessor` component Create `client/src/components/ocr/OCRProcessor.jsx`. Accepts props: `file` (File | null), `onResult({ ocrData, processedBlob })`, `onError(message)`. When `file` changes to a non-null value, it: 1. Sets internal status to `'preprocessing'` 2. Calls `preprocessImage(file)` → stores the resulting blob as `processedBlob` 3. Sets status to `'scanning'` 4. Calls `extractPedigree(processedBlob)` 5. Calls `onResult` with `{ ocrData, processedBlob }` — both are needed by the caller (ocrData to build `initialState`, processedBlob to retain for the zone picker) 6. Sets status to `'done'` Renders a loading indicator with status-appropriate text while processing (`'preprocessing'` → "Preparing image…", `'scanning'` → "Reading pedigree…"). Renders nothing when idle or done. On any error, calls `onError` with a human-readable message. **Test:** Mount `OCRProcessor` with a real pedigree image `File`. Confirm the loading states render in sequence. Confirm `onResult` is called with a non-null `ocrData` object containing the seven ancestor keys and a valid `processedBlob`. Confirm no errors on a clean image. --- ### Step 4.3 — Create `ZonePicker` component Create `client/src/components/ocr/ZonePicker.jsx`. Accepts props: `image` (Blob — the **preprocessed** image), `fieldLabel` (string), `onConfirm(rawText)`, `onCancel()`. There is no `initialCrop` prop — bounding boxes are not retained by `extractPedigree`, so no pre-drawn rectangle is shown. The user always draws the crop region from scratch. Renders: - The preprocessed image displayed via `react-image-crop` with no initial selection - A "Re-scan" button that crops the image canvas to the drawn rectangle, runs `runOCR` on the crop, extracts the raw text from the result, and calls `onConfirm(rawText)` - A "Cancel" button that calls `onCancel` `ZonePicker` does not parse or apply field values — it returns raw OCR text only. The caller is responsible for parsing and updating state. **Test:** Mount `ZonePicker` with a preprocessed pedigree image. Confirm the image renders in the crop UI. Draw a rectangle around a single line of text on the pedigree. Click "Re-scan". Confirm `onConfirm` is called with a string that matches the text in the drawn region. --- ## Phase 5 — Integrate into the Add Rabbit wizard ### Step 5.1 — Add "Scan Pedigree" entry point to Step 1 In `AddRabbit.jsx`, add a "Scan Pedigree" button to Step 1 (the pedigree tree step), above the existing tree controls. Add the following state to `AddRabbit.jsx`: - `ocrFile` — the raw `File` selected by the user (null until a scan is triggered) - `ocrData` — the structured result from `extractPedigree` (null until OCR completes) - `processedBlob` — the preprocessed image blob retained for the zone picker (null until OCR completes) - `ocrStatus` — `'idle' | 'preprocessing' | 'scanning' | 'done' | 'error'` Clicking "Scan Pedigree" renders `ImageCapture`. On file selection, set `ocrFile`. `OCRProcessor` watches `ocrFile` and runs the pipeline. On `onResult`, store `ocrData` and `processedBlob` in state and set `ocrStatus` to `'done'`. The existing pedigree tree controls remain unchanged and fully functional. **Test:** Open the Add Rabbit wizard. Confirm "Scan Pedigree" is visible on Step 1. Confirm clicking it reveals the image picker. Confirm selecting an image triggers the OCR pipeline (loading states appear). Confirm the existing tree controls are still usable after the button is added. --- ### Step 5.2 — Build `initialState` from OCR result and mount `PedigreeTree` `PedigreeTree` must not mount until `initialState` is ready. Block its render while `ocrStatus` is `'preprocessing'` or `'scanning'`. When `ocrData` is available, build `initialState` for `PedigreeTree` as follows: **Gen1 (sire/dam) — herd matching (see Step 5.3 for matching logic):** - Confident herd match found → seed as `{ mode: 'linked', id: }` - No match → seed as `{ mode: 'manual', name: serializeManualAncestor(remapped) }` **Gen2/gen3 — always manual:** - Seed as `{ mode: 'manual', name: serializeManualAncestor(remapped) }` **Field remapping for `serializeManualAncestor`** (apply to every manual node): | OCR field | `serializeManualAncestor` field | |---|---| | `rabbitName` | `name` | | `farmName` | `farm` | | `earNumber` | `ear_number` | | `weight` (null → `0`) | `weight_lbs` | | `dob` (null → `"Unknown"`) | `dob` | Once `initialState` is built, mount `PedigreeTree` with it. If the user rescans, bump `PedigreeTree`'s `key` prop to remount with a fresh `initialState` built from the new `ocrData`. **Test:** Scan a pedigree. Confirm `PedigreeTree` does not render during preprocessing/scanning. Confirm it mounts with the correct ancestor data once `ocrData` is ready. Rescan with a different pedigree — confirm the tree remounts and shows the new data. --- ### Step 5.3 — Gen1 herd matching For each of the sire and dam nodes independently, attempt a herd match before building `initialState`: 1. Call `GET /api/rabbits/quick-search?fields=ear_number&q=` (skip if `earNumber` is null) 2. Filter results client-side to an exact `ear_number` match (case-insensitive) 3. If exactly one exact match → use that rabbit's id for a linked node 4. If no exact ear number match → call `GET /api/rabbits/quick-search?fields=name&q=` (skip if `rabbitName` is null) 5. Filter results client-side to an exact `name` match 6. If exactly one exact name match → use that rabbit's id for a linked node 7. If still no match → fall back to a manual node Ear number takes precedence. Both searches must complete before `initialState` is built and `PedigreeTree` is allowed to mount. **Test:** - Add a rabbit to the herd with a known ear number. Scan a pedigree where that ear number appears as the sire. Confirm the sire slot mounts as a linked node showing the matched rabbit's name. - Scan a pedigree where neither sire nor dam exists in the herd. Confirm both gen1 slots mount as manual nodes with the OCR-extracted name and ear number. --- ### Step 5.4 — Wire subject fields into `details` state When `ocrData` is available, apply the subject fields to the wizard's `details` state slice: | OCR field | `details` field | |---|---| | `subject.farmName` | `farm` | | `subject.rabbitName` | `name` | | `subject.earNumber` | `ear_number` | | `subject.dob` | `dob` (already `YYYY-MM-DD`) | | `subject.weight` | `weight_lbs` | For any field where the OCR result is `null`, leave the existing `details` value unchanged. Do not overwrite with null. Use the "adjusting state during render" pattern (a `useRef` flag) consistent with `AddRabbit.jsx`'s existing one-shot default-fill pattern (e.g. how `details.farm` is seeded from `settings.rabbitry_name`). Do not use a `useEffect` that calls `setState`. **Test:** Scan a pedigree. Navigate to Step 2. Confirm the subject's farm, name, ear number, DOB, and weight are pre-filled. Confirm null fields are blank (not showing "null"). Confirm already-typed details values are not overwritten by null OCR results. --- ### Step 5.5 — Highlight null OCR fields After OCR data is applied, any field that received a `null` OCR result should display a yellow background and a small "?" indicator alongside it. This applies to: - Individual fields within each ancestor slot's manual editor in `PedigreeTree` - The five subject fields in the Step 2 details form The highlight is visual only — the field remains fully editable. Remove the highlight when the user types a non-empty value into that field. **Test:** Scan a pedigree where DOB is absent from one ancestor. Open that ancestor's manual editor. Confirm the DOB field has a yellow highlight and "?" indicator. Type a value. Confirm the highlight disappears. --- ### Step 5.6 — Show OCR status banner After OCR completes, display a dismissible banner above the pedigree tree on Step 1: - **Success (some fields populated):** *"Pedigree scanned — review the tree and check highlighted fields before continuing."* - **All fields null (no text found):** *"No text detected. Try retaking the photo with better lighting."* - **Tesseract error:** *"Couldn't read this pedigree. Check the image quality or enter manually."* The wizard must remain fully usable for manual entry in all three cases. **Test:** - Scan a clean pedigree. Confirm the success banner appears. - Pass a completely black or blank image. Confirm the no-text banner appears. - Temporarily move the Tesseract WASM out of `public/tesseract/` to force a load failure. Confirm the error banner appears and the wizard is still usable for manual entry. --- ## Phase 6 — Zone picker correction flow ### Step 6.1 — Add zone picker trigger to ancestor slot editors In the manual-entry editor for each ancestor slot (the panel that opens when the user clicks a manual node in `PedigreeTree.jsx`), add a small scan/camera icon button next to each of the five fields (farm name, rabbit name, ear number, DOB, weight). Clicking the icon opens the `ZonePicker` panel. Pass it: - `image` — the `processedBlob` from `AddRabbit` state Do not pass `initialCrop` — the prop does not exist. The user draws the crop region from scratch. - `fieldLabel` — a human-readable label for the field being corrected Track which field triggered the picker in local state (`activeZoneField`). The `ZonePicker` panel should render alongside or below the editor, not in a modal, so the user can see both simultaneously. **Test:** After scanning a pedigree, click an ancestor slot to open its editor. Confirm each field has a scan icon button. Click the scan icon next to the ear number field. Confirm the `ZonePicker` opens showing the preprocessed pedigree image with no pre-drawn selection. Confirm the user can draw a crop rectangle freely. --- ### Step 6.2 — Apply zone picker result to the ancestor editor When `ZonePicker` calls `onConfirm(rawText)`: 1. Run `rawText` through the appropriate parser from `ocrParse.js` for the active field (ear number regex, possessive name split, date parser, or weight normalizer) 2. Update the relevant field(s) in the ancestor editor's local state 3. Close `ZonePicker` (clear `activeZoneField`) 4. Remove the yellow highlight from the updated field When `ZonePicker` calls `onCancel`, close the panel without modifying any state. Note: for a name re-scan, the possessive split may produce both `farmName` and `rabbitName`. Update both fields if the scan icon was on either name field. **Test:** After scanning a pedigree where ear number was null, click the ear number scan icon. Draw a rectangle over the ear number. Click "Re-scan". Confirm the ear number field updates with the correct value. Confirm the yellow highlight is gone. Confirm no other fields in the editor changed. --- ### Step 6.3 — Add zone picker trigger to Step 2 subject fields In the Step 2 details form, add scan icon buttons next to the five OCR-populated fields: farm name, rabbit name, ear number, DOB, weight. These work identically to Steps 6.1–6.2 but update `details` state in `AddRabbit` instead of an ancestor editor's local state. **Test:** After scanning a pedigree where subject DOB was null (yellow highlight visible on DOB field), navigate to Step 2. Click the scan icon next to DOB. Draw a rectangle over the DOB on the pedigree image. Click "Re-scan". Confirm the DOB field updates with the correct value and the yellow highlight is removed. --- ## Phase 7 — End-to-end validation ### Step 7.1 — Full happy-path scan and save Using a real printed pedigree (ARBA-format or non-standard): 1. Open Add Rabbit wizard 2. Click "Scan Pedigree" and photograph or upload the pedigree 3. Wait for OCR to complete 4. Review Step 1 — confirm the pedigree tree is populated with ancestor data 5. Click Next 6. Review Step 2 — confirm subject fields are pre-filled 7. Fill in breed and any null fields manually 8. Save **Test:** Confirm the saved rabbit appears in the herd with correct name, ear number, and pedigree ancestors. Confirm no console errors during the entire flow. --- ### Step 7.2 — Linked gen1 cascade behavior Scan a pedigree where the sire exists in the herd (matched by ear number). **Test:** Confirm the sire slot in the tree shows as a linked node. Confirm the sire's grandparent branch (sire_sire, sire_dam) is locked and derives from the linked rabbit's own stored pedigree — not from the OCR-extracted grandparent text. Confirm the dam branch (if unmatched) retains OCR-extracted manual text. --- ### Step 7.3 — Zone picker correction round-trip 1. Scan a pedigree where at least one field is wrong or null 2. Use the zone picker to correct that field 3. Complete the wizard and save **Test:** Confirm the corrected value — not the original OCR extraction — is saved to the database. --- ### Step 7.4 — Offline scan 1. Load the app once while online 2. Disconnect from the internet 3. Complete a full scan and save **Test:** Confirm OCR completes without network errors. Confirm the saved rabbit appears in the herd. Confirm no console errors. --- ### Step 7.5 — Manual entry fallback after failed scan 1. Scan an unreadable image (blank paper, very dark photo) 2. Confirm the appropriate error banner appears 3. Dismiss the banner and complete the wizard via manual entry **Test:** Confirm the wizard is fully usable for manual entry after a failed scan. Confirm no state corruption from the failed OCR attempt — the tree and details form behave as if no scan was attempted. --- ## Out of Scope (do not implement in this phase) - Deskew / perspective correction - Great-grandparent (4th generation) extraction - Handwriting-specific handling - Multi-page pedigrees - Bulk scanning multiple rabbits at once