27 KiB

Raw Permalink Blame History

MyKindlebox — OCR Pedigree Scanning: Implementation Plan

Superseded 2026-10-28. This Tesseract plan was abandoned after Phase 3 in favor of server-side Claude vision extraction — see the note at the top of FEATURE_OCR_Pedigree_Scan.md. Kept for history only.

Feature: OCR-assisted pedigree entry with manual zone correction fallback
Reference: FEATURE_OCR_Pedigree_Scan.md, architecture.md
Scope: Base feature only — image capture, OCR pipeline, form pre-fill, zone picker correction.

Overview

This plan wires a "Scan Pedigree" button into the existing Add Rabbit wizard (Step 1). Tapping it lets the user photograph or upload a printed pedigree; the app extracts up to five fields per ancestor cell and pre-fills the wizard. The user reviews, corrects with a zone picker if needed, then proceeds through the normal wizard flow.

All OCR processing is client-side. No server changes are required for the OCR feature itself (Phase 5 adds one new API call for herd matching).

Architectural notes

PedigreeTree injection: PedigreeTree reads initialState once at mount and owns its state internally. OCR results must be ready before PedigreeTree mounts. The pattern mirrors the existing "Add Kit" flow: block PedigreeTree render in AddRabbit.jsx until the OCR initialState object is fully built, then mount. If the user rescans, bump PedigreeTree's key prop to remount with a fresh initialState. Do not refactor PedigreeTree to accept controlled post-mount updates.

Chain-authority cascade: PedigreeTree cascades live — the moment a gen1 node is mode: 'linked', fetchCascade fires and its grandparent branch locks immediately. There is no deferred-link model. If a gen1 node matches a herd rabbit, seed it as mode: 'linked' in initialState; the OCR-extracted grandparents on that branch are discarded at mount and replaced by the linked rabbit's stored pedigree. This is correct behavior — the herd record is authoritative.

Herd matching endpoint: Use GET /api/rabbits/quick-search?fields=ear_number&q=<earNumber> and filter client-side to an exact ear_number match. If no exact ear number match, fall back to quick-search?fields=name&q=<rabbitName> with a single exact name match. Do not use /search — it is name-only substring and will not match ear numbers.

Field remapping: OCR emits { farmName, rabbitName, earNumber, weight, dob }. serializeManualAncestor expects { name, farm, ear_number, weight_lbs, dob }. Always remap explicitly: rabbitName → name, farmName → farm, earNumber → ear_number, weight → weight_lbs. Never pass OCR output to serializeManualAncestor directly.

Zone picker image: The zone picker displays the preprocessed image blob (not the original). All stored bounding box coordinates are in preprocessed (≤1600px) coordinate space and only align correctly against the preprocessed image. Discard the original file after preprocessing completes — only the preprocessed blob is retained in state.

Null farmName and the Save guard: The manual-entry editor requires farm + name + ear number before enabling Save. An ancestor whose OCR-extracted farmName is null will pre-fill with an empty farm field and the Save button will be disabled until the user fills it in. The yellow highlight on the farm field signals this. This is intentional — do not weaken the existing Save guard.

Phase 1 — Install and vendor OCR dependencies

Step 1.1 — Install packages

Install the two required packages into the client:

tesseract.js
react-image-crop

Do not install jimp — all image preprocessing is handled via the canvas API (see Phase 2).

Test: Run the client dev server. Confirm no import errors in the browser console. Confirm both packages appear in client/package.json dependencies.

Step 1.2 — Vendor Tesseract assets locally

Copy the following Tesseract runtime assets into client/public/tesseract/:

The Tesseract WASM core file
The Tesseract worker script
The English language training data file (eng.traineddata)

These files must be served from the app itself — no CDN calls at runtime. The app is self-hosted and must work offline.

Configure tesseract.js (wherever it is initialized) to point workerPath, corePath, and langPath at ./tesseract/ relative to the app's public root.

Test: Disconnect the dev machine from the internet. Load the app. Trigger a Tesseract recognize call on any test image (a photo of text). Confirm it completes successfully without any network requests to jsDelivr, unpkg, or any external CDN. Check the Network tab in DevTools to verify.

Step 1.3 — Add Tesseract assets to service worker cache

In client/public/sw.js, add the tesseract/ directory assets to the app-shell cache list alongside other static assets. Use cache-first strategy for these files — they are static, versioned, and large.

Test: Load the app once online. Open DevTools → Application → Cache Storage. Confirm the Tesseract WASM, worker, and eng.traineddata files are present in the cache. Disconnect from the internet and reload. Confirm the scanner still initializes without errors.

Phase 2 — Build the preprocessing utility

Step 2.1 — Create `client/src/lib/ocrPreprocess.js`

Create a new utility module using only the browser canvas API (no third-party image library). It exports a single async function preprocessImage(file) that accepts a File or Blob and returns a processed image blob ready for Tesseract.

The function must perform these steps in order:

Decode the file into an HTMLImageElement via URL.createObjectURL
Draw it onto a canvas, scaling down proportionally if width exceeds 1600px
Read pixel data via getImageData and convert each pixel to grayscale (standard luminance formula: 0.299R + 0.587G + 0.114B)
Apply a binarization pass: convert each grayscale pixel to black (0) or white (255) based on a luminance threshold (start at 128, tune if needed)
Write the result back to the canvas and return it as a PNG Blob via canvas.toBlob

Test: Import the function in a temporary test component. Pass it a phone photo of printed text. Render the output blob in an <img> tag. Confirm the result is black-and-white, high-contrast (text clearly black on white background), and no wider than 1600px.

Step 2.2 — Validate preprocessing on a real pedigree photo

Using a real printed pedigree photographed at a slight angle or in non-ideal lighting, run it through preprocessImage. Confirm the output is readable.

Test: Visually inspect the output image. Text must be legible and clearly distinct from the background. If binarization produces large black blobs obscuring text, lower the threshold value. If text is faint or missing, raise it.

Phase 3 — Build the OCR and layout analysis utilities

Step 3.1 — Create `client/src/lib/ocrRun.js`

Create a utility module that exports runOCR(imageBlob). It accepts a preprocessed image blob and returns the raw Tesseract result object containing words (with bounding boxes and confidence scores) and lines.

Initialize Tesseract using the local vendor paths configured in Step 1.2.

Before returning, filter out any word objects with a confidence score below 60.

Test: Pass a preprocessed pedigree image into runOCR. Log the returned words array. Confirm word objects contain text, bbox (with x0, y0, x1, y1), and confidence. Confirm low-confidence noise words are absent from the result.

Step 3.2 — Create `client/src/lib/ocrLayout.js` — column detection

Create a utility module for layout analysis. First export: detectColumns(words, imageWidth).

This function divides the image width into 3 equal bands and assigns each word to a column (0 = subject, 1 = parents, 2 = grandparents) based on its x0 position. Returns an array of three word arrays, one per column.

Test: Run detectColumns on a word list from a real pedigree OCR result. Log the three column arrays. Confirm the subject column contains the leftmost text, and the grandparent column contains the rightmost text.

Step 3.3 — Add sire/dam split to `ocrLayout.js`

Add export: splitSireDam(columnWords).

For a given column's word array, sort by y0. Find the largest vertical gap between adjacent words. Return { sire: wordsAboveGap, dam: wordsBelowGap }.

Test: Run splitSireDam on the parents column from a real pedigree OCR result. Confirm the top half contains the sire entry and the bottom half contains the dam entry. The split point should fall in the blank space between the two ancestor cells.

Step 3.4 — Add cell text aggregation to `ocrLayout.js`

Add export: aggregateCellText(words).

For a given array of words belonging to one cell, sort by y0 then x0. Group words whose y0 values are within ~10px of each other onto the same line. Join lines with newline characters.

Returns a plain multiline string representing everything OCR'd in that cell.

Test: Run aggregateCellText on the sire word array from a real pedigree. Confirm the output reads as natural lines of text matching what is printed in that cell on the pedigree.

Step 3.5 — Create `client/src/lib/ocrParse.js` — field extraction

Create a utility module that exports parseCellFields(cellText).

It accepts a multiline string (output of aggregateCellText) and returns:

{
  farmName: string | null,
  rabbitName: string | null,
  earNumber: string | null,
  dob: string | null,       // YYYY-MM-DD or null
  weight: number | null,    // decimal lbs or null
}

Rules per field:

Name (farmName + rabbitName): Take the first non-label line of the cell. Apply the possessive split: match ^(.+?'s?)\s+(.+)$. If matched, farmName = first capture group, rabbitName = second. If no possessive pattern is found, set farmName = null and rabbitName = the full first line.
Ear number: Scan all lines for a label (Tattoo:, Ear #:, Ear No:, or similar) followed by a value matching [A-Z0-9]{2,8}. Also accept a bare [A-Z0-9]{2,8} token on its own line when no label is present.
DOB: Scan all lines for date patterns: MM/DD/YYYY, MM-DD-YYYY, MM/DD/YY, M/D/YY. Convert any match to YYYY-MM-DD. Return null if no date is found or parsing fails. Do not use "Unknown" here — that sentinel is only written at save time.
Weight: Scan all lines for weight patterns and normalize to a decimal float in lbs: "8 lbs 4 oz" → 8.25, "8.5 lbs" → 8.5, "8.5" with a trailing lbs/lb → 8.5. Return null if no weight is found.

Test: Write at least three test cases using real cell text strings extracted from pedigree OCR results:

A cell with all five fields present and labelled
A cell with only name and ear number (DOB and weight absent)
A cell where the name has no possessive (rabbit name only, no farm)

Confirm outputs match expected values for each case.

Step 3.6 — Create `client/src/lib/ocrPipeline.js` — full pipeline orchestrator

Create a utility that exports extractPedigree(imageBlob). It:

Calls runOCR(imageBlob) to get the word list
Calls detectColumns to split words into subject / parents / grandparents columns
Subject column: calls aggregateCellText on all words, then parseCellFields
Parents column: calls splitSireDam, then aggregateCellText + parseCellFields on each half
Grandparents column: divides into four quadrants at the horizontal and vertical midpoints of the column (top-left = sire_sire, bottom-left = sire_dam, top-right = dam_sire, bottom-right = dam_dam), then aggregateCellText + parseCellFields on each
Returns the full output shape:

{
  subject:   { farmName, rabbitName, earNumber, dob, weight },
  sire:      { farmName, rabbitName, earNumber, dob, weight },
  dam:       { farmName, rabbitName, earNumber, dob, weight },
  sire_sire: { farmName, rabbitName, earNumber, dob, weight },
  sire_dam:  { farmName, rabbitName, earNumber, dob, weight },
  dam_sire:  { farmName, rabbitName, earNumber, dob, weight },
  dam_dam:   { farmName, rabbitName, earNumber, dob, weight },
}

Test: Run extractPedigree on a preprocessed real pedigree image. Log the output. Confirm all seven keys are present. Confirm at minimum the subject name and sire name are correctly extracted. Null values on sparse fields are acceptable — confirm no crashes or thrown errors.

Phase 4 — Build the OCR UI components

Step 4.1 — Create `ImageCapture` component

Create client/src/components/ocr/ImageCapture.jsx.

Renders a styled button labelled "Scan Pedigree". When clicked, it triggers a hidden file input that accepts images and requests the rear camera on mobile (accept="image/*" capture="environment").

On file selection, calls an onImage(file) prop with the selected File object. Does not perform any processing — the caller handles that.

Test: Mount ImageCapture in isolation. Click the button. Confirm the file picker opens. Select an image file. Confirm onImage is called with the correct File object. In Chrome's mobile device emulator, confirm the camera capture option appears.

Step 4.2 — Create `OCRProcessor` component

Create client/src/components/ocr/OCRProcessor.jsx.

Accepts props: file (File | null), onResult({ ocrData, processedBlob }), onError(message).

When file changes to a non-null value, it:

Sets internal status to 'preprocessing'
Calls preprocessImage(file) → stores the resulting blob as processedBlob
Sets status to 'scanning'
Calls extractPedigree(processedBlob)
Calls onResult with { ocrData, processedBlob } — both are needed by the caller (ocrData to build initialState, processedBlob to retain for the zone picker)
Sets status to 'done'

Renders a loading indicator with status-appropriate text while processing ('preprocessing' → "Preparing image…", 'scanning' → "Reading pedigree…"). Renders nothing when idle or done.

On any error, calls onError with a human-readable message.

Test: Mount OCRProcessor with a real pedigree image File. Confirm the loading states render in sequence. Confirm onResult is called with a non-null ocrData object containing the seven ancestor keys and a valid processedBlob. Confirm no errors on a clean image.

Step 4.3 — Create `ZonePicker` component

Create client/src/components/ocr/ZonePicker.jsx.

Accepts props: image (Blob — the preprocessed image), fieldLabel (string), onConfirm(rawText), onCancel().

There is no initialCrop prop — bounding boxes are not retained by extractPedigree, so no pre-drawn rectangle is shown. The user always draws the crop region from scratch.

Renders:

The preprocessed image displayed via react-image-crop with no initial selection
A "Re-scan" button that crops the image canvas to the drawn rectangle, runs runOCR on the crop, extracts the raw text from the result, and calls onConfirm(rawText)
A "Cancel" button that calls onCancel

ZonePicker does not parse or apply field values — it returns raw OCR text only. The caller is responsible for parsing and updating state.

Test: Mount ZonePicker with a preprocessed pedigree image. Confirm the image renders in the crop UI. Draw a rectangle around a single line of text on the pedigree. Click "Re-scan". Confirm onConfirm is called with a string that matches the text in the drawn region.

Phase 5 — Integrate into the Add Rabbit wizard

Step 5.1 — Add "Scan Pedigree" entry point to Step 1

In AddRabbit.jsx, add a "Scan Pedigree" button to Step 1 (the pedigree tree step), above the existing tree controls.

Add the following state to AddRabbit.jsx:

ocrFile — the raw File selected by the user (null until a scan is triggered)
ocrData — the structured result from extractPedigree (null until OCR completes)
processedBlob — the preprocessed image blob retained for the zone picker (null until OCR completes)
ocrStatus — 'idle' | 'preprocessing' | 'scanning' | 'done' | 'error'

Clicking "Scan Pedigree" renders ImageCapture. On file selection, set ocrFile. OCRProcessor watches ocrFile and runs the pipeline. On onResult, store ocrData and processedBlob in state and set ocrStatus to 'done'.

The existing pedigree tree controls remain unchanged and fully functional.

Test: Open the Add Rabbit wizard. Confirm "Scan Pedigree" is visible on Step 1. Confirm clicking it reveals the image picker. Confirm selecting an image triggers the OCR pipeline (loading states appear). Confirm the existing tree controls are still usable after the button is added.

Step 5.2 — Build `initialState` from OCR result and mount `PedigreeTree`

PedigreeTree must not mount until initialState is ready. Block its render while ocrStatus is 'preprocessing' or 'scanning'.

When ocrData is available, build initialState for PedigreeTree as follows:

Gen1 (sire/dam) — herd matching (see Step 5.3 for matching logic):

Confident herd match found → seed as { mode: 'linked', id: <matchedRabbitId> }
No match → seed as { mode: 'manual', name: serializeManualAncestor(remapped) }

Gen2/gen3 — always manual:

Seed as { mode: 'manual', name: serializeManualAncestor(remapped) }

Field remapping for serializeManualAncestor (apply to every manual node):

OCR field	`serializeManualAncestor` field
`rabbitName`	`name`
`farmName`	`farm`
`earNumber`	`ear_number`
`weight` (null → `0`)	`weight_lbs`
`dob` (null → `"Unknown"`)	`dob`

Once initialState is built, mount PedigreeTree with it. If the user rescans, bump PedigreeTree's key prop to remount with a fresh initialState built from the new ocrData.

Test: Scan a pedigree. Confirm PedigreeTree does not render during preprocessing/scanning. Confirm it mounts with the correct ancestor data once ocrData is ready. Rescan with a different pedigree — confirm the tree remounts and shows the new data.

Step 5.3 — Gen1 herd matching

For each of the sire and dam nodes independently, attempt a herd match before building initialState:

Call GET /api/rabbits/quick-search?fields=ear_number&q=<earNumber> (skip if earNumber is null)
Filter results client-side to an exact ear_number match (case-insensitive)
If exactly one exact match → use that rabbit's id for a linked node
If no exact ear number match → call GET /api/rabbits/quick-search?fields=name&q=<rabbitName> (skip if rabbitName is null)
Filter results client-side to an exact name match
If exactly one exact name match → use that rabbit's id for a linked node
If still no match → fall back to a manual node

Ear number takes precedence. Both searches must complete before initialState is built and PedigreeTree is allowed to mount.

Test:

Add a rabbit to the herd with a known ear number. Scan a pedigree where that ear number appears as the sire. Confirm the sire slot mounts as a linked node showing the matched rabbit's name.
Scan a pedigree where neither sire nor dam exists in the herd. Confirm both gen1 slots mount as manual nodes with the OCR-extracted name and ear number.

Step 5.4 — Wire subject fields into `details` state

When ocrData is available, apply the subject fields to the wizard's details state slice:

OCR field	`details` field
`subject.farmName`	`farm`
`subject.rabbitName`	`name`
`subject.earNumber`	`ear_number`
`subject.dob`	`dob` (already `YYYY-MM-DD`)
`subject.weight`	`weight_lbs`

For any field where the OCR result is null, leave the existing details value unchanged. Do not overwrite with null.

Use the "adjusting state during render" pattern (a useRef flag) consistent with AddRabbit.jsx's existing one-shot default-fill pattern (e.g. how details.farm is seeded from settings.rabbitry_name). Do not use a useEffect that calls setState.

Test: Scan a pedigree. Navigate to Step 2. Confirm the subject's farm, name, ear number, DOB, and weight are pre-filled. Confirm null fields are blank (not showing "null"). Confirm already-typed details values are not overwritten by null OCR results.

Step 5.5 — Highlight null OCR fields

After OCR data is applied, any field that received a null OCR result should display a yellow background and a small "?" indicator alongside it.

This applies to:

Individual fields within each ancestor slot's manual editor in PedigreeTree
The five subject fields in the Step 2 details form

The highlight is visual only — the field remains fully editable. Remove the highlight when the user types a non-empty value into that field.

Test: Scan a pedigree where DOB is absent from one ancestor. Open that ancestor's manual editor. Confirm the DOB field has a yellow highlight and "?" indicator. Type a value. Confirm the highlight disappears.

Step 5.6 — Show OCR status banner

After OCR completes, display a dismissible banner above the pedigree tree on Step 1:

Success (some fields populated): "Pedigree scanned — review the tree and check highlighted fields before continuing."
All fields null (no text found): "No text detected. Try retaking the photo with better lighting."
Tesseract error: "Couldn't read this pedigree. Check the image quality or enter manually."

The wizard must remain fully usable for manual entry in all three cases.

Test:

Scan a clean pedigree. Confirm the success banner appears.
Pass a completely black or blank image. Confirm the no-text banner appears.
Temporarily move the Tesseract WASM out of public/tesseract/ to force a load failure. Confirm the error banner appears and the wizard is still usable for manual entry.

Phase 6 — Zone picker correction flow

Step 6.1 — Add zone picker trigger to ancestor slot editors

In the manual-entry editor for each ancestor slot (the panel that opens when the user clicks a manual node in PedigreeTree.jsx), add a small scan/camera icon button next to each of the five fields (farm name, rabbit name, ear number, DOB, weight).

Clicking the icon opens the ZonePicker panel. Pass it:

image — the processedBlob from AddRabbit state Do not pass initialCrop — the prop does not exist. The user draws the crop region from scratch.
fieldLabel — a human-readable label for the field being corrected

Track which field triggered the picker in local state (activeZoneField).

The ZonePicker panel should render alongside or below the editor, not in a modal, so the user can see both simultaneously.

Test: After scanning a pedigree, click an ancestor slot to open its editor. Confirm each field has a scan icon button. Click the scan icon next to the ear number field. Confirm the ZonePicker opens showing the preprocessed pedigree image with no pre-drawn selection. Confirm the user can draw a crop rectangle freely.

Step 6.2 — Apply zone picker result to the ancestor editor

When ZonePicker calls onConfirm(rawText):

Run rawText through the appropriate parser from ocrParse.js for the active field (ear number regex, possessive name split, date parser, or weight normalizer)
Update the relevant field(s) in the ancestor editor's local state
Close ZonePicker (clear activeZoneField)
Remove the yellow highlight from the updated field

When ZonePicker calls onCancel, close the panel without modifying any state.

Note: for a name re-scan, the possessive split may produce both farmName and rabbitName. Update both fields if the scan icon was on either name field.

Test: After scanning a pedigree where ear number was null, click the ear number scan icon. Draw a rectangle over the ear number. Click "Re-scan". Confirm the ear number field updates with the correct value. Confirm the yellow highlight is gone. Confirm no other fields in the editor changed.

Step 6.3 — Add zone picker trigger to Step 2 subject fields

In the Step 2 details form, add scan icon buttons next to the five OCR-populated fields: farm name, rabbit name, ear number, DOB, weight.

These work identically to Steps 6.1–6.2 but update details state in AddRabbit instead of an ancestor editor's local state.

Test: After scanning a pedigree where subject DOB was null (yellow highlight visible on DOB field), navigate to Step 2. Click the scan icon next to DOB. Draw a rectangle over the DOB on the pedigree image. Click "Re-scan". Confirm the DOB field updates with the correct value and the yellow highlight is removed.

Phase 7 — End-to-end validation

Step 7.1 — Full happy-path scan and save

Using a real printed pedigree (ARBA-format or non-standard):

Open Add Rabbit wizard
Click "Scan Pedigree" and photograph or upload the pedigree
Wait for OCR to complete
Review Step 1 — confirm the pedigree tree is populated with ancestor data
Click Next
Review Step 2 — confirm subject fields are pre-filled
Fill in breed and any null fields manually
Save

Test: Confirm the saved rabbit appears in the herd with correct name, ear number, and pedigree ancestors. Confirm no console errors during the entire flow.

Step 7.2 — Linked gen1 cascade behavior

Scan a pedigree where the sire exists in the herd (matched by ear number).

Test: Confirm the sire slot in the tree shows as a linked node. Confirm the sire's grandparent branch (sire_sire, sire_dam) is locked and derives from the linked rabbit's own stored pedigree — not from the OCR-extracted grandparent text. Confirm the dam branch (if unmatched) retains OCR-extracted manual text.

Step 7.3 — Zone picker correction round-trip

Scan a pedigree where at least one field is wrong or null
Use the zone picker to correct that field
Complete the wizard and save

Test: Confirm the corrected value — not the original OCR extraction — is saved to the database.

Step 7.4 — Offline scan

Load the app once while online
Disconnect from the internet
Complete a full scan and save

Test: Confirm OCR completes without network errors. Confirm the saved rabbit appears in the herd. Confirm no console errors.

Step 7.5 — Manual entry fallback after failed scan

Scan an unreadable image (blank paper, very dark photo)
Confirm the appropriate error banner appears
Dismiss the banner and complete the wizard via manual entry

Test: Confirm the wizard is fully usable for manual entry after a failed scan. Confirm no state corruption from the failed OCR attempt — the tree and details form behave as if no scan was attempted.

Out of Scope (do not implement in this phase)

Deskew / perspective correction
Great-grandparent (4th generation) extraction
Handwriting-specific handling
Multi-page pedigrees
Bulk scanning multiple rabbits at once

27 KiB Raw Permalink Blame History Unescape Escape