hsa-app/receiptscan/README.md

receiptscan

Sends a receipt photo to Claude, classifies the receipt into a budget category, and (for groceries) locates + categorizes every line item, then renders an annotated PNG: each line item is lightly shaded by food category with a category tag in the margin, plus a per-category subtotal sidebar.

Run

/home/jm/.local/go/bin/go -C /home/jm/programming/HSAmanager/receiptscan run .

• Image path is hard-coded to /home/jm/aldi.jpg (or pass one as an argument).
• Reads CLAUDE_API_KEY from ../.env (or ANTHROPIC_API_KEY in the env).
• Output: <image>_annotated.png next to the input.
• Go is installed locally at /home/jm/.local/go (no root). GOTOOLCHAIN=local avoids auto-downloading a toolchain.

Useful env flags

• RECEIPT_CACHE=read — re-render from the saved <image>.cache.json without calling the API (fast iteration on rendering).
• RECEIPT_DEBUG=1 — print the raw API response, the row-fit, and per-item row assignments to stderr.
• RECEIPT_ANALYZE=1 — dump detected text rows vs. model boxes and exit.

Model query

• Model: claude-opus-4-8 (returns true 1:1 image-pixel coordinates; Haiku does not localize accurately and produced vertically-stretched boxes).
• Adaptive thinking + output_config.effort: high. Without thinking the model often returned a placeholder/empty list.
• Structured output via output_config.format (json_schema, see schema in main.go → analyze): receipt_type, currency, total_amount, and line_items[] of {name, price, category, box{x,y,width,height}}.
• The model intermittently returns an empty line_items for a grocery receipt, so the call is retried up to 4× until items come back.

Prompt

You are reading a photographed grocery store receipt. The image is exactly
<W> pixels wide and <H> pixels tall. All coordinates you return must be in
pixels in that space, with (0,0) at the top-left.

1. Set receipt_type to "groceries" and read the grand total (total_amount)
   with its currency.

2. This receipt has many purchased line items (dozens). Return EVERY one of
   them - do not stop early and do not return an empty list. For each line item:
   - name and price exactly as printed,
   - a tight pixel rectangle (box) enclosing that whole printed line, from the
     item name on the left through its price on the right,
   - a food category, exactly one of: dairy, meat, bakery_bread, berries,
     non_berry_fruits, vegetables, desserts, processed_food, pet_stuff,
     other_groceries.
   Use other_groceries only when nothing more specific fits. Do NOT include
   subtotal, tax, payment, or total rows as line items.

Return only the structured object.

Budget categories for receipt_type: house_maintenance, activities, restaurant, groceries, hobbies, alcohol, medical, education, other.

How box alignment works

The model's returned y-coordinates are vertically stretched (~1.35× the true line pitch) and only roughly placed, so boxes are not drawn at the model coordinates. Instead the image itself is used to place them:

1. Detect horizontal text-row bands (dark-pixel projection in the item column).
2. Keep only item rows: bands with ink in the price column (right) AND the item-code column (far left), at or below the model's first item. This excludes centered headers, weight/quantity sub-lines, paper-edge shadows, and the payment/total block.
3. The first N price-bearing rows (N = item count from the model) are the N items, in order — map item i → row i.

This keeps alignment correct regardless of the model's coordinate drift.