81 lines
3.6 KiB
Markdown
81 lines
3.6 KiB
Markdown
|
# receiptscan
|
|||
|
|
|
|||
|
|
Sends a receipt photo to Claude, classifies the receipt into a budget category,
|
|||
|
|
and (for groceries) locates + categorizes every line item, then renders an
|
|||
|
|
annotated PNG: each line item is lightly shaded by food category with a category
|
|||
|
|
tag in the margin, plus a per-category subtotal sidebar.
|
|||
|
|
|
|||
|
|
## Run
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
/home/jm/.local/go/bin/go -C /home/jm/programming/HSAmanager/receiptscan run .
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- Image path is hard-coded to `/home/jm/aldi.jpg` (or pass one as an argument).
|
|||
|
|
- Reads `CLAUDE_API_KEY` from `../.env` (or `ANTHROPIC_API_KEY` in the env).
|
|||
|
|
- Output: `<image>_annotated.png` next to the input.
|
|||
|
|
- Go is installed locally at `/home/jm/.local/go` (no root). `GOTOOLCHAIN=local`
|
|||
|
|
avoids auto-downloading a toolchain.
|
|||
|
|
|
|||
|
|
### Useful env flags
|
|||
|
|
- `RECEIPT_CACHE=read` — re-render from the saved `<image>.cache.json` without
|
|||
|
|
calling the API (fast iteration on rendering).
|
|||
|
|
- `RECEIPT_DEBUG=1` — print the raw API response, the row-fit, and per-item row
|
|||
|
|
assignments to stderr.
|
|||
|
|
- `RECEIPT_ANALYZE=1` — dump detected text rows vs. model boxes and exit.
|
|||
|
|
|
|||
|
|
## Model query
|
|||
|
|
|
|||
|
|
- Model: `claude-opus-4-8` (returns true 1:1 image-pixel coordinates; Haiku does
|
|||
|
|
not localize accurately and produced vertically-stretched boxes).
|
|||
|
|
- Adaptive thinking + `output_config.effort: high`. Without thinking the model
|
|||
|
|
often returned a placeholder/empty list.
|
|||
|
|
- Structured output via `output_config.format` (`json_schema`, see schema in
|
|||
|
|
`main.go` → `analyze`): `receipt_type`, `currency`, `total_amount`, and
|
|||
|
|
`line_items[]` of `{name, price, category, box{x,y,width,height}}`.
|
|||
|
|
- The model intermittently returns an empty `line_items` for a grocery receipt,
|
|||
|
|
so the call is retried up to 4× until items come back.
|
|||
|
|
|
|||
|
|
### Prompt
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
You are reading a photographed grocery store receipt. The image is exactly
|
|||
|
|
<W> pixels wide and <H> pixels tall. All coordinates you return must be in
|
|||
|
|
pixels in that space, with (0,0) at the top-left.
|
|||
|
|
|
|||
|
|
1. Set receipt_type to "groceries" and read the grand total (total_amount)
|
|||
|
|
with its currency.
|
|||
|
|
|
|||
|
|
2. This receipt has many purchased line items (dozens). Return EVERY one of
|
|||
|
|
them - do not stop early and do not return an empty list. For each line item:
|
|||
|
|
- name and price exactly as printed,
|
|||
|
|
- a tight pixel rectangle (box) enclosing that whole printed line, from the
|
|||
|
|
item name on the left through its price on the right,
|
|||
|
|
- a food category, exactly one of: dairy, meat, bakery_bread, berries,
|
|||
|
|
non_berry_fruits, vegetables, desserts, processed_food, pet_stuff,
|
|||
|
|
other_groceries.
|
|||
|
|
Use other_groceries only when nothing more specific fits. Do NOT include
|
|||
|
|
subtotal, tax, payment, or total rows as line items.
|
|||
|
|
|
|||
|
|
Return only the structured object.
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Budget categories for `receipt_type`: house_maintenance, activities, restaurant,
|
|||
|
|
groceries, hobbies, alcohol, medical, education, other.
|
|||
|
|
|
|||
|
|
## How box alignment works
|
|||
|
|
|
|||
|
|
The model's returned y-coordinates are vertically stretched (~1.35× the true
|
|||
|
|
line pitch) and only roughly placed, so boxes are **not** drawn at the model
|
|||
|
|
coordinates. Instead the image itself is used to place them:
|
|||
|
|
|
|||
|
|
1. Detect horizontal text-row bands (dark-pixel projection in the item column).
|
|||
|
|
2. Keep only **item rows**: bands with ink in the price column (right) AND the
|
|||
|
|
item-code column (far left), at or below the model's first item. This
|
|||
|
|
excludes centered headers, weight/quantity sub-lines, paper-edge shadows, and
|
|||
|
|
the payment/total block.
|
|||
|
|
3. The first N price-bearing rows (N = item count from the model) are the N
|
|||
|
|
items, in order — map item *i* → row *i*.
|
|||
|
|
|
|||
|
|
This keeps alignment correct regardless of the model's coordinate drift.
|