5acef6869a
All checks were successful
Build and Test / build-and-test (push) Successful in 39s
Pilot tool that sends a receipt photo to Claude, classifies the budget category, and for groceries locates + categorizes each line item, rendering an annotated PNG with per-category subtotals. Uses the Claude cloud API (claude-opus-4-8) for pixel-accurate bounding boxes; reads CLAUDE_API_KEY from ../.env. Complementary to the local-only OCR/ tools (ollama, no cloud). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
80 lines
3.6 KiB
Markdown
80 lines
3.6 KiB
Markdown
# receiptscan
|
||
|
||
Sends a receipt photo to Claude, classifies the receipt into a budget category,
|
||
and (for groceries) locates + categorizes every line item, then renders an
|
||
annotated PNG: each line item is lightly shaded by food category with a category
|
||
tag in the margin, plus a per-category subtotal sidebar.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
/home/jm/.local/go/bin/go -C /home/jm/programming/HSAmanager/receiptscan run .
|
||
```
|
||
|
||
- Image path is hard-coded to `/home/jm/aldi.jpg` (or pass one as an argument).
|
||
- Reads `CLAUDE_API_KEY` from `../.env` (or `ANTHROPIC_API_KEY` in the env).
|
||
- Output: `<image>_annotated.png` next to the input.
|
||
- Go is installed locally at `/home/jm/.local/go` (no root). `GOTOOLCHAIN=local`
|
||
avoids auto-downloading a toolchain.
|
||
|
||
### Useful env flags
|
||
- `RECEIPT_CACHE=read` — re-render from the saved `<image>.cache.json` without
|
||
calling the API (fast iteration on rendering).
|
||
- `RECEIPT_DEBUG=1` — print the raw API response, the row-fit, and per-item row
|
||
assignments to stderr.
|
||
- `RECEIPT_ANALYZE=1` — dump detected text rows vs. model boxes and exit.
|
||
|
||
## Model query
|
||
|
||
- Model: `claude-opus-4-8` (returns true 1:1 image-pixel coordinates; Haiku does
|
||
not localize accurately and produced vertically-stretched boxes).
|
||
- Adaptive thinking + `output_config.effort: high`. Without thinking the model
|
||
often returned a placeholder/empty list.
|
||
- Structured output via `output_config.format` (`json_schema`, see schema in
|
||
`main.go` → `analyze`): `receipt_type`, `currency`, `total_amount`, and
|
||
`line_items[]` of `{name, price, category, box{x,y,width,height}}`.
|
||
- The model intermittently returns an empty `line_items` for a grocery receipt,
|
||
so the call is retried up to 4× until items come back.
|
||
|
||
### Prompt
|
||
|
||
```
|
||
You are reading a photographed grocery store receipt. The image is exactly
|
||
<W> pixels wide and <H> pixels tall. All coordinates you return must be in
|
||
pixels in that space, with (0,0) at the top-left.
|
||
|
||
1. Set receipt_type to "groceries" and read the grand total (total_amount)
|
||
with its currency.
|
||
|
||
2. This receipt has many purchased line items (dozens). Return EVERY one of
|
||
them - do not stop early and do not return an empty list. For each line item:
|
||
- name and price exactly as printed,
|
||
- a tight pixel rectangle (box) enclosing that whole printed line, from the
|
||
item name on the left through its price on the right,
|
||
- a food category, exactly one of: dairy, meat, bakery_bread, berries,
|
||
non_berry_fruits, vegetables, desserts, processed_food, pet_stuff,
|
||
other_groceries.
|
||
Use other_groceries only when nothing more specific fits. Do NOT include
|
||
subtotal, tax, payment, or total rows as line items.
|
||
|
||
Return only the structured object.
|
||
```
|
||
|
||
Budget categories for `receipt_type`: house_maintenance, activities, restaurant,
|
||
groceries, hobbies, alcohol, medical, education, other.
|
||
|
||
## How box alignment works
|
||
|
||
The model's returned y-coordinates are vertically stretched (~1.35× the true
|
||
line pitch) and only roughly placed, so boxes are **not** drawn at the model
|
||
coordinates. Instead the image itself is used to place them:
|
||
|
||
1. Detect horizontal text-row bands (dark-pixel projection in the item column).
|
||
2. Keep only **item rows**: bands with ink in the price column (right) AND the
|
||
item-code column (far left), at or below the model's first item. This
|
||
excludes centered headers, weight/quantity sub-lines, paper-edge shadows, and
|
||
the payment/total block.
|
||
3. The first N price-bearing rows (N = item count from the model) are the N
|
||
items, in order — map item *i* → row *i*.
|
||
|
||
This keeps alignment correct regardless of the model's coordinate drift.
|