Compare commits
No commits in common. "OCR" and "main" have entirely different histories.
9 changed files with 0 additions and 1665 deletions
263
OCR/README.md
263
OCR/README.md
|
|
@ -1,263 +0,0 @@
|
||||||
I want to create an OCR feature (ollama, whatever) that's
|
|
||||||
|
|
||||||
1) local (assume I have a GPU with 24GB VRAM)
|
|
||||||
2) works on receipts (scanned or digital)
|
|
||||||
3) given a receipt, return an ordered list of rows of text in that receipt
|
|
||||||
4) example /home/jm/programming/HSAmanager/OCR/example/data/aldi.jpg
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Getting started (working prototype)
|
|
||||||
|
|
||||||
Local, GPU-backed, no cloud / no API key. Engine: **Qwen2.5-VL 7B** served by
|
|
||||||
**ollama**. The model transcribes the receipt top-to-bottom and returns one
|
|
||||||
string per physical row, columns joined left-to-right.
|
|
||||||
|
|
||||||
### One-time setup (already done on this machine)
|
|
||||||
|
|
||||||
- ollama binary installed (no root) at `~/.local/bin/ollama`.
|
|
||||||
- Model pulled: `ollama pull qwen2.5vl:7b` (~6 GB).
|
|
||||||
|
|
||||||
### Run
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. start the server (leave running; ~6 GB VRAM when the model loads)
|
|
||||||
~/.local/bin/ollama serve &
|
|
||||||
|
|
||||||
# 2. transcribe a receipt -> ordered, numbered rows
|
|
||||||
python3 OCR/receipt_ocr.py OCR/example/data/aldi.jpg
|
|
||||||
|
|
||||||
# raw JSON {"rows": [...]} instead of numbered text
|
|
||||||
python3 OCR/receipt_ocr.py OCR/example/data/aldi.jpg --json
|
|
||||||
```
|
|
||||||
|
|
||||||
No third-party Python deps — `receipt_ocr.py` uses the stdlib and the ollama
|
|
||||||
HTTP API. Override with `OLLAMA_MODEL` / `OLLAMA_HOST` env vars.
|
|
||||||
|
|
||||||
### Notes / next steps
|
|
||||||
|
|
||||||
- Works on the ALDI example with near-perfect line ordering and column joining.
|
|
||||||
- Caveat: the model sometimes *expands* thermal-print abbreviations
|
|
||||||
(`Fire Roasted Tom` → `Fire Roasted Tomatoes`). Helpful for readability but
|
|
||||||
not byte-for-byte faithful; tighten the prompt if you need verbatim text.
|
|
||||||
- Accuracy upgrade: `qwen2.5vl:32b` (q4 ≈ 20 GB, fits the 3090 but tighter/slower).
|
|
||||||
- This is local-only and complementary to `../receiptscan/`, which uses the
|
|
||||||
Claude cloud API for line-item categorization + annotated rendering.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Operating ollama (runbook)
|
|
||||||
|
|
||||||
ollama is the local model server both scripts talk to over HTTP
|
|
||||||
(`http://127.0.0.1:11434`). It must be running before you call either script.
|
|
||||||
Installed **without root** at `~/.local/bin/ollama` (the binary was extracted
|
|
||||||
from the `ollama-linux-amd64.tar.zst` GitHub release; `~/.local/bin` is on PATH,
|
|
||||||
so plain `ollama` works too).
|
|
||||||
|
|
||||||
### Start / check / stop
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Start the server (does NOT survive a reboot or terminal close — restart it):
|
|
||||||
ollama serve & # or: nohup ollama serve > /tmp/ollama.log 2>&1 &
|
|
||||||
|
|
||||||
# Is it up? (prints a JSON version string if so)
|
|
||||||
curl -fsS http://127.0.0.1:11434/api/version
|
|
||||||
|
|
||||||
# Is the process alive?
|
|
||||||
pgrep -af "ollama serve"
|
|
||||||
|
|
||||||
# Stop it:
|
|
||||||
pkill -f "ollama serve"
|
|
||||||
```
|
|
||||||
|
|
||||||
After a reboot the server is **not** running — just `ollama serve &` again. The
|
|
||||||
pulled models persist on disk (under `~/.ollama`), so you don't re-download them.
|
|
||||||
|
|
||||||
### Models
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ollama list # models on disk
|
|
||||||
ollama ps # models currently loaded in VRAM (and how long until idle-unload)
|
|
||||||
ollama pull <model> # download a model, e.g. ollama pull qwen2.5:14b
|
|
||||||
ollama rm <model> # delete a model to reclaim disk
|
|
||||||
```
|
|
||||||
|
|
||||||
Currently pulled: `qwen2.5vl:7b` (OCR), `mistral-small3.2:24b` (grouping, default),
|
|
||||||
plus `qwen2.5:7b` / `qwen2.5:14b` (earlier grouping experiments — `ollama rm`
|
|
||||||
them if you want the disk back).
|
|
||||||
|
|
||||||
### VRAM / GPU notes
|
|
||||||
|
|
||||||
- A model loads into VRAM on first request and **auto-unloads after ~5 min idle**
|
|
||||||
(so `ollama ps` is often empty — that's normal; the next call reloads it in a
|
|
||||||
few seconds).
|
|
||||||
- Only run one large model at a time on the 24 GB 3090. The OCR model (~6 GB) and
|
|
||||||
the grouping model (~15 GB) can coexist, but bumping grouping to
|
|
||||||
`qwen2.5:32b`/`qwen2.5vl:32b` (~20 GB) leaves little room — expect a reload
|
|
||||||
when switching between OCR and grouping.
|
|
||||||
- Force-unload now (free VRAM without stopping the server):
|
|
||||||
`ollama stop <model>`.
|
|
||||||
- Keep a model resident longer / shorter via the request `keep_alive` field, or
|
|
||||||
globally with `OLLAMA_KEEP_ALIVE` (e.g. `OLLAMA_KEEP_ALIVE=30m ollama serve`).
|
|
||||||
|
|
||||||
### Pointing the scripts elsewhere
|
|
||||||
|
|
||||||
Both scripts honor `OLLAMA_HOST` and `OLLAMA_MODEL`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
OLLAMA_MODEL=qwen2.5:14b python3 OCR/receipt_group.py # try another model
|
|
||||||
OLLAMA_HOST=http://other-box:11434 python3 OCR/receipt_ocr.py img.jpg # remote server
|
|
||||||
```
|
|
||||||
|
|
||||||
### Logs / troubleshooting
|
|
||||||
|
|
||||||
- If a script hangs or errors connecting, the server is probably down — check
|
|
||||||
with the `curl` above and restart.
|
|
||||||
- Startup logs (GPU discovery, VRAM, errors) go wherever you redirected `serve`
|
|
||||||
(e.g. `/tmp/ollama.log`); tail that if a model fails to load.
|
|
||||||
- "model not found" → you haven't `ollama pull`ed it (or typo'd the tag); see
|
|
||||||
`ollama list`.
|
|
||||||
|
|
||||||
|
|
||||||
# grouping the OCR results
|
|
||||||
next task is ... given an OCR output, like:
|
|
||||||
```text
|
|
||||||
1 ALDI
|
|
||||||
2 Store #145
|
|
||||||
3 1501 Rockville Pike
|
|
||||||
4 Rockville, MD
|
|
||||||
5 https://help.aldi.us
|
|
||||||
6 Your cashier today was Wendy
|
|
||||||
7 382175 Unsalted Peanuts 2.29 FA
|
|
||||||
8 382175 Unsalted Peanuts 2.29 FA
|
|
||||||
9 384773 4 lb. Sugar 2.79 FA
|
|
||||||
10 384773 4 lb. Sugar 2.79 FA
|
|
||||||
11 382437 Fire Roasted Tomatoes 2.30 FA
|
|
||||||
12 2 @ 1.15
|
|
||||||
13 371607 Canned Cat Food 2.28 NB
|
|
||||||
14 4 @ 0.57
|
|
||||||
15 371607 Canned Cat Food 0.57 NB
|
|
||||||
16 416787 1% Milk, Gallon 3.38 FA
|
|
||||||
17 343557 Protein Powder 18.49 FA
|
|
||||||
18 416645 Chocolate Milk 1.91 FA
|
|
||||||
19 382260 Plain NF Greek Yogurt 2.79 FA
|
|
||||||
20 382260 Plain NF Greek Yogurt 2.79 FA
|
|
||||||
21 297956 PureandSimple Bars 3.99 FA
|
|
||||||
22 356508 Broccoli Crowns 2.21 FA
|
|
||||||
23 1.17 lb x 1.89/lb
|
|
||||||
24 469529 Corn Tortillas 1.95 FA
|
|
||||||
25 609748 Assorted Cashews 6.49 FA
|
|
||||||
26 634568 Chicken Skewers 7.49 FA
|
|
||||||
27 382473 Shredded Mozzarella 3.29 FA
|
|
||||||
28 341878 Yellow Onions 1.85 FA
|
|
||||||
29 535290 ABF B/S Thighs 6.61 FA
|
|
||||||
30 272135 Salmon Portions 9.35 FA
|
|
||||||
31 272135 Salmon Portions 8.54 FA
|
|
||||||
32 382310 Feta Crumbles 1.29 FA
|
|
||||||
33 356508 Broccoli Crowns 2.53 FA
|
|
||||||
34 1.34 lb x 1.89/lb
|
|
||||||
35 262137 Stuffed Olives 2.89 FA
|
|
||||||
36 356490 Bagged Avocados 2.99 FA
|
|
||||||
37 356607 Red Delic Apples 2.75 FA
|
|
||||||
38 356527 Celery 1.89 FA
|
|
||||||
39 382653 Indian Sauces 3.69 FA
|
|
||||||
40 356628 Seedless Cucumber 0.89 FA
|
|
||||||
41 356684 WildTwist Apf LRW 5.44 FA
|
|
||||||
42 (G) 3.10 lb - (T) 0.06 lb
|
|
||||||
43 (N) 3.04 lb x 1.79/lb
|
|
||||||
44 282119 Pineapples 1.89 FA
|
|
||||||
45 356419 Mangoes 3.80 FA
|
|
||||||
46 4 @ 0.95
|
|
||||||
47 356691 Zucchini 2.01 FA
|
|
||||||
48 1.69 lb x 1.19/lb
|
|
||||||
49 356522 Cantaloupe 1.89 FA
|
|
||||||
50 356504 Blueberries 9.95 FA
|
|
||||||
51 5 @ 1.99
|
|
||||||
52 388137 Large Eggs 2.92 FA
|
|
||||||
53 2 @ 1.46
|
|
||||||
54 262747 Bananas LRW 1.45 FA
|
|
||||||
55 (G) 2.97 lb - (T) 0.01 lb
|
|
||||||
56 (N) 2.96 lb x 0.49/lb
|
|
||||||
57 262747 Bananas LRW 1.13 FA
|
|
||||||
58 (G) 2.32 lb - (T) 0.01 lb
|
|
||||||
59 (N) 2.31 lb x 0.49/lb
|
|
||||||
60 356427 Multi-Peppers 3pk. 2.69 FA
|
|
||||||
61 341876 Red Grapes LRW 5.48 FA
|
|
||||||
62 (G) 3.96 lb - (T) 0.02 lb
|
|
||||||
63 (N) 3.94 lb x 1.39/lb
|
|
||||||
64 479744 Campari Tomatoes 2.99 FA
|
|
||||||
65 479744 Campari Tomatoes 2.99 FA
|
|
||||||
66 569269 Protein Bread 3.99 FA
|
|
||||||
67 343989 Family AsstCookie 1.99 FA
|
|
||||||
68 356646 Strawberries 2.09 FA
|
|
||||||
69 356646 Strawberries 2.09 FA
|
|
||||||
70 356646 Strawberries 2.09 FA
|
|
||||||
71 VISA 172.42
|
|
||||||
72 ************0294 ONLINE
|
|
||||||
73 06/21/26 12:43 Ref/Seq # 0
|
|
||||||
74 Auth# 00401D
|
|
||||||
75 AID A000000031010
|
|
||||||
76 TVR 0000000000
|
|
||||||
77 IAD 06021203A00000
|
|
||||||
78 TSI 0000 ARC 00 EntryMode 07
|
|
||||||
79 ++APPROVED++
|
|
||||||
80 SUBTOTAL 172.24
|
|
||||||
81 B-Taxable @6.00% 0.18
|
|
||||||
82 A-Taxable @0.00% 0.00
|
|
||||||
83 AMOUNT DUE 172.42
|
|
||||||
84 TOTAL $172.42
|
|
||||||
85 60 ITEMS
|
|
||||||
86 Credit Card $ 172.42
|
|
||||||
87 *7367 L411/005/064 06/21/26 12:43PM
|
|
||||||
88 ************
|
|
||||||
89 Sign up for ALDI emails
|
|
||||||
90 for a sneak peek on the weekly ad!
|
|
||||||
91 www.aldi.us/signup
|
|
||||||
```
|
|
||||||
|
|
||||||
write a LLM call, still using my local ollama (you can switch the model). I want the output to be an object roughly like that:
|
|
||||||
|
|
||||||
store name:
|
|
||||||
store branch:
|
|
||||||
store address:
|
|
||||||
list of rows that belong together as one item purchased
|
|
||||||
total in dollars.
|
|
||||||
(anything else is relevant here)?
|
|
||||||
|
|
||||||
you can hardcode the lines for this test and let's call the LLM to see what it does
|
|
||||||
|
|
||||||
the output needs to be a new script (receipt_group.py) that (for now) read this examle as is from an example file that you will create, and with the correct prompt organizes the information correctly.
|
|
||||||
|
|
||||||
## Grouping — implemented
|
|
||||||
|
|
||||||
`receipt_group.py` reads OCR rows (default
|
|
||||||
`example/data/aldi_ocr.txt`) and asks a local ollama text model to reconstruct
|
|
||||||
the receipt: store name/branch/address, line items (merging multi-row items
|
|
||||||
such as weighed produce and `qty @ price` lines), total, and metadata
|
|
||||||
(cashier, datetime, payment, subtotal, tax, item count).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 OCR/receipt_group.py # groups only: each item + its rows
|
|
||||||
python3 OCR/receipt_group.py OCR/example/data/aldi_ocr.txt --json # full object + reconciliation
|
|
||||||
```
|
|
||||||
|
|
||||||
- Default model: `mistral-small3.2:24b` (~15 GB). Override with `OLLAMA_MODEL`.
|
|
||||||
Tried qwen2.5:7b (many merge errors) and qwen2.5:14b (~90%); Mistral Small got
|
|
||||||
the grouping right. Swapping models is just the env var — no code change.
|
|
||||||
- **Why an LLM groups, not a regex**: detecting item vs detail rows is trivial,
|
|
||||||
but deciding *which* item a detail row attaches to is not — wrapped
|
|
||||||
descriptions, discount lines, and qty/weight lines can sit above or below the
|
|
||||||
item, so a positional rule breaks. The prompt tells the model to attach each
|
|
||||||
detail row to the item its **math** reconciles with (count × unit, weight ×
|
|
||||||
rate = the item's charged price), never by a fixed above/below position.
|
|
||||||
- **Completeness clause** in the prompt: every input row must land in exactly one
|
|
||||||
item's `rows`, copied verbatim, and the model must walk the input top-to-bottom
|
|
||||||
to confirm nothing was dropped. This fixed the model silently swallowing Large
|
|
||||||
Eggs' `2 @ 1.46` line. Prompting lowers the odds of a drop but cannot guarantee
|
|
||||||
it — hence the backstop below.
|
|
||||||
- **Deterministic reconciliation pass** (the part you *can* trust), shown in
|
|
||||||
`--json` under `validation`: Python recomputes each item from its detail math,
|
|
||||||
sums items vs the printed subtotal, checks total = subtotal + tax, and sums
|
|
||||||
unit quantities vs the printed `N ITEMS`. When the model slips (e.g. dropping a
|
|
||||||
`qty @` line), the unit-count check flags it (59 vs 60) even though dollars
|
|
||||||
still balance.
|
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 438 KiB |
|
|
@ -1,91 +0,0 @@
|
||||||
1 ALDI
|
|
||||||
2 Store #145
|
|
||||||
3 1501 Rockville Pike
|
|
||||||
4 Rockville, MD
|
|
||||||
5 https://help.aldi.us
|
|
||||||
6 Your cashier today was Wendy
|
|
||||||
7 382175 Unsalted Peanuts 2.29 FA
|
|
||||||
8 382175 Unsalted Peanuts 2.29 FA
|
|
||||||
9 384773 4 lb. Sugar 2.79 FA
|
|
||||||
10 384773 4 lb. Sugar 2.79 FA
|
|
||||||
11 382437 Fire Roasted Tomatoes 2.30 FA
|
|
||||||
12 2 @ 1.15
|
|
||||||
13 371607 Canned Cat Food 2.28 NB
|
|
||||||
14 4 @ 0.57
|
|
||||||
15 371607 Canned Cat Food 0.57 NB
|
|
||||||
16 416787 1% Milk, Gallon 3.38 FA
|
|
||||||
17 343557 Protein Powder 18.49 FA
|
|
||||||
18 416645 Chocolate Milk 1.91 FA
|
|
||||||
19 382260 Plain NF Greek Yogurt 2.79 FA
|
|
||||||
20 382260 Plain NF Greek Yogurt 2.79 FA
|
|
||||||
21 297956 PureandSimple Bars 3.99 FA
|
|
||||||
22 356508 Broccoli Crowns 2.21 FA
|
|
||||||
23 1.17 lb x 1.89/lb
|
|
||||||
24 469529 Corn Tortillas 1.95 FA
|
|
||||||
25 609748 Assorted Cashews 6.49 FA
|
|
||||||
26 634568 Chicken Skewers 7.49 FA
|
|
||||||
27 382473 Shredded Mozzarella 3.29 FA
|
|
||||||
28 341878 Yellow Onions 1.85 FA
|
|
||||||
29 535290 ABF B/S Thighs 6.61 FA
|
|
||||||
30 272135 Salmon Portions 9.35 FA
|
|
||||||
31 272135 Salmon Portions 8.54 FA
|
|
||||||
32 382310 Feta Crumbles 1.29 FA
|
|
||||||
33 356508 Broccoli Crowns 2.53 FA
|
|
||||||
34 1.34 lb x 1.89/lb
|
|
||||||
35 262137 Stuffed Olives 2.89 FA
|
|
||||||
36 356490 Bagged Avocados 2.99 FA
|
|
||||||
37 356607 Red Delic Apples 2.75 FA
|
|
||||||
38 356527 Celery 1.89 FA
|
|
||||||
39 382653 Indian Sauces 3.69 FA
|
|
||||||
40 356628 Seedless Cucumber 0.89 FA
|
|
||||||
41 356684 WildTwist Apf LRW 5.44 FA
|
|
||||||
42 (G) 3.10 lb - (T) 0.06 lb
|
|
||||||
43 (N) 3.04 lb x 1.79/lb
|
|
||||||
44 282119 Pineapples 1.89 FA
|
|
||||||
45 356419 Mangoes 3.80 FA
|
|
||||||
46 4 @ 0.95
|
|
||||||
47 356691 Zucchini 2.01 FA
|
|
||||||
48 1.69 lb x 1.19/lb
|
|
||||||
49 356522 Cantaloupe 1.89 FA
|
|
||||||
50 356504 Blueberries 9.95 FA
|
|
||||||
51 5 @ 1.99
|
|
||||||
52 388137 Large Eggs 2.92 FA
|
|
||||||
53 2 @ 1.46
|
|
||||||
54 262747 Bananas LRW 1.45 FA
|
|
||||||
55 (G) 2.97 lb - (T) 0.01 lb
|
|
||||||
56 (N) 2.96 lb x 0.49/lb
|
|
||||||
57 262747 Bananas LRW 1.13 FA
|
|
||||||
58 (G) 2.32 lb - (T) 0.01 lb
|
|
||||||
59 (N) 2.31 lb x 0.49/lb
|
|
||||||
60 356427 Multi-Peppers 3pk. 2.69 FA
|
|
||||||
61 341876 Red Grapes LRW 5.48 FA
|
|
||||||
62 (G) 3.96 lb - (T) 0.02 lb
|
|
||||||
63 (N) 3.94 lb x 1.39/lb
|
|
||||||
64 479744 Campari Tomatoes 2.99 FA
|
|
||||||
65 479744 Campari Tomatoes 2.99 FA
|
|
||||||
66 569269 Protein Bread 3.99 FA
|
|
||||||
67 343989 Family AsstCookie 1.99 FA
|
|
||||||
68 356646 Strawberries 2.09 FA
|
|
||||||
69 356646 Strawberries 2.09 FA
|
|
||||||
70 356646 Strawberries 2.09 FA
|
|
||||||
71 VISA 172.42
|
|
||||||
72 ************0294 ONLINE
|
|
||||||
73 06/21/26 12:43 Ref/Seq # 0
|
|
||||||
74 Auth# 00401D
|
|
||||||
75 AID A000000031010
|
|
||||||
76 TVR 0000000000
|
|
||||||
77 IAD 06021203A00000
|
|
||||||
78 TSI 0000 ARC 00 EntryMode 07
|
|
||||||
79 ++APPROVED++
|
|
||||||
80 SUBTOTAL 172.24
|
|
||||||
81 B-Taxable @6.00% 0.18
|
|
||||||
82 A-Taxable @0.00% 0.00
|
|
||||||
83 AMOUNT DUE 172.42
|
|
||||||
84 TOTAL $172.42
|
|
||||||
85 60 ITEMS
|
|
||||||
86 Credit Card $ 172.42
|
|
||||||
87 *7367 L411/005/064 06/21/26 12:43PM
|
|
||||||
88 ************
|
|
||||||
89 Sign up for ALDI emails
|
|
||||||
90 for a sneak peek on the weekly ad!
|
|
||||||
91 www.aldi.us/signup
|
|
||||||
|
|
@ -1,284 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""Group raw receipt-OCR rows into a structured object with a local LLM.
|
|
||||||
|
|
||||||
Takes the text output of receipt_ocr.py (one row per line, optionally prefixed
|
|
||||||
with a line number) and asks a local ollama model to reconstruct the receipt:
|
|
||||||
store identity, the line items (merging multi-row items such as weighed produce
|
|
||||||
and "qty @ price" lines), the total, and any other useful metadata.
|
|
||||||
|
|
||||||
Local-only: no cloud, no API key. Uses a text model (default qwen2.5:7b).
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python3 receipt_group.py [ocr_rows.txt] # pretty summary
|
|
||||||
python3 receipt_group.py ocr_rows.txt --json # raw structured JSON
|
|
||||||
|
|
||||||
Env:
|
|
||||||
OLLAMA_HOST default http://127.0.0.1:11434
|
|
||||||
OLLAMA_MODEL default qwen2.5:7b
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
DEFAULT_INPUT = os.path.join(os.path.dirname(__file__), "example", "data", "aldi_ocr.txt")
|
|
||||||
HOST = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434").rstrip("/")
|
|
||||||
MODEL = os.environ.get("OLLAMA_MODEL", "mistral-small3.2:24b")
|
|
||||||
|
|
||||||
PROMPT = """You are given the OCR transcription of a single retail receipt, one \
|
|
||||||
physical row per line. Each line may start with a row number (e.g. "23 ...") — \
|
|
||||||
ignore the row number itself, it is not part of the receipt text.
|
|
||||||
|
|
||||||
Reconstruct the receipt as structured data. Rules:
|
|
||||||
|
|
||||||
- A purchased ITEM begins on a row that has a product code (a run of digits) and \
|
|
||||||
a price. EVERY row that begins with a product code starts a NEW item — never \
|
|
||||||
attach a code-bearing row to the item above it, even when the name and price are \
|
|
||||||
identical to the row above. Two identical consecutive code rows are TWO separate \
|
|
||||||
purchases, each its own entry. Do not merge or de-duplicate them.
|
|
||||||
- A DETAIL row is any row that does NOT begin with a product code, e.g.:
|
|
||||||
* quantity lines like "2 @ 1.15" (count @ unit price)
|
|
||||||
* weight lines like "1.17 lb x 1.89/lb"
|
|
||||||
* gross/tare/net lines like "(G) 3.10 lb - (T) 0.06 lb" and "(N) 3.04 lb x 1.79/lb"
|
|
||||||
Attach each detail row to the item it actually describes. Decide that by the \
|
|
||||||
MATH, not by position: count x unit price, or weight x rate, equals that item's \
|
|
||||||
charged price. (A gross/tare line that has no price of its own stays with the \
|
|
||||||
net line it accompanies.) The describing item is usually adjacent, but verify by \
|
|
||||||
reconciliation rather than assuming it is the one above or below.
|
|
||||||
- The item's "price" is the dollar amount printed on the item's own (first) row — \
|
|
||||||
the amount actually charged. Keep two decimals.
|
|
||||||
- Put the exact original row strings that belong to each item in its "rows" array, \
|
|
||||||
in order, WITHOUT the leading row number.
|
|
||||||
- Header rows (store name, branch, address, website, cashier) are NOT items.
|
|
||||||
- Payment, auth, subtotal, tax, total, item-count and marketing/footer rows are \
|
|
||||||
NOT items — pull the useful values into the metadata fields instead.
|
|
||||||
- Do not invent values. If something is not present, omit it.
|
|
||||||
|
|
||||||
COMPLETENESS — this is critical:
|
|
||||||
- Every item row AND every detail row from the input must appear in exactly ONE \
|
|
||||||
item's "rows" array, copied VERBATIM (same digits, same spacing, same price). \
|
|
||||||
You may not drop, skip, summarize, or rewrite any row.
|
|
||||||
- NEVER discard a quantity line ("2 @ 1.46"), a weight line ("1.17 lb x 1.89/lb"), \
|
|
||||||
or a gross/tare/net line. Attach each to the item its math reconciles with.
|
|
||||||
- Before you answer, walk the input top to bottom and confirm that every row is \
|
|
||||||
either placed in an item's "rows" or is a header/payment/footer row that you \
|
|
||||||
deliberately excluded. No item or detail row may be missing.
|
|
||||||
|
|
||||||
Return STRICT JSON only, matching the requested schema."""
|
|
||||||
|
|
||||||
SCHEMA = {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"store_name": {"type": "string"},
|
|
||||||
"store_branch": {"type": "string"},
|
|
||||||
"store_address": {"type": "string"},
|
|
||||||
"items": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"name": {"type": "string"},
|
|
||||||
"code": {"type": "string"},
|
|
||||||
"price": {"type": "number"},
|
|
||||||
"rows": {"type": "array", "items": {"type": "string"}},
|
|
||||||
},
|
|
||||||
"required": ["name", "price", "rows"],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"total": {"type": "number"},
|
|
||||||
"metadata": {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"cashier": {"type": "string"},
|
|
||||||
"purchase_datetime": {"type": "string"},
|
|
||||||
"payment_method": {"type": "string"},
|
|
||||||
"subtotal": {"type": "number"},
|
|
||||||
"tax": {"type": "number"},
|
|
||||||
"item_count": {"type": "integer"},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"required": ["store_name", "items", "total"],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def group(rows_text):
|
|
||||||
payload = {
|
|
||||||
"model": MODEL,
|
|
||||||
"messages": [
|
|
||||||
{"role": "user", "content": PROMPT + "\n\n--- RECEIPT OCR ---\n" + rows_text},
|
|
||||||
],
|
|
||||||
"stream": False,
|
|
||||||
"format": SCHEMA,
|
|
||||||
"options": {"temperature": 0, "num_ctx": 8192},
|
|
||||||
}
|
|
||||||
req = urllib.request.Request(
|
|
||||||
f"{HOST}/api/chat",
|
|
||||||
data=json.dumps(payload).encode("utf-8"),
|
|
||||||
headers={"Content-Type": "application/json"},
|
|
||||||
)
|
|
||||||
with urllib.request.urlopen(req, timeout=600) as resp:
|
|
||||||
body = json.load(resp)
|
|
||||||
return json.loads(body["message"]["content"])
|
|
||||||
|
|
||||||
|
|
||||||
# --- Deterministic reconciliation -------------------------------------------
|
|
||||||
# The LLM decides the grouping; arithmetic decides whether to trust it. Detail
|
|
||||||
# rows carry their own math, so we recompute each item's price from its details
|
|
||||||
# and balance the whole receipt against the printed subtotal / total / count.
|
|
||||||
|
|
||||||
QTY_RE = re.compile(r"(\d+)\s*@\s*\$?(\d+(?:\.\d+)?)") # "4 @ 0.57"
|
|
||||||
WEIGHT_RE = re.compile(r"(\d+(?:\.\d+)?)\s*lb\s*[xX]\s*\$?(\d+(?:\.\d+)?)\s*/\s*lb") # "1.17 lb x 1.89/lb"
|
|
||||||
TOL = 0.02 # cents of slack for rounding on weighed items
|
|
||||||
|
|
||||||
|
|
||||||
def check_item(it):
|
|
||||||
"""Reconcile one item from its detail rows.
|
|
||||||
|
|
||||||
Returns (status, qty, note) where status is 'ok' | 'warn' | 'none'.
|
|
||||||
'none' = no math-bearing detail row, so there's nothing to verify.
|
|
||||||
"""
|
|
||||||
price = float(it.get("price", 0))
|
|
||||||
qty = 1
|
|
||||||
expected = None
|
|
||||||
note = ""
|
|
||||||
for row in it.get("rows", [])[1:]: # skip the anchor row; details only
|
|
||||||
m = QTY_RE.search(row)
|
|
||||||
if m:
|
|
||||||
qty = int(m.group(1))
|
|
||||||
expected = round(qty * float(m.group(2)), 2)
|
|
||||||
note = f"{m.group(1)} @ {m.group(2)} = {expected:.2f}"
|
|
||||||
continue
|
|
||||||
w = WEIGHT_RE.search(row)
|
|
||||||
if w:
|
|
||||||
expected = round(float(w.group(1)) * float(w.group(2)), 2)
|
|
||||||
note = f"{w.group(1)} lb x {w.group(2)}/lb = {expected:.2f}"
|
|
||||||
if expected is None:
|
|
||||||
return ("none", qty, "")
|
|
||||||
if abs(expected - price) <= TOL:
|
|
||||||
return ("ok", qty, note)
|
|
||||||
return ("warn", qty, f"{note} != charged {price:.2f}")
|
|
||||||
|
|
||||||
|
|
||||||
def validate(obj):
|
|
||||||
items = obj.get("items", [])
|
|
||||||
meta = obj.get("metadata", {}) or {}
|
|
||||||
|
|
||||||
per_item = [check_item(it) for it in items]
|
|
||||||
items_sum = round(sum(float(it.get("price", 0)) for it in items), 2)
|
|
||||||
unit_count = sum(q for _, q, _ in per_item)
|
|
||||||
|
|
||||||
subtotal = meta.get("subtotal")
|
|
||||||
tax = meta.get("tax") or 0
|
|
||||||
total_printed = obj.get("total")
|
|
||||||
count_printed = meta.get("item_count")
|
|
||||||
|
|
||||||
report = {
|
|
||||||
"per_item": [{"status": s, "qty": q, "note": n} for s, q, n in per_item],
|
|
||||||
"items_sum": items_sum,
|
|
||||||
"subtotal_printed": subtotal,
|
|
||||||
"subtotal_ok": subtotal is not None and abs(items_sum - subtotal) <= TOL,
|
|
||||||
"total_computed": round(items_sum + tax, 2),
|
|
||||||
"total_printed": total_printed,
|
|
||||||
"total_ok": total_printed is not None and abs(items_sum + tax - total_printed) <= TOL,
|
|
||||||
"unit_count": unit_count,
|
|
||||||
"unit_count_printed": count_printed,
|
|
||||||
"unit_count_ok": count_printed is not None and unit_count == count_printed,
|
|
||||||
}
|
|
||||||
return report
|
|
||||||
|
|
||||||
|
|
||||||
# --- Rendering ----------------------------------------------------------------
|
|
||||||
|
|
||||||
MARK = {"ok": "✓", "warn": "⚠", "none": " "}
|
|
||||||
|
|
||||||
|
|
||||||
def yn(ok):
|
|
||||||
return "✓" if ok else "⚠"
|
|
||||||
|
|
||||||
|
|
||||||
def pretty(obj, report):
|
|
||||||
out = []
|
|
||||||
out.append(f"Store: {obj.get('store_name', '')}")
|
|
||||||
if obj.get("store_branch"):
|
|
||||||
out.append(f"Branch: {obj['store_branch']}")
|
|
||||||
if obj.get("store_address"):
|
|
||||||
out.append(f"Address: {obj['store_address']}")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
items = obj.get("items", [])
|
|
||||||
checks = report["per_item"]
|
|
||||||
out.append(f"Items ({len(items)}):")
|
|
||||||
for it, chk in zip(items, checks):
|
|
||||||
code = it.get("code", "")
|
|
||||||
head = f" {MARK[chk['status']]} {it.get('name', '')} ${it.get('price', 0):.2f}"
|
|
||||||
if code:
|
|
||||||
head += f" [{code}]"
|
|
||||||
out.append(head)
|
|
||||||
for extra in it.get("rows", [])[1:]: # detail rows only
|
|
||||||
out.append(f" · {extra}")
|
|
||||||
if chk["status"] == "warn":
|
|
||||||
out.append(f" ⚠ {chk['note']}")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
# Receipt-level balance
|
|
||||||
out.append("Reconciliation:")
|
|
||||||
out.append(
|
|
||||||
f" Items sum: ${report['items_sum']:.2f}"
|
|
||||||
+ (
|
|
||||||
f" (printed subtotal ${report['subtotal_printed']:.2f}) {yn(report['subtotal_ok'])}"
|
|
||||||
if report["subtotal_printed"] is not None
|
|
||||||
else ""
|
|
||||||
)
|
|
||||||
)
|
|
||||||
if report["total_printed"] is not None:
|
|
||||||
out.append(
|
|
||||||
f" + tax/total: ${report['total_computed']:.2f}"
|
|
||||||
f" (printed TOTAL ${report['total_printed']:.2f}) {yn(report['total_ok'])}"
|
|
||||||
)
|
|
||||||
if report["unit_count_printed"] is not None:
|
|
||||||
out.append(
|
|
||||||
f" Unit count: {report['unit_count']}"
|
|
||||||
f" (printed {report['unit_count_printed']} ITEMS) {yn(report['unit_count_ok'])}"
|
|
||||||
)
|
|
||||||
|
|
||||||
meta = obj.get("metadata", {})
|
|
||||||
if meta:
|
|
||||||
out.append("")
|
|
||||||
out.append("Metadata:")
|
|
||||||
for k, v in meta.items():
|
|
||||||
out.append(f" {k}: {v}")
|
|
||||||
return "\n".join(out)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
args = [a for a in sys.argv[1:] if not a.startswith("--")]
|
|
||||||
as_json = "--json" in sys.argv
|
|
||||||
path = args[0] if args else DEFAULT_INPUT
|
|
||||||
if not os.path.exists(path):
|
|
||||||
sys.exit(f"input not found: {path}")
|
|
||||||
|
|
||||||
with open(path, encoding="utf-8") as f:
|
|
||||||
rows_text = f.read()
|
|
||||||
|
|
||||||
obj = group(rows_text)
|
|
||||||
|
|
||||||
if as_json:
|
|
||||||
obj["validation"] = validate(obj)
|
|
||||||
print(json.dumps(obj, ensure_ascii=False, indent=2))
|
|
||||||
else:
|
|
||||||
# Just the groups: each purchased item, with the rows that belong together.
|
|
||||||
items = obj.get("items", [])
|
|
||||||
width = len(str(len(items)))
|
|
||||||
for i, it in enumerate(items, 1):
|
|
||||||
rows = it.get("rows", [])
|
|
||||||
head = rows[0] if rows else f"{it.get('name', '')} {it.get('price', 0):.2f}"
|
|
||||||
print(f"{str(i).rjust(width)} {head}")
|
|
||||||
for extra in rows[1:]:
|
|
||||||
print(f"{' ' * width} {extra}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
@ -1,87 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""Local receipt OCR: given a receipt image, print an ordered list of rows of text.
|
|
||||||
|
|
||||||
Uses a local Qwen2.5-VL model served by ollama. No cloud calls, no API key.
|
|
||||||
Zero third-party dependencies — talks to the ollama HTTP API with the stdlib.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python3 receipt_ocr.py [image.jpg] # pretty, numbered rows
|
|
||||||
python3 receipt_ocr.py image.jpg --json # raw JSON {"rows": [...]}
|
|
||||||
|
|
||||||
Env:
|
|
||||||
OLLAMA_HOST default http://127.0.0.1:11434
|
|
||||||
OLLAMA_MODEL default qwen2.5vl:7b
|
|
||||||
"""
|
|
||||||
|
|
||||||
import base64
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
DEFAULT_IMAGE = os.path.join(os.path.dirname(__file__), "example", "data", "aldi.jpg")
|
|
||||||
HOST = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434").rstrip("/")
|
|
||||||
MODEL = os.environ.get("OLLAMA_MODEL", "qwen2.5vl:7b")
|
|
||||||
|
|
||||||
PROMPT = (
|
|
||||||
"You are an OCR engine for retail receipts. Transcribe EVERY line of text on "
|
|
||||||
"this receipt, from top to bottom, exactly as printed. Each physical row of the "
|
|
||||||
"receipt becomes one string in the output, with its columns joined left-to-right "
|
|
||||||
"and separated by single spaces (e.g. an item code, its name, and its price all "
|
|
||||||
"go in the same row). Do not merge or split rows, do not reorder, do not add "
|
|
||||||
"commentary, do not correct spelling, do not invent text. Preserve numbers and "
|
|
||||||
"punctuation verbatim. Return strict JSON only."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Structured-output schema: forces the model to return {"rows": ["...", ...]}.
|
|
||||||
SCHEMA = {
|
|
||||||
"type": "object",
|
|
||||||
"properties": {"rows": {"type": "array", "items": {"type": "string"}}},
|
|
||||||
"required": ["rows"],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def ocr(image_path):
|
|
||||||
with open(image_path, "rb") as f:
|
|
||||||
b64 = base64.b64encode(f.read()).decode("ascii")
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"model": MODEL,
|
|
||||||
"messages": [{"role": "user", "content": PROMPT, "images": [b64]}],
|
|
||||||
"stream": False,
|
|
||||||
"format": SCHEMA,
|
|
||||||
"options": {"temperature": 0, "num_ctx": 8192},
|
|
||||||
}
|
|
||||||
|
|
||||||
req = urllib.request.Request(
|
|
||||||
f"{HOST}/api/chat",
|
|
||||||
data=json.dumps(payload).encode("utf-8"),
|
|
||||||
headers={"Content-Type": "application/json"},
|
|
||||||
)
|
|
||||||
with urllib.request.urlopen(req, timeout=600) as resp:
|
|
||||||
body = json.load(resp)
|
|
||||||
|
|
||||||
content = body["message"]["content"]
|
|
||||||
return json.loads(content)["rows"]
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
args = [a for a in sys.argv[1:] if not a.startswith("--")]
|
|
||||||
as_json = "--json" in sys.argv
|
|
||||||
image_path = args[0] if args else DEFAULT_IMAGE
|
|
||||||
|
|
||||||
if not os.path.exists(image_path):
|
|
||||||
sys.exit(f"image not found: {image_path}")
|
|
||||||
|
|
||||||
rows = ocr(image_path)
|
|
||||||
|
|
||||||
if as_json:
|
|
||||||
print(json.dumps({"rows": rows}, ensure_ascii=False, indent=2))
|
|
||||||
else:
|
|
||||||
width = len(str(len(rows)))
|
|
||||||
for i, row in enumerate(rows, 1):
|
|
||||||
print(f"{str(i).rjust(width)} {row}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
@ -1,80 +0,0 @@
|
||||||
# receiptscan
|
|
||||||
|
|
||||||
Sends a receipt photo to Claude, classifies the receipt into a budget category,
|
|
||||||
and (for groceries) locates + categorizes every line item, then renders an
|
|
||||||
annotated PNG: each line item is lightly shaded by food category with a category
|
|
||||||
tag in the margin, plus a per-category subtotal sidebar.
|
|
||||||
|
|
||||||
## Run
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/home/jm/.local/go/bin/go -C /home/jm/programming/HSAmanager/receiptscan run .
|
|
||||||
```
|
|
||||||
|
|
||||||
- Image path is hard-coded to `/home/jm/aldi.jpg` (or pass one as an argument).
|
|
||||||
- Reads `CLAUDE_API_KEY` from `../.env` (or `ANTHROPIC_API_KEY` in the env).
|
|
||||||
- Output: `<image>_annotated.png` next to the input.
|
|
||||||
- Go is installed locally at `/home/jm/.local/go` (no root). `GOTOOLCHAIN=local`
|
|
||||||
avoids auto-downloading a toolchain.
|
|
||||||
|
|
||||||
### Useful env flags
|
|
||||||
- `RECEIPT_CACHE=read` — re-render from the saved `<image>.cache.json` without
|
|
||||||
calling the API (fast iteration on rendering).
|
|
||||||
- `RECEIPT_DEBUG=1` — print the raw API response, the row-fit, and per-item row
|
|
||||||
assignments to stderr.
|
|
||||||
- `RECEIPT_ANALYZE=1` — dump detected text rows vs. model boxes and exit.
|
|
||||||
|
|
||||||
## Model query
|
|
||||||
|
|
||||||
- Model: `claude-opus-4-8` (returns true 1:1 image-pixel coordinates; Haiku does
|
|
||||||
not localize accurately and produced vertically-stretched boxes).
|
|
||||||
- Adaptive thinking + `output_config.effort: high`. Without thinking the model
|
|
||||||
often returned a placeholder/empty list.
|
|
||||||
- Structured output via `output_config.format` (`json_schema`, see schema in
|
|
||||||
`main.go` → `analyze`): `receipt_type`, `currency`, `total_amount`, and
|
|
||||||
`line_items[]` of `{name, price, category, box{x,y,width,height}}`.
|
|
||||||
- The model intermittently returns an empty `line_items` for a grocery receipt,
|
|
||||||
so the call is retried up to 4× until items come back.
|
|
||||||
|
|
||||||
### Prompt
|
|
||||||
|
|
||||||
```
|
|
||||||
You are reading a photographed grocery store receipt. The image is exactly
|
|
||||||
<W> pixels wide and <H> pixels tall. All coordinates you return must be in
|
|
||||||
pixels in that space, with (0,0) at the top-left.
|
|
||||||
|
|
||||||
1. Set receipt_type to "groceries" and read the grand total (total_amount)
|
|
||||||
with its currency.
|
|
||||||
|
|
||||||
2. This receipt has many purchased line items (dozens). Return EVERY one of
|
|
||||||
them - do not stop early and do not return an empty list. For each line item:
|
|
||||||
- name and price exactly as printed,
|
|
||||||
- a tight pixel rectangle (box) enclosing that whole printed line, from the
|
|
||||||
item name on the left through its price on the right,
|
|
||||||
- a food category, exactly one of: dairy, meat, bakery_bread, berries,
|
|
||||||
non_berry_fruits, vegetables, desserts, processed_food, pet_stuff,
|
|
||||||
other_groceries.
|
|
||||||
Use other_groceries only when nothing more specific fits. Do NOT include
|
|
||||||
subtotal, tax, payment, or total rows as line items.
|
|
||||||
|
|
||||||
Return only the structured object.
|
|
||||||
```
|
|
||||||
|
|
||||||
Budget categories for `receipt_type`: house_maintenance, activities, restaurant,
|
|
||||||
groceries, hobbies, alcohol, medical, education, other.
|
|
||||||
|
|
||||||
## How box alignment works
|
|
||||||
|
|
||||||
The model's returned y-coordinates are vertically stretched (~1.35× the true
|
|
||||||
line pitch) and only roughly placed, so boxes are **not** drawn at the model
|
|
||||||
coordinates. Instead the image itself is used to place them:
|
|
||||||
|
|
||||||
1. Detect horizontal text-row bands (dark-pixel projection in the item column).
|
|
||||||
2. Keep only **item rows**: bands with ink in the price column (right) AND the
|
|
||||||
item-code column (far left), at or below the model's first item. This
|
|
||||||
excludes centered headers, weight/quantity sub-lines, paper-edge shadows, and
|
|
||||||
the payment/total block.
|
|
||||||
3. The first N price-bearing rows (N = item count from the model) are the N
|
|
||||||
items, in order — map item *i* → row *i*.
|
|
||||||
|
|
||||||
This keeps alignment correct regardless of the model's coordinate drift.
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
module receiptscan
|
|
||||||
|
|
||||||
go 1.26
|
|
||||||
|
|
||||||
require golang.org/x/image v0.21.0
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
golang.org/x/image v0.21.0 h1:c5qV36ajHpdj4Qi0GnE0jUc/yuo33OLFaa0d+crTD5s=
|
|
||||||
golang.org/x/image v0.21.0/go.mod h1:vUbsLavqK/W303ZroQQVKQ+Af3Yl6Uz1Ppu5J/cLz78=
|
|
||||||
|
|
@ -1,853 +0,0 @@
|
||||||
// Command receiptscan sends a receipt image to Claude, asks it to classify the
|
|
||||||
// receipt into a budget category and (for groceries) to locate and categorize
|
|
||||||
// every line item, then renders an annotated image: each line item is lightly
|
|
||||||
// shaded by its food category, labeled, and summarized with per-category
|
|
||||||
// subtotals in a sidebar.
|
|
||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"encoding/base64"
|
|
||||||
"encoding/json"
|
|
||||||
"fmt"
|
|
||||||
"image"
|
|
||||||
"image/color"
|
|
||||||
"image/draw"
|
|
||||||
_ "image/jpeg"
|
|
||||||
"image/png"
|
|
||||||
"io"
|
|
||||||
"math"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"sort"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"golang.org/x/image/font"
|
|
||||||
"golang.org/x/image/font/basicfont"
|
|
||||||
"golang.org/x/image/math/fixed"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
model = "claude-opus-4-8"
|
|
||||||
endpoint = "https://api.anthropic.com/v1/messages"
|
|
||||||
)
|
|
||||||
|
|
||||||
// ---------- structured response from Claude ----------
|
|
||||||
|
|
||||||
type Box struct {
|
|
||||||
X int `json:"x"`
|
|
||||||
Y int `json:"y"`
|
|
||||||
Width int `json:"width"`
|
|
||||||
Height int `json:"height"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type LineItem struct {
|
|
||||||
Name string `json:"name"`
|
|
||||||
Price float64 `json:"price"`
|
|
||||||
Category string `json:"category"`
|
|
||||||
Box Box `json:"box"`
|
|
||||||
}
|
|
||||||
|
|
||||||
type Receipt struct {
|
|
||||||
ReceiptType string `json:"receipt_type"`
|
|
||||||
Currency string `json:"currency"`
|
|
||||||
TotalAmount float64 `json:"total_amount"`
|
|
||||||
LineItems []LineItem `json:"line_items"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// Food categories and their pale highlight colors (used for groceries).
|
|
||||||
var categoryColor = map[string]color.RGBA{
|
|
||||||
"dairy": {165, 205, 255, 255}, // light blue
|
|
||||||
"meat": {255, 150, 150, 255}, // red
|
|
||||||
"bakery_bread": {235, 205, 150, 255}, // tan
|
|
||||||
"berries": {220, 175, 235, 255}, // light purple
|
|
||||||
"non_berry_fruits": {255, 245, 150, 255}, // pale yellow
|
|
||||||
"vegetables": {180, 235, 175, 255}, // light green
|
|
||||||
"desserts": {255, 190, 220, 255}, // pink
|
|
||||||
"processed_food": {210, 200, 235, 255}, // lavender
|
|
||||||
"pet_stuff": {215, 200, 185, 255}, // light brown
|
|
||||||
"other_groceries": {220, 220, 220, 255}, // light grey
|
|
||||||
}
|
|
||||||
|
|
||||||
var categoryOrder = []string{
|
|
||||||
"dairy", "meat", "bakery_bread", "berries", "non_berry_fruits",
|
|
||||||
"vegetables", "desserts", "processed_food", "pet_stuff", "other_groceries",
|
|
||||||
}
|
|
||||||
|
|
||||||
func prettyCat(c string) string {
|
|
||||||
switch c {
|
|
||||||
case "bakery_bread":
|
|
||||||
return "Bakery / Bread"
|
|
||||||
case "non_berry_fruits":
|
|
||||||
return "Non-berry fruits"
|
|
||||||
case "processed_food":
|
|
||||||
return "Processed food"
|
|
||||||
case "pet_stuff":
|
|
||||||
return "Pet stuff"
|
|
||||||
case "other_groceries":
|
|
||||||
return "Other groceries"
|
|
||||||
}
|
|
||||||
return strings.Title(c)
|
|
||||||
}
|
|
||||||
|
|
||||||
func main() {
|
|
||||||
imgPath := "/home/jm/aldi.jpg"
|
|
||||||
if len(os.Args) > 1 {
|
|
||||||
imgPath = os.Args[1]
|
|
||||||
}
|
|
||||||
|
|
||||||
apiKey, err := loadAPIKey()
|
|
||||||
if err != nil {
|
|
||||||
fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
raw, err := os.ReadFile(imgPath)
|
|
||||||
if err != nil {
|
|
||||||
fatal(fmt.Errorf("read image: %w", err))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Decode once to learn the true pixel dimensions; Claude (Opus 4.7+) returns
|
|
||||||
// coordinates 1:1 with actual image pixels for images up to 2576px on the
|
|
||||||
// long edge, so we hand it the real dimensions to ground the boxes.
|
|
||||||
srcImg, _, err := image.Decode(bytes.NewReader(raw))
|
|
||||||
if err != nil {
|
|
||||||
fatal(fmt.Errorf("decode image: %w", err))
|
|
||||||
}
|
|
||||||
w, h := srcImg.Bounds().Dx(), srcImg.Bounds().Dy()
|
|
||||||
fmt.Printf("Image %s is %dx%d px. Asking %s...\n", filepath.Base(imgPath), w, h, model)
|
|
||||||
|
|
||||||
// Cache the model response so rendering can be iterated without re-calling.
|
|
||||||
cacheFile := strings.TrimSuffix(imgPath, filepath.Ext(imgPath)) + ".cache.json"
|
|
||||||
var receipt *Receipt
|
|
||||||
if os.Getenv("RECEIPT_CACHE") == "read" {
|
|
||||||
receipt, err = loadReceipt(cacheFile)
|
|
||||||
if err != nil {
|
|
||||||
fatal(fmt.Errorf("load cache: %w", err))
|
|
||||||
}
|
|
||||||
fmt.Printf("Loaded cached analysis from %s\n", cacheFile)
|
|
||||||
} else {
|
|
||||||
// The model occasionally returns an empty line_items list for a grocery
|
|
||||||
// receipt; retry until it enumerates them (or attempts are exhausted).
|
|
||||||
for attempt := 1; attempt <= 4; attempt++ {
|
|
||||||
receipt, err = analyze(apiKey, raw, w, h)
|
|
||||||
if err != nil {
|
|
||||||
fatal(err)
|
|
||||||
}
|
|
||||||
if receipt.ReceiptType != "groceries" || len(receipt.LineItems) > 0 {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
fmt.Printf(" (attempt %d returned no line items, retrying...)\n", attempt)
|
|
||||||
}
|
|
||||||
if b, e := json.MarshalIndent(receipt, "", " "); e == nil {
|
|
||||||
_ = os.WriteFile(cacheFile, b, 0644)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Detected text rows from the image; used to snap model boxes onto real rows.
|
|
||||||
rows := detectTextRows(srcImg, receipt)
|
|
||||||
|
|
||||||
if os.Getenv("RECEIPT_ANALYZE") != "" {
|
|
||||||
analyzeAlignment(receipt, rows)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
alignBoxes(srcImg, receipt, rows)
|
|
||||||
printSummary(receipt)
|
|
||||||
|
|
||||||
outPath := strings.TrimSuffix(imgPath, filepath.Ext(imgPath)) + "_annotated.png"
|
|
||||||
if err := render(srcImg, receipt, outPath); err != nil {
|
|
||||||
fatal(err)
|
|
||||||
}
|
|
||||||
fmt.Printf("\nAnnotated image written to %s\n", outPath)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- API call ----------
|
|
||||||
|
|
||||||
func analyze(apiKey string, img []byte, w, h int) (*Receipt, error) {
|
|
||||||
schema := map[string]any{
|
|
||||||
"type": "object",
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": []string{"receipt_type", "currency", "total_amount", "line_items"},
|
|
||||||
"properties": map[string]any{
|
|
||||||
"receipt_type": map[string]any{
|
|
||||||
"type": "string",
|
|
||||||
"enum": []string{"house_maintenance", "activities", "restaurant",
|
|
||||||
"groceries", "hobbies", "alcohol", "medical", "education", "other"},
|
|
||||||
"description": "Best-fitting budget category for the whole receipt.",
|
|
||||||
},
|
|
||||||
"currency": map[string]any{"type": "string", "description": "ISO-ish currency symbol or code, e.g. $, USD, EUR."},
|
|
||||||
"total_amount": map[string]any{"type": "number", "description": "Grand total paid."},
|
|
||||||
"line_items": map[string]any{
|
|
||||||
"type": "array",
|
|
||||||
"description": "One entry per purchased line item. Empty unless the receipt is groceries.",
|
|
||||||
"items": map[string]any{
|
|
||||||
"type": "object",
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": []string{"name", "price", "category", "box"},
|
|
||||||
"properties": map[string]any{
|
|
||||||
"name": map[string]any{"type": "string", "description": "Item name as printed."},
|
|
||||||
"price": map[string]any{"type": "number", "description": "Line item price."},
|
|
||||||
"category": map[string]any{
|
|
||||||
"type": "string",
|
|
||||||
"enum": categoryOrder,
|
|
||||||
},
|
|
||||||
"box": map[string]any{
|
|
||||||
"type": "object",
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": []string{"x", "y", "width", "height"},
|
|
||||||
"description": "Pixel rectangle tightly enclosing the whole line (name + price).",
|
|
||||||
"properties": map[string]any{
|
|
||||||
"x": map[string]any{"type": "integer", "description": "Left edge, pixels from left."},
|
|
||||||
"y": map[string]any{"type": "integer", "description": "Top edge, pixels from top."},
|
|
||||||
"width": map[string]any{"type": "integer"},
|
|
||||||
"height": map[string]any{"type": "integer"},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
prompt := fmt.Sprintf(`You are reading a photographed grocery store receipt. The image is exactly %d pixels wide and %d pixels tall. All coordinates you return must be in pixels in that space, with (0,0) at the top-left.
|
|
||||||
|
|
||||||
1. Set receipt_type to "groceries" and read the grand total (total_amount) with its currency.
|
|
||||||
|
|
||||||
2. This receipt has many purchased line items (dozens). Return EVERY one of them - do not stop early and do not return an empty list. For each line item:
|
|
||||||
- name and price exactly as printed,
|
|
||||||
- a tight pixel rectangle (box) enclosing that whole printed line, from the item name on the left through its price on the right,
|
|
||||||
- a food category, exactly one of: dairy, meat, bakery_bread, berries, non_berry_fruits, vegetables, desserts, processed_food, pet_stuff, other_groceries.
|
|
||||||
Use other_groceries only when nothing more specific fits. Do NOT include subtotal, tax, payment, or total rows as line items.
|
|
||||||
|
|
||||||
Return only the structured object.`, w, h)
|
|
||||||
|
|
||||||
reqBody := map[string]any{
|
|
||||||
"model": model,
|
|
||||||
"max_tokens": 24000,
|
|
||||||
"output_config": map[string]any{
|
|
||||||
"format": map[string]any{"type": "json_schema", "schema": schema},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
// Per-model thinking config: Haiku 4.5 (a 4.x model) uses budgeted extended
|
|
||||||
// thinking and rejects the effort param; 4.6+ models use adaptive thinking.
|
|
||||||
if strings.Contains(model, "haiku") {
|
|
||||||
reqBody["thinking"] = map[string]any{"type": "enabled", "budget_tokens": 8000}
|
|
||||||
} else {
|
|
||||||
reqBody["thinking"] = map[string]any{"type": "adaptive"}
|
|
||||||
reqBody["output_config"].(map[string]any)["effort"] = "high"
|
|
||||||
}
|
|
||||||
reqBody["messages"] = []any{
|
|
||||||
map[string]any{
|
|
||||||
"role": "user",
|
|
||||||
"content": []any{
|
|
||||||
map[string]any{
|
|
||||||
"type": "image",
|
|
||||||
"source": map[string]any{
|
|
||||||
"type": "base64",
|
|
||||||
"media_type": "image/jpeg",
|
|
||||||
"data": base64.StdEncoding.EncodeToString(img),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
map[string]any{"type": "text", "text": prompt},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
buf, _ := json.Marshal(reqBody)
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
|
||||||
defer cancel()
|
|
||||||
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(buf))
|
|
||||||
req.Header.Set("x-api-key", apiKey)
|
|
||||||
req.Header.Set("anthropic-version", "2023-06-01")
|
|
||||||
req.Header.Set("content-type", "application/json")
|
|
||||||
|
|
||||||
resp, err := (&http.Client{Timeout: 6 * time.Minute}).Do(req)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("request: %w", err)
|
|
||||||
}
|
|
||||||
defer resp.Body.Close()
|
|
||||||
body, _ := io.ReadAll(resp.Body)
|
|
||||||
if os.Getenv("RECEIPT_DEBUG") != "" {
|
|
||||||
fmt.Fprintf(os.Stderr, "[debug] http=%d body=%s\n", resp.StatusCode, string(body))
|
|
||||||
}
|
|
||||||
|
|
||||||
var parsed struct {
|
|
||||||
StopReason string `json:"stop_reason"`
|
|
||||||
Content []struct {
|
|
||||||
Type string `json:"type"`
|
|
||||||
Text string `json:"text"`
|
|
||||||
} `json:"content"`
|
|
||||||
Error *struct {
|
|
||||||
Type string `json:"type"`
|
|
||||||
Message string `json:"message"`
|
|
||||||
} `json:"error"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(body, &parsed); err != nil {
|
|
||||||
return nil, fmt.Errorf("decode response: %w", err)
|
|
||||||
}
|
|
||||||
if parsed.Error != nil {
|
|
||||||
return nil, fmt.Errorf("api error (%s): %s", parsed.Error.Type, parsed.Error.Message)
|
|
||||||
}
|
|
||||||
if parsed.StopReason == "refusal" {
|
|
||||||
return nil, fmt.Errorf("model refused the request")
|
|
||||||
}
|
|
||||||
|
|
||||||
var jsonText string
|
|
||||||
for _, b := range parsed.Content {
|
|
||||||
if b.Type == "text" {
|
|
||||||
jsonText = b.Text
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if os.Getenv("RECEIPT_DEBUG") != "" {
|
|
||||||
fmt.Fprintf(os.Stderr, "[debug] stop_reason=%s json_len=%d\n%s\n", parsed.StopReason, len(jsonText), jsonText)
|
|
||||||
}
|
|
||||||
if jsonText == "" {
|
|
||||||
return nil, fmt.Errorf("no text block in response")
|
|
||||||
}
|
|
||||||
|
|
||||||
var r Receipt
|
|
||||||
if err := json.Unmarshal([]byte(jsonText), &r); err != nil {
|
|
||||||
return nil, fmt.Errorf("parse structured output: %w\n%s", err, jsonText)
|
|
||||||
}
|
|
||||||
return &r, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- row detection & snapping ----------
|
|
||||||
|
|
||||||
type rowBand struct{ top, bottom, center int }
|
|
||||||
|
|
||||||
// detectTextRows finds horizontal bands of printed text within the column that
|
|
||||||
// the model's line-item boxes occupy, by projecting dark pixels per scanline.
|
|
||||||
func detectTextRows(img image.Image, r *Receipt) []rowBand {
|
|
||||||
b := img.Bounds()
|
|
||||||
w, h := b.Dx(), b.Dy()
|
|
||||||
|
|
||||||
minX, maxX := w, 0
|
|
||||||
for _, it := range r.LineItems {
|
|
||||||
if it.Box.X < minX {
|
|
||||||
minX = it.Box.X
|
|
||||||
}
|
|
||||||
if it.Box.X+it.Box.Width > maxX {
|
|
||||||
maxX = it.Box.X + it.Box.Width
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if minX < 0 {
|
|
||||||
minX = 0
|
|
||||||
}
|
|
||||||
if maxX > w || maxX <= minX {
|
|
||||||
maxX = w
|
|
||||||
}
|
|
||||||
span := maxX - minX
|
|
||||||
if span <= 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
dark := make([]float64, h)
|
|
||||||
for y := 0; y < h; y++ {
|
|
||||||
cnt := 0
|
|
||||||
for x := minX; x < maxX; x++ {
|
|
||||||
rr, gg, bb, _ := img.At(b.Min.X+x, b.Min.Y+y).RGBA()
|
|
||||||
lum := (299*rr + 587*gg + 114*bb) / 1000 >> 8 // 0..255
|
|
||||||
if lum < 150 {
|
|
||||||
cnt++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
dark[y] = float64(cnt) / float64(span)
|
|
||||||
}
|
|
||||||
|
|
||||||
const thr = 0.03
|
|
||||||
var bands []rowBand
|
|
||||||
inBand, start := false, 0
|
|
||||||
for y := 0; y < h; y++ {
|
|
||||||
if dark[y] > thr {
|
|
||||||
if !inBand {
|
|
||||||
inBand, start = true, y
|
|
||||||
}
|
|
||||||
} else if inBand {
|
|
||||||
inBand = false
|
|
||||||
if y-start >= 8 {
|
|
||||||
bands = append(bands, rowBand{start, y, (start + y) / 2})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if inBand && h-start >= 8 {
|
|
||||||
bands = append(bands, rowBand{start, h, (start + h) / 2})
|
|
||||||
}
|
|
||||||
return bands
|
|
||||||
}
|
|
||||||
|
|
||||||
func analyzeAlignment(r *Receipt, rows []rowBand) {
|
|
||||||
fmt.Printf("\n%d detected text bands:\n", len(rows))
|
|
||||||
for i, rw := range rows {
|
|
||||||
fmt.Printf(" band %2d: y %4d..%4d center %4d (h%2d)\n", i, rw.top, rw.bottom, rw.center, rw.bottom-rw.top)
|
|
||||||
}
|
|
||||||
fmt.Printf("\n%d model items:\n", len(r.LineItems))
|
|
||||||
for i, it := range r.LineItems {
|
|
||||||
fmt.Printf(" item %2d: top %4d center %4d %s\n", i, it.Box.Y, it.Box.Y+it.Box.Height/2, it.Name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// alignBoxes places each line item onto its true printed row. Item rows are the
|
|
||||||
// rows that carry a price in the right-hand column; headers, weight/quantity
|
|
||||||
// sub-lines, and totals don't, so the first N price-bearing rows (top to bottom)
|
|
||||||
// are exactly the N items, in order. This avoids fitting unreliable model
|
|
||||||
// coordinates. If price detection doesn't yield enough rows, fall back to the
|
|
||||||
// coordinate-fit snap.
|
|
||||||
func alignBoxes(img image.Image, r *Receipt, rows []rowBand) {
|
|
||||||
n := len(r.LineItems)
|
|
||||||
if n == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
itemRows := pricedRows(img, rows, r)
|
|
||||||
if os.Getenv("RECEIPT_DEBUG") != "" {
|
|
||||||
fmt.Fprintf(os.Stderr, "[align] items=%d pricedRows=%d\n", n, len(itemRows))
|
|
||||||
}
|
|
||||||
if len(itemRows) >= n {
|
|
||||||
for i := 0; i < n; i++ {
|
|
||||||
b := itemRows[i]
|
|
||||||
if os.Getenv("RECEIPT_DEBUG") != "" {
|
|
||||||
fmt.Fprintf(os.Stderr, " item %2d -> row center %4d %s\n", i, b.center, r.LineItems[i].Name)
|
|
||||||
}
|
|
||||||
r.LineItems[i].Box.Y = b.top - 1
|
|
||||||
r.LineItems[i].Box.Height = b.bottom - b.top + 2
|
|
||||||
}
|
|
||||||
return
|
|
||||||
}
|
|
||||||
snapBoxes(r, rows) // fallback
|
|
||||||
}
|
|
||||||
|
|
||||||
// pricedRows returns the bands that look like item lines: ink in the price
|
|
||||||
// column (right end of the boxes) AND ink in the item-code column (just left of
|
|
||||||
// the item name). Requiring both excludes centered header lines (which can reach
|
|
||||||
// the price column but have nothing at the far left). Top-to-bottom order.
|
|
||||||
func pricedRows(img image.Image, all []rowBand, r *Receipt) []rowBand {
|
|
||||||
b := img.Bounds()
|
|
||||||
right, nameLeft, minY := 0, b.Dx(), b.Dy()
|
|
||||||
for _, it := range r.LineItems {
|
|
||||||
if x := it.Box.X + it.Box.Width; x > right {
|
|
||||||
right = x
|
|
||||||
}
|
|
||||||
if it.Box.X < nameLeft {
|
|
||||||
nameLeft = it.Box.X
|
|
||||||
}
|
|
||||||
if it.Box.Y < minY {
|
|
||||||
minY = it.Box.Y
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if right <= 0 || right > b.Dx() {
|
|
||||||
right = b.Dx()
|
|
||||||
}
|
|
||||||
priceL := clamp(right-110, 0, b.Dx())
|
|
||||||
codeL := clamp(nameLeft-95, 0, b.Dx())
|
|
||||||
codeR := clamp(nameLeft-15, 0, b.Dx())
|
|
||||||
// The model is accurate near the top, so rows above its first item are header
|
|
||||||
// or paper-edge artifacts, not items.
|
|
||||||
floorY := minY - 30
|
|
||||||
|
|
||||||
var out []rowBand
|
|
||||||
for _, rw := range all {
|
|
||||||
if rw.center < floorY {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if inkFrac(img, codeL, codeR, rw.top, rw.bottom) > 0.03 &&
|
|
||||||
inkFrac(img, priceL, right, rw.top, rw.bottom) > 0.04 {
|
|
||||||
out = append(out, rw)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
func inkFrac(img image.Image, x0, x1, y0, y1 int) float64 {
|
|
||||||
b := img.Bounds()
|
|
||||||
cnt, tot := 0, 0
|
|
||||||
for y := y0; y < y1; y++ {
|
|
||||||
for x := x0; x < x1; x++ {
|
|
||||||
rr, gg, bb, _ := img.At(b.Min.X+x, b.Min.Y+y).RGBA()
|
|
||||||
if (299*rr+587*gg+114*bb)/1000>>8 < 150 {
|
|
||||||
cnt++
|
|
||||||
}
|
|
||||||
tot++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if tot == 0 {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
return float64(cnt) / float64(tot)
|
|
||||||
}
|
|
||||||
|
|
||||||
func clamp(v, lo, hi int) int {
|
|
||||||
if v < lo {
|
|
||||||
return lo
|
|
||||||
}
|
|
||||||
if v > hi {
|
|
||||||
return hi
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
|
|
||||||
// snapBoxes corrects the model's boxes onto the real text rows. The model
|
|
||||||
// returns vertically-stretched coordinates (line pitch ~1.35x the true pitch),
|
|
||||||
// so we first recover the best linear map a*y+b that lands the item centers on
|
|
||||||
// detected text bands (a 1-D iterative-closest-point fit), then snap each item
|
|
||||||
// to its band with a monotonic (strictly increasing) constraint so sub-lines,
|
|
||||||
// headers and the payment block are skipped automatically.
|
|
||||||
func snapBoxes(r *Receipt, rows []rowBand) {
|
|
||||||
n := len(r.LineItems)
|
|
||||||
if n == 0 || len(rows) < n {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
M := make([]float64, n)
|
|
||||||
for i, it := range r.LineItems {
|
|
||||||
M[i] = float64(it.Box.Y) + float64(it.Box.Height)/2
|
|
||||||
}
|
|
||||||
bandC := make([]float64, len(rows))
|
|
||||||
for i, b := range rows {
|
|
||||||
bandC[i] = float64(b.center)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The scale is pinned to the ratio of true line pitch to the model's line
|
|
||||||
// pitch (robust to a few double-gaps from sub-lines). Searching scale freely
|
|
||||||
// lets the fit march the tail items onto the evenly-spaced payment rows.
|
|
||||||
a0 := medianDiff(bandC) / medianDiff(M)
|
|
||||||
if a0 <= 0 || math.IsNaN(a0) {
|
|
||||||
a0 = 0.75
|
|
||||||
}
|
|
||||||
|
|
||||||
bestA, bestB, bestCost := a0, 0.0, math.Inf(1)
|
|
||||||
for a := a0 * 0.9; a <= a0*1.1; a += a0 * 0.004 {
|
|
||||||
for b := -350.0; b <= 250.0; b += 2 {
|
|
||||||
if c, ok := assignCost(a, b, M, rows); ok && c < bestCost {
|
|
||||||
bestCost, bestA, bestB = c, a, b
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if os.Getenv("RECEIPT_DEBUG") != "" {
|
|
||||||
fmt.Fprintf(os.Stderr, "[snap] a0=%.3f -> a=%.3f b=%.1f avgResidual=%.1fpx\n",
|
|
||||||
a0, bestA, bestB, bestCost/float64(n))
|
|
||||||
}
|
|
||||||
|
|
||||||
idxs, _ := assignBands(bestA, bestB, M, rows)
|
|
||||||
for i := range r.LineItems {
|
|
||||||
band := rows[idxs[i]]
|
|
||||||
r.LineItems[i].Box.Y = band.top - 1
|
|
||||||
r.LineItems[i].Box.Height = band.bottom - band.top + 2
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// assignBands maps each item (in order) to a strictly-increasing band index via
|
|
||||||
// greedy nearest-after-previous. Returns the indices and total residual.
|
|
||||||
func assignBands(a, b float64, M []float64, rows []rowBand) ([]int, float64) {
|
|
||||||
idxs := make([]int, len(M))
|
|
||||||
prev, total := -1, 0.0
|
|
||||||
for i := range M {
|
|
||||||
y := a*M[i] + b
|
|
||||||
idx, bd := -1, math.Inf(1)
|
|
||||||
for j := prev + 1; j < len(rows); j++ {
|
|
||||||
if d := math.Abs(float64(rows[j].center) - y); d < bd {
|
|
||||||
bd, idx = d, j
|
|
||||||
} else if float64(rows[j].center) > y {
|
|
||||||
break // distances only grow past y
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if idx < 0 {
|
|
||||||
idx = len(rows) - 1
|
|
||||||
}
|
|
||||||
idxs[i], prev, total = idx, idx, total+bd
|
|
||||||
}
|
|
||||||
return idxs, total
|
|
||||||
}
|
|
||||||
|
|
||||||
// assignCost returns the residual of a monotonic assignment; ok is false if the
|
|
||||||
// transform can't fit all items into strictly-increasing bands.
|
|
||||||
func assignCost(a, b float64, M []float64, rows []rowBand) (float64, bool) {
|
|
||||||
if len(rows) < len(M) {
|
|
||||||
return 0, false
|
|
||||||
}
|
|
||||||
prev := -1
|
|
||||||
for i := range M {
|
|
||||||
y := a*M[i] + b
|
|
||||||
idx, bd := -1, math.Inf(1)
|
|
||||||
for j := prev + 1; j < len(rows); j++ {
|
|
||||||
if d := math.Abs(float64(rows[j].center) - y); d < bd {
|
|
||||||
bd, idx = d, j
|
|
||||||
} else if float64(rows[j].center) > y {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if idx < 0 || len(rows)-idx < len(M)-i {
|
|
||||||
return 0, false // ran out of bands for the remaining items
|
|
||||||
}
|
|
||||||
prev = idx
|
|
||||||
_ = bd
|
|
||||||
}
|
|
||||||
_, total := assignBands(a, b, M, rows)
|
|
||||||
return total, true
|
|
||||||
}
|
|
||||||
|
|
||||||
func medianDiff(xs []float64) float64 {
|
|
||||||
if len(xs) < 2 {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
d := make([]float64, 0, len(xs)-1)
|
|
||||||
for i := 1; i < len(xs); i++ {
|
|
||||||
if xs[i]-xs[i-1] > 0 {
|
|
||||||
d = append(d, xs[i]-xs[i-1])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if len(d) == 0 {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
sort.Float64s(d)
|
|
||||||
return d[len(d)/2]
|
|
||||||
}
|
|
||||||
|
|
||||||
func loadReceipt(path string) (*Receipt, error) {
|
|
||||||
b, err := os.ReadFile(path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
var r Receipt
|
|
||||||
if err := json.Unmarshal(b, &r); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return &r, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- rendering ----------
|
|
||||||
|
|
||||||
const sidebarW = 420
|
|
||||||
|
|
||||||
func render(src image.Image, r *Receipt, outPath string) error {
|
|
||||||
b := src.Bounds()
|
|
||||||
w, h := b.Dx(), b.Dy()
|
|
||||||
|
|
||||||
canvas := image.NewRGBA(image.Rect(0, 0, w+sidebarW, h))
|
|
||||||
// white background
|
|
||||||
draw.Draw(canvas, canvas.Bounds(), image.NewUniform(color.White), image.Point{}, draw.Src)
|
|
||||||
// the receipt photo on the left
|
|
||||||
draw.Draw(canvas, image.Rect(0, 0, w, h), src, b.Min, draw.Src)
|
|
||||||
|
|
||||||
subtotals := map[string]float64{}
|
|
||||||
|
|
||||||
for _, it := range r.LineItems {
|
|
||||||
col, ok := categoryColor[it.Category]
|
|
||||||
if !ok {
|
|
||||||
col = categoryColor["other_groceries"]
|
|
||||||
}
|
|
||||||
subtotals[it.Category] += it.Price
|
|
||||||
|
|
||||||
rect := clampRect(it.Box, w, h)
|
|
||||||
// pale category highlight over the whole printed line
|
|
||||||
fill := color.RGBA{col.R, col.G, col.B, 110}
|
|
||||||
draw.Draw(canvas, rect, image.NewUniform(fill), image.Point{}, draw.Over)
|
|
||||||
drawBorder(canvas, rect, col, 1)
|
|
||||||
|
|
||||||
// category tag in the margin to the right of the line (no overlap with text)
|
|
||||||
yMid := (rect.Min.Y + rect.Max.Y) / 2
|
|
||||||
drawTag(canvas, rect.Max.X+8, yMid, prettyCat(it.Category), col)
|
|
||||||
}
|
|
||||||
|
|
||||||
drawSidebar(canvas, w, h, r, subtotals)
|
|
||||||
|
|
||||||
f, err := os.Create(outPath)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
return png.Encode(f, canvas)
|
|
||||||
}
|
|
||||||
|
|
||||||
func drawSidebar(canvas *image.RGBA, w, h int, r *Receipt, subtotals map[string]float64) {
|
|
||||||
x0 := w + 1
|
|
||||||
// faint divider
|
|
||||||
draw.Draw(canvas, image.Rect(w, 0, w+1, h), image.NewUniform(color.RGBA{200, 200, 200, 255}), image.Point{}, draw.Src)
|
|
||||||
|
|
||||||
black := color.RGBA{20, 20, 20, 255}
|
|
||||||
y := 28
|
|
||||||
drawText(canvas, x0+16, y, "RECEIPT SUMMARY", black)
|
|
||||||
y += 26
|
|
||||||
drawText(canvas, x0+16, y, fmt.Sprintf("Type: %s", prettyCat(r.ReceiptType)), black)
|
|
||||||
y += 20
|
|
||||||
drawText(canvas, x0+16, y, fmt.Sprintf("Total: %s%.2f", sym(r.Currency), r.TotalAmount), black)
|
|
||||||
y += 30
|
|
||||||
drawText(canvas, x0+16, y, "Category subtotals", black)
|
|
||||||
y += 8
|
|
||||||
draw.Draw(canvas, image.Rect(x0+16, y, w+sidebarW-16, y+1), image.NewUniform(color.RGBA{200, 200, 200, 255}), image.Point{}, draw.Src)
|
|
||||||
y += 20
|
|
||||||
|
|
||||||
var catSum float64
|
|
||||||
for _, c := range categoryOrder {
|
|
||||||
v, ok := subtotals[c]
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
catSum += v
|
|
||||||
col := categoryColor[c]
|
|
||||||
// swatch
|
|
||||||
draw.Draw(canvas, image.Rect(x0+16, y-11, x0+34, y+3), image.NewUniform(col), image.Point{}, draw.Src)
|
|
||||||
drawBorder(canvas, image.Rect(x0+16, y-11, x0+34, y+3), black, 1)
|
|
||||||
drawText(canvas, x0+42, y, fmt.Sprintf("%-16s %s%6.2f", prettyCat(c), sym(r.Currency), v), black)
|
|
||||||
y += 22
|
|
||||||
}
|
|
||||||
|
|
||||||
y += 6
|
|
||||||
draw.Draw(canvas, image.Rect(x0+16, y, w+sidebarW-16, y+1), image.NewUniform(color.RGBA{200, 200, 200, 255}), image.Point{}, draw.Src)
|
|
||||||
y += 20
|
|
||||||
drawText(canvas, x0+16, y, fmt.Sprintf("Items subtotal: %s%.2f", sym(r.Currency), catSum), black)
|
|
||||||
y += 20
|
|
||||||
drawText(canvas, x0+16, y, fmt.Sprintf("(taxes/fees not itemized)"), color.RGBA{120, 120, 120, 255})
|
|
||||||
}
|
|
||||||
|
|
||||||
// drawTag draws a solid colored chip with a category name, vertically centered
|
|
||||||
// on y, with its left edge at x. Placed in the margin beside each line.
|
|
||||||
func drawTag(dst *image.RGBA, x, y int, s string, col color.RGBA) {
|
|
||||||
wpx := len(s)*7 + 10
|
|
||||||
r := image.Rect(x, y-9, x+wpx, y+8)
|
|
||||||
draw.Draw(dst, r, image.NewUniform(color.RGBA{col.R, col.G, col.B, 255}), image.Point{}, draw.Over)
|
|
||||||
drawBorder(dst, r, darken(col), 1)
|
|
||||||
drawText(dst, x+5, y+4, s, color.RGBA{30, 30, 30, 255})
|
|
||||||
}
|
|
||||||
|
|
||||||
func darken(c color.RGBA) color.RGBA {
|
|
||||||
return color.RGBA{c.R * 7 / 10, c.G * 7 / 10, c.B * 7 / 10, 255}
|
|
||||||
}
|
|
||||||
|
|
||||||
func drawText(dst *image.RGBA, x, y int, s string, c color.Color) {
|
|
||||||
d := &font.Drawer{
|
|
||||||
Dst: dst,
|
|
||||||
Src: image.NewUniform(c),
|
|
||||||
Face: basicfont.Face7x13,
|
|
||||||
Dot: fixed.P(x, y),
|
|
||||||
}
|
|
||||||
d.DrawString(s)
|
|
||||||
}
|
|
||||||
|
|
||||||
func drawBorder(dst *image.RGBA, r image.Rectangle, c color.Color, thick int) {
|
|
||||||
u := image.NewUniform(c)
|
|
||||||
draw.Draw(dst, image.Rect(r.Min.X, r.Min.Y, r.Max.X, r.Min.Y+thick), u, image.Point{}, draw.Over)
|
|
||||||
draw.Draw(dst, image.Rect(r.Min.X, r.Max.Y-thick, r.Max.X, r.Max.Y), u, image.Point{}, draw.Over)
|
|
||||||
draw.Draw(dst, image.Rect(r.Min.X, r.Min.Y, r.Min.X+thick, r.Max.Y), u, image.Point{}, draw.Over)
|
|
||||||
draw.Draw(dst, image.Rect(r.Max.X-thick, r.Min.Y, r.Max.X, r.Max.Y), u, image.Point{}, draw.Over)
|
|
||||||
}
|
|
||||||
|
|
||||||
func clampRect(box Box, w, h int) image.Rectangle {
|
|
||||||
x0, y0 := box.X, box.Y
|
|
||||||
x1, y1 := box.X+box.Width, box.Y+box.Height
|
|
||||||
if x0 < 0 {
|
|
||||||
x0 = 0
|
|
||||||
}
|
|
||||||
if y0 < 0 {
|
|
||||||
y0 = 0
|
|
||||||
}
|
|
||||||
if x1 > w {
|
|
||||||
x1 = w
|
|
||||||
}
|
|
||||||
if y1 > h {
|
|
||||||
y1 = h
|
|
||||||
}
|
|
||||||
if x1 <= x0 {
|
|
||||||
x1 = x0 + 1
|
|
||||||
}
|
|
||||||
if y1 <= y0 {
|
|
||||||
y1 = y0 + 1
|
|
||||||
}
|
|
||||||
return image.Rect(x0, y0, x1, y1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------- helpers ----------
|
|
||||||
|
|
||||||
func printSummary(r *Receipt) {
|
|
||||||
fmt.Printf("\nReceipt type: %s\nTotal: %s%.2f\n", prettyCat(r.ReceiptType), sym(r.Currency), r.TotalAmount)
|
|
||||||
if len(r.LineItems) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
fmt.Printf("\n%d line items:\n", len(r.LineItems))
|
|
||||||
subtotals := map[string]float64{}
|
|
||||||
for _, it := range r.LineItems {
|
|
||||||
subtotals[it.Category] += it.Price
|
|
||||||
fmt.Printf(" %-28s %s%6.2f %s\n", trunc(it.Name, 28), sym(r.Currency), it.Price, prettyCat(it.Category))
|
|
||||||
}
|
|
||||||
fmt.Println("\nCategory subtotals:")
|
|
||||||
for _, c := range categoryOrder {
|
|
||||||
if v, ok := subtotals[c]; ok {
|
|
||||||
fmt.Printf(" %-18s %s%6.2f\n", prettyCat(c), sym(r.Currency), v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func trunc(s string, n int) string {
|
|
||||||
if len(s) <= n {
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
return s[:n-1] + "…"
|
|
||||||
}
|
|
||||||
|
|
||||||
func sym(currency string) string {
|
|
||||||
switch strings.ToUpper(strings.TrimSpace(currency)) {
|
|
||||||
case "USD", "$", "":
|
|
||||||
return "$"
|
|
||||||
case "EUR":
|
|
||||||
return "€"
|
|
||||||
case "GBP":
|
|
||||||
return "£"
|
|
||||||
case "CAD":
|
|
||||||
return "C$"
|
|
||||||
}
|
|
||||||
return currency + " "
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadAPIKey reads CLAUDE_API_KEY (or ANTHROPIC_API_KEY) from the environment,
|
|
||||||
// falling back to a .env file in the working directory or this program's folder.
|
|
||||||
func loadAPIKey() (string, error) {
|
|
||||||
for _, k := range []string{"CLAUDE_API_KEY", "ANTHROPIC_API_KEY"} {
|
|
||||||
if v := os.Getenv(k); v != "" {
|
|
||||||
return v, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
candidates := []string{".env"}
|
|
||||||
if exe, err := os.Executable(); err == nil {
|
|
||||||
candidates = append(candidates, filepath.Join(filepath.Dir(exe), ".env"))
|
|
||||||
}
|
|
||||||
candidates = append(candidates,
|
|
||||||
"/home/jm/programming/HSAmanager/.env",
|
|
||||||
"/home/jm/programming/HSAmanager/receiptscan/.env")
|
|
||||||
for _, p := range candidates {
|
|
||||||
if v := keyFromEnvFile(p); v != "" {
|
|
||||||
return v, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return "", fmt.Errorf("no CLAUDE_API_KEY / ANTHROPIC_API_KEY in env or .env")
|
|
||||||
}
|
|
||||||
|
|
||||||
func keyFromEnvFile(path string) string {
|
|
||||||
f, err := os.Open(path)
|
|
||||||
if err != nil {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
sc := bufio.NewScanner(f)
|
|
||||||
for sc.Scan() {
|
|
||||||
line := strings.TrimSpace(sc.Text())
|
|
||||||
line = strings.TrimPrefix(line, "export ")
|
|
||||||
for _, k := range []string{"CLAUDE_API_KEY", "ANTHROPIC_API_KEY"} {
|
|
||||||
if strings.HasPrefix(line, k+"=") {
|
|
||||||
v := strings.TrimSpace(line[len(k)+1:])
|
|
||||||
v = strings.Trim(v, `"'`)
|
|
||||||
if v != "" {
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
func fatal(err error) {
|
|
||||||
fmt.Fprintln(os.Stderr, "error:", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
Loading…
Reference in a new issue