hsa-app/OCR/README.md

I want to create an OCR feature (ollama, whatever) that's

1. local (assume I have a GPU with 24GB VRAM)
2. works on receipts (scanned or digital)
3. given a receipt, return an ordered list of rows of text in that receipt
4. example /home/jm/programming/HSAmanager/OCR/example/data/aldi.jpg

---

Getting started (working prototype)

Local, GPU-backed, no cloud / no API key. Engine: Qwen2.5-VL 7B served by ollama. The model transcribes the receipt top-to-bottom and returns one string per physical row, columns joined left-to-right.

One-time setup (already done on this machine)

• ollama binary installed (no root) at ~/.local/bin/ollama.
• Model pulled: ollama pull qwen2.5vl:7b (~6 GB).

Run

# 1. start the server (leave running; ~6 GB VRAM when the model loads)
~/.local/bin/ollama serve &

# 2. transcribe a receipt -> ordered, numbered rows
python3 OCR/receipt_ocr.py OCR/example/data/aldi.jpg

# raw JSON {"rows": [...]} instead of numbered text
python3 OCR/receipt_ocr.py OCR/example/data/aldi.jpg --json

No third-party Python deps — receipt_ocr.py uses the stdlib and the ollama HTTP API. Override with OLLAMA_MODEL / OLLAMA_HOST env vars.

Notes / next steps

• Works on the ALDI example with near-perfect line ordering and column joining.
• Caveat: the model sometimes expands thermal-print abbreviations (Fire Roasted Tom → Fire Roasted Tomatoes). Helpful for readability but not byte-for-byte faithful; tighten the prompt if you need verbatim text.
• Accuracy upgrade: qwen2.5vl:32b (q4 ≈ 20 GB, fits the 3090 but tighter/slower).
• This is local-only and complementary to ../receiptscan/, which uses the Claude cloud API for line-item categorization + annotated rendering.

---

Operating ollama (runbook)

ollama is the local model server both scripts talk to over HTTP (http://127.0.0.1:11434). It must be running before you call either script. Installed without root at ~/.local/bin/ollama (the binary was extracted from the ollama-linux-amd64.tar.zst GitHub release; ~/.local/bin is on PATH, so plain ollama works too).

Start / check / stop

# Start the server (does NOT survive a reboot or terminal close — restart it):
ollama serve &                     # or: nohup ollama serve > /tmp/ollama.log 2>&1 &

# Is it up? (prints a JSON version string if so)
curl -fsS http://127.0.0.1:11434/api/version

# Is the process alive?
pgrep -af "ollama serve"

# Stop it:
pkill -f "ollama serve"

After a reboot the server is not running — just ollama serve & again. The pulled models persist on disk (under ~/.ollama), so you don't re-download them.

Models

ollama list            # models on disk
ollama ps              # models currently loaded in VRAM (and how long until idle-unload)
ollama pull <model>    # download a model, e.g. ollama pull qwen2.5:14b
ollama rm <model>      # delete a model to reclaim disk

Currently pulled: qwen2.5vl:7b (OCR), mistral-small3.2:24b (grouping, default), plus qwen2.5:7b / qwen2.5:14b (earlier grouping experiments — ollama rm them if you want the disk back).

VRAM / GPU notes

• A model loads into VRAM on first request and auto-unloads after ~5 min idle (so ollama ps is often empty — that's normal; the next call reloads it in a few seconds).
• Only run one large model at a time on the 24 GB 3090. The OCR model (~6 GB) and the grouping model (~15 GB) can coexist, but bumping grouping to qwen2.5:32b/qwen2.5vl:32b (~20 GB) leaves little room — expect a reload when switching between OCR and grouping.
• Force-unload now (free VRAM without stopping the server): ollama stop <model>.
• Keep a model resident longer / shorter via the request keep_alive field, or globally with OLLAMA_KEEP_ALIVE (e.g. OLLAMA_KEEP_ALIVE=30m ollama serve).

Pointing the scripts elsewhere

Both scripts honor OLLAMA_HOST and OLLAMA_MODEL:

OLLAMA_MODEL=qwen2.5:14b python3 OCR/receipt_group.py        # try another model
OLLAMA_HOST=http://other-box:11434 python3 OCR/receipt_ocr.py img.jpg  # remote server

Logs / troubleshooting

• If a script hangs or errors connecting, the server is probably down — check with the curl above and restart.
• Startup logs (GPU discovery, VRAM, errors) go wherever you redirected serve (e.g. /tmp/ollama.log); tail that if a model fails to load.
• "model not found" → you haven't ollama pulled it (or typo'd the tag); see ollama list.

grouping the OCR results

next task is ... given an OCR output, like:

 1  ALDI
 2  Store #145
 3  1501 Rockville Pike
 4  Rockville, MD
 5  https://help.aldi.us
 6  Your cashier today was Wendy
 7  382175 Unsalted Peanuts 2.29 FA
 8  382175 Unsalted Peanuts 2.29 FA
 9  384773 4 lb. Sugar 2.79 FA
10  384773 4 lb. Sugar 2.79 FA
11  382437 Fire Roasted Tomatoes 2.30 FA
12  2 @ 1.15
13  371607 Canned Cat Food 2.28 NB
14  4 @ 0.57
15  371607 Canned Cat Food 0.57 NB
16  416787 1% Milk, Gallon 3.38 FA
17  343557 Protein Powder 18.49 FA
18  416645 Chocolate Milk 1.91 FA
19  382260 Plain NF Greek Yogurt 2.79 FA
20  382260 Plain NF Greek Yogurt 2.79 FA
21  297956 PureandSimple Bars 3.99 FA
22  356508 Broccoli Crowns 2.21 FA
23  1.17 lb x 1.89/lb
24  469529 Corn Tortillas 1.95 FA
25  609748 Assorted Cashews 6.49 FA
26  634568 Chicken Skewers 7.49 FA
27  382473 Shredded Mozzarella 3.29 FA
28  341878 Yellow Onions 1.85 FA
29  535290 ABF B/S Thighs 6.61 FA
30  272135 Salmon Portions 9.35 FA
31  272135 Salmon Portions 8.54 FA
32  382310 Feta Crumbles 1.29 FA
33  356508 Broccoli Crowns 2.53 FA
34  1.34 lb x 1.89/lb
35  262137 Stuffed Olives 2.89 FA
36  356490 Bagged Avocados 2.99 FA
37  356607 Red Delic Apples 2.75 FA
38  356527 Celery 1.89 FA
39  382653 Indian Sauces 3.69 FA
40  356628 Seedless Cucumber 0.89 FA
41  356684 WildTwist Apf LRW 5.44 FA
42  (G) 3.10 lb - (T) 0.06 lb
43  (N) 3.04 lb x 1.79/lb
44  282119 Pineapples 1.89 FA
45  356419 Mangoes 3.80 FA
46  4 @ 0.95
47  356691 Zucchini 2.01 FA
48  1.69 lb x 1.19/lb
49  356522 Cantaloupe 1.89 FA
50  356504 Blueberries 9.95 FA
51  5 @ 1.99
52  388137 Large Eggs 2.92 FA
53  2 @ 1.46
54  262747 Bananas LRW 1.45 FA
55  (G) 2.97 lb - (T) 0.01 lb
56  (N) 2.96 lb x 0.49/lb
57  262747 Bananas LRW 1.13 FA
58  (G) 2.32 lb - (T) 0.01 lb
59  (N) 2.31 lb x 0.49/lb
60  356427 Multi-Peppers 3pk. 2.69 FA
61  341876 Red Grapes LRW 5.48 FA
62  (G) 3.96 lb - (T) 0.02 lb
63  (N) 3.94 lb x 1.39/lb
64  479744 Campari Tomatoes 2.99 FA
65  479744 Campari Tomatoes 2.99 FA
66  569269 Protein Bread 3.99 FA
67  343989 Family AsstCookie 1.99 FA
68  356646 Strawberries 2.09 FA
69  356646 Strawberries 2.09 FA
70  356646 Strawberries 2.09 FA
71  VISA 172.42
72  ************0294 ONLINE
73  06/21/26 12:43 Ref/Seq # 0
74  Auth# 00401D
75  AID A000000031010
76  TVR 0000000000
77  IAD 06021203A00000
78  TSI 0000 ARC 00 EntryMode 07
79  ++APPROVED++
80  SUBTOTAL 172.24
81  B-Taxable @6.00% 0.18
82  A-Taxable @0.00% 0.00
83  AMOUNT DUE 172.42
84  TOTAL $172.42
85  60 ITEMS
86  Credit Card $ 172.42
87  *7367 L411/005/064 06/21/26 12:43PM
88  ************
89  Sign up for ALDI emails
90  for a sneak peek on the weekly ad!
91  www.aldi.us/signup

write a LLM call, still using my local ollama (you can switch the model). I want the output to be an object roughly like that:

store name: store branch: store address: list of rows that belong together as one item purchased total in dollars. (anything else is relevant here)?

you can hardcode the lines for this test and let's call the LLM to see what it does

the output needs to be a new script (receipt_group.py) that (for now) read this examle as is from an example file that you will create, and with the correct prompt organizes the information correctly.

Grouping — implemented

receipt_group.py reads OCR rows (default example/data/aldi_ocr.txt) and asks a local ollama text model to reconstruct the receipt: store name/branch/address, line items (merging multi-row items such as weighed produce and qty @ price lines), total, and metadata (cashier, datetime, payment, subtotal, tax, item count).

python3 OCR/receipt_group.py                       # groups only: each item + its rows
python3 OCR/receipt_group.py OCR/example/data/aldi_ocr.txt --json   # full object + reconciliation

• Default model: mistral-small3.2:24b (~15 GB). Override with OLLAMA_MODEL. Tried qwen2.5:7b (many merge errors) and qwen2.5:14b (~90%); Mistral Small got the grouping right. Swapping models is just the env var — no code change.
• Why an LLM groups, not a regex: detecting item vs detail rows is trivial, but deciding which item a detail row attaches to is not — wrapped descriptions, discount lines, and qty/weight lines can sit above or below the item, so a positional rule breaks. The prompt tells the model to attach each detail row to the item its math reconciles with (count × unit, weight × rate = the item's charged price), never by a fixed above/below position.
• Completeness clause in the prompt: every input row must land in exactly one item's rows, copied verbatim, and the model must walk the input top-to-bottom to confirm nothing was dropped. This fixed the model silently swallowing Large Eggs' 2 @ 1.46 line. Prompting lowers the odds of a drop but cannot guarantee it — hence the backstop below.
• Deterministic reconciliation pass (the part you can trust), shown in --json under validation: Python recomputes each item from its detail math, sums items vs the printed subtotal, checks total = subtotal + tax, and sums unit quantities vs the printed N ITEMS. When the model slips (e.g. dropping a qty @ line), the unit-count check flags it (59 vs 60) even though dollars still balance.