Receipt OCR and Expense Categorisation: Why Vendor Memory Beats a Smarter Model

Reading a receipt is the easy part. Modern OCR pulls the merchant, the date, the amount and the currency off a photo or a PDF reliably enough that it is rarely where a startup's books go wrong. The part that goes wrong is the next step: deciding which account the expense belongs to, and deciding it the same way every month.

That second step is categorisation, and the instinct is to fix it with a smarter model. In practice the fix is memory. A system that remembers what a vendor was last time beats a cleverer system that guesses fresh every time.

What OCR actually solves

OCR turns an image into fields. Merchant, date, total, tax, currency. A good extractor also says how sure it is about each field, which matters more than the headline accuracy number, because it tells you which receipts to look at.

The failure cases are ordinary and specific. A crumpled thermal receipt. A date written as 22.06.2026, which is common on Malaysian and Thai receipts and trips up parsers built for one format. An emailed receipt that arrives as an HTML page rather than an image. None of these are exotic, and each one is handled by better plumbing, not by a better model.

If you want the honest range on how often automated bookkeeping gets things right, we went through it in how accurate AI bookkeeping actually is.

Why categorisation is where books drift

Categorisation is a judgment, and judgments drift when nobody writes them down. A model looking at a receipt from a hotel can reasonably say Travel, Accommodation, or Meals if the bill was mostly the restaurant. Each answer is defensible. The problem is getting a different defensible answer in March and in April.

That drift is invisible month to month and obvious at year end. Travel looks low, office costs look high, and the management accounts tell a story about the business that is really a story about inconsistent guessing. It is one of the quieter reasons a management report ends up weeks late: someone has to go back and recategorise before anyone trusts it.

The same vendor, under several names

The first thing vendor memory has to handle is that vendors do not arrive under one name. Card networks and terminals produce descriptors: the same cafe can appear with a store number prefix one month and without it the next. A ride-hailing app can arrive under three slightly different strings depending on the payment method.

A rule that matches the exact string that arrived last time will miss the next one. The fix is to write the rule against the part of the name that does not change, the brand rather than the terminal code. That is a small, boring discipline, and it does more for clean books than any model upgrade.

Vendor memory beats a smarter model

Once a vendor has been categorised and someone has confirmed it, that decision should win over anything the model suggests next time. The model is useful for the first receipt from a new vendor. After that, the remembered decision is better, because it carries the one thing the model cannot know: how this specific business chose to treat this specific supplier.

That ordering is how YourBooks works. Vendor rules are checked first and override the AI's suggestion; the model only proposes a category where no rule exists. The practical effect is that the books get more consistent the longer you use them, rather than staying exactly as consistent as the model happens to be that day.

Low confidence should wait, not post

The other half of the design is what happens when the system is unsure. A receipt with an unreadable amount or an ambiguous date should go into a review queue, not into the ledger. Posting a guess and fixing it later sounds efficient. It means last month's P&L was wrong for however long it took someone to notice.

A related detail: when a category is corrected, the journal entry should move with it. Fixing the category in one place and leaving the old entry in the ledger is how duplicate costs appear at year end.

What to do with your own receipts

If you are still categorising by hand or in a spreadsheet, the same principles apply without any software:

1. Keep a short vendor list with the account each one maps to, and use it every month.
2. Write the vendor name the way it appears on your card statement, using the part that stays the same.
3. Put anything uncertain in a separate tab until it is confirmed, rather than guessing.
4. Convert foreign receipts once, at the rate on the transaction date, and keep the original amount beside it.

Whether a person or a system does this matters less than whether it happens the same way every month. We covered that trade-off in AI bookkeeping versus hiring a bookkeeper.

Takeaway

OCR reads the receipt. Memory keeps the books consistent. If your categories change month to month, the fix is not a smarter guesser, it is writing down each vendor decision once and letting it win every time after that, with anything uncertain held back until someone confirms it.