Categorisation Isn't Insight
87% is a good number. It's also almost irrelevant to whether MoneyFlow was actually useful to anyone, and figuring that out the hard way is the most valuable thing that project taught me about building with AI.
Our transaction categorisation accuracy (correctly labelling a bank transaction as “groceries” or “transport” or “subscriptions” without a human correcting it) landed at 87% by the end of testing. That's a solid engineering result for a small team running a locally-hosted model on real, messy, three-different-banks'-worth of statement data. But categorisation accuracy measures whether the model sorted the data correctly. It says nothing about whether a user looked at their categorised spending and understood anything they didn't already know.
The metric that's actually load-bearing
The number that mattered more was our AI insight relevance score: 4.1 out of 5, based on usability testers rating whether the AI-generated commentary on their spending actually told them something useful. That's a much harder thing to get right, because it's not a classification problem, it's a judgment problem. Sorting “Tesco: £34.50” into “groceries” is pattern matching. Telling a student that their subscription spend has crept up 40% over three months in a way that's specific to their actual life, not a generic “you're spending a lot on subscriptions” template, is reasoning.
Most “AI-powered” finance products stop at the first problem. It's the one that's easy to build a demo around: show a screen full of neatly labelled transactions, ship it, call the product AI-powered. It's also the one that's easy to benchmark, because you can measure categorisation accuracy against a labelled dataset in an afternoon. Insight quality doesn't have that luxury. There's no ground-truth dataset for “was this observation actually useful to a 20-year-old trying to understand their own spending.” You have to go find that out from real people, which is slower and less flattering than reporting an accuracy number.
Sorting data is a solved problem dressed up as a product. Insight is the unsolved problem hiding behind it.
We only found the gap between those two numbers because we tested with real usability sessions: 14 of 15 testers completing the full workflow unaided, watching where the AI commentary landed and where it fell flat. The commentary that scored well wasn't the most technically sophisticated output the model produced. It was the output that referenced something specific and true about that person's actual data: a category, a trend, a comparison to their own history, not a comparison to some abstract “average user.” The commentary that scored badly was generic: correct, but the kind of correct that could have applied to anyone.
That's the trap. A model can be accurate and still be useless, because accuracy and relevance are answering different questions. Accuracy asks “did you sort this correctly.” Relevance asks “did this change what the person understood about their own situation.” You can hit the first one with a decent classifier and a labelled dataset. You can only hit the second one by actually understanding what the person in front of you doesn't already know, which means talking to them, testing with them, and being willing to find out that your technically-correct output landed flat.
The fix wasn't a smarter model. It was rewriting what we asked the model to do. Early versions of the insight engine summarised categories: “you spent £340 on eating out this month.” True, and worth almost nothing, because the user already knew that from looking at their own bank app. What actually moved the relevance score was forcing every insight to reference a comparison the user couldn't already see at a glance: this month against their own three-month average, this category against the rest of their spending, a trend rather than a snapshot. Same categorised data underneath, completely different perceived usefulness, because the second version required the model to reason about change over time instead of just describing a total.
It also changed what we tested for. Once we knew relevance and accuracy were separate problems, we stopped treating usability sessions as a bug hunt for miscategorised transactions and started treating them as a bug hunt for boring insights: commentary that was technically true but told the person nothing they'd act on. That's a harder thing to catch than a wrong label, because a wrong label is obviously wrong and a boring insight just quietly fails to land, with no error message and no test that flags it. You only find it by watching someone read the output and shrug.
If I'm being honest about what I'd tell a team building an “AI-powered” anything: report your categorisation number if you want, but don't let it stand in for the harder number. The 87% told us the plumbing worked. The 4.1 told us whether we'd actually built something worth using. Most teams ship the first number and never go collect the second, and that's exactly why so many AI features feel technically impressive and practically forgettable.