Why Financial Services Is the Hardest Place to Actually Ship AI
Everyone treats regulation like it's the tax you pay for being in financial services: the thing that slows the real work down. I don't buy that framing anymore, and two summers of watching it up close changed my mind: at Severn Trent, regulation wasn't friction bolted onto the work, it was the work. Every dashboard, every model, every insight had to survive scrutiny it wasn't built by me and wouldn't be judged by me. That's a completely different design problem than shipping software that only has to satisfy your own product team, and it's the reason financial services (and regulated utilities, which run on the same logic) are genuinely the hardest place to ship AI well.
The ODI framework Severn Trent operates under ties real financial incentives (in our case, a £75m target) to outcomes that get independently assessed: customer experience, environmental performance, service reliability. Nothing I built during that internship existed in a vacuum where “it works on my machine” was good enough. Every dashboard I built had to be defensible to someone outside the building, using a methodology I didn't write, against evidence I had to be able to produce on demand. That's not a compliance checkbox at the end of the process. It's a constraint that shapes the process from the first line of the spec.
Most AI teams don't build like this by default. They ship a model that performs well on a held-out test set, watch the metric go up, and call it done. Whether the system's reasoning would survive an external audit, whether its data lineage is traceable, whether a non-technical reviewer could understand why it made a specific call: none of that gets forced into the process unless something outside the team is going to check. In an unregulated product, nothing is going to check, so most teams don't build the muscle for it until they're forced to, usually after something has already gone wrong.
The MoneyFlow decision nobody made us make
We ran into a version of this problem on MoneyFlow, even though a student personal finance tracker has nowhere near the regulatory weight of a water utility. The obvious way to build an AI insight engine in 2025 is to send transaction data to a hosted model (GPT-4, Claude, whatever API is cheapest that week) and let it do the reasoning server-side. We didn't do that. We ran Mistral 7B locally, on-device, specifically so that no user's bank statement data ever left their machine.
Nobody made us do this. There's no regulator checking a final-year group project. We did it because Leo, who owned the engineering, and I, as the person accountable for the product decisions, both understood the same thing Severn Trent had already taught me: if you can't explain your data handling to someone skeptical of you by default, you don't actually have a defensible product, you have a liability with a nice UI. Building local-first cost us real engineering time (quantising and running a 7B model on modest hardware is not the easy path), but it meant the trust question was answered by the architecture, not by a privacy policy nobody reads.
Regulation doesn't just constrain what you can ship. Done seriously, it teaches you to ask the one question most AI teams skip: how would I prove this is right to someone who doesn't trust me by default?
The standard complaint about regulated industries is that they move slowly because compliance adds friction to every decision. I think that's true and also not the interesting point. Friction that forces you to answer “how would I prove this” before you ship is friction that's doing useful work. The teams that skip it aren't actually faster in any way that matters: they're just deferring the cost of that question to whenever a regulator, a journalist, or a user finally asks it for them, at which point the answer is a lot more expensive to produce.
Concretely, that means the boring plumbing most consumer AI teams treat as an afterthought (versioned methodology, a traceable path from raw input to output, a record of who signed off on what and when) has to exist before the feature ships, not after an incident makes leadership ask for it. At Severn Trent, an assessment wasn't finished when the analysis was correct. It was finished when someone else could pick it up, follow the reasoning without me in the room, and reproduce the same conclusion from the same evidence. That standard is inconvenient. It's also exactly the standard that keeps a system trustworthy once the person who built it has moved on, which every system eventually has to survive.
My honest prediction is that the AI systems built inside regulated environments (the ones forced to build audit trails, explainability, and provenance in from day one) end up structurally more robust than the ones built by teams optimising for shipping speed. Not because regulated teams are smarter, but because they were never allowed to skip the step where you prove your system deserves to be trusted. The rest of the industry is going to have to retrofit that discipline eventually. Financial services and regulated utilities are just the ones who had no choice but to build it in from the start.