In short
Most AI platforms sold for fraud analyst workflows score alerts. Scoring tells an analyst which case to open. It does not do any of the work inside the case.
The measured difference between approaches is large. Across roughly 10,000 runs and 150 outcome scenarios, work broken into defined steps reached 93% coverage and 82% success, against 79% and 71% for an open-ended approach on the best available model.
Three things sold under one description
Ask for an AI platform that handles fraud analyst workflows and you will be shown three different products.
| Type | What it does | What the analyst still does |
|---|---|---|
| Alert scoring and prioritisation | Ranks which cases to open first | Everything inside the case |
| Copilots and assistants | Drafts text and summarises what it is shown | Finds and supplies the evidence |
| Execution layers | Assembles evidence and prepares the response | Reviews, decides and approves |
All three are legitimate. Only the third one reduces the number of hours a case costs, because the other two leave retrieval where it was.
Why scoring does not shrink a backlog
Prioritisation reorders a queue. It does not shorten it. An NCRP case still needs the same evidence whether it is worked first or last.
If each case costs forty minutes of evidence gathering, working them in a better order still costs the same total hours. The best cases get worked first, which is worth something, and the backlog at the end of the month is the same size.
In one Indian banking operation, complaint response ran at about three days and almost none of that was decision-making. Better ordering would not have touched those three days.
Why a copilot only helps the last step
Copilots are good at writing, and writing is a real part of this job. Drafting a fixed-format reply to a law enforcement request takes time and attention.
But the analyst still opens five systems first: the core banking portal, the API, the data lake, the internal service desk and the email inbox. Transaction history sits in one, KYC records in another. A copilot sitting inside one of them cannot see the other four.
So the drafting gets faster - and the case does not. This is the single most common disappointment in AI pilots for case operations, and it is predictable from the architecture.
Most AI pilots fail when they hit real operational constraints: messy inputs, exceptions, permissions, audit requirements, and legacy systems.
The measured gap between designs
Patched published a benchmark in 2026 comparing two designs across roughly 10,000 runs and 150 distinct outcome scenarios.
| Approach | Coverage | Success rate |
|---|---|---|
| Open-ended, best model | 79% | 71% |
| Defined steps, best model | 93% | 82% |
| Defined steps, cheaper model | 91% | 80% |
The third row carries the argument. Defined steps on a cheaper model beat an open-ended approach on the best model available, on both measures.
For a buyer this means the model licence is not the lever. The design is.
The coverage question nobody asks in a demo
Every demo runs on the scenarios that demo well. Real queues are not shaped that way.
One Indian banking queue of I4C and LEA work held 16,736 cases across 189 distinct scenarios. The largest single scenario was 9.9% of volume, and the top ten together covered 48.2%.
So a platform demonstrating five scenarios is showing you roughly a third of your volume. Getting to 87.5% coverage takes 50 scenarios, which is ten times the demo.
Ask for the resolution rate by scenario rather than blended. A blended figure carried by two or three easy high-volume scenarios looks excellent and changes nothing.
What degrades, and how fast
A platform that works on the day it is installed is not the same as one that works in month seven.
Patched replayed roughly 1,000 production runs against logic frozen 30 days earlier: 210 of 1,151 comparable executions diverged, a rate of 18.25%. At 60 days it was 288 of 1,294, or 22.26%. On document validation the divergence reached 39.58%.
One production deployment needed 637 improvements against 490k workflow runs to hold its results. Ask any vendor how many changes they deployed on their last account.
Five questions that separate the three types
- After the score, what happens? If the answer is that an analyst takes it from there, you bought prioritisation.
- How many of my systems does it read? One is a copilot. Five is an execution layer.
- Resolution rate by scenario? A blended number hides the tail, and the tail is half the queue.
- What does it do when confidence is low? A draft and an escalation is the correct answer. An answer for every case is not.
- Who keeps it current? If the answer is your team, the maintenance is now a headcount line.
Why pilots of the first two types look successful
There is a measurement trap here that catches careful teams.
Scoring and copilot pilots are usually measured on analyst satisfaction and on time saved per task. Both improve, genuinely, and neither predicts what happens to the queue.
An analyst who drafts a reply in four minutes instead of nine is happier and faster at drafting. If the case still costs forty minutes of retrieval, the queue is 11% cheaper and the business case promised more.
The measure that survives is analyst hours per case, end to end, on a representative sample including tail scenarios. Any other framing will flatter the first two categories.
A fair comparison table for your shortlist
| What to compare | Weak answer | Answer worth buying |
|---|---|---|
| Systems read per case | One, through a plugin | All the sources the case depends on |
| Coverage measure | A blended resolution rate | Rate by scenario, tail included |
| Low-confidence behaviour | Produces an answer anyway | Draft plus categorised escalation |
| Change ownership | Self-service configuration | Vendor-maintained, with a change count you can audit |
| Record produced | A log of what succeeded | Inputs, checks, decisions, timestamps, outputs, what was skipped |
Sources
- Patched, published benchmark, 2026: 10,000 runs, 150 outcome scenarios, 79%/71% against 93%/82% and 91%/80%.
- Patched, published drift benchmark, 2026: 18.25% at 30 days, 22.26% at 60 days, 39.58% on document validation.
- Scenario breakdown from one Indian banking operation, 2026: 16,736 cases, 189 scenarios, 48.2% in the top ten.
