Can AI do the job? Test it before you rely on it.
In short
AI is being sold for customer work. I test whether it can actually do the job, on real tasks, and publish what I find.
A confident wrong answer costs more than no answer.
A missed deadline or a non-compliant reply can go to the ombudsman: £680 a case once a firm’s free allowance is used.
Published test
Scorensic: a test pack for real jobs.
Your tasks in, a one-page report card out: right, confidently wrong, and whether your data left the building. Letters are checked against what they must and must never say. Complaints come first; customer service, invoices and bookings are next.
First test: UK complaints handling
Early results62 tasks from invented complaints, run on two small AI models.
- 0 of 8 deadlines worked out correctly by either model.
- 0 of 2 regulated replies that met the ombudsman rules. One sounded compliant but wasn’t.
- 8 of 8 letters refused by one model. It called a gym cancellation reply a possible scam.
Lesson: calculate dates with ordinary software, check every letter, keep a person in charge.
Larger models are not tested yet. A first result, not a verdict.
Use it or get involved
Bring 10 to 20 real examples.
I’ll show you where AI helps, where it doesn’t, and what a safe set-up looks like.
For engineers: how the test works
Scorensic checks its own scoring against deliberately wrong answers before testing other systems. The source code and first recorded results are public, so anyone can inspect the method and limits.