MAZ WORKSManazir Hussain
Maz Works / Open-source research

Scorensic. Can AI do this job?

Open-source test packs for real business work, with marking that can’t be fooled.

Abstract

Put real tasks and right answers in a sheet. Scorensic runs them on any AI model and gives a one-page report card: right, confidently wrong, didn’t know, consistent, and whether data left the building. Before scoring, it tries to fool its own marking. The first pack covers UK complaints handling. Underneath is an engine that checks the graders behind AI benchmarks.

Read the first resultsView the repository ↗Status: MVP working (research and development)
The problem

A benchmark score is only as good as the code that marks the answers.

Leaderboards compare models, but the grader itself is rarely tested. When it accepts empty answers, rejects correct ones or lets the agent read the answer key, the score measures the grader’s bug, not the model.

What we’re investigating

Feed the grader answers whose verdict is already known.

Scorensic gives a grader the correct answer, harmless reformattings, empty output, refusals, known-wrong and slightly altered answers. It repeats each one in separate runs, and searches what the agent can see for the answer key. Every disagreement is reported as a specific input you can rerun, not a single score.

Where it fits among existing tools

Where it fits

Built for the gaps other tools leave.

Good tools already exist. Scorensic credits them and aims at what they don’t cover.

Existing workDoes wellLeaves openScorensic’s aim
BenchJackScans benchmarks for exploits using a coding agent.Needs Claude Code or Codex to run.Runs offline, with no model and no API key.
evalmutRigorous mutation testing of grader verdicts.No checks across repeated runs or for leaked answers.Re-scores each input in separate processes and scans what the agent can read.
Harbor oracle and no-op agentsChecks that the reference passes and an empty agent fails.Its own manual says a passing oracle does not prove wrong answers fail.A reusable set of junk, hedged and known-wrong answers for every task.
Hand-written fixesMaintainers patch each bug after it is found.One-off. The same bug class returns in the next benchmark.One command that runs the same checks against any grader.

Current experiments

What has been tested, and what hasn’t.

Synthetic grader faults

Done

Six toy graders: a clean control and five with one deliberate fault each (accepts wrong answers, rejects valid ones, inconsistent scoring, leaked answers, a do-nothing agent passes). All five faults detected; the clean control shows no findings. Synthetic only.

Reproduce a real scorer bug

Done

Against the inspect_evals scorers from PR #2334, it caught 2 of the 3 known problem types and raised no false alarms on the fixed versions.

UK complaints pack on real models

First results

54 short tasks and 8 reply letters on two small local models. Neither worked out a single deadline correctly, and neither wrote a regulated reply that met the ombudsman rules. Invented data, small models only.

Benchmark adapters

In progress

Inspect AI is connected. Harbor verifiers are next.

Real grader failures found by others

Research evidence

Real grader failures, from primary sources.

Found by others and linked to the original report. These are the motivation, not Scorensic results.

Development progress

Stage 3 of 5.

A stage only moves forward when the evidence for it is published in the repository.

Open source under Apache-2.0. Runs offline on a 16 GB laptop. No API key needed.

  1. DoneResearch initiated
  2. DoneFirst reproducible tests
  3. CurrentMVP working
  4. Not yetExternal benchmark validation
  5. Not yetPublic beta
Known limitations

What this does not show yet.

  • Grader checks are validated on one real scorer bug so far, not across benchmarks.
  • Business results so far cover two small models on invented complaints. Larger models are untested.
  • Rules check what a letter says, not its tone. A person should still read it.
  • Passing every check does not prove a grader is correct. It only means these probes passed.
  • The leaked-answer search is a plain text match. It cannot yet see git history, encoded or paraphrased leaks.
  • It runs grader code in a separate process with a timeout. That is not a security sandbox.