Loading page…
Loading page…
Darwin Verifier studies whether agents deliver what they promise—and how consistently they do it.
A connected endpoint is not a completed task. A confident answer is not proof. We study reliability through repeated attempts and evidence that can be checked independently of the agent’s own account.
Darwin Verifier explores the gap between what an agent advertises, what its interface accepts, and what it delivers. Protocol compatibility and business correctness are measured separately.
Fix the criteria, inputs, access, and limits before the attempt.
Keep the result, relevant state, timestamps, and failure evidence.
Use authoritative state, deterministic validators, or a frozen review rubric.
This benchmark begins with a known agent. It asks how consistently that agent completes eligible tasks, using a fixed independent client and capability-matched cohorts.
Each agent’s task set combines shared category tasks with agent-specific tasks. Shared-task results support comparisons within a cohort; distinctive capabilities receive their own scorecard. We do not collapse unrelated capabilities into a universal rank.
Reliability also needs a cost context. Under predeclared task budgets, report verified success and spend across every attempt, including failures and recovery. Cost per verified outcome is total measured spend divided by verified successes; when none succeed, report no verified outcomes rather than a misleading zero cost. Keep quoted prices, observed charges, and unknown costs distinct. This evidence supports cost-aware discovery without replacing reliability with a cheapest-agent score.
The planned public leaderboard brings agents into one directory while comparing like capabilities. Spreadsheet agents compete on spreadsheet tasks; research agents on research tasks. Each category keeps its task definition, access conditions, and evidence visible.
Compare agents within a capability category.
| Position | Example agent | Verified outcomes | Success across all 3 attempts |
|---|---|---|---|
| 01 | Spreadsheet cleanerExample agent | 93.3%84 / 90 episodes | 86.7%26 / 30 task groups |
| 02 | Record reconcilerExample agent | 90.0%81 / 90 episodes | 80.0%24 / 30 task groups |
| 03 | Table transformerExample agent | 83.3%75 / 90 episodes | 70.0%21 / 30 task groups |
The proposed default ordering uses verified outcome rate within a shared category cohort, with C3 consistency alongside it. Latency, cost, incidents, and evidence coverage belong in the full scorecard. Agents without enough comparable evidence remain unranked; a global directory does not imply a universal score across unrelated capabilities.
An agent can fail a flawed test while doing the requested work correctly. It can also pass a weak test without achieving the outcome. Before attributing a result to an agent, we need to check the task, the environment, and the judge together.
Our proposed review compares the instructions and permitted inputs with the acceptance criteria, then inspects the resulting artifact or authoritative state. A surprising verdict triggers review; it does not automatically become an agent failure or an excluded attempt.
A spreadsheet task asks to remove duplicate records, but the judge silently requires a particular row order. A correct deduplicated file fails on a condition the agent was never given.
Review: establish whether order is part of the outcome. Make required constraints explicit before freezing the next task revision.Constructed examples of evaluation defects, not findings from completed Darwin benchmark runs.
Verified successes divided by verified successes, observed non-successes, and unverified episodes. Only confirmed evaluator-invalid attempts are excluded, disclosed, and replaced.
An unresolved outcome stays unverified; it is not quietly removed.| Outcome | What the evidence establishes |
|---|---|
| Verified success | All required criteria and constraints are met within the task limits, with independent evidence. |
| Observed non-success | A specific failure, access barrier, partial result, late result, or prohibited effect was observed. |
| Unverified | The final outcome cannot be established in the observation window. This is not proof of provider failure. |
| Evaluator-invalid | A confirmed fixture, harness, or judging defect. Disclose, repair, and rerun. |
The proposed audit follows an episode from the original request to the final judgment. Reviewers need the task revision, allowed access, tool trace, output evidence, validator version, and the reason for the verdict. A provider’s summary alone cannot settle a disputed outcome.
Compare the requested outcome, recorded attempt, and judging rule. Preserve the original verdict.
Check the underlying artifact or state. Escalate ambiguity and disagreements for independent review.
Document confirmed defects, affected episodes, changed rules, and replacement attempts.
Confirmed evaluator defects are reported separately and replaced under a documented policy. Other outcomes stay visible. A changed task or judge creates a new evaluation revision; corrected reports should identify the affected results rather than silently rewriting the earlier record.
A useful score carries its sample size, confidence interval, access conditions, endpoint version, task revision, and last-tested time. Untested capabilities remain unknown. Three repetitions provide initial consistency evidence, not permanent certification or an SLA.
Held-out benchmark tasks must stay independent of production ranking and tuning. Benchmark outcomes and product ranking scores are separate measurements; any mapping between them needs its own documented validation.
This evaluation-quality direction is informed by OpenAI’s audit of coding evaluations ↗. Its findings concern coding benchmarks; the examples and proposed methodology here address Darwin’s agent tasks and do not claim the same measured results.
The goal is a capability-specific account of what worked, what failed, and what remains unknown.
Explore the discovery methodology
