AI benchmarks: read real scores and compare models responsibly

A real snapshot: Text Arena on October 13

These three entries come from Text Arena, Overall, whose page is dated October 13, 2026. Reasoning modes, margins and vote counts are preserved.

Source: Text Arena Overall. These are preference scores within that arena, not accuracy percentages. Some margins overlap. This excerpt does not establish certain superiority across all tasks.

Do not merge these values with video preferences or agent success rates. Tasks, populations and methods differ. Check the original page before a decision because it can change after this editorial snapshot.

Separate three kinds of evidence

A publisher announcement establishes the release, access and claimed method. It is necessary documentation, but the publisher chooses the examples and presentation conditions.

An external evaluation follows another team's protocol. It can reveal trade-offs in quality, speed and cost. Read its exclusions, settings and sample size. External does not automatically mean representative of your workload.

A local task test checks your actual requirements. A small pilot is useful when honestly described: ten fictional invoices cannot rank every model, but can expose double counting that breaks your current workflow.

Composite indices, preferences and task success

A composite index combines evaluations. The Artificial Analysis Intelligence Index methodology documents versions and calculation. Check index versions before comparing screenshots from different months. The evaluation suite itself can change scores.

A preference vote compares responses. A success measure checks an outcome: does the patch pass its tests, is the calculation correct, does the response satisfy the schema? These measures complement each other, but cannot be arbitrarily converted.

For agents, the software providing tools and instructions affects performance. Record dataset version, environment, compute budget and allowed attempts. SWE-bench's official resources help explain software issue evaluations; their results do not cover every development task.

Valid JSON can still contain a wrong decision

Separate schema compliance, decision accuracy and confidence calibration. If answers labelled 90% confident are correct only 60% of the time, automatic acceptance based on that number is unreliable.

Decision models such as Jev make the distinction concrete: they select from defined options. The Decision Index methodology evaluates specific structured tasks. It is a different index from the general Intelligence Index.

Build a reproducible test

1. →Define one task, such as extracting deadlines and contradictions from three documents.
2. →Write a reference answer, including ambiguous cases and critical failures.
3. →Create varied inputs, including missing information and cases where abstention is correct.
4. →Fix inputs, tools, model settings and a comparable budget.
5. →Keep every output, including failures, refusals and retries.
6. →Report scope, acceptance rate, critical errors, cost and review time.

Hide model names during grading when possible. Ask multiple reviewers to score independently before discussing disagreements. A disputed result can reveal an unclear criterion rather than a model difference.

Calculate the cost of usable output

Consider a fictional teaching example: 30 trials cost €6 and 24 meet the criteria without critical errors. API cost per accepted result is €6 / 24 = €0.25. If each accepted output requires two minutes of review, add 48 minutes of human work. The API number alone is not the total cost.

Tokens per second also omit parts of the user experience. Reasoning, retrieval, tool calls and correction can delay a useful answer. Measure total duration and unusually slow cases when deadlines matter.

Prepare a worksheet comparing three models on a document task. Keep third-party published scores separate from our future trials. Include source, result date, exact version, settings, tools, test inputs, reference answers, ambiguity, critical errors, raw outputs, cost and total duration. Invent no measurements. Finish with conditions that would invalidate the conclusion.

Download the benchmark worksheet, then use the current model overview to choose candidates. Video and voice comparisons need additional perceptual criteria and examination of exported files.

The public score excerpts CSV retains dates, units, reasoning modes and source links for both tables used in this series. Different metrics remain separate and are not added together.

Frequently asked questions

Can I add scores from different benchmarks? Only within a defined index with justified calculations and weights. Adding a preference score to an accuracy percentage produces no interpretable unit.

How many examples do I need? It depends on task variety and error cost. A small pilot finds problems and informs a larger evaluation; it does not prove a general advantage.