# Evaluation methods

Measure organizational outcomes, continuity, intervention, and resource use with explicit scope.

Concept ID: axiomao:evaluation-methods
Language: en
Status: reference
Reference edition: 0.1
Canonical: https://axiomao.org/en/evaluation-methods/
Published: 2026-10-14

## Evaluate outcomes, not declarations

An evaluation should begin with a question and a specified outcome. For an individual task, define what counts as successful completion and what evidence establishes it. For an organization, include continuing obligations: whether accepted work remains correct, deadlines are handled, and unresolved issues receive an appropriate next action.

TheAgentCompany offers a controlled environment for professional tasks, while Vending-Bench examines long-term coherence in a simulated business. These settings test different capabilities. Neither a favorable benchmark score nor a working service endpoint demonstrates the complete operation of an autonomous organization in another environment.


## Test interruption and sustained work

Longitudinal evaluation should inspect sequences rather than isolated answers. Useful cases include restart, delayed input, changed instructions, exhausted resources, unavailable agents, and an external operation with an uncertain outcome. Observe whether the system preserves obligations, reconciles state, and respects authority while recovering.

Report interventions and failed attempts alongside completed work. Record elapsed time, compute or token use where available, external costs, and human review effort. A system that completes a task after substantial hidden assistance should not be described as having completed it autonomously under the same conditions as an unassisted run.


## Interpret time horizons carefully

METR’s 50% task-completion horizon expresses task difficulty through the time an appropriately skilled human typically needs for tasks a system can solve with that success probability. It does not mean the AI operates continuously for that duration, nor that it succeeds on half of arbitrary business activities.

Use the metric with its task distribution, system setup, and version. Organizational evaluation additionally needs evidence about persistent authority, commitments, and recovery. Quantitative comparisons should preserve their denominators and uncertainty; a few successful demonstrations cannot establish a general reliability rate or justify an unrestricted autonomy claim.


## Sources

- [Xu et al. (2024). TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://arxiv.org/abs/2412.14161)
- [Backlund & Petersson (2025). Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://arxiv.org/abs/2502.15840)
- [Andon Labs (2025). Vending-Bench 2](https://andonlabs.com/evals/vending-bench-2)
- [Kwa et al. (2025; revised 2026). Measuring AI Ability to Complete Long Software Tasks](https://arxiv.org/abs/2503.14499)

## Related

- https://axiomao.org/en/organization-profiles/
- https://axiomao.org/en/research-agenda/
- https://axiomao.org/en/autonomy-and-accountability/