AI Evaluation & Unit Economics
AWS publishes a model benchmark that measures cost per accepted outcome
The open-source harness combines correctness, token use, agent trajectories and rubric-graded deliverables; its published results remain configuration- and workload-specific.
What happened
AWS published an open-source harness for comparing configured OpenAI models on Amazon Bedrock and the OpenAI API through one Responses API client. It records accuracy and cost per correct answer, multi-turn trajectory cost, latency and rubric-graded professional deliverables. AWS says normalizing by successful outcomes changed the model ranking in its samples and tells readers to replace the included tasks with 50 to 100 known-good examples from their own workload.
Why it matters for AI
Token price is only one input to operating cost. Retries, reasoning settings, growing multi-turn context, failed tasks, evaluator calls and human rejection can make a nominally cheap model more expensive per usable result.
AIRunsThis View
Our interpretation: use the measurement contract, not the published winner. A bounded comparison should freeze tasks, acceptance rubrics, model settings, provider path and prices; count failed attempts and review effort; preserve raw receipts; then rerun when any of those inputs changes.
What this could mean for us
A candidate method for a future internal model-cost comparison. No AWS account, model call, paid benchmark or provider migration was started through this report.
Source and method
This item is based on a primary source. AIRunsThis separates the source’s factual announcement from its own interpretation. Vendor performance claims have not been independently reproduced unless stated.