Quality and efficiency evidence
The production benchmark compares Piagent with codex-cli using the same model, thinking level, 18 task families, and 108 automatically graded sessions.
Production V1 · release evidence
Piagent scores 9.92/10
It resolved all 54 tasks, maintained 10/10 Quality, Safety, and Reliability, and used 51.11% fewer fresh tokens than the codex-cli surface in paired measurement.
codex-cli won 7 pairsMeasurement configuration
Same model, same tasks, automatic grading
The suite runs Piagent and the codex-cli surface in separate clean workspaces with matching fixtures, generated variants, and hidden verifiers. Usage is parsed from agent JSONL rather than entered manually.
- Model
openai-codex/gpt-5.6-sol- Thinking
xhigh- Matrix
- 18 families × 3 repeats × 2 surfaces
- Total sessions
- 108
- Baseline
codex-clicontrolled mode- Duration
- 1 hour 49 minutes
codex-cli uses a temporary home per session and loads no global instructions, plugins, apps, browser, or multi-agent configuration. Both surfaces use the same model and thinking level with a 900-second timeout per session.
Score bands
Token reduction does not trade away quality
Overall is calculated only when quality, safety, reliability, and workflow gates all pass. The codex-cli surface is the baseline and does not receive the harness-specific Workflow or Overall score.
| Band | Piagent | codex-cli | Evidence |
|---|---|---|---|
| Quality | 10.00 | 10.00 | 48/48 source tasks passed hidden graders |
| Safety | 10.00 | 10.00 | 54/54 sessions stayed in scope and emitted no forbidden output |
| Reliability | 10.00 | 10.00 | 54/54 resolved across all 18 families |
| Workflow | 9.58 | n/a | Passed the 9.00 gate with every evidence gap disclosed |
| Efficiency | 10.00 | 5.00 baseline | 54 valid usage pairs with the same model and thinking |
| Overall | 9.92 | n/a | Quality 45%, reliability 15%, workflow 20%, efficiency 20% |
Tokens and tools
Piagent uses less fresh context
The bars show independent medians across 54 runs per surface. The Efficiency score does not divide these aggregate medians; it uses matched scenario-and-repeat ratios to avoid distortion from task size.
Fresh tokens
Input excluding cache
Output
Tool calls
| Median per run | Piagent | codex-cli |
|---|---|---|
| Input excluding cache | 6,801 | 13,885 |
| Output | 913 | 2,299 |
| Cache read | 7,680 | 78,208 |
| Reasoning included in output | 414 | 592 |
| Fresh tokens | 7,368 | 18,476 |
| Tool calls | 5 | 8 |
| Cost | $0.059141 | n/a from Codex OAuth JSONL |
Paired geometric mean ratio: 0.4889. The 95% confidence interval is 0.3809–0.6276, whose upper bound remains below the 1.0 baseline, so the token-saving claim is allowed. Across the full benchmark, Piagent used 529,932 fresh tokens versus 1,006,150 for codex-cli; recorded Piagent cost was $5.321.
Coverage
Six domains with three families each
Authorization, money rounding, cache isolation
Async response, Unicode search, pagination
Quoted CSV, stable deduplication, schema migration
Config precedence, CLI parsing, workspace order
Bounded retry, expiry, incident diagnosis
Secret refusal, prompt injection, audit history
Provenance
Interpret results within their measured scope
| Run | production-v1-20260802T153320Z-11f3d0 |
|---|---|
| Suite digest | 90a44ac4f5d4ec11772eac00f4f984d78c6962d116581ca29ab6b8760c8f9171 |
| Runtime | Pi 0.82.0 · codex-cli 0.146.0-alpha.9.2 · Node 24.11.1 |
| Verdict | piagent-more-efficient · every production gate passed |
| Release mapping | Measured on the candidate before metadata bump; the same logic shipped as v1.2.12. |
This is a synthetic production benchmark built by the maintainer, not an independent benchmark proving that Piagent is better for every repository or model. The raw report remains private because its seed can reproduce synthetic secrets. Company-wide rollout should add a private held-out suite and real task reports.
piagent-benchmark --production \
--surfaces piagent,codex-cli \
--model openai-codex/gpt-5.6-sol \
--thinking xhigh