Null resultAI usage (tokens)
Handing a step to a cheaper model or to code
Does moving a lookup step off the main assistant use less AI usage?
measured-local2026-09-22da8b53a
- docs/experiments/DELEGATION-2026-09-22.md · da8b53a · 2026-09-22
Headline figures
-88.1%
Arm B, mean main-session token reduction; negative is dearer
measured-local · 2026-09-22
-1.56
Arm B, mean total token reduction including the delegate
measured-local · 2026-09-22
-2.18
Arm B, mean cost reduction
measured-local · 2026-09-22
No
Arm B, quality held on every task
measured-local · 2026-09-22
-3.01
Arm C, mean main-session token reduction; negative is dearer
measured-local · 2026-09-22
-3.01
Arm C, mean total token reduction including the delegate
measured-local · 2026-09-22
-1.51
Arm C, mean cost reduction
measured-local · 2026-09-22
No
Arm C, quality held on every task
measured-local · 2026-09-22
Notes
- One first-party pilot skill and two tasks, so this measures the mechanism, not a corpus-wide rate.
- Arm B hands the lookup to Haiku 4.5; arm C hands it to a lookup script. The delegate's own tokens and dollars are counted apart from the main session and only joined in the total columns.
- Both arms lost because the agent called the delegated lookup several times and every extra call re-bills the whole context.
Tables
n: Repetitions per arm. 6 rows.
| task_id | arm | label | score | quality_ok | main_tokens | main_token_reduction | total_tokens | total_token_reduction | cost_reduction | tool_calls | tool_call_change | delegate_calls | delegate_tokens | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| product-table | A | original, all on the main model | 1 | — | 42,858 | — | 42,858 | — | — | 3 | — | 0 | — | 3 | measured-local | 2026-09-22 | da8b53ad95e89365b23f2d5316e2e056d3a5e7b2 |
| product-table | B | delegated to model (Sonnet + haiku) | 0.8 | No | 63,649 | -0.4851 | 90,116 | -1.1027 | -0.532 | 6 | 3 | 4 | 26,467 | 3 | measured-local | 2026-09-22 | da8b53ad95e89365b23f2d5316e2e056d3a5e7b2 |
| product-table | C | delegated to code (lookup script, no model) | 0.8 | No | 176,628 | -3.1212 | 176,628 | -3.1212 | -0.9839 | 13 | 10 | 0 | — | 3 | measured-local | 2026-09-22 | da8b53ad95e89365b23f2d5316e2e056d3a5e7b2 |
| invoice-guard | A | original, all on the main model | 0.8 | — | 35,619 | — | 35,619 | — | — | 3 | — | 0 | — | 3 | measured-local | 2026-09-22 | da8b53ad95e89365b23f2d5316e2e056d3a5e7b2 |
| invoice-guard | B | delegated to model (Sonnet + haiku) | 0.8 | Yes | 81,071 | -1.2761 | 107,336 | -2.0134 | -3.8187 | 6 | 3 | 4 | 26,652 | 3 | measured-local | 2026-09-22 | da8b53ad95e89365b23f2d5316e2e056d3a5e7b2 |
| invoice-guard | C | delegated to code (lookup script, no model) | 0.8 | Yes | 138,539 | -2.8895 | 138,539 | -2.8895 | -2.0265 | 11 | 8 | 0 | — | 3 | measured-local | 2026-09-22 | da8b53ad95e89365b23f2d5316e2e056d3a5e7b2 |