ResultAI usage (tokens)
Prepared facts
Does gathering a skill's routine facts once, before the assistant starts, use less AI usage with the same results?
measured-local2026-09-2652363d2
- docs/experiments/PRECOMPUTE-2026-09-26.md · 52363d2 · 2026-09-26
- data: experiments/airmarket-6k/campaign/round-evidence/r9-precompute · 52363d2
Headline figures
31
Skills compared
measured-local · 2026-09-26
7of 31
Accepted
95% interval 11.4% to 39.8%
measured-local · n=31 · 2026-09-26
16.6%
Median token reduction over every compared skill
measured-local · 2026-09-26
18
Inconclusive: the three repetitions disagree in sign
measured-local · 2026-09-26
1
Inconclusive: too few repetitions
measured-local · 2026-09-26
4
Rejected for a mission loss
measured-local · 2026-09-26
1
Rejected, not cheaper
measured-local · 2026-09-26
7
Compared skills whose mission regressed on some task, whatever the decision
measured-local · 2026-09-26
Notes
- SKILL.md stays byte-identical except one appended block of fixed read-only commands that Claude Code expands at skill load; every component is justified by the original arm's traces, never by the prose.
- Picking did not help: the cohorts table compares Jev's pick, the trace pick and a random draw, and their intervals overlap.
- 118 of the 174 skills that beat the base model in earlier studies were plannable; that count is in the source document and is not derived from these rows.
Superseded figures
A figure this project published and later corrected, kept beside what replaced it (CLAUDE.md, "Correcting a measurement instrument").
Outcome split of the 24 skills not accepted
17 inconclusive, 5 rejected for a mission loss → 18 inconclusive by sign, 4 rejected for a mission loss, as the records decide them
The source document's split differs from the recorded decisions by one skill, and the records do not show which skill the document moved. The accepted count, its interval and the median saving are unchanged. mission_preserved is carried on every row, because two inconclusive skills and the not-cheaper one also scored below the original on some task.
docs/experiments/PRECOMPUTE-2026-09-26.md
Tables
n: Runs per arm behind the row: repetitions times tasks compared, or the graded runs of the eligibility check when the skill was not compared. 31 rows.
| skill | name | shape | installs | status | status_superseded | decision | accepted | artifacts | skill_md_unchanged | token_reduction | spread_min | spread_max | straddles_zero | net_call_change | tokens_none | tokens_original | tokens_optimized | score_none | score_original | score_optimized | mission_preserved | retention_band | R | tasks | cohorts | components | jev_score | subsumable_calls_per_run | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| product-vision | Product Vision | reference-prose | 2,585 | compared | — | accepted | Yes | precompute | No | 0.4375 | 0.4114 | 0.4455 | No | -1 | 41,596 | 53,419 | 30,050 | 0.6 | 0.8 | 0.8 | Yes | full | 3 | 2 | random | listing;inputs | 0.27 | 1.5 | 6 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| approachability-audit | Approachability Audit | reference-prose | — | compared | — | accepted | Yes | precompute | No | 0.3241 | 0.2996 | 0.3326 | No | -1 | 21,831 | 24,913 | 16,838 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | trace | listing;inputs | 0.23 | 2.333 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| marquee-loop | Marquee Skill | workflow | — | compared | — | accepted | Yes | precompute | No | 0.2259 | 0.201 | 0.274 | No | -1 | 27,831 | 30,362 | 23,503 | 0.75 | 1 | 1 | Yes | full | 3 | 1 | random | listing;inputs | 0.21 | 2 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| github-project-triage | GitHub Project Triage | reference-lookup | 89 | compared | — | accepted | Yes | precompute | No | 0.2257 | 0.2136 | 0.2569 | No | 0 | 21,476 | 51,049 | 39,527 | 0.6 | 0.8 | 1 | Yes | full | 3 | 1 | jev | listing;inputs | 0.55 | 1.333 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| rds-sqlserver | Amazon RDS for SQL Server | workflow | 3,348 | compared | — | accepted | Yes | precompute | No | 0.2241 | 0.2186 | 0.3783 | No | -1 | 16,663 | 76,880 | 59,653 | 0.2 | 1 | 1 | Yes | full | 3 | 1 | random | listing | 0.23 | 1 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| jq | jq - Built-in JSON Processor | reference-lookup | — | compared | — | accepted | Yes | precompute | No | 0.214 | 0.2089 | 0.2143 | No | -1 | 33,675 | 40,683 | 31,977 | 0.4 | 1 | 1 | Yes | full | 3 | 1 | random | listing;inputs | 0.1 | 2 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| writing-core | Academic Writing Core | reference-lookup | — | compared | — | accepted | Yes | precompute | No | 0.1989 | 0.1728 | 0.2622 | No | -1 | 30,902 | 51,324 | 41,117 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | trace | listing;inputs | 0.23 | 2.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| ads-plan | Paid Media Plan | workflow | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.4581 | -0.1803 | 0.7078 | Yes | -3 | 37,854 | 59,864 | 32,440 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | trace | listing;inputs | 0.25 | 2.333 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| git-advanced-workflows | Git Advanced Workflows | reference-lookup | 17,900 | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.393 | -0.0745 | 0.4552 | Yes | -5 | 114,182 | 247,955 | 150,498 | 0 | 0.6 | 0 | No | lost | 3 | 1 | trace | listing;inputs;git | 0.2 | 3 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| programming | Programming | reference-lookup | 11 | compared | — | inconclusive_insufficient_repetitions | No | precompute | No | 0.3894 | 0.3642 | 0.6266 | No | -7 | 309,224 | 776,690 | 474,251 | 0.6 | 1 | 1 | Yes | full | 2 | 1 | jev | listing;bundle:references/python/README.md | 0.49 | 2 | 2 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| abp-application-layer | ABP Application Layer Patterns | reference-lookup | 16 | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.3842 | -0.0629 | 0.6203 | Yes | -3 | 141,465 | 98,749 | 60,808 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | trace | listing;inputs | 0.28 | 2.333 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| aws-well-architected-review | AWS Well-Architected Review | reference-lookup | 108 | compared | — | rejected_mission_failed | No | precompute | No | 0.363 | 0.3445 | 0.5339 | No | -1 | 22,054 | 30,236 | 19,259 | 0.6 | 0.8 | 0.6 | No | lost | 3 | 1 | jev | listing;inputs | 0.48 | 2 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| axiom-audit-iap | In-App Purchase Auditor Agent | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.3377 | -0.1677 | 0.5364 | Yes | -2 | 23,701 | 98,612 | 65,310 | 0.25 | 0.75 | 0.75 | Yes | full | 3 | 1 | trace | listing;inputs | 0.25 | 2.333 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| debug-gradient-flow | Debugging Gradient Flow in Training | reference-lookup | 3 | compared | — | rejected_mission_failed | No | precompute | No | 0.3033 | 0.1418 | 0.3377 | No | -2 | 100,148 | 128,778 | 89,724 | 0.25 | 0.75 | 0.5 | No | partial | 3 | 2 | trace | listing;inputs | 0.2 | 2.667 | 6 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| xcode-project-analyzer | Xcode Project Analyzer | reference-prose | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.2769 | -0.068 | 0.4219 | Yes | -1 | 20,846 | 38,336 | 27,721 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | jev | listing | 0.41 | 1.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| blog-analyze | Blog Analyze | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.1661 | -0.3141 | 0.5742 | Yes | -1 | 19,355 | 89,798 | 74,886 | 0.4 | 1 | 1 | Yes | full | 3 | 1 | jev | listing;inputs | 0.42 | 1 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| prp-commit | Commit Intended Work | workflow | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.145 | -0.137 | 0.145 | Yes | -1 | 47,001 | 65,366 | 55,890 | 0 | 0.2 | 0.4 | Yes | full | 3 | 1 | jev | listing | 0.4 | 1 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| stat-research-orchestrator | Statistical Research Orchestrator | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.1326 | -0.4325 | 0.2051 | Yes | -1 | 46,463 | 77,532 | 67,253 | 0 | 1 | 1 | Yes | full | 3 | 1 | random | inputs | 0.2 | 1 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| dmr-from-django-ninja | DMR from django-ninja | workflow | 45 | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.1325 | -0.9535 | 0.2918 | Yes | -4 | 53,954 | 396,121 | 343,650 | 0 | 0.8 | 0.8 | Yes | full | 3 | 1 | jev;trace | listing;inputs | 0.55 | 2.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| unity-netcode | Unity Netcode for GameObjects Skills | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.1312 | -0.6627 | 0.3321 | Yes | -1 | 518,656 | 548,296 | 476,348 | 0.8 | 1 | 1 | Yes | full | 3 | 2 | trace | listing | 0.26 | 3 | 6 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| catchup | Context Catchup | workflow | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.1187 | -0.7617 | 0.12 | Yes | -1 | 46,465 | 74,318 | 65,497 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | jev | inputs;git | 0.78 | 1.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| gh-aw | GitHub Agentic Workflows | reference-lookup | 36 | compared | — | rejected_mission_failed | No | precompute | No | 0.1118 | 0.0388 | 0.3452 | No | -1 | 46,230 | 138,152 | 122,700 | 0.2 | 1 | 0.8 | No | strong | 3 | 1 | trace | listing | 0.31 | 3 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| animation-principles | Animation Principles | reference-prose | — | compared | — | rejected_mission_failed | No | precompute | No | 0.0847 | 0.0758 | 0.1106 | No | -0.5 | 64,638 | 62,169 | 56,902 | 0 | 1 | 0.75 | No | strong | 3 | 2 | random | listing;inputs | 0.21 | 2 | 6 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| code-review-expert | Code Review Expert | reference-lookup | 13,180 | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | 0.0765 | -0.0236 | 0.126 | Yes | 0 | 23,913 | 44,482 | 41,079 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | jev | listing | 0.46 | 2 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| threejs-interaction | Three.js Interaction | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | -0.0158 | -0.0198 | 0.2373 | Yes | 0 | 49,991 | 68,340 | 69,419 | 0.4 | 0.6 | 0.6 | Yes | full | 3 | 1 | trace | listing;inputs | 0.18 | 2.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| modeling-product-usage-metrics | Modeling product-usage metrics | reference-lookup | — | compared | — | rejected_not_cheaper | No | precompute | No | -0.0549 | -0.3406 | -0.0269 | No | 0 | 22,892 | 45,362 | 47,854 | 0.6 | 0.8 | 0.4 | No | lost | 3 | 1 | random | listing | 0.22 | 1.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| publish-project-to-github | Publish Project to GitHub | workflow | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | -0.0592 | -0.3886 | 0.1502 | Yes | 0.5 | 92,614 | 336,990 | 356,942 | 0 | 0.2 | 0 | No | lost | 3 | 2 | jev | listing | 0.41 | 1.5 | 6 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| retrospective | Sprint Retrospective | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | -0.1684 | -0.3988 | 0.0061 | Yes | 1 | 33,580 | 59,947 | 70,043 | 0.6 | 0.8 | 0.8 | Yes | full | 3 | 1 | trace | listing | 0.34 | 2.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| update-pr | Update PR | workflow | 14 | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | -0.2156 | -0.2217 | 0.1959 | Yes | 2 | 28,572 | 50,381 | 61,241 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | jev | listing | 0.56 | 1 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| draw-io-diagram-generator | Draw.io Diagram Generator | reference-lookup | 2,724 | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | -0.3902 | -0.8863 | 0.1497 | Yes | 1 | 63,891 | 153,082 | 212,807 | 0.6 | 1 | 1 | Yes | full | 3 | 1 | random | bundle:assets/templates/architecture.drawio | 0.12 | 1 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| wiki-lint | Lint the wiki | workflow | — | compared | — | inconclusive_repetitions_straddle_zero | No | precompute | No | -0.475 | -0.5623 | 0.07 | Yes | 3 | 87,993 | 114,576 | 168,998 | 0.75 | 1 | 1 | Yes | full | 3 | 1 | jev | listing | 0.67 | 1.667 | 3 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
n: Skills compared in the cohort. 3 rows.
| cohort | skills | compared | accepted | interval_low | interval_high | median_token_reduction | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|---|
| jev | 12 | 12 | 1 | 0.0149 | 0.3539 | 0.1387 | 12 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| trace | 12 | 12 | 2 | 0.047 | 0.448 | 0.2511 | 12 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |
| random | 8 | 8 | 4 | 0.2152 | 0.7848 | 0.1733 | 8 | measured-local | 2026-09-26 | 52363d25071579bcd00029e20f4278c965578645 |