Null resultAI usage (tokens)
Rule checks from absolute prohibitions
Does a rule check pay when the prohibition has no condition?
measured-local2026-09-1980e4a13
- docs/experiments/DETECTION-2026-09-19.md · 80e4a13 · 2026-09-20
Headline figures
20
Skills attempted
measured-local · 2026-09-19
12of 20
Reached a three-arm comparison
measured-local · n=20 · 2026-09-19
0of 12
Accepted
95% interval 0.0% to 24.3%
measured-local · n=12 · 2026-09-19
5.7%
Mean token reduction over compared skills; negative is dearer
measured-local · 2026-09-19
11of 12
Compared skills whose repetitions disagree in sign
measured-local · n=12 · 2026-09-19
4
Stopped as no_baseline_advantage
measured-local · 2026-09-19
1
Stopped as tasks_not_discriminating
measured-local · 2026-09-19
3
Stopped as unmeasurable
measured-local · 2026-09-19
Notes
- An absolute rule check has no cost asymmetry between a compliant run and a violating one, so there is little to save.
- The remaining 214 absolute candidates were not run: the plan pre-registered a skip below about 15 percent acceptance.
Superseded figures
A figure this project published and later corrected, kept beside what replaced it (CLAUDE.md, "Correcting a measurement instrument").
Compared skills whose repetitions straddle zero
every one of the 12 → 11 of 12
Recounted from the records: one compared skill was rejected as not cheaper with all three repetitions dearer, so it does not straddle zero. Nothing was accepted either way.
docs/experiments/DETECTION-2026-09-19.md
Tables
n: Runs per arm behind the row: repetitions times tasks compared, or the graded runs of the eligibility check when the skill was not compared. 20 rows.
| skill | name | shape | installs | status | status_superseded | decision | accepted | artifacts | skill_md_unchanged | token_reduction | spread_min | spread_max | straddles_zero | net_call_change | tokens_none | tokens_original | tokens_optimized | score_none | score_original | score_optimized | mission_preserved | retention_band | R | tasks | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cpp-testing | C++ Testing (Agent Skill) | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.2896 | -0.7719 | 0.6059 | Yes | -2 | 77,589 | 128,874 | 91,550 | 0.25 | 0.5 | 0.25 | No | lost | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| activitykit | ActivityKit | reference-lookup | 2,887 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.2422 | -0.3127 | 0.2787 | Yes | -2 | 56,843 | 81,162 | 61,501 | 0.6 | 0.8 | 0.8 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| fix-merge-conflicts | Fix merge conflicts | workflow | 2,250 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.238 | -0.0185 | 0.2419 | Yes | -1 | 26,351 | 38,657 | 29,456 | 0.75 | 1 | 1 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| pr-screenshots | PR Screenshots | reference-lookup | 139 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.2274 | -0.0079 | 0.2274 | Yes | 0 | 22,457 | 34,330 | 26,523 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| webiny-full-stack-architect | Full-Stack Extension Skeleton | reference-lookup | 2 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.2065 | -0.9018 | 0.3923 | Yes | 0 | 272,428 | 155,881 | 123,685 | 0 | 0.6 | 0.6 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| quantum-qiskit | Qiskit Quantum Machine Learning | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.1625 | -0.2275 | 0.3626 | Yes | 0 | 39,830 | 83,175 | 69,656 | 0 | 0.5 | 0.5 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| auto-repo-setup | Auto Repo Setup | workflow | 373 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.065 | -0.1517 | 0.1592 | Yes | -1.5 | 114,026 | 215,334 | 201,340 | 0 | 1 | 1 | Yes | full | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| app-clips | App Clips | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.0059 | -0.1413 | 0.0131 | Yes | 0 | 19,907 | 14,812 | 14,724 | 0.4 | 1 | 0.8 | No | partial | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| terraform-skill | Terraform Skill for Claude | reference-lookup | 6,130 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | 0.0038 | -0.2444 | 0.2146 | Yes | -1 | 39,552 | 73,371 | 73,091 | 0.4 | 0.6 | 0.8 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| shopify-review-triage | Shopify Review Triage | reference-prose | 23 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | -0.013 | -0.047 | 0.0057 | Yes | 0 | 19,230 | 33,247 | 33,680 | 0 | 0.8 | 0.8 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| axiom-data | Data & Persistence | workflow | — | compared | — | rejected_not_cheaper | No | hook | Yes | -0.3372 | -0.314 | -0.1559 | No | 1.5 | 57,078 | 97,396 | 130,240 | 0.8 | 1 | 1 | Yes | full | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| webiny-admin-architect | Admin Architecture Patterns | reference-lookup | 1 | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | -0.4074 | -0.8902 | 0.1087 | Yes | 1 | 126,850 | 82,191 | 115,675 | 0 | 1 | 1 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| arize-trace | Arize Trace Skill | reference-lookup | — | unmeasurable | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 0 | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| computer-use | Computer Use | reference-lookup | 156,523 | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| cpp-coding-standards | C++ Coding Standards (C++ Core Guidelines) | reference-lookup | — | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| no-comments | No comments | workflow | 1,533 | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| redis-expert | Redis Data Store Expertise | workflow | — | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| shepherd-pr | Review PR | workflow | 73 | unmeasurable | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 0 | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| supersede-pr | Supersede PR | workflow | 14 | unmeasurable | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 0 | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| webiny-mailer-smtp | Mailer SMTP Configuration | reference-lookup | 2 | tasks_not_discriminating | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |