Null resultAI usage (tokens)
Rule checks on file paths
Does a rule check that blocks writes to named files pay?
measured-local2026-09-1980e4a13
- docs/experiments/DETECTION-2026-09-19.md · 80e4a13 · 2026-09-20
Headline figures
10
Skills attempted
measured-local · 2026-09-19
4of 10
Reached a three-arm comparison
measured-local · n=10 · 2026-09-19
0of 4
Accepted
95% interval 0.0% to 49.0%
measured-local · n=4 · 2026-09-19
-64.9%
Mean token reduction over compared skills; negative is dearer
measured-local · 2026-09-19
1of 4
Compared skills whose repetitions disagree in sign
measured-local · n=4 · 2026-09-19
4
Stopped as no_baseline_advantage
measured-local · 2026-09-19
2
Stopped as unmeasurable
measured-local · 2026-09-19
Notes
- A seeded sample of 10 of the 21 unmeasured path-scoped candidates.
- Of an 80-item sample of write prohibitions, 14 were decidable from a file path alone; the rest name content, a phase or a role a rule check cannot resolve. That classification is in the source document and is not derived from these rows.
Tables
n: Runs per arm behind the row: repetitions times tasks compared, or the graded runs of the eligibility check when the skill was not compared. 10 rows.
| skill | name | shape | installs | status | status_superseded | decision | accepted | artifacts | skill_md_unchanged | token_reduction | spread_min | spread_max | straddles_zero | net_call_change | tokens_none | tokens_original | tokens_optimized | score_none | score_original | score_optimized | mission_preserved | retention_band | R | tasks | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| research-lit | Research Literature Review | reference-lookup | — | compared | — | inconclusive_repetitions_straddle_zero | No | hook | Yes | -0.004 | -0.025 | 0.0144 | Yes | 0 | 21,526 | 88,817 | 89,175 | 0.8 | 1 | 1 | Yes | full | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| code-splitting | Code Splitting | reference-lookup | — | compared | — | rejected_not_cheaper | No | hook | Yes | -0.013 | -0.1179 | -0.0075 | No | 0 | 109,206 | 92,586 | 93,787 | 0.8 | 1 | 1 | Yes | full | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| prp-spike | PRP Spike - Prove or Disprove | reference-prose | — | compared | — | rejected_not_cheaper | No | hook | Yes | -0.5886 | -4.0852 | -0.2122 | No | 5 | 44,315 | 260,679 | 414,116 | 0.2 | 0.8 | 0.6 | No | partial | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| maple-onboarding-style | Maple OTel onboarding style | workflow | — | compared | — | rejected_not_cheaper | No | hook | Yes | -1.9922 | -10.357 | -1.1909 | No | 11 | 170,345 | 195,358 | 584,544 | 0 | 1 | 0.8 | No | strong | 3 | 1 | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| baoyu-url-to-markdown | URL to Markdown | reference-lookup | — | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| harbor | Harbor CLI | reference-lookup | — | unmeasurable | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 0 | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| presentation | Presentation | workflow | 1,877 | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| tech-search | Tech Search | reference-lookup | — | unmeasurable | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 0 | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| triage | Triage | workflow | 798,050 | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| workflow | Standard Workflow | workflow | — | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |