Null resultAI usage (tokens)
Helpers built from documented steps
Does moving a skill's documented creation steps into a helper reach a test at all?
measured-local2026-09-195658cf4
- docs/experiments/COMPILER-2026-09-19.md · 5658cf4 · 2026-09-19
Headline figures
15
Skills attempted
measured-local · 2026-09-19
0of 15
Reached a three-arm comparison
measured-local · n=15 · 2026-09-19
0of 0
Accepted
measured-local · n=0 · 2026-09-19
6
Every task already solved unaided
measured-local · 2026-09-19
4
No baseline advantage
measured-local · 2026-09-19
3
Eligible, but none of the documented steps appeared in a real run
measured-local · 2026-09-19
1
Too few repetitions survived
measured-local · 2026-09-19
1
Provider error
measured-local · 2026-09-19
Notes
- All three eligible skills were refused because none of their documented creation steps appeared in a real run: 0 of 4, 0 of 4 and 0 of 5.
- The binding constraint is upstream of the compiler: no corpus skill of this static shape performed the steps it documents.
Tables
n: Runs per arm behind the row: repetitions times tasks compared, or the graded runs of the eligibility check when the skill was not compared. 15 rows.
| skill | name | shape | installs | status | status_superseded | decision | accepted | artifacts | skill_md_unchanged | token_reduction | spread_min | spread_max | straddles_zero | net_call_change | tokens_none | tokens_original | tokens_optimized | score_none | score_original | score_optimized | mission_preserved | retention_band | R | tasks | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| block-no-verify-hook | Block No-Verify Hook | reference-lookup | 5,474 | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 1 | 3 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| check-context | — | — | — | error | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | 0 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| code-audit | Code Audit Patterns | reference-lookup | — | no_artifact | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 1 | 3 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| deserialization-insecure | Insecure Deserialization | reference-lookup | 3,091 | tasks_not_discriminating | insufficient_repetitions | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 4 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| fuzzing-dictionary | Fuzzing Dictionary | reference-lookup | 4,651 | tasks_not_discriminating | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| invoice-organizer | Invoice Organizer | reference-prose | — | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| ios-ettrace-performance | iOS ETTrace Performance | reference-lookup | 850 | insufficient_repetitions | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 2 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| jndi-injection | JNDI Injection | reference-lookup | 3,059 | tasks_not_discriminating | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| kernel-exploitation | Kernel Exploitation | reference-lookup | 3,095 | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| memory-forensics | Memory Forensics | workflow | 9,993 | tasks_not_discriminating | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| minecraft-modpack-server | Minecraft Modpack Server Setup | reference-lookup | — | no_artifact | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 1 | 3 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| offensive-toctou | TOCTOU - Time-of-Check / Time-of-Use Exploitation | reference-lookup | — | no_artifact | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 4 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| ppt-editing-skill | Editing Presentations | reference-lookup | — | no_baseline_advantage | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| steganography-techniques | Steganography Techniques | workflow | 2,994 | tasks_not_discriminating | — | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 6 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |
| traffic-analysis-pcap | PCAP Traffic Analysis | workflow | 3,145 | tasks_not_discriminating | insufficient_repetitions | — | No | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3 | 2 | 4 | measured-local | 2026-09-19 | 5658cf4666bb9158f16708aa0a48dfb3c59d752e |