Null resultAI usage (tokens)
The mission rule, out of sample
Does refusing a rule check that forbids the skill's own purpose hold on skills it never saw?
measured-local2026-09-1980e4a13
- docs/experiments/DETECTION-2026-09-19.md · 80e4a13 · 2026-09-20
Headline figures
35
Unseen skills the rule could see
measured-local · 2026-09-19
2
Rule check refused by the rule before measurement
measured-local · 2026-09-19
18
Comparisons the rule let through
measured-local · 2026-09-19
5
Let through, and the mission regressed
measured-local · 2026-09-19
0of 5
Of those, accepted anyway
measured-local · n=5 · 2026-09-19
Notes
- Precision is unobservable here: a refused rule check never ran its improved arm. Re-measuring the two refusals with the rule off would recreate the mission-breaking runs lane K already paid for, so it was not run.
- The five regressions it let through are a different failure, the agent thrashing against the rule check; the net-call and cheaper-than-original conditions rejected all five.
Tables
n: Runs per arm behind a comparison; 0 when the skill was not compared. 35 rows.
| skill | band | status | rule_outcome | accepted | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|
| activitykit | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| app-clips | absolute | compared | passed_mission_regressed | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| arize-trace | absolute | unmeasurable | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| auto-repo-setup | absolute | compared | passed_mission_preserved | No | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| axiom-data | absolute | compared | passed_mission_preserved | No | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| computer-use | absolute | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| cpp-coding-standards | absolute | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| cpp-testing | absolute | compared | passed_mission_regressed | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| fix-merge-conflicts | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| no-comments | absolute | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| pr-screenshots | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| quantum-qiskit | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| redis-expert | absolute | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| shepherd-pr | absolute | unmeasurable | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| shopify-review-triage | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| supersede-pr | absolute | unmeasurable | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| terraform-skill | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| webiny-admin-architect | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| webiny-full-stack-architect | absolute | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| webiny-mailer-smtp | absolute | tasks_not_discriminating | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| atmos-secrets | conditional-rest | artifact_refused | refused_by_rule | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| dmr-from-django-ninja | conditional-rest | compared | passed_mission_regressed | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| publish-project-to-github | conditional-rest | artifact_refused | refused_by_rule | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| review-pr | conditional-rest | compared | passed_mission_preserved | No | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| terraform | conditional-rest | tasks_not_discriminating | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| baoyu-url-to-markdown | path-scoped | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| code-splitting | path-scoped | compared | passed_mission_preserved | No | 6 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| harbor | path-scoped | unmeasurable | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| maple-onboarding-style | path-scoped | compared | passed_mission_regressed | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| presentation | path-scoped | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| prp-spike | path-scoped | compared | passed_mission_regressed | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| research-lit | path-scoped | compared | passed_mission_preserved | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| tech-search | path-scoped | unmeasurable | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| triage | path-scoped | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| workflow | path-scoped | no_baseline_advantage | not_reached | No | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |