Null resultShare of skills
Tasks the assistant already solves
Does a stricter task prompt reduce tasks the assistant solves perfectly without the skill?
measured-local2026-09-1980e4a13
- docs/experiments/DETECTION-2026-09-19.md · 80e4a13 · 2026-09-20
Headline figures
10
Skills drawn
measured-local · 2026-09-19
2
Skills that cannot run offline, no task graded
measured-local · 2026-09-19
16
Tasks graded without the skill
measured-local · 2026-09-19
6of 16
Tasks the unaided host scored 1.000
measured-local · n=16 · 2026-09-19
37.5%
Share already solved unaided
measured-local · 2026-09-19
32.4%
The same share under the earlier prompt, from the beats-base-model study
measured-local · 2026-09-19
Notes
- A fresh seeded draw of different skills, so the prompt change is confounded with the draw.
- Floor-perfect has more than one cause: some checks are satisfiable by a competent unaided answer whatever the prompt says.
Tables
n: Runs without the skill behind the row. 18 rows.
| skill | task_id | graded | repetitions | score_none_median | floor_perfect | n | evidence_class | date | commit |
|---|---|---|---|---|---|---|---|---|---|
| ce-polish-beta | polish-login-form | Yes | 3 | 0.6 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| ce-polish-beta | polish-product-grid | Yes | 3 | 0.8 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| design-negotiation | timeline-compression-negotiation | Yes | 3 | 1 | Yes | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| design-negotiation | scope-reduction-already-built | Yes | 3 | 1 | Yes | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| enrich-lead | — | No | — | — | — | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| gh-create-issue | file-bug-report-issue | Yes | 3 | 0.3333 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| gh-create-issue | file-feature-request-issue | Yes | 3 | 1 | Yes | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| github-codespaces-efficiency | audit-wasteful-devcontainer | Yes | 3 | 0.2 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| github-codespaces-efficiency | optimize-machine-and-prebuilds | Yes | 3 | 0.8 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| memory-reviewer | — | No | — | — | — | 0 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| paperkit | ios-markup-editor | Yes | 3 | 0.2 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| paperkit | macos-markup-safe-load | Yes | 3 | 0.4 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| spec | create-cache-spec | Yes | 3 | 1 | Yes | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| spec | amend-spec-task | Yes | 3 | 0.8 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| validate-data | sql-query-denominator-validation | Yes | 3 | 1 | Yes | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| validate-data | multi-issue-analysis-validation | Yes | 3 | 0.6667 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| validating-and-publishing-canvases | canvas-capability-declarations | Yes | 3 | 0 | No | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |
| validating-and-publishing-canvases | canvas-version-conflict-merge | Yes | 3 | 1 | Yes | 3 | measured-local | 2026-09-19 | 80e4a13fb96ebb96d8efff8690a8c77215657590 |