SkillOptimizer
Research hub
ResultShare of skills

Does the skill beat the base model?

What share of skills do better than the assistant alone on at least one of their own tasks?

measured-local2026-09-1933c5c78
  • docs/experiments/BASELINE-2026-09-19.md · 33c5c78 · 2026-09-19

Headline figures

60

Skills drawn, stratified and seeded

measured-local · 2026-09-19

49of 60

Decided

measured-local · n=60 · 2026-09-19

34of 49

Beat the base model

measured-local · n=49 · 2026-09-19

69.4%(34 of 49)

Share of decided skills that beat the base model

95% interval 55.5% to 80.5%

measured-local · n=49 · 2026-09-19

7

No baseline advantage

measured-local · 2026-09-19

8

Every task already solved unaided

measured-local · 2026-09-19

9

Excluded: cannot run offline

measured-local · 2026-09-19

2

Excluded: too few repetitions survived

measured-local · 2026-09-19

74

Tasks graded

measured-local · 2026-09-19

24of 74

Tasks the unaided host scored 1.000

measured-local · n=74 · 2026-09-19

32.4%

Share of graded tasks already solved unaided

measured-local · 2026-09-19

Notes

  • A skill beats the base model when, on some task with its full three repetitions, the median grade with the skill exceeds the median without it by more than 0.05.
  • Per skill and existential: this is not the campaign's per-task figure of 27 of 63 tasks, and neither may be quoted as the other.
  • A skill excluded as unmeasurable is not a skill that failed: whether it beats the base model is unknown.
  • Groups are reported as a comparison, never as a fitted rule; a group under 8 skills is not quotable.

Superseded figures

A figure this project published and later corrected, kept beside what replaced it (CLAUDE.md, "Correcting a measurement instrument").

Skills that beat the base model

33 of 48, 68.8% → 34 of 49, 69.4%

The first eligibility rule let one failed grade on a second task veto a task with three good repetitions. The rule is existential over tasks; terracotta moved from insufficient repetitions to beating the base model. The status_superseded column carries the old scoring.

docs/experiments/BASELINE-2026-09-19.md

Group tables in the source document, by shape, size and installs

workflow 9 of 12, medium 16 of 19, mid installs 7 of 12 → workflow 10 of 13, medium 17 of 20, mid installs 8 of 13

The document's group tables were computed under the superseded rule; the groups table here is computed from the corrected scoring.

docs/experiments/BASELINE-2026-09-19.md

Tables

Every drawn skill, with its outcome

n: Graded runs per arm behind the row: the repetitions of every graded task. 60 rows.

skillnamestatusstatus_supersededdecidedbeats_baseshapebytessize_bandinstallsinstall_bandtasks_gradedtasks_floor_perfecttasks_beating_basebest_advantagemean_advantageRnevidence_classdatecommit
alphagbm-tepper-signalAlphaGBM Tepper Panic-Buy Signaleligible—YesYesreference-lookup4,865medium—tail1011133measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
feedbackFeedbackeligible—YesYesreference-prose13,654large134tail1011133measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
n8n-error-handlingn8n Error Handlingeligible—YesYesreference-lookup19,074large—tail2110.750.37536measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
research-agentResearch Agent Authoring Playbookeligible—YesYesreference-prose8,300large—tail1010.60.633measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
typescript-serverSpacetimeDB TypeScript SDK Referenceeligible—YesYesreference-lookup15,653large—tail1010.60.633measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
optimize-simplicite-logsOptimize Simplicite Logseligible—YesYesworkflow3,743medium75tail2110.50.2536measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
architecture-designerArchitecture Designereligible—YesYesreference-prose4,750medium—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
cost-efficiency-analyzerCost Efficiency Analyzereligible—YesYesreference-prose3,927medium—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
debugging-network-issuesDebugging Network Issueseligible—YesYesreference-lookup49,668large576mid2110.40.236measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
eol-messageEOL Messageeligible—YesYesreference-lookup17,721large—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
geo-fundamentalsGEO Fundamentalseligible—YesYesreference-lookup3,445medium—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
landing-page-conversion-auditLanding Page Conversion Auditeligible—YesYesdialogue7,379large29tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
offensive-reportingPenetration Test Reporting - Professional Methodologyeligible—YesYesreference-lookup14,720large—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
terracottaTerracotta Design System Skill (Universal)eligibleinsufficient_repetitionsYesYesworkflow3,778medium1,120mid2010.40.435measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
unity-primetweenPrimeTween Skillseligible—YesYesworkflow3,260medium—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
unity-script-rolesUnity Script Roleseligible—YesYesloader1,443small—tail1010.40.433measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
caveman-commitCaveman Commiteligible—YesYesreference-lookup2,588medium—tail1010.250.2533measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
compose-modifier-and-layout-styleCompose modifier and layout styleeligible—YesYesreference-lookup17,475large11tail1010.250.2533measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
dotnet-local-tools.NET Local Toolseligible—YesYesreference-lookup8,912large—tail1010.250.2533measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
llm-councilLLM Council Skilleligible—YesYesworkflow4,850medium1,163mid2010.25-0.12536measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
wiki-lintLint the wikieligible—YesYesworkflow3,029medium—tail1010.250.2533measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
ads-planPaid Media Planeligible—YesYesworkflow1,518small—tail1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
catchupContext Catchupeligible—YesYesworkflow3,082medium—tail1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
data-analystData Analysis Experteligible—YesYesworkflow2,930medium—tail2010.20.136measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
event-store-designEvent Store Designeligible—YesYesreference-prose4,148medium9,783mid1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
friendlyFriendly Design System Skill (Universal)eligible—YesYesworkflow3,536medium1,217mid1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
hermes-agent-skill-authoringAuthoring Hermes-Agent Skills (in-repo)eligible—YesYesworkflow14,438large—tail1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
latent-briefingLatent Briefing and KV Cache Memory Sharingeligible—YesYesreference-prose13,029large—tail1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
modeling-product-usage-metricsModeling product-usage metricseligible—YesYesreference-lookup5,312medium—tail1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
napkinNapkin - Visual Whiteboard for Copilot CLIeligible—YesYesreference-lookup7,394large2,246mid1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
product-visionProduct Visioneligible—YesYesreference-prose2,518medium2,585mid2010.20.136measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
triton-kernelWriting Optimized Triton GPU Kernelseligible—YesYesreference-lookup4,528medium—tail1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
vision-sftVision-Language SFTeligible—YesYesreference-lookup7,872large1,508mid1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
web-quality-auditWeb quality auditeligible—YesYesreference-lookup7,426large26,967top1010.20.233measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
color-systemColor Systemno_baseline_advantage—YesNoreference-prose1,682small—tail2100036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
compose-expertCompose Multiplatform Expertno_baseline_advantage—YesNoreference-lookup15,966large1,259mid2100036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
conducting-user-interviewsConducting User Interviewsno_baseline_advantage—YesNoreference-prose4,937medium—tail2100-0.536measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
em-dashem dashtasks_not_discriminating—YesNoreference-prose9,668large81tail2200036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
folder-structure-blueprint-generatorProject Folder Structure Blueprint Generatorno_baseline_advantage—YesNoreference-prose13,880large9,434mid2100036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
git-checkpointGit Checkpoint and Rollbackno_baseline_advantage—YesNoreference-lookup1,259small—tail2000036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
gtm-technical-product-pricingTechnical Product Pricingtasks_not_discriminating—YesNoreference-prose13,108large1,714mid2200036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
make-pr-easy-to-reviewMake PR Easy to Reviewno_baseline_advantage—YesNoworkflow2,275small2,294mid2000036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
plano-observability-debuggingPlano Observability and Debuggingtasks_not_discriminating—YesNoworkflow967small3tail2200036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
protocol-reverse-engineeringProtocol Reverse Engineeringtasks_not_discriminating—YesNoreference-lookup12,626large11,203top2200036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
sprites-and-imagesSprites and Imagestasks_not_discriminating—YesNoreference-lookup14,942large786mid2200036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
step-functionsAWS Step Functionstasks_not_discriminating—YesNoreference-lookup9,709large106tail2200036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
theme-factoryTheme Factory Skilltasks_not_discriminating—YesNoreference-prose3,124medium85,182top2200-0.236measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
v8-logV8 Log Viewer CLIno_baseline_advantage—YesNoreference-lookup4,159medium—tail2100036measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
verifyVerifytasks_not_discriminating—YesNoworkflow621small—tail2200-0.2536measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
tech-resume-optimizerTech Resume Optimizerinsufficient_repetitions—NoNoreference-lookup10,556large—tail2010.20.135measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
xena-tcga-gene-query🦀 xena-tcga-gene-queryinsufficient_repetitions—NoNoreference-lookup19,276large—tail1000032measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
clawpathy-autoresearchClawpathy Autoresearchunmeasurable—NoNoreference-lookup5,516medium—tail000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
copilot-spacesCopilot Spacesunmeasurable—NoNoreference-lookup8,726large4,492mid000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
creating-amazon-aurora-db-cluster-with-instancesCreating Amazon Aurora DB Cluster with Instancesunmeasurable—NoNoworkflow1,610small4,762mid000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
gws-gmail-sendgmail +sendunmeasurable—NoNoloader2,626medium70,499top000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
muapi-photo-pack-generatorMuAPI Photo Pack Generatorunmeasurable—NoNoreference-prose3,881medium—tail000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
persona-it-adminIT Administratorunmeasurable—NoNoworkflow917small27,745top000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
span-codingSpan Codingunmeasurable—NoNoworkflow3,983medium—tail000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
threads-profile-searchThreads - Profile Searchunmeasurable—NoNoreference-lookup7,200large6tail000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
tui-qaTUI QAunmeasurable—NoNoreference-lookup8,503large5tail000——30measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9

Beating the base model by shape, size and installs

n: Decided skills in the group. 11 rows.

dimensiongroupdecidedbeats_baseshareinterval_lowinterval_highquotablenevidence_classdatecommit
shapereference-lookup21150.71430.50040.8619Yes21measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
shapereference-prose1370.53850.29140.7679Yes13measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
shapeworkflow13100.76920.49740.9182Yes13measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
shapedialogue1110.20651No1measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
shapeloader1110.20651No1measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
size_bandlarge22150.68180.47320.8364Yes22measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
size_bandmedium20170.850.63960.9476Yes20measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
size_bandsmall720.28570.08220.6411No7measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
install_bandtail33250.75760.58980.8717Yes33measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
install_bandmid1380.61540.35520.8229Yes13measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9
install_bandtop310.33330.06150.7923No3measured-local2026-09-1933c5c78cc2e4fccc4f55a54ded91f0c1c6d1aed9