SkillOptimizer
Research hub
ResultMoney

Cheaper on a smaller model

Does a skill let a smaller model match the default model's results for less money?

measured-local2026-09-2080e4a13
  • docs/experiments/DETECTION-2026-09-19.md · 80e4a13 · 2026-09-20

Headline figures

34

Skills that beat the base model, measured on the smaller model

measured-local · 2026-09-20

16of 34

Held quality on every comparable task and cost less

measured-local · n=34 · 2026-09-20

65.7%

Median cost on the smaller model as a share of the default model's, downgradeable skills

measured-local · 2026-09-20

40.8%

Lowest cost ratio among downgradeable skills

measured-local · 2026-09-20

94.0%

Highest cost ratio among downgradeable skills

measured-local · 2026-09-20

41

Tasks with a comparable default-model run

measured-local · 2026-09-20

27of 41

Comparable tasks where quality held

measured-local · n=41 · 2026-09-20

15of 16

Downgradeable skills where the skill also lifted the smaller model above no skill

measured-local · n=16 · 2026-09-20

13

Skills whose quality dropped on the smaller model

measured-local · 2026-09-20

5

Skills that held quality but were not cheaper

measured-local · 2026-09-20

Notes

  • A cost result on its own axis: it does not reduce token count and is never added to a token figure.
  • Smaller model: Haiku 4.5 with the skill. Default model: Sonnet 5 with the skill, from the lane L1 runs. A skill held when the smaller model matched within 0.05 on every comparable task, which says nothing about unseen tasks.
  • A skill's cost ratio is the upper median of its tasks' ratios, as compiler-site.mjs computes it.

Superseded figures

A figure this project published and later corrected, kept beside what replaced it (CLAUDE.md, "Correcting a measurement instrument").

median_cost_ratio in the study's summary.json

0.42 → 0.657, the median over downgradeable skills

The summary's median runs over every task row, and a task with no comparable default-model run records a cost ratio of 0. The reported figure is per skill, over skills that held quality and cost less; cost_ratio_recorded keeps the raw value.

docs/experiments/DETECTION-2026-09-19.md

Split of the 18 skills that are not downgradeable

14 dropped quality, 4 held quality but were not cheaper → 13 dropped quality, 5 held quality but were not cheaper

Recounted from the per-task records with the document's own rule, held means quality held on every comparable task. The document's split does not reproduce from those records, and it names terracotta as a skill that held quality, while terracotta held on 1 of its 2 comparable tasks. The 16 downgradeable skills and their cost ratios are unchanged.

docs/experiments/DETECTION-2026-09-19.md

Tables

Cheaper on a smaller model, per skill

n: Runs on the smaller model behind the row: comparable tasks times three repetitions. 34 rows.

skilldowngradeablequality_held_allcheaperskill_lifts_haikucost_ratiotaskstasks_comparabletasks_heldnevidence_classdatecommit
web-quality-auditYesYesYesYes0.4082113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
unity-script-rolesYesYesYesYes0.4332113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
typescript-serverYesYesYesYes0.5362113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
napkinYesYesYesYes0.5412113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
landing-page-conversion-auditYesYesYesYes0.5732113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
compose-modifier-and-layout-styleYesYesYesYes0.62113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
llm-councilYesYesYesNo0.6232226measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
hermes-agent-skill-authoringYesYesYesYes0.6562113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
catchupYesYesYesYes0.6582113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
cost-efficiency-analyzerYesYesYesYes0.7482113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
eol-messageYesYesYesYes0.7512113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
unity-primetweenYesYesYesYes0.7532113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
optimize-simplicite-logsYesYesYesYes0.7762226measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
geo-fundamentalsYesYesYesYes0.7992113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
caveman-commitYesYesYesYes0.8422113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
research-agentYesYesYesYes0.942113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
offensive-reportingNoNoYesNo0.2162103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
debugging-network-issuesNoNoYesNo0.3762206measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
feedbackNoNoYesYes0.4072103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
product-visionNoNoYesYes0.5442216measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
wiki-lintNoNoYesNo0.5872103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
modeling-product-usage-metricsNoNoYesYes0.6522103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
data-analystNoNoYesNo0.7142216measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
ads-planNoNoYesYes0.8592103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
n8n-error-handlingNoNoYesYes0.9792216measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
triton-kernelNoNoNoYes1.0072103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
event-store-designNoYesNoYes1.022113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
vision-sftNoYesNoYes1.0362113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
latent-briefingNoYesNoYes1.0942113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
dotnet-local-toolsNoYesNoYes1.2562113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
alphagbm-tepper-signalNoNoNoYes1.4662103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
architecture-designerNoYesNoYes1.5242113measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
friendlyNoNoNoYes2.1812103measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
terracottaNoNoNoYes2.5452216measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590

Cheaper on a smaller model, per task

n: Repetitions per arm. 68 rows.

skilltask_idcomparablescore_haiku_skillscore_haiku_nonescore_sonnet_skillquality_heldskill_lifts_haikucost_ratiocost_ratio_recordednevidence_classdatecommit
ads-planfitness-app-media-planYes0.80.61NoYes0.8590.8593measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
ads-planhealthdash-optimizationNo10.8——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
alphagbm-tepper-signaltepper-quality-filter-validationNo0.80——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
alphagbm-tepper-signaltepper-signal-multi-scenarioYes0.811NoNo1.4661.4663measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
architecture-designerecommerce-microservices-designYes10.61YesYes1.5241.5243measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
architecture-designermonolith-to-microservices-reviewNo11——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
catchupcontext-restoration-after-breakYes10.61YesYes0.6580.6583measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
catchupprioritize-work-from-stateNo0.80.8——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
caveman-commitbreaking-api-endpointNo0.750.75——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
caveman-commitperf-query-optYes0.750.250.75YesYes0.8420.8423measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
compose-modifier-and-layout-stylefix-composable-modifier-parameterYes10.751YesYes0.60.63measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
compose-modifier-and-layout-stylerefactor-modifier-chain-styleNo11——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
cost-efficiency-analyzercost-efficiency-current-quarterYes10.41YesYes0.7480.7483measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
cost-efficiency-analyzercost-efficiency-trend-analysisNo0.60.4——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
data-analystcompare-campaign-performanceYes0.60.60.8NoNo0.550.553measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
data-analystexplore-customer-datasetYes0.80.80.8YesNo0.7140.7143measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
debugging-network-issuesisolate-slow-download-throughputYes0.20.40.8NoNo0.3140.3143measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
debugging-network-issuestimeout-root-cause-multi-layerYes00.751NoNo0.3760.3763measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
dotnet-local-toolsfix-broken-local-tools-configNo10.5——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
dotnet-local-toolssetup-local-tools-freshYes10.51YesYes1.2561.2563measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
eol-messageeol-message-feature-replacementYes0.80.60.6YesYes0.7510.7513measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
eol-messageeol-message-graceful-exitNo0.80.2——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
event-store-designevent-store-implementationNo11——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
event-store-designevent-store-system-designYes10.81YesYes1.021.023measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
feedbackextract-quiet-feedbackNo10.5——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
feedbackspecific-feedback-sbiYes001NoNo0.4070.4073measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
friendlydropdown-design-guidelinesYes0.60.40.8NoYes2.1812.1813measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
friendlyimprove-incomplete-button-guidelinesNo10.4——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
geo-fundamentalsaudit-blog-postsYes10.41YesYes0.7990.7993measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
geo-fundamentalsimprove-article-for-ai-citationNo0.80.2——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
hermes-agent-skill-authoringcreate-bundled-skillYes0.80.60.8YesYes0.6560.6563measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
hermes-agent-skill-authoringfix-noncompliant-skillNo0.80.2——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
landing-page-conversion-auditcheckout-bottleneck-diagnosisNo0.80.6——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
landing-page-conversion-auditlanding-page-message-match-auditYes10.41YesYes0.5730.5733measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
latent-briefinglatent-briefing-config-tuningNo11——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
latent-briefinglatent-briefing-designYes10.61YesYes1.0941.0943measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
llm-councilpayment-error-refactorYes0.250.250.25YesNo0.6230.6233measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
llm-councilsearch-filter-planYes0.250.750YesNo0.4610.4613measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
modeling-product-usage-metricsmetrics-map-product-questionsNo0.80——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
modeling-product-usage-metricsretention-mobile-app-configYes0.40.60.8NoNo0.6520.6523measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
n8n-error-handlingscheduled-sync-retry-alertYes0.7501NoYes0.9790.9793measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
n8n-error-handlingwebhook-silent-errorYes111YesNo0.3520.3523measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
napkinactivate-napkin-whiteboardYes0.800.4YesYes0.5410.5413measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
napkincheck-unsaved-napkinNo0.80.4——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
offensive-reportingformat-raw-findingYes000.6NoNo0.2160.2163measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
offensive-reportingwrite-executive-summaryNo0.80.8——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
optimize-simplicite-logsfind-and-analyze-errors-in-logYes10.751YesYes0.7760.7763measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
optimize-simplicite-logsparse-and-filter-simplicite-logYes111YesNo0.6060.6063measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
product-visioncreate-vision-from-scratchYes000.8NoNo0.3870.3873measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
product-visionrefine-flat-visionYes10.80.8YesYes0.5440.5443measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
research-agentauthor-competitor-research-agentYes10.41YesYes0.940.943measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
research-agentauthor-regulation-monitoring-agentNo10.4——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
terracottafix-input-guidelinesYes0.60.40.6YesYes0.8720.8723measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
terracottamodal-guidelinesYes0.60.60.9NoNo2.5452.5453measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
triton-kernelfix-softmax-masking-bugNo10.8——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
triton-kernelfused-vector-op-kernelYes0.80.81NoNo1.0071.0073measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
typescript-serverproduct-inventoryYes10.20.8YesYes0.5360.5363measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
typescript-serverscheduled-notificationsNo0.60.2——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
unity-primetweengenerate-sequence-scriptNo0.60.4——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
unity-primetweengenerate-tween-scriptYes10.81YesYes0.7530.7533measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
unity-script-rolescollectible-system-architectureYes10.61YesYes0.4330.4333measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
unity-script-rolesquest-dialogue-system-architectureNo10.8——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
vision-sftvlm-config-technical-diagramsYes0.800.4YesYes1.0361.0363measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
vision-sftvlm-debug-silent-killersNo11——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
web-quality-auditcomprehensive-website-auditYes10.81YesYes0.4080.4083measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
web-quality-auditlaunch-blocker-auditNo10.8——Yes—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
wiki-lintvault-audit-with-metadataNo0.20.6——No—03measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590
wiki-lintvault-health-checkYes0.511NoNo0.5870.5873measured-local2026-09-2080e4a13fb96ebb96d8efff8690a8c77215657590