How to evaluate agent skills is the missing test loop for reusable instruction folders that agents load on demand. A skill is not a prompt you vibe-check once. It is a packaged capability that fires—or fails—non-deterministically, so you cannot tell a bad skill from a hard task without evals.
That gap is the core of Philipp Schmid’s AI Engineer talk Don’t Ship Skills Without Evals, and of his written companion on testing agent skills. Skills Bench shows skills can lift model performance, human-written skills beat generated ones, and most published skills still ship with no tests at all. This post is the practitioner version: what a skill is, how to build a small harness, and when to delete the skill because the model no longer needs it.
If you are still choosing workflows versus agents, start with how to build effective AI agents. If you need the production stack around scoring and traces, read how to take AI from demo to production. What follows is the layer between those two: proving a single skill is worth the tokens it costs.
What Is an Agent Skill?
An agent skill is a folder—at minimum a SKILL.md file, plus optional scripts, examples, and reference docs—that gives a model the context to finish a task without retraining. The model does not memorize the whole folder on every turn. Skills use progressive disclosure: load a little, then more, only if the task needs it.
- Layer 1 — description. A short trigger. The model uses it to decide whether to open the skill at all. This text is a tax on every call, so keep it tight and specific.
- Layer 2 — body. The instructions: which API, which conventions, what to avoid. Directives beat essays.
- Layer 3 — external references. Deep files the agent reads only for a branch of the work—cloud-specific deploy notes, a proprietary schema, a house style guide.
Skills fire in two ways. The user can invoke them explicitly (a slash command or a named mention). More often the model decides from the description. That second path is where silent failure lives: a vague description never loads, or it loads on every coding prompt.
Why Can't You Vibe-Check Agent Skills?
You cannot vibe-check agent skills because the same prompt can pass once and fail the next run. A single “looks good” demo does not tell you whether the skill helped, the model got lucky, or the task was easier than you thought.
Agents are non-deterministic. Without a scored suite you cannot attribute a miss. Was the skill poorly written? Did it fail to trigger? Did the model ignore a directive? Was the task simply too hard for this model? Vibe checks collapse those four causes into one feeling.
Skills Bench (v1.1) is the uncomfortable evidence. Skills generally help versus no skill. Human-written skills outperform AI-generated ones, and generated skills often sit close enough to the baseline that you would never notice the harm without an ablation. Length is a smell too: once a skill body stretches past a few hundred lines, progressive disclosure stops being cheap—split it.
If you would not merge application code without tests, do not merge a skill without evals. The artifact is different. The risk is the same.
Capability Skills vs Preference Skills: Which Need Evals?
Both need evals. They fail on different clocks. A capability skill teaches the model something it cannot yet do consistently—an internal API, a proprietary format, a new SDK. A preference skill encodes how your team works: Tailwind here, components in this folder, never call the v1 endpoint.
Capability skills can retire. Models improve. When the unloaded score matches the loaded score, the skill is paying a description tax for a capability the base model already has. Preference skills do not retire that way. Foundation models will not learn your private conventions. Those skills are durable infrastructure, and a silent regression in one of them drifts every file the agent touches.
Design the checks to match the type. Capability evals ask “did the outcome work?” Preference evals ask “did it pick our stack, our names, our forbidden paths?” Mixing those questions is how you celebrate a passing run that still shipped the wrong SDK.
How to Build a Skill Eval Harness
A skill eval harness is three boring pieces: a file of test cases, a script that runs the agent, and validators that score the output. You do not need a vendor platform to start. Schmid’s DeepMind-style loop is small enough to live next to the skill folder.
1. Test cases
Put 10–20 prompts in JSON or YAML. Each case needs a user prompt, the language or environment you care about, a should_trigger flag, and the checks that must pass if the skill did its job. Cover happy paths, edge cases, and no-ops. Grade outcomes, not the exact path the agent took—agents find weird routes to the right answer, and you should not fail them for that.
2. Execution
Run the agent the way users run it: CLI, IDE harness, or API. Isolate every trial in a clean workspace. Shared state is how evals cheat—yesterday’s files, a leftover conversation, a skill body sitting on disk from a previous case. Capture the output (and traces if you have them) for scoring.
3. Validators
Start cheap. Regex and Python asserts catch the things that actually matter: the right SDK import, a current model id, no deprecated endpoint, the file that must exist. Use LLM-as-a-judge only when “correct” is a rubric—design quality, tone, whether the architecture matches the house style. Constrain the judge to a typed pass/fail schema. Do not make a second model the default grader for checks a regex can finish in milliseconds.
Because runs jitter, do not trust one trial. Schmid’s rule of thumb is three to six trials per case and a pass-rate distribution, not a victory lap on the best sample. When you change the skill, merge only if the suite improves or you added cases that cover the change. That is eval-driven skill development, not prompt folklore.
How to Stop Skills From Over-Triggering
Stop over-triggering by writing negative test cases—prompts where the skill must stay unloaded—and by tightening the description until those cases pass. A React-component skill that fires on every “build a website” request is not helpful. It is a token leak with extra steps.
Five writing rules that show up in eval scores, not in style guides:
- Write directives, not essays. “Use the Interactions API” changes behavior. “The Interactions API is recommended” often does not.
- Keep the description lean. It is loaded whether the skill is needed or not. Vague keywords (“web development”, “write code”) are how you get false triggers.
- Delete no-ops. Filler like “write high-quality code” costs tokens and moves no metric. If a line would not change a check, cut it.
- Document when not to use it. Everyone writes the happy path. Almost nobody writes the exclusion. Negative examples in the description are load-bearing.
- Test across harnesses. A skill that passes in one IDE can fail in another. If your users live in Cursor, VS Code, and a CLI, the suite has to run in each.
Monday-morning version if you have nothing yet: take the skill your team actually uses. Write five prompts where it should fire and five where it should not. Run both sets with the skill on and off. That ten-prompt grid is already more signal than a week of vibe checks.
When Should You Retire an Agent Skill?
Retire a capability skill when ablation tests—the same cases with the skill loaded and unloaded—show the base model matching the skill-on pass rate. You are then paying description tokens and maintenance for a capability the model absorbed. Delete the skill. Keep the evals. They become regression tests for the next model drop.
Re-run the suite when a new model ships. That is the retirement clock. A skill that was essential on last quarter’s model can be dead weight on this one—or the new model can regress a preference you thought was safe. Ablation is the only honest signal of usefulness. Absolute pass rate without a baseline measures the model, not the skill.
Do not retire preference skills on the same rule. If the unloaded agent stops using Tailwind, or starts calling the v1 endpoint, the skill is still earning its keep. The eval’s job there is to block the merge that “simplified” the file and quietly dropped the constraint.
FAQ: Evaluating Agent Skills
What is a skill eval harness?
A skill eval harness is a small loop that runs fixed prompts through your agent and scores the result. It is a case file, a runner, and validators—regex first, LLM-as-judge only for qualitative checks. Isolation per run is part of the harness, not an optional nicety.
How many test cases do you need to evaluate agent skills?
Start with 10–20 prompts per skill: happy paths, edges, and negatives. Five on / five off is enough to expose a broken trigger. Grow the set from real failures, not from synthetic volume. Repeat each case three to six times so one lucky pass cannot greenlight a merge.
Should you use LLM-as-judge for every skill eval?
No. Use deterministic checks whenever the success criterion is a string, a file, an import, or a model id. Reach for a judged rubric when the skill encodes taste—layout, naming, architecture—and pin the judge to a schema so the score is parseable.
Do AI-generated skills need evals more than human-written ones?
Yes, in practice. Generated skills are faster to draft and weaker on average; without ablation you cannot see whether they help, hurt, or no-op. Human-written skills still need the same suite. Quality is a measured delta, not an authorship badge.
The CSY Takeaway
Do not ship skills without evals. Write the description as a trigger, keep the body as directives, score happy paths and negatives, and run ablation until you know the skill is the reason the score moved. When the model catches up, retire the capability and keep the tests.
At CSY SimplifAI Solution, we help startups turn agent skills into shipping systems—eval harnesses next to the skill folder, orchestration you can replay, and a merge bar that is a number instead of a vibe. If you are past the demo and need agents whose skills you can prove, let’s design the smallest loop that still holds up.
