What Claude Code plugin evals are
Claude Code plugin evals are automated tests for how a plugin changes Claude's behavior. You write a set of cases, each one a realistic prompt plus one or more graders, and the claude plugin eval command runs them and scores the results. A grader is a pass or fail check on what Claude produced, such as a regex over the reply, whether a particular tool was called, or a rubric a second model judges the reply against.
This is different from claude plugin validate. Validate checks the plugin's files for syntax and schema errors. Evals check behavior: does Claude pick up your skill when someone asks for the thing it does, and is the answer better than what Claude gives with no plugin at all. For an internal marketplace, that second question is the one nobody can answer today without guessing.
What you need before you run an eval
- Claude Code v2.1.269 or later. Check with claude --version and upgrade with claude update.
- Git 2.31 or later if git is installed. With an older git, the command stops before running any case. Without git, it runs normally.
- A plugin directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin.
- The same sign-in and model provider your normal sessions use. Eval runs, judge graders and the init interview all call the model with your credentials, so they count against your plan's usage limits or your API bill.
The command itself runs in a terminal. If you work in the Claude Code desktop app, you do not have to leave it: open a session at the plugin's root folder and ask Claude to run claude plugin eval init for you. Claude runs the command and asks you the setup questions in that conversation.
How an eval run scores a plugin
For each run of a case, Claude Code starts a fresh, isolated, non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case's turn or time limit. Each grader then checks the final reply, the transcript, or a file Claude created.
- Each case runs three times by default, because one run of a non-deterministic agent tells you very little.
- A run's score is the fraction of its graders that passed, weighted if you set weights. A case's score is the mean across its runs.
- A case passes when its score meets --threshold, which defaults to 1.0. That default means one imperfect case fails the whole command, so set a threshold on purpose.
Then comes the part that makes this useful for a team. The same runs repeat with no plugin loaded, and the summary shows WITH, W/OUT and the difference between them. A positive difference means the plugin raised the score. If a case scores 1.0 both with and without the plugin, the plugin is not what made it pass, and you are maintaining something Claude already does on its own.
Create your first suite with claude plugin eval init
- From the plugin root, run claude plugin eval init. Answer y if it asks to trust the directory.
- A Claude Code session opens. Claude reads your plugin, asks what a good result looks like, proposes prompts that should and should not trigger the plugin, designs graders for each, trial-runs them once, and writes one case folder per prompt under evals/.
- Exit that session, then run claude plugin eval . to run every case.
- Read the summary table, then open the report to see each grader's verdict and explanation for every run.
Each case is a folder with a prompt.md, a graders folder and optionally a case.yaml. The prompt.md frontmatter sets the run limits, such as max_turns (default 10) and timeout_seconds (default 300), and the allowed_tools the case may use. The body is the message Claude receives, written the way a person would type it rather than naming the skill.
The most common first finding is a difference near zero with the skill-invoked grader failing. That means Claude is not choosing your skill on natural phrasing. Fix the skill's description, run the suite again and compare. That loop alone justifies setting this up, because a skill nobody triggers is the quietest way an internal plugin fails.
Pick graders that give a stable signal
There are six grader types. Four are computed from the transcript and files and cost nothing. Two call a judge model and add to the bill. There are no custom-code graders.
The six grader types
| Grader | Passes when | Costs a model call |
|---|---|---|
| regex | A JavaScript regex is found in the target, or is absent with match: not_contains, or appears exactly N times | No |
| tool_used | A tool was called between min and max times, optionally with input matching a regex | No |
| tool_order | One tool call came before another | No |
| file_exists | A file Claude created during the run matches a path glob | No |
| llm | A judge model votes PASS on your rubric in at least two of three votes | Yes |
| baseline | A judge finds the run at least as good as a reference transcript | Yes |
My rule for a team suite: every case gets one grader on the result and one on the steps. The result grader checks the final message or a file the plugin produced. The steps grader, usually tool_used or tool_order, checks that your plugin is what produced it. Together they tell you whether the answer was right and whether the plugin deserves the credit.
- Grade long output, like a generated file, with a regex over the file's contents. An llm judge drifts more the longer the text it reads.
- Keep llm rubrics short and written as concrete PASS and FAIL conditions.
- If the skill-invoked grader passes but the difference is negative, suspect the judge before the plugin. Re-run with --judge-model sonnet and tighten the rubric so formatting does not decide the verdict.
- To check a build or test inside the run, have the prompt ask Claude to run it and write the outcome to a file, then grade that file.
What a run can and cannot touch
Each run gets a temporary home directory, working directory and Claude Code configuration. Your user settings, hooks, CLAUDE.md files, MCP servers, other plugins, memory and skills are all absent, so ship anything a case depends on inside the plugin under test. The agent under test cannot read the eval folder, so Claude never sees the case's graders or sibling cases.
- Only read-only tools run by default. Bash, Write, Edit, WebFetch and WebSearch need an explicit --allow-tools grant, and a granted shell runs inside Claude Code's OS-level sandbox.
- Your plugin's real MCP servers do not start. You answer their tools from mock files under evals/mocks/, or opt in to real servers with --allow-real-servers or --mocks off.
- Managed settings an administrator deployed still apply inside a run, so a managed laptop and an unmanaged one can score the same suite differently.
Gate plugin changes in CI
For a company marketplace, the value of evals is a required check on every pull request that touches a plugin. Run the suite with --json, fail the build on the exit code, and pin everything that could move the score for reasons other than your change.
CI flags and why each one matters
| Flag | Why |
|---|---|
| --trust-plugin | The job never waits at the first-run trust prompt. Without it, a run that cannot ask is refused with exit 1 |
| --json results.json | Writes the versioned result document for archiving and scripts |
| --threshold 0.8 | Sets the pass bar on purpose instead of the default 1.0 |
| --model and --judge-model | Pins both models so a model rollout is not mistaken for a plugin regression |
| --no-publish | Keeps the HTML report local to the job |
| --max-cost-usd | A ceiling on the list-price cost estimate. Once reached, no new run starts and the command exits 2 with partial results |
Exit code 0 means every case met the threshold. Exit 1 means a case fell short, a case file failed to load, or a run ended on an API error such as a usage limit. Exit 2 is a partial run from the cost ceiling or a rejected credential. The difference against the no-plugin baseline is reported but never changes the exit code, so if you want to block a change that makes the plugin pointless, read cases[].aggregates.delta from the JSON in your own script.
To keep the bill predictable, give the every-change suite only free graders, use --ablation none where you do not need the baseline, and run the full two-arm suite with judge graders on a schedule or before a release. Leave any result with partial set to true out of trend charts. The headless mode guide covers the credentials a CI runner needs.
The CCC Plugin Eval Gate
Here is the rollout I would use for a team marketplace. I call it the CCC Plugin Eval Gate: no plugin enters or changes in the marketplace without a suite that proves it beats no plugin.
- Every plugin ships with an evals/ folder in the same pull request. No suite, no listing.
- Each suite has at least one case that should trigger the plugin and one that should not, so you catch a skill that fires on everything.
- Every case pairs a result grader with a steps grader. Judge graders only where a regex cannot decide.
- The PR check runs free graders with --ablation none, pinned models, a threshold and a cost ceiling.
- Before a release, run the full suite with the baseline. A plugin whose mean difference is at or below zero gets fixed or removed, not promoted.
- After release, compare against real adoption in plugin usage, and fold what you learn into the team cost picture.
The baseline is the part I would never skip. Internal plugins tend to accumulate because each one sounded useful when someone wrote it. A measured difference against no plugin is the only honest answer to whether it still pulls its weight, especially after a new model ships and Claude starts doing on its own what the plugin used to teach it. For a refresher on what a plugin can contain, see Claude Code plugins explained.
Short, practical drops on skills, MCP, agents, prompts, and more. No spam, unsubscribe anytime.
Frequently asked questions
How do I test a Claude Code plugin?
Run claude plugin eval init from the plugin root to have Claude write a suite of cases and graders, then run claude plugin eval . to score it. Each case runs with your plugin and again without it, so you see what the plugin added.
Do Claude Code plugin evals cost money?
Yes. Every eval run and every judge grader is a real model call on your account, counted against your plan's usage limits or your API bill. Use free graders, --ablation none and --max-cost-usd to keep it under control.
What is the difference between claude plugin validate and claude plugin eval?
Validate checks a plugin's files for syntax and schema errors. Eval checks behavior: whether Claude uses the plugin on realistic prompts and whether the result beats running with no plugin.
What does the no-plugin baseline show?
Each case is repeated with no plugin loaded. The difference between the with-plugin and without-plugin scores is what the plugin contributed. A case that scores the same both ways is not being helped by the plugin.
Can I run Claude Code plugin evals in CI?
Yes. Run the suite with --trust-plugin, --json, a --threshold, pinned --model and --judge-model, --no-publish and --max-cost-usd, and fail the build on a non-zero exit code. The runner needs a Claude Code install and credentials in its environment.
Why does my plugin eval show a difference near zero?
Most often Claude is not choosing your skill on natural phrasing, which shows up as the skill-invoked grader failing. Rewrite the skill's description and run the suite again. If the plugin never loaded, add plugins: ["../.."] to the case.
Last reviewed by David Iya on November 21, 2026


