Claude Code Plugin Evals: Test a Plugin Before Your Org Ships It

David IyaDavid Iya November 21, 2026 9 min read
A row of five brass pressure gauges on a dark lab bench beside a wooden rack of empty glass test tubes, one gauge lit by a single desk lamp
Original image, Claude Code Club

What Claude Code plugin evals are

Claude Code plugin evals are automated tests for how a plugin changes Claude's behavior. You write a set of cases, each one a realistic prompt plus one or more graders, and the claude plugin eval command runs them and scores the results. A grader is a pass or fail check on what Claude produced, such as a regex over the reply, whether a particular tool was called, or a rubric a second model judges the reply against.

This is different from claude plugin validate. Validate checks the plugin's files for syntax and schema errors. Evals check behavior: does Claude pick up your skill when someone asks for the thing it does, and is the answer better than what Claude gives with no plugin at all. For an internal marketplace, that second question is the one nobody can answer today without guessing.

What you need before you run an eval

  • Claude Code v2.1.269 or later. Check with claude --version and upgrade with claude update.
  • Git 2.31 or later if git is installed. With an older git, the command stops before running any case. Without git, it runs normally.
  • A plugin directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin.
  • The same sign-in and model provider your normal sessions use. Eval runs, judge graders and the init interview all call the model with your credentials, so they count against your plan's usage limits or your API bill.

The command itself runs in a terminal. If you work in the Claude Code desktop app, you do not have to leave it: open a session at the plugin's root folder and ask Claude to run claude plugin eval init for you. Claude runs the command and asks you the setup questions in that conversation.

How an eval run scores a plugin

For each run of a case, Claude Code starts a fresh, isolated, non-interactive session with only your plugin loaded, sends the prompt, and lets Claude work until it finishes or hits the case's turn or time limit. Each grader then checks the final reply, the transcript, or a file Claude created.

  • Each case runs three times by default, because one run of a non-deterministic agent tells you very little.
  • A run's score is the fraction of its graders that passed, weighted if you set weights. A case's score is the mean across its runs.
  • A case passes when its score meets --threshold, which defaults to 1.0. That default means one imperfect case fails the whole command, so set a threshold on purpose.

Then comes the part that makes this useful for a team. The same runs repeat with no plugin loaded, and the summary shows WITH, W/OUT and the difference between them. A positive difference means the plugin raised the score. If a case scores 1.0 both with and without the plugin, the plugin is not what made it pass, and you are maintaining something Claude already does on its own.

Create your first suite with claude plugin eval init

  1. From the plugin root, run claude plugin eval init. Answer y if it asks to trust the directory.
  2. A Claude Code session opens. Claude reads your plugin, asks what a good result looks like, proposes prompts that should and should not trigger the plugin, designs graders for each, trial-runs them once, and writes one case folder per prompt under evals/.
  3. Exit that session, then run claude plugin eval . to run every case.
  4. Read the summary table, then open the report to see each grader's verdict and explanation for every run.

Each case is a folder with a prompt.md, a graders folder and optionally a case.yaml. The prompt.md frontmatter sets the run limits, such as max_turns (default 10) and timeout_seconds (default 300), and the allowed_tools the case may use. The body is the message Claude receives, written the way a person would type it rather than naming the skill.

The most common first finding is a difference near zero with the skill-invoked grader failing. That means Claude is not choosing your skill on natural phrasing. Fix the skill's description, run the suite again and compare. That loop alone justifies setting this up, because a skill nobody triggers is the quietest way an internal plugin fails.

Pick graders that give a stable signal

There are six grader types. Four are computed from the transcript and files and cost nothing. Two call a judge model and add to the bill. There are no custom-code graders.

The six grader types

GraderPasses whenCosts a model call
regexA JavaScript regex is found in the target, or is absent with match: not_contains, or appears exactly N timesNo
tool_usedA tool was called between min and max times, optionally with input matching a regexNo
tool_orderOne tool call came before anotherNo
file_existsA file Claude created during the run matches a path globNo
llmA judge model votes PASS on your rubric in at least two of three votesYes
baselineA judge finds the run at least as good as a reference transcriptYes

My rule for a team suite: every case gets one grader on the result and one on the steps. The result grader checks the final message or a file the plugin produced. The steps grader, usually tool_used or tool_order, checks that your plugin is what produced it. Together they tell you whether the answer was right and whether the plugin deserves the credit.

  • Grade long output, like a generated file, with a regex over the file's contents. An llm judge drifts more the longer the text it reads.
  • Keep llm rubrics short and written as concrete PASS and FAIL conditions.
  • If the skill-invoked grader passes but the difference is negative, suspect the judge before the plugin. Re-run with --judge-model sonnet and tighten the rubric so formatting does not decide the verdict.
  • To check a build or test inside the run, have the prompt ask Claude to run it and write the outcome to a file, then grade that file.

What a run can and cannot touch

Each run gets a temporary home directory, working directory and Claude Code configuration. Your user settings, hooks, CLAUDE.md files, MCP servers, other plugins, memory and skills are all absent, so ship anything a case depends on inside the plugin under test. The agent under test cannot read the eval folder, so Claude never sees the case's graders or sibling cases.

  • Only read-only tools run by default. Bash, Write, Edit, WebFetch and WebSearch need an explicit --allow-tools grant, and a granted shell runs inside Claude Code's OS-level sandbox.
  • Your plugin's real MCP servers do not start. You answer their tools from mock files under evals/mocks/, or opt in to real servers with --allow-real-servers or --mocks off.
  • Managed settings an administrator deployed still apply inside a run, so a managed laptop and an unmanaged one can score the same suite differently.

Gate plugin changes in CI

For a company marketplace, the value of evals is a required check on every pull request that touches a plugin. Run the suite with --json, fail the build on the exit code, and pin everything that could move the score for reasons other than your change.

CI flags and why each one matters

FlagWhy
--trust-pluginThe job never waits at the first-run trust prompt. Without it, a run that cannot ask is refused with exit 1
--json results.jsonWrites the versioned result document for archiving and scripts
--threshold 0.8Sets the pass bar on purpose instead of the default 1.0
--model and --judge-modelPins both models so a model rollout is not mistaken for a plugin regression
--no-publishKeeps the HTML report local to the job
--max-cost-usdA ceiling on the list-price cost estimate. Once reached, no new run starts and the command exits 2 with partial results

Exit code 0 means every case met the threshold. Exit 1 means a case fell short, a case file failed to load, or a run ended on an API error such as a usage limit. Exit 2 is a partial run from the cost ceiling or a rejected credential. The difference against the no-plugin baseline is reported but never changes the exit code, so if you want to block a change that makes the plugin pointless, read cases[].aggregates.delta from the JSON in your own script.

To keep the bill predictable, give the every-change suite only free graders, use --ablation none where you do not need the baseline, and run the full two-arm suite with judge graders on a schedule or before a release. Leave any result with partial set to true out of trend charts. The headless mode guide covers the credentials a CI runner needs.

The CCC Plugin Eval Gate

Here is the rollout I would use for a team marketplace. I call it the CCC Plugin Eval Gate: no plugin enters or changes in the marketplace without a suite that proves it beats no plugin.

  1. Every plugin ships with an evals/ folder in the same pull request. No suite, no listing.
  2. Each suite has at least one case that should trigger the plugin and one that should not, so you catch a skill that fires on everything.
  3. Every case pairs a result grader with a steps grader. Judge graders only where a regex cannot decide.
  4. The PR check runs free graders with --ablation none, pinned models, a threshold and a cost ceiling.
  5. Before a release, run the full suite with the baseline. A plugin whose mean difference is at or below zero gets fixed or removed, not promoted.
  6. After release, compare against real adoption in plugin usage, and fold what you learn into the team cost picture.

The baseline is the part I would never skip. Internal plugins tend to accumulate because each one sounded useful when someone wrote it. A measured difference against no plugin is the only honest answer to whether it still pulls its weight, especially after a new model ships and Claude starts doing on its own what the plugin used to teach it. For a refresher on what a plugin can contain, see Claude Code plugins explained.

Free Claude Code drops, straight to your inbox

Short, practical drops on skills, MCP, agents, prompts, and more. No spam, unsubscribe anytime.

Frequently asked questions

How do I test a Claude Code plugin?

Run claude plugin eval init from the plugin root to have Claude write a suite of cases and graders, then run claude plugin eval . to score it. Each case runs with your plugin and again without it, so you see what the plugin added.

Do Claude Code plugin evals cost money?

Yes. Every eval run and every judge grader is a real model call on your account, counted against your plan's usage limits or your API bill. Use free graders, --ablation none and --max-cost-usd to keep it under control.

What is the difference between claude plugin validate and claude plugin eval?

Validate checks a plugin's files for syntax and schema errors. Eval checks behavior: whether Claude uses the plugin on realistic prompts and whether the result beats running with no plugin.

What does the no-plugin baseline show?

Each case is repeated with no plugin loaded. The difference between the with-plugin and without-plugin scores is what the plugin contributed. A case that scores the same both ways is not being helped by the plugin.

Can I run Claude Code plugin evals in CI?

Yes. Run the suite with --trust-plugin, --json, a --threshold, pinned --model and --judge-model, --no-publish and --max-cost-usd, and fail the build on a non-zero exit code. The runner needs a Claude Code install and credentials in its environment.

Why does my plugin eval show a difference near zero?

Most often Claude is not choosing your skill on natural phrasing, which shows up as the skill-invoked grader failing. Rewrite the skill's description and run the suite again. If the plugin never loaded, add plugins: ["../.."] to the case.

Last reviewed by David Iya on November 21, 2026

David Iya

Written by

David Iya

Forbes 30 Under 30 · Y Combinator

Keep reading

Claude CodeExtensions

What is the difference between skills, MCP, subagents and hooks?

A skill is a saved set of instructions, MCP connects Claude Code to other apps, a subagent is a helper that does a side job in its own workspace, and a hook is an automatic rule that runs every time. Here is each one explained with a plain analogy, a comparison table, a quick way to decide which you need, and how they work together.

Duncan Rogoff 9 min
Read article
Claude CodeGetting Started

How long does it take to learn Claude Code?

You can get Claude Code working and build your first real thing in a few evenings, and nobody can promise you a number beyond that. Here is a first-week plan built for someone with a full-time job: just over three hours total, split into small blocks, with one finished result at the end.

Duncan Rogoff 9 min
Read article

Ready to build it yourself?

Join Claude Code Club, the #1 community for learning claude code, for $9/month.

← Back to the blog