Claude Code Plugin Eval: The New Way to Prove Your Skills Actually Work

Key Takeaways

• Claude Code 2.1.269 adds claude plugin eval, a native evaluation command for running plugin test suites and producing scored, reproducible JSON and HTML reports. ([GitHub][1])
• The most important idea is with-plugin versus without-plugin evaluation. A high score is not enough; the useful number is the improvement the plugin creates over the same task without it.
• The announcement demonstrates this as an ablation test: the same cases are run with the plugin or Skill enabled, then again without it, so developers can measure the actual contribution rather than relying on subjective impressions. ([sotwe][2])
• This is especially valuable after model upgrades. A Skill that once added substantial value may become redundant when the base model improves.
• Strong eval suites should test outcomes, negative cases, trigger precision, regressions, cost, and variance, not merely whether a Skill was invoked.
• For proprietary code or prompts, verify the options in your installed build before running large suites. Current build-inspection references describe local JSON/HTML reports, with --no-publish available to keep reports local. ([GitHub][3])

Claude Code plugins and Skills have become increasingly capable, but capability created a measurement problem: how can a developer prove that a plugin actually makes Claude better?

Anthropic's new claude plugin eval command is a direct answer. Instead of judging a Skill from a few successful demos, developers can create a repeatable benchmark, run the same tasks with and without the extension, score the results, and compare the difference.

That changes plugin development from "this prompt seems better" to "this extension improved this benchmark by a measurable amount."

What Is claude plugin eval?

Anthropic added claude plugin eval in Claude Code v2.1.269, released on September 11, 2026. The release notes describe it as a way to run a plugin's eval suite against Claude Code and receive scored, reproducible results in JSON and HTML. ([GitHub][1])

The official Claude Developers announcement adds the most important practical detail: developers can create test cases, run a plugin or Skill against those cases, score the runs, and then repeat the cases without the plugin to see the difference. ([sotwe][2])

Conceptually, the workflow is:

Test case
   |
   +--> Claude Code WITH plugin ----> score
   |
   +--> Claude Code WITHOUT plugin -> score
                                  |
                                  v
                         delta = with - without

This is an ablation test. The extension is removed while the rest of the task remains as comparable as possible.

That distinction matters because modern coding models are already strong. A plugin can produce an excellent result and still contribute almost nothing if Claude would have produced the same result without it.

Why the Delta Matters More Than the Raw Score

The launch screenshot makes the idea concrete.

One example case shows approximately:

WITH     0.92
W/OUT    0.17
DELTA   +0.75

That is a strong signal. The plugin appears to transform a weak baseline into a high-quality result.

Another type of case can look like:

WITH     1.00
W/OUT    1.00
DELTA    0.00

The second result is not necessarily bad. It means the task is already handled well by the base system, so the plugin may be unnecessary for that case.

This leads to a better way to interpret eval results:

A plugin should therefore be evaluated on incremental value, not on whether its best demo looks impressive.

Why This Is a Big Change for Claude Code Skills

Claude Code plugins can package Skills, agents, hooks, MCP servers, LSP integrations, and other reusable behaviors. Anthropic's plugin documentation positions plugins as the shareable, versioned format for distributing Claude Code customizations across projects and teams. ([Claude][4])

That makes testing increasingly important.

Traditional software already has familiar quality gates:

code
-> unit tests
-> integration tests
-> regression tests
-> CI
-> release

Agentic software needs a similar layer:

Skill or plugin
-> realistic prompts
-> outcome graders
-> baseline comparison
-> repeated runs
-> regression threshold
-> release

The difference is that LLM behavior is probabilistic. A single successful run does not prove reliability.

A serious eval suite therefore measures both quality and stability.

A Practical Mental Model: Tests for Behavior, Not Text

The wrong way to test an AI Skill is to expect one exact answer.

For example, this is usually too brittle:

Expected response:
"The vulnerability is SQL injection on line 42."

Claude may identify the same problem using different wording, line references, or remediation details.

A better evaluation asks whether the important outcome happened:

• Was the vulnerability found?
• Was its severity reasonable?
• Was the vulnerable data flow explained?
• Was a concrete remediation proposed?
• Did the plugin avoid unrelated changes?
• Did it use required tools when the task genuinely required them?

This is closer to testing a competent engineer than snapshot-testing a string.

How to Design a Useful Eval Suite

A strong suite should cover more than the happy path.

A practical starting matrix is:

The inclusion of should-not-fire cases is particularly important.

A Skill that activates on every vaguely related request may look powerful in demos while degrading the overall product through unnecessary context, token usage, tool calls, or over-constrained answers.

Current Eval Suite Structure

Anthropic's public v2.1.269 release note confirms the command and report formats, while current build-inspection references describe the runner as accepting eval cases such as case.yaml or a prose-oriented prompt.md plus grader files under graders/. ([GitHub][5])

A representative layout looks like:

my-plugin/
├── .claude-plugin/
│   └── plugin.json
├── skills/
│   └── code-review/
│       └── SKILL.md
└── evals/
    ├── 01-security-review/
    │   ├── prompt.md
    │   └── graders/
    │       └── criteria.md
    ├── 02-simple-review/
    │   ├── prompt.md
    │   └── graders/
    │       └── criteria.md
    └── 03-should-not-fire/
        ├── prompt.md
        └── graders/
            └── criteria.md

Because this surface is new and parts of the runner have previously been described as early access in build-inspection material, treat claude plugin eval --help in the installed Claude Code version as the authoritative interface for available flags. ([GitHub][3])

A Sensible First Run

Start small.

The official release explicitly points users to:

claude plugin eval --help

Then run the smallest useful suite before paying for a large benchmark.

Current runner references show workflows based on commands such as:

claude plugin eval .

For builds that expose the corresponding options, a low-cost pilot can use a single repetition with a with-versus-without comparison:

claude plugin eval . --runs 1 --ablation with-without --no-scaffold --no-publish

The important caveat is version sensitivity: confirm every option with --help before copying a command into CI. Current build-inspection references describe these flags, but the public release note only guarantees the top-level eval command and its JSON/HTML reporting behavior. ([GitHub][6])

What Should the Graders Measure?

The best eval suites combine several types of checks.

Deterministic checks

Use deterministic checks for facts that should not require another model to interpret:

• A required file exists.
• A forbidden file was not modified.
• A particular tool was or was not called.
• A required command completed successfully.
• A result matches a stable pattern.
• A generated artifact parses as valid JSON, YAML, or another schema.

These checks are cheap and reproducible.

Semantic outcome checks

Use model-based grading when the quality criterion is genuinely semantic:

• Is the explanation technically correct?
• Did the review find the important security issue?
• Is the plan actionable rather than generic?
• Did the response preserve the user's constraints?
• Is the generated migration safe and complete?

Model-based graders should have narrow, explicit rubrics. A vague instruction such as "score quality" invites inconsistent judging.

Side-channel checks

Quality is not the only outcome.

A plugin can improve a score while making the workflow impractically expensive.

Track:

• total tokens;
• wall-clock latency;
• number of turns;
• number of tool calls;
• API or model cost;
• files changed;
• retries and failures.

The useful question is not simply "Did quality improve?"

It is:

"Did quality improve enough to justify the added complexity and cost?"

Do Not Confuse Invocation With Success

A common testing mistake is to grade the plugin based on whether the Skill was triggered.

That only proves routing.

Consider this sequence:

Prompt -> Skill invoked -> wrong answer

The trigger worked, but the plugin failed.

A mature suite separates at least two questions:

1. Routing: Did Claude invoke the Skill when it should, and avoid it when it should not?
2. Outcome: Did the final result actually improve?

This distinction is also reflected in Anthropic's current Skill guidance, which recommends evaluating both whether a Skill is invoked on appropriate prompts and whether the resulting output meets expectations. ([GitHub][7])

claude plugin eval vs the Skill Creator Evaluation Workflow

Claude Code already has a Skill evaluation workflow through Anthropic's skill-creator plugin.

Current Claude Code Skill documentation describes skill-creator as supporting:

• test cases stored in evals/evals.json;
• isolated runs;
• assertion grading;
• with-Skill versus without-Skill benchmarks;
• blind A/B comparison between Skill versions;
• trigger-description tuning;
• HTML review interfaces. ([GitHub][7])

The new native command should be viewed as a complementary layer rather than simply "Skill Creator with a new name."

For Skill authors, skill-creator remains useful for authoring and refinement. For plugin maintainers, claude plugin eval provides a clearer path toward repeatable release testing.

The Most Valuable Use Case: Model-Upgrade Regression Testing

AI extensions have an unusual maintenance problem: the platform beneath them keeps getting smarter.

Suppose a Skill originally produces:

WITH     0.88
W/OUT    0.42
DELTA   +0.46

After a major model upgrade, the same benchmark becomes:

WITH     0.90
W/OUT    0.86
DELTA   +0.04

The Skill did not necessarily get worse.

The base model got better.

That creates several possible actions:

• remove instructions that are now redundant;
• shorten the Skill to reduce context cost;
• focus the Skill on the remaining hard cases;
• replace prompt instructions with deterministic tooling;
• retire the Skill entirely if the measurable benefit has disappeared.

This is a major advantage of keeping a stable eval suite: it reveals when an extension has become obsolete.

How to Use Plugin Evals in CI Without Fooling Yourself

It is tempting to add an eval command to GitHub Actions immediately.

That is useful only after the benchmark itself is trustworthy.

A safer workflow is:

1. Define representative cases
2. Calibrate graders manually
3. Run a cheap pilot
4. Inspect failures
5. Repeat enough times to estimate variance
6. Pin the tested model where possible
7. Establish a baseline
8. Add a release threshold
9. Revisit the suite when the model changes

Do not set a CI threshold from one lucky run.

For example, a rule such as:

mean score >= 0.85

can be misleading if the baseline is already 0.84.

A better release policy evaluates at least three dimensions:

absolute quality
+
with-vs-without delta
+
variance / failure rate

For mature plugins, cost can become a fourth gate.

Why Multiple Runs Matter

LLM outputs are non-deterministic.

Even with the same prompt, model, repository, and Skill, results can vary.

If one case produces:

run 1: pass
run 2: fail
run 3: pass
run 4: pass
run 5: fail

the useful signal is not "it passed once."

The useful signal is that the behavior may only be reliable around 60% of the time.

Repeated runs help distinguish:

• a consistently strong Skill;
• a brittle Skill that occasionally looks excellent;
• a grader that is itself unstable;
• a task whose success depends too heavily on stochastic tool choices.

For development, one run may be enough for a quick smoke test. For release decisions, repeated runs are much more informative.

Cost Can Grow Faster Than Expected

A with-versus-without benchmark doubles the execution arms.

For example:

10 cases
x 3 runs
x 2 arms
= 60 agent executions

If semantic graders require additional model calls, the total can increase further.

The launch screenshot itself includes a cost column, which is an important design signal: evaluation quality has to be balanced against evaluation expense.

A practical strategy is:

• During development: run one repetition on a small subset.
• Before merge: run the most important regression cases.
• Before release: run the full suite with multiple repetitions.
• After a major model change: rerun the baseline and compare deltas.

This produces better economics than treating every edit as a full benchmark.

Privacy and Report Publishing

Evaluation cases often contain the most sensitive material in a plugin project:

• private prompts;
• internal repositories;
• security test cases;
• unreleased product requirements;
• customer workflows;
• proprietary scoring rubrics.

Current build-inspection references describe local output such as aggregate-result.json and report.html, and also describe a --no-publish option for keeping reports local. ([Wehub][8])

For sensitive suites, check the behavior of the installed version before running the benchmark and prefer local-only reporting where available.

The safe principle is simple:

Treat eval datasets like production test data, not disposable prompts.

Common Mistakes

1. Testing only easy cases

If Claude already scores near-perfectly without the plugin, the test provides little evidence of value.

Include tasks where the plugin is expected to create a real advantage.

2. Optimizing the Skill to the benchmark

Once authors know every eval case, it is easy to overfit.

Maintain a hidden or rotating holdout set for important plugins.

3. Using vague graders

"Good answer" is not a rubric.

Prefer criteria such as:

• identifies the root cause;
• names the affected component;
• proposes a fix that preserves backward compatibility;
• does not modify unrelated files.

4. Ignoring negative delta

A negative result is valuable information.

Do not hide it by reporting only average scores.

5. Comparing across changing models without recording the model

If the base model changes, the baseline changes.

Version the model, the plugin, the eval suite, and ideally the judge configuration alongside each result.

6. Treating trigger rate as product quality

A Skill that fires frequently is not necessarily useful.

Measure whether it improves outcomes.

7. Running expensive suites too often

Separate smoke tests from release benchmarks.

The first should be fast. The second should be statistically meaningful.

A Recommended Evaluation Scorecard

For a serious public plugin, a compact scorecard could look like this:

Publishing even part of this scorecard would be more informative than marketing claims such as "improves code quality" or "makes Claude more accurate."

What This Could Mean for the Plugin Ecosystem

Anthropic has already positioned plugins as a standard way to package and share Claude Code customizations. The company describes plugins as reusable bundles that can include Skills, agents, hooks, MCP servers, and other extensions, distributed across teams or marketplaces. ([Claude][4])

Native eval infrastructure creates the foundation for a more mature ecosystem.

Future plugin quality signals could reasonably include:

• tested Claude Code version;
• tested model;
• number of eval cases;
• with-plugin score;
• without-plugin baseline;
• mean delta;
• reliability across repeated runs;
• benchmark cost;
• date of last regression run.

That would make plugin discovery more useful because users could evaluate not only what a plugin claims to do, but what measurable value it adds.

This marketplace direction is an inference, not an announced Anthropic policy. But the technical prerequisite now exists: a reproducible way to compare behavior with and without an extension.

Suggested Visuals for Publication

• Hero: use the official Claude Developers "Run evals on your plugin" score-table image from the announcement. It immediately communicates WITH, W/OUT, delta, run count, and cost.
• Plugin context: Anthropic's official "Customize Claude Code with plugins" article contains a product screenshot of the Claude Code plugin manager, useful for explaining where plugins fit in the ecosystem. ([Claude][9])
• Skill context: Anthropic's "Introducing Agent Skills" announcement includes the Skills interface and is a useful secondary visual when explaining the difference between Skills and plugin-level evaluation.
• Optional technical visual: create a simple original diagram showing same prompt -> WITH plugin / WITHOUT plugin -> grader -> delta. This is more reusable than another terminal screenshot and clearly explains the ablation concept.

Conclusion

claude plugin eval is important because it changes the question plugin authors should ask.

The old question was:

"Does this Skill produce a good result?"

The better question is:

"Does this Skill produce a meaningfully better result than Claude would produce without it?"

Claude Code v2.1.269 now provides a native path toward answering that question with scored, reproducible reports. ([GitHub][1])

For plugin and Skill authors, the most effective next step is straightforward: start with five to ten realistic cases, include at least one should-not-fire case, run a with-versus-without baseline, inspect the delta, and keep the suite as a regression benchmark.

If an extension consistently creates a large positive delta, the eval demonstrates its value. If the delta approaches zero as Claude improves, that is equally useful evidence: the plugin can be simplified, refocused, or retired.

That is the shift from prompt craftsmanship to measurable agent engineering.