Blog

How do you know your AI skill works?

Publish date: November 17, 2026
How do you know your AI skill works?
  • Skills fail quietly. Either the wrong skill gets picked, or the right one gives a bad answer. Nothing crashes.
  • Test the pick first. It's cheap, and every failure points at our wording.
  • Then test the answer, with and without the skill. Our catalog migration skill went from a 38% to an 87% average pass rate.
  • Results wobble. The same test scored 50% on one run and 100% on the next, so a test only fails if two of three runs fail.
  • A pass isn't a guarantee in your workspace. That part is yours, and our guardrails post shows how to handle it.

A skill can fail without an error

In our own builds on commercetools for Builders, the two skills whose names matched the work were never loaded. Two unexpected ones did the heavy lifting instead. Nothing crashed, and the answers looked fine. (We cover this in Building with AI: Best practices for guardrails.)

That raised two questions, one for each side of the table. 

  • For you, the builder: How do you ensure the right skill is used in your workspace?
  • For us, the people who write skills: Are our descriptions clear enough to be picked, and do our instructions actually improve the answer?

The guardrails post answers the first question. This post answers the second.

commercetools for Builders

Build with AI

commercetools for Builders

Describe what you need in plain language and let AI skills handle the commerce foundations.

Start building

Why skills need tests

An AI coding agent does two things with a skill. First, it picks one based on a short description of each skill’s purpose. Then it follows the skill’s instructions.

We ship skills for agents we don’t run, so both steps occur outside our visibility. Linters and link checks can tell us a skill is well-formed. They can’t tell us whether an agent picks it or follows it.

That leaves two quiet failures:

  • The wrong skill gets picked. Rewording a description can steal requests from a neighboring skill, or stop a skill from being picked at all. The more skills exist, the more often this happens.
  • The right skill gives a bad answer. The agent invents a custom connector when a certified one already exists. Or it picks a setting that can’t be undone, without asking first.

Both cost money. A needless custom connector is work someone pays for later. A silently chosen setting is a risk the customer never agreed to.

Code gets tested automatically on every change, so a small edit can’t quietly break something. We wanted the same for our skills, so we built two automated checks.

Check #1: Is the right skill picked?

The question: Are our skill descriptions clear enough that the right skill gets chosen?

How it works:

  • The test agent sees only each skill’s header: Its name, its description and a few other fields. It gets no project files, no extra tools and no other skills.
  • A script, not another AI model, holds the expected answers and marks each pick. The agent only ever sees the prompt, so the answer key can't leak into a pick.

Why we keep it so bare: With everything else removed, a wrong pick can only come from our own wording. That means every recurring failure has a clear fix.

What a test case can say:

  • “This request should pick these skills.” Connecting a payment provider like Stripe requires two of our skills to work together, so one case expects both.
  • “This request must not pick that skill.” Naming forbidden skills tests overlaps directly between neighboring skills.

What it costs: Very little. The check runs only when a skill's header changes, and a full run is far cheaper than Check #2. That's why it runs first.

Check #2: Is the answer what a reviewer would expect?

The question: Once the skill is picked, does it improve the answer?

How it works:

1. One agent answers each test case with the skill loaded.

2. A second agent marks every expectation as pass or fail.

3. A case passes when it meets 80% of its expectations.

Why we run misses again: Results wobble. The same case scored anywhere from 50% to 100% across identical runs, so a single run tells you very little. Now, a case that misses anything is rerun twice, and it fails only if two of its three runs fail.

Does the skill earn its keep? We compare each case against a baseline, the same test run without the skill. On our catalog migration skill, the average pass rate rose from 38% to 87%. One case scored 100% both with and without the skill, so the skill only added cost in that case. That’s useful to know, because it lets us ask whether a skill is worth running and get a real answer.

How to read the gain: It shows how closely answers follow the skill’s rules, not how good an answer is in general. To keep the test fair, the marking agent never sees the skill's text.

One bias to watch: If the marking agent is the same AI model as the agents it grades, it may favor answers written in its own style. Strict, specific marking criteria reduce this, and so does using a different model.

Both checks are advisory for now. They don’t block anyone’s work until they have a stable track record.

What our checks can’t tell you

Check #1 is a controlled test, so a failure tells us exactly what to fix. A pass is weaker evidence.

Real workspaces are crowded with project notes, company tools and other vendors’ skills. In that crowd, a coding tool may not show our skill descriptions to the agent at all. Better wording can’t fix that, because the tool makes that choice. Your own skills also compete with ours, and we don't control those either.

We don’t test a crowded workspace on purpose. Every workspace is cluttered differently, so when a test there fails, we can’t tell which part caused it, and we can’t fix what we can’t explain.

The takeaway: We make sure our descriptions are clear and distinct. What happens in your workspace is where your guardrails come in.

Where your guardrails pick up

Our checks and the five guardrails from our best-practices post cover opposite halves of the same problem.

What our checks cover, or can't The guardrail that covers the rest
We test that the right skill gets picked, but a crowded workspace can hide it Map each task to a named skill before you build (Best practice #3)
We test that the answer follows the skill, but a passing answer isn’t proof something worked Verify outcomes, not confirmations, through a different path than the write (#4)
A script, not an AI, marks the picks, so the answer key can’t leak Put guardrails outside the model, in permissions rather than instructions (#1)
Skills are incomplete, and we can’t find every gap ourselves Keep a corrections file for each skill (#5)

The pattern is the same on both sides. Don’t ask the model to grade itself, and don’t trust a success message. Check the result with something independent.

Before you rely on a skill

What we do:

1. Test that the right skill gets picked whenever a skill’s description changes.

2. Test that the answer is better with the skill than without it.

3. Re-run misses before calling a failure, and tell you what each check leaves open.

What we suggest you do:

1. Map each task to the skill that should handle it before you start building.

2. Verify results through a path independent of the one that wrote them.

3. Keep a corrections file for each skill.

It takes a little longer to set up. But each step is cheaper to do first than to add after something has quietly gone wrong.

Explore the commercetools for Builders documentation — and start building in days. Included in our 60-day free trial. 

commercetools for Builders

Build with AI

commercetools for Builders

Describe what you need in plain language and let AI skills handle the commerce foundations.

Start building
About the authors
Islam Farg
Islam Farg
Staff Engineer, AI Developer Experience, commercetools

You may also be interested in

View all blogs
Building with AI: Best practices for guardrails from real-world development
September 22, 2026
Build with AI

Building with AI: Best practices for guardrails from real-world development

commercetools for Builders: From building faster to lowering the barrier of moving to commercetools
September 29, 2026
Build with AI

commercetools for Builders: From building faster to lowering the barrier of moving to commercetools

commercetools for Builders: From creating digital storefronts to further solving what blocks modernization
August 31, 2026
Build with AI

commercetools for Builders: From creating digital storefronts to further solving what blocks modernization

Get in touch

Stay informed

Receive monthly updates on trends, products and growth insights in your inbox.