Stay informed
Receive monthly updates on trends, products and growth insights in your inbox.

In our own builds on commercetools for Builders, the two skills whose names matched the work were never loaded. Two unexpected ones did the heavy lifting instead. Nothing crashed, and the answers looked fine. (We cover this in Building with AI: Best practices for guardrails.)
That raised two questions, one for each side of the table.
The guardrails post answers the first question. This post answers the second.

Build with AI
commercetools for Builders
Describe what you need in plain language and let AI skills handle the commerce foundations.
An AI coding agent does two things with a skill. First, it picks one based on a short description of each skill’s purpose. Then it follows the skill’s instructions.
We ship skills for agents we don’t run, so both steps occur outside our visibility. Linters and link checks can tell us a skill is well-formed. They can’t tell us whether an agent picks it or follows it.
That leaves two quiet failures:
Both cost money. A needless custom connector is work someone pays for later. A silently chosen setting is a risk the customer never agreed to.
Code gets tested automatically on every change, so a small edit can’t quietly break something. We wanted the same for our skills, so we built two automated checks.
The question: Are our skill descriptions clear enough that the right skill gets chosen?
How it works:
Why we keep it so bare: With everything else removed, a wrong pick can only come from our own wording. That means every recurring failure has a clear fix.
What a test case can say:
What it costs: Very little. The check runs only when a skill's header changes, and a full run is far cheaper than Check #2. That's why it runs first.
The question: Once the skill is picked, does it improve the answer?
How it works:
1. One agent answers each test case with the skill loaded.
2. A second agent marks every expectation as pass or fail.
3. A case passes when it meets 80% of its expectations.
Why we run misses again: Results wobble. The same case scored anywhere from 50% to 100% across identical runs, so a single run tells you very little. Now, a case that misses anything is rerun twice, and it fails only if two of its three runs fail.
Does the skill earn its keep? We compare each case against a baseline, the same test run without the skill. On our catalog migration skill, the average pass rate rose from 38% to 87%. One case scored 100% both with and without the skill, so the skill only added cost in that case. That’s useful to know, because it lets us ask whether a skill is worth running and get a real answer.
How to read the gain: It shows how closely answers follow the skill’s rules, not how good an answer is in general. To keep the test fair, the marking agent never sees the skill's text.
One bias to watch: If the marking agent is the same AI model as the agents it grades, it may favor answers written in its own style. Strict, specific marking criteria reduce this, and so does using a different model.
Both checks are advisory for now. They don’t block anyone’s work until they have a stable track record.
Check #1 is a controlled test, so a failure tells us exactly what to fix. A pass is weaker evidence.
Real workspaces are crowded with project notes, company tools and other vendors’ skills. In that crowd, a coding tool may not show our skill descriptions to the agent at all. Better wording can’t fix that, because the tool makes that choice. Your own skills also compete with ours, and we don't control those either.
We don’t test a crowded workspace on purpose. Every workspace is cluttered differently, so when a test there fails, we can’t tell which part caused it, and we can’t fix what we can’t explain.
The takeaway: We make sure our descriptions are clear and distinct. What happens in your workspace is where your guardrails come in.
Our checks and the five guardrails from our best-practices post cover opposite halves of the same problem.
The pattern is the same on both sides. Don’t ask the model to grade itself, and don’t trust a success message. Check the result with something independent.
What we do:
1. Test that the right skill gets picked whenever a skill’s description changes.
2. Test that the answer is better with the skill than without it.
3. Re-run misses before calling a failure, and tell you what each check leaves open.
What we suggest you do:
1. Map each task to the skill that should handle it before you start building.
2. Verify results through a path independent of the one that wrote them.
3. Keep a corrections file for each skill.
It takes a little longer to set up. But each step is cheaper to do first than to add after something has quietly gone wrong.
Explore the commercetools for Builders documentation — and start building in days. Included in our 60-day free trial.

Build with AI
commercetools for Builders
Describe what you need in plain language and let AI skills handle the commerce foundations.
Receive monthly updates on trends, products and growth insights in your inbox.