The same skill behaves differently on every platform
Agent Skills promise write-once portability across 25+ platforms. This site measures where that promise holds and where it breaks: the same probe skills, run on real platforms, every finding cited to a transcript.
The Checks
40 checks across 10 categories, from loading timing to validation strictness. Each asks one testable question about platform behavior and explains why the answer matters to skill authors.
Browse the catalog →Platform Reports
Automated, transcript-cited findings for every tested platform, a comparison table of where they agree and diverge, and a summary of where each platform contradicts the Agent Skills specification.
Read the reports →Authoring Guidance
The findings turned into practice: rules for writing skills that survive platform differences, each backed by the checks that motivated it, plus a glossary of the terms used throughout.
Get the guidance →One skill, different outcomes #
Agent Skills define a portable format, but the spec leaves most loading, validation, and permission behavior open, and platforms filled the gaps differently. The same probe skills, run against the platforms with headless modes, come back with different answers:
Frontmatter can vanish
Claude Code strips YAML frontmatter before injecting a skill. On Codex CLI and Antigravity, the model sees it only if it reads the raw file. Load-bearing information that lives only in frontmatter may never reach the model.
See the finding →Grouped skills disappear
Organize skills in subfolders and Codex CLI still finds them; Claude Code and Antigravity list direct children only. The grouped skills vanish from their catalogs with no error anywhere.
See the comparison →Names resolve differently
Install the same skill name at project and user scope and Codex CLI and Antigravity load the project variant. Claude Code loads the user variant, against the implementation guide's "universal convention."
See the comparison →A skill that works perfectly where you wrote it can misbehave everywhere else, with no error and no way to tell from the outside. Every claim above traces to a transcript-cited finding.
How the testing works #
The benchmark skills are 33 fixtures (spec-compliant skills plus deliberate rule-breakers) seeded with unique canary phrases. By asking the model whether it knows a canary phrase, we can tell exactly what a platform loaded and when, without trusting the model’s self-reporting about its own context. The benchmark runner drives the checks headlessly, using skillxp to install the fixtures, invoke each harness, and parse the resulting transcripts, and cites every finding to an archived transcript.
Contributing #
We need empirical data from real platforms, and even partial data from a single platform beats speculation about all of them. Install the benchmark skills, run any of the 40 checks, and submit findings with the platform template. The GitHub repository has full instructions.
Related Research #
- Agent Skill Report: Analysis of 673+ skills examining how authors actually write skills in practice.
- skill-validator: CLI tool for validating Agent Skills against the spec.
- Agent Ecosystem: The research program behind this project.
License #
This work is licensed under CC BY 4.0.