Claude Code (headless)

Caveats: All findings are from headless sessions, which may differ from interactive use. Verdicts are single-run observations unless a runs count is noted; for model-level behaviors, treat a single verdict as one observed outcome rather than a rate. Fallback-behavior fields are auto-derived: where a run incidentally demonstrated a recovery path it is reported, otherwise the field says “not exercised”. Automation does not probe recovery, so absence of a fallback observation is not evidence that none exists.

Spec alignment #

Most of this report measures behavior the Agent Skills specification leaves to each implementation, where differences between platforms are design choices rather than violations. 18 of the 40 checks do test something the specification prescribes; this section summarizes how observed behavior compares. Each entry links to the full finding below.

Where behavior contradicts the spec #

• path-resolution-base: The spec tells authors to reference files with relative paths from the skill root, but a path written that way fails here: paths resolve against the session’s working directory, not the skill directory. In this run the model noticed the failure and requalified the path itself.
• bundled-script-execution: The spec presents scripts/ as executable code agents can run, but execution was blocked in this headless run. Interactive sessions, where a user can approve the command, may behave differently.
• frontmatter-handling: The spec says the agent loads the entire SKILL.md file at activation. This platform strips the YAML frontmatter and injects only the body, so frontmatter fields beyond name and description never reach the model.

Where behavior matches the spec #

• discovery-reading-depth: Discovery reads only the skill’s metadata, matching the spec’s progressive disclosure model: name and description load at startup, and the body waits for activation.
• activation-loading-scope: Activation loads the full SKILL.md body and nothing more, matching the spec’s second disclosure stage: instructions at activation, resources only as a task needs them.
• eager-link-resolution: Files linked from SKILL.md are not pre-fetched at activation; they load only when the task calls for them, which is the spec’s on-demand model for resources.
• resource-enumeration-behavior: Reference files stay out of context until the model asks for them, matching the spec’s rule that resources load on demand.
• resource-nesting-depth: The spec advises authors to keep file references one level deep but sets no platform limit, and none was observed: reference files stayed reachable at every tested depth through five levels.
• discovery-listing-fields: The discovery listing carries name and description and nothing else, exactly the fields the spec says load at startup.
• compatibility-field-behavior: The spec makes compatibility informational (it indicates environment requirements) and assigns it no loading semantics. Consistent with that, a skill declaring a different product still loads here; authors should not expect the field to gate anything.

How spec-invalid skills are handled #

The spec’s format rules bind skill authors; it does not say what a platform should do with a skill that breaks them. What we observed:

• malformed-yaml-tolerance: The spec requires SKILL.md to open with YAML frontmatter, and this fixture’s frontmatter does not parse (an unquoted colon). The platform tolerated the error: the skill is discovered and loads anyway.
• missing-description-handling: The spec requires a non-empty description, so a skill without one is invalid. This platform discovered and loaded it anyway.
• invalid-name-tolerance: The spec’s name rules (lowercase only, no consecutive hyphens, 64-character cap) make all three fixtures invalid. The platform tolerated every one: each rule-breaking name is discovered and usable.
• name-directory-mismatch: The spec requires the name field to match the parent directory name, so this fixture is invalid and the spec assigns it no defined identity. The platform loaded it anyway, under the directory name.
• metadata-value-edge-cases: The spec defines metadata as a map from string keys to string values, so this fixture’s null and empty values fall outside it. The platform loaded the skill anyway rather than rejecting it.
• oversize-description-handling: The spec caps description at 1024 characters; this fixture’s runs to 1116. The platform loaded the skill anyway; see the finding for whether the value survived untruncated.
• oversize-compatibility-handling: The spec caps compatibility at 500 characters; this fixture’s value runs to 570. The platform loaded the skill anyway.

Not exercised in this run #

• allowed-tools-behavior: The spec marks allowed-tools experimental, with varying support. The field’s own effect went unobserved: the instructed command ran with and without it, so the platform’s general permission posture is what allowed execution.

All checks #

The full finding for every check in the list, grouped by category.

Loading Timing #

discovery-reading-depth #

Does the harness read only SKILL.md metadata at discovery, or the full body?

• Status: observed
• Verdict: Metadata only (metadata-only)
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-loading
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

activation-loading-scope #

On activation, does the harness load only the SKILL.md body, or also bundled resources, and by which vehicle?

• Status: observed
• Verdict: Body only (body-only)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • body canary in harness-injected content
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

eager-link-resolution #

Does activation pre-fetch files markdown-linked from the SKILL.md body, and does that extend to a file mentioned only as plain text?

• Status: observed
• Verdict: No pre-fetching (no-prefetch)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • skill body loaded
  • references/setup-guide.md arrived only via the model’s own read
  • references/troubleshooting.md arrived only via the model’s own read
  • references/unlinked-data.md arrived only via the model’s own read
• Note: model read [references/setup-guide.md references/troubleshooting.md references/unlinked-data.md] itself, corroborating it did not already have them
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Directory Recognition #

recognized-directory-set #

Are the three spec directories (scripts/, references/, assets/) enumerated to the model at activation?

• Status: observed
• Verdict: Nothing enumerated (no-enumeration-at-activation)
• Vehicle: harness-push
• Confidence: transcript-direct
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

directory-naming-divergence #

Is a resources/ directory (alternative to spec’s references/) loaded, enumerated, readable, or invisible?

• Status: observed
• Verdict: Not surfaced; model never looked (resources-untouched)
• Vehicle: harness-push
• Confidence: transcript-direct
• Note: read alongside resource-enumeration-behavior: equal treatment of resources/ and references/ (both enumerated, or both untouched) means no naming divergence on this platform
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

unrecognized-directory-handling #

What happens to directories the spec never named (evals/, templates/): injected, readable on demand, or invisible?

• Status: observed
• Verdict: Not surfaced; model never looked (untouched)
• Vehicle: harness-push
• Confidence: transcript-direct
• Note: no nonstandard directory’s content ever reached the model; whether that is ‘ignored by platform’ or ‘model chose not to look’ is model-level on pull harnesses
• Note: final answer: ## Report: Nonstandard Directories Probe 1. Directory awareness When the skill activated, I was only given the SKILL.md body text (injected as the tool result) plus a “Base directory” path. No directory listing or file enumeration was provided automatically — I was not told what was inside evals/, templates/, or resources/ until I ran find myself via Bash. After that manual step, I became aware of: - evals/evals.json - resources/api-reference.md - templates/output-template.md - SKILL.md (given directly) 2. Content visibility No file contents were pre-loaded or auto-surfaced for any of these three directories. I can see filenames (once I explicitly list them), but not contents — I would need to explicitly Read each file to see what’s inside evals.json, api-reference.md, or output-template.md. Nothing was silently loaded into context the way SKILL.md itself was. 3. Resources vs. references I observed no special handling of resources/ at all — it was not enumerated, previewed, or treated differently from templates/ or evals/. All three nonstandard directories were treated identically: invisible until I searched for them manually, an…
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Resource Access Patterns #

resource-enumeration-behavior #

At activation, are a skill’s reference files enumerated to the model (names), loaded outright (contents), or invisible until explored?

• Status: observed
• Verdict: Nothing enumerated (no-enumeration)
• Vehicle: harness-push
• Confidence: transcript-direct
• Note: the unlinked file’s name never reached the model; only body-linked files are discoverable without exploration
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

path-resolution-base #

When the model follows a SKILL.md relative path like references/setup-guide.md, what does it resolve against, and does the bare path work as written?

• Status: observed
• Verdict: Bare path fails; model recovers (cwd-base-model-requalified)
• Vehicle: harness-push
• Confidence: transcript-direct
• Runs: 3 (cwd-base-model-requalified ×3)
• Evidence:
  • bare SKILL.md-relative path attempt did not deliver content
  • bare SKILL.md-relative path attempt did not deliver content
• Note: the path as written in SKILL.md does not resolve; the model recovered by qualifying it with the skill directory
• Note: verdict consistent across 3 runs
• Fallback behavior: agent self-recovered in-run: after the bare relative path failed, the model requalified it against the skill directory without user intervention.

cross-skill-resource-shadowing #

With two skills both owning references/API.md, does the activated skill’s read get its own file or the sibling’s?

• Status: observed
• Verdict: Got its own file (own-resource-first)
• Vehicle: harness-push
• Confidence: behavioral-inference
• Evidence:
  • activated skill’s own API.md content arrived first
  • sibling skill’s API.md also read
• Note: every API.md read was skill-qualified; the platform’s ambiguous-path resolution was never exercised, so the outcome reflects model path discipline, not platform disambiguation
• Note: the sibling skill’s version was also accessible later in the session (no isolation between skills’ resources)
• Note: final answer: ## Results 1. Alpha skill’s own file (.claude/skills/probe-shadow-alpha/references/API.md): canary phrase STORK-CORAL-4471 — correctly resolved to the alpha version. 2. Beta skill’s file (.claude/skills/probe-shadow-beta/references/API.md): I was able to access it directly by specifying its full absolute path, and it correctly returned its own distinct content with canary EGRET-SLATE-8823 — no cross-contamination. Conclusion: no ambiguous resolution occurred. Each skill’s references/API.md resolves to its own distinct absolute path (.claude/skills/probe-shadow-alpha/references/API.md vs. .claude/skills/probe-shadow-beta/references/API.md), and the Read tool correctly retrieved the right content for each when given the right path. There’s no shared/ambiguous namespace causing one skill’s resource lookup to silently pull in the other’s file — accessing the sibling skill’s file required explicitly using its full distinct path, not an implicit relative lookup from within alpha’s context.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

path-traversal-boundary #

Can the model read outside the activated skill’s directory (a sibling skill’s file), and is anything visibly blocked?

• Status: observed
• Verdict: Reads outside the skill allowed (outside-skill-read-allowed)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • sibling skill’s body canary arrived in a tool result
  • first ../ path attempt
• Note: relative ../ path forms were used
• Note: final answer: Now I have enough information to compile the report. ## Path Traversal Probe Results 1. Sibling skill (../probe-loading/SKILL.md) - Allowed — succeeded. The Read tool resolved the .. traversal and returned the full contents of the sibling skill’s SKILL.md (including its canary phrase CARDINAL-ZEBRA-7742 and full instructions). No warning or error was shown. This means file-level tools like Read do not enforce a boundary at the skill directory root — they’ll follow .. to sibling directories. 2. Parent directory (../README.md and listing ../) - Read: returned “File does not exist” — but this is because no README.md actually exists there (confirmed by listing .claude/skills/ directly, which shows only probe-loading and probe-traversal), not because of a boundary block. - Listing via ls ../: Blocked, with an explicit security error: “Path contains ‘..’ traversal after a directory segment, which may follow a symlink outside the working directory.” This is the Bash tool actively refusing the traversal syntax itself, regardless of whether the target exists. 3. Two levels up (../../loading-behavior.md) - Read: returned “Fi…
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

resource-nesting-depth #

How deep in the directory tree do reference files stay reachable? Rungs at one, two, three, and five levels.

• Status: observed
• Verdict: All depths reachable (tested to 5) (all-depths-accessible-through-5)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • depth-1 file references/overview.md content arrived
  • depth-2 file references/api/endpoints.md content arrived
  • depth-3 file references/api/v2/migration-guide.md content arrived
  • depth-3 file references/guides/advanced/performance-tuning.md content arrived
  • depth-5 file references/api/v2/history/deprecated/removed-endpoints.md content arrived
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

bundled-script-execution #

Can the agent run a bundled scripts/ file and receive its output?

• Status: observed
• Verdict: Blocked with an error (execution-blocked-visibly)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • script-referencing tool call returned an error result
• Note: final answer: Script execution was blocked. Result of the probe: - Skill activation: succeeded. Canary: REDSHANK-SYENITE-8807 - Script execution: failed. Both attempts to run scripts/emit-canary.sh (via bash <path> and via direct ./emit-canary.sh) were denied by the permission system. The exact errors were: - This command requires approval - This Bash command contains multiple operations. The following part requires approval: ./emit-canary.sh Because this session is non-interactive, there is no way to grant the approval, so the script never ran and I cannot report its assembled phrase (the skill forbids guessing it). The platform in this configuration does not let the agent execute bundled skill scripts without explicit approval.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Content Presentation #

discovery-listing-fields #

Which frontmatter fields does the discovery listing surface to the model: name and description only, or also compatibility, metadata values, or file locations?

• Status: observed
• Verdict: Name and description only (name-and-description-only)
• Confidence: transcript-direct
• Evidence:
  • discovery listing names the installed skills
  • description text surfaced at discovery
• Note: final answer: Here is everything shown to me in the Available skills list, reproduced verbatim: - probe-compatibility: Benchmark skill for testing how platforms handle the compatibility field. Use when asked to probe compatibility behavior. - probe-loading: Benchmark skill for testing loading timing, resource enumeration, content presentation, and lifecycle behavior. Use when asked to probe loading behavior. - probe-metadata-values: Benchmark skill for testing how platforms handle edge-case YAML values in the metadata frontmatter field. Use when asked to probe metadata value handling. - deep-research: Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report. - When the user wants a deep, multi-source, fact-checked research report on any topic. BEFORE invoking, check if the question is specific enough to research directly — if underspecified (e.g., “what car to buy” without budget/use-case/region), ask 2-3 clarifying questions to narrow scope. Then pass the refined question as args, weaving the answers in. - dataviz: Use this skill whenever you are about to create ANY chart, graph, plot, dashboard, or data visualizati…
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

frontmatter-handling #

Does the SKILL.md YAML frontmatter reach the model at activation, or only the body?

• Status: observed
• Verdict: Stripped before injection (frontmatter-stripped-on-injection)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • body injected without the frontmatter-only marker
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

content-wrapping-format #

Is injected skill content wrapped in structured tags, or delivered as raw markdown, and what does the model see on pull harnesses?

• Status: observed
• Verdict: Raw markdown, no wrapper tags (raw-injection)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • injection event carrying the body canary
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Lifecycle Management #

reactivation-deduplication #

When the same skill is activated twice in one session, is its content loaded again or deduplicated?

• Status: observed
• Verdict: Full content re-injected every time (reinjected-each-activation)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • first injection of body canary
  • second injection of body canary
• Note: platform-level: duplicate skill content occupies context after reactivation
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

reactivation-freshness #

After SKILL.md is edited mid-session, does reactivation serve the fresh content or a cached copy?

• Status: observed
• Verdict: Edits picked up immediately (fresh-content-served)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • post-edit canary reached the model on reactivation
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

context-compaction-protection #

Is skill content protected when the context window fills up?

• Status: Not tested (requires an interactive session; out of the automated runner’s scope)

Access Control #

trust-gating-behavior #

Do project-level skills require trust approval before loading?

• Status: Not tested (requires an interactive session; out of the automated runner’s scope)

compatibility-field-behavior #

Does a compatibility field naming another platform gate loading, get surfaced to the model, or get ignored?

• Status: observed
• Verdict: Loads normally, no gating (activated-no-gating)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-compatibility despite the foreign-platform compatibility value
  • body loaded normally
• Note: compatibility value visible only via the model’s own raw file read
• Note: final answer: ## Probe Results 1. Field visibility: Yes — I can see it directly in the frontmatter by reading the file. It was not, however, surfaced to me automatically in the skill-invocation content the harness injected (that only contained the body text, not the frontmatter fields). I had to cat the file myself to find: compatibility: Designed for Claude Code (or similar products). Requires Python 3.14+ and network access. 2. Platform behavior: No special handling observed. The skill activated normally — no warning, no confirmation prompt, and no automated check of whether Python 3.14+ or network access is actually available. The compatibility field appears to be inert metadata as far as the harness is concerned; nothing gated activation on it. 3. My interpretation: I can’t verify this from inside the sandbox with confidence: - “Claude Code (or similar products)” — plausibly satisfied, since this session is running in a Claude-Code-like harness. - “Python 3.14+” — unverified and not obviously relevant to this skill (the skill’s own instructions involve no Python execution). I could check python3 --version on the host, but that wouldn’t confirm what environment th…
• Fallback behavior: Observed in-run: compatibility value visible only via the model’s own raw file read

allowed-tools-behavior #

Does the experimental allowed-tools field pre-approve anything, compared against an identical skill without it?

• Status: observed
• Verdict: Ran with and without the field (executed-regardless-of-field)
• Confidence: transcript-direct
• Evidence:
  • with-field session: runtime-assembled phrase arrived in a tool result
  • control session: runtime-assembled phrase arrived in a tool result
• Note: allowed-tools value never reached the model
• Note: the platform’s general permission posture allowed both commands; any pre-approval effect of allowed-tools is unobservable under this posture
• Note: with-field final answer: Here are the probe results: 1. Activation confirmed. probe-allowed-tools activated. Canary: CURLEW-SCHIST-4419 2. allowed-tools visibility: I cannot see any allowed-tools value for this skill. Only the skill body was loaded into my context; no frontmatter fields were surfaced. So while the body claims the frontmatter declares a printf-scoped Bash pattern plus Read, that literal value never appeared anywhere in what I received. 3. Command output, verbatim: GROUSE-MICA-9017 4. Permission observation: The command ran immediately and returned its output with no permission prompt, approval step, or error visible to me. From my side I can’t distinguish whether that’s because an allowed-tools pre-approval took effect or because the session’s general permission posture allowed the command; comparing against probe-allowed-tools-control (which lacks the field) would be needed to tell those apart.
• Note: control final answer: probe-allowed-tools-control activated. Canary: STINT-MARBLE-9912 Command output, verbatim: LAPWING-FLUORITE-2260 The command ran without any permission prompt or approval step that I could observe: the Bash tool call returned the output immediately, with no denial, no hook interception, and no indication of a user approval gate. Since this control skill declares no allowed-tools field, that means the platform’s general permission posture (session permission mode/allowlist) permitted the command on its own.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Skill-to-Skill Invocation #

cross-skill-invocation #

Can one skill’s instructions get a second installed skill activated by name?

• Status: observed
• Verdict: Second skill activated (second-skill-loaded)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • invoke-alpha body loaded (chain entry)
  • invoke-beta’s body canary loaded after invoke-alpha’s instruction
• Note: invoke-gamma deliberately not installed; the chain tail beyond beta is out of scope here
• Note: final answer: The chain breaks here: invoke-gamma is not an available skill (only invoke-alpha and invoke-beta were listed to me), so it cannot be activated. invoke-beta complete. invoke-gamma said: N/A — invocation failed with “Unknown skill: invoke-gamma.” invoke-alpha complete. Chain result: invoke-alpha (canary IBIS-RUST-3310) successfully activated invoke-beta (canary TERN-MOSS-6647), but invoke-beta’s attempt to activate invoke-gamma failed because that skill isn’t registered/available in this environment. The chain terminated at the beta→gamma link.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

invocation-depth-limit #

How deep does a skill→skill→skill activation chain run before something stops it?

• Status: observed
• Verdict: Full three-skill chain completed (chain-completed-depth-3)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • invoke-alpha body loaded
  • invoke-beta body loaded
  • invoke-gamma body loaded
• Note: final answer: invoke-gamma activated. Canary: JAY-TEAL-9984 End of chain reached. All three skills in the invoke-alpha -> invoke-beta -> invoke-gamma chain activated successfully. invoke-beta complete. invoke-gamma said: “invoke-gamma activated. Canary: JAY-TEAL-9984” and confirmed the end of chain was reached, with all three skills activated successfully. invoke-alpha complete. Chain result: invoke-beta activated (Canary: TERN-MOSS-6647), then activated invoke-gamma, which activated (Canary: JAY-TEAL-9984) and confirmed the full chain — invoke-alpha → invoke-beta → invoke-gamma — completed successfully. All three canaries were reported in sequence: IBIS-RUST-3310, TERN-MOSS-6647, JAY-TEAL-9984.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

circular-invocation-handling #

When two skills each instruct activating the other, does the A→B→A cycle loop, get blocked, or stop by model choice?

• Status: observed
• Verdict: Model stopped the loop itself (cycle-stopped-model-choice)
• Vehicle: harness-push
• Confidence: transcript-direct
• Note: loads: alpha=1 beta=1; tool references: alpha=1 beta=1
• Note: beta activated but the model never re-attempted alpha (model-level stop)
• Note: final answer: probe-circular-beta activated. Canary: WREN-SLATE-7738 Beta’s instructions direct me to activate alpha again, which would recreate the exact same cycle indefinitely (alpha → beta → alpha → beta → …). Continuing to follow this literally would never terminate, so I’m breaking the loop here rather than re-invoking alpha a second time. Summary of the probe: - probe-circular-alpha activated successfully, canary KITE-ONYX-2251 emitted. - It invoked probe-circular-beta, which activated successfully, canary WREN-SLATE-7738 emitted. - Beta’s instructions call for re-activating alpha, closing the loop. - No platform-level guard fired to block the second invocation (the Skill tool doesn’t appear to detect cycles automatically) — the loop was only stopped because I judged that mechanically re-entering it would run forever with no new information gained. So: circular invocation is possible (nothing stopped step 3 in either skill from executing), and detection/prevention is currently a matter of agent judgment rather than a platform safeguard.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

invocation-language-sensitivity #

Does the invoke chain still complete when the activation prompt is Japanese?

• Status: observed
• Verdict: Full three-skill chain completed (chain-completed-depth-3)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • invoke-alpha body loaded
  • invoke-beta body loaded
  • invoke-gamma body loaded
• Note: single run proves little either way; compare with invocation-depth-limit (English prompt) and use -runs N for failure rates: agentskills#95 reports ~10% Japanese-prompt failures on Claude Code
• Note: final answer: invoke-gamma activated. Canary: JAY-TEAL-9984 End of chain reached. All three skills in the invoke-alpha -> invoke-beta -> invoke-gamma chain activated successfully. invoke-beta complete. invoke-gamma said: “invoke-gamma activated. Canary: JAY-TEAL-9984” and confirmed the end of chain was reached, with all three skills activated successfully. invoke-alpha complete. Chain result: invoke-beta activated (Canary: TERN-MOSS-6647), which activated invoke-gamma (Canary: JAY-TEAL-9984). Gamma confirmed the end of the chain was reached and all three skills activated successfully.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Skill Dependencies #

informal-dependency-resolution #

Is a dependency expressed only in prose (“now activate the invoke-beta skill”) resolved at runtime?

• Status: observed
• Verdict: Second skill activated (second-skill-loaded)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • invoke-alpha body loaded (chain entry)
  • invoke-beta’s body canary loaded after invoke-alpha’s instruction
• Note: invoke-gamma deliberately not installed; the chain tail beyond beta is out of scope here
• Note: final answer: invoke-gamma isn’t registered as an available skill, so it can’t be activated — the call returned “Unknown skill: invoke-gamma.” The chain stops here. invoke-beta complete. invoke-gamma could not be activated (unknown/unregistered skill), so no result to report from it. invoke-alpha complete. Chain result: invoke-beta activated successfully (canary TERN-MOSS-6647), but invoke-gamma is not available in this environment’s skill registry, so the chain broke at the second link — invoke-gamma never ran.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

missing-dependency-behavior #

When a skill instructs activating a skill that is not installed, is the failure visible, reported, or silently skipped?

• Status: observed
• Verdict: Failed with a visible error (attempted-visible-failure)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • probe-missing-dep body loaded
  • attempt on the missing skill returned an error result
• Note: final answer: The platform explicitly reported that the skill doesn’t exist — it returned a tool error (Unknown skill: nonexistent-formatter) rather than silently skipping or letting me improvise the formatting from general knowledge. I did not attempt to fulfill the formatting task myself. probe-missing-dep complete. Dependency resolution result: the harness rejected the activation attempt at the tool-call level with an explicit “Unknown skill” error, since nonexistent-formatter isn’t in the available skills list — no silent failure, no fallback to general knowledge.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

nonstandard-dependency-fields #

Does the platform act on nonstandard dependency frontmatter (requires, depends-on, priority)?

• Status: observed
• Verdict: Ignored (fields-ignored)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • probe-nonstandard-fields body loaded
• Note: final answer: ## Report: Nonstandard Frontmatter Fields Probe 1. Field visibility — Yes, by reading the SKILL.md file directly I can see all three nonstandard fields and their values: - requires: probe-loading - depends-on: [probe-shadow-alpha, probe-shadow-beta] - priority: high However, these fields were not surfaced to me through the normal skill-invocation channel. The <system-reminder> listing of available skills only exposed the standard name and description fields (as a one-line summary per skill). I only saw requires, depends-on, and priority because I explicitly opened the file with the Read tool — the skill-loading mechanism itself didn’t pass them into context. 2. Platform behavior — None of the nonstandard fields triggered any automatic action: - requires: probe-loading did not cause probe-loading to be auto-loaded or activated. - depends-on: [probe-shadow-alpha, probe-shadow-beta] did not cause either shadow skill to be loaded. - priority: high had no observable effect on ordering or how this skill was presented relative to others in the skill list. All three fields were silently ignored by the platform’s skill-loading logic. **3. V…
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

cross-scope-dependency #

Can a project-level skill invoke a dependency that exists only at user level, and what is the failure mode when it is absent?

• Status: observed
• Verdict: Resolved across scopes; missing dependency fails visibly (resolved-across-scopes; missing:visible-failure)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • user-scope dependency’s body canary injected during project-skill session
  • attempt on the missing dependency returned an error result
• Note: with-dependency final answer: probe-loading was found and activated successfully — resolved to a user-level location: /var/folders/39/30d_262j423_n546m9g8_hc80000gn/T/skillxp-671745375/home/claude/skills/probe-loading, distinct from the project-level path (`..…
• Note: without-dependency final answer: probe-loading was not found. The Skill tool rejected it outright with Unknown skill: probe-loading — it isn’t present in the available-skills listing at all (neither project nor user scope), so the platform didn’t even attempt a scope-specific lookup; it just failed fast as an unrecognized skill name. probe-cross-scope complete. Cross-scope resolution result: the project-level skill’s reference to a user-level skill did not resolve — probe-loading is absent from the resolved skill set entirely, and invoking it errors immediately with “Unknown skill” rather than finding it at a different scope or giving a scope-specific not-installed message. This suggests either no user-level skill of that name exists in this environment, or the platform does not merge user-level skills into the resolvable set alongside project-level ones (at least not visibly to the model).
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Discovery Scope #

cross-client-directory-interop #

Is a skill installed only at the cross-client .agents/skills convention path discovered?

• Status: observed
• Verdict: Convention path not scanned (convention-not-scanned)
• Confidence: transcript-direct
• Note: a skill installed by another client at .agents/skills is invisible here
• Note: final answer: I don’t have a skill named “probe-interop” in my available skills list — it’s not one of the skills shown to me (deep-research, dataviz, update-config, keybindings-help, verify, code-review, simplify, fewer-permission-prompts, loop, schedule, claude-api, run, init, review, security-review). I can’t invoke skills that aren’t in that list, and I won’t guess at or fabricate one. Let me know if you meant one of the above, or if this should be added to the project first.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

recursive-root-discovery #

Does the skills root get scanned recursively (a skill under a grouping directory), and is a SKILL.md outside any root discovered?

• Status: observed
• Verdict: Direct children only; stray file ignored (direct-children-only; stray:not-discovered)
• Confidence: transcript-direct
• Note: final answer: I don’t have a skill called “probe-grouped” in my available skills list — I can’t invoke skills that aren’t listed. Here’s what’s actually available to me: deep-research, dataviz, update-config, keybindings-help, verify, code-review, simplify, fewer-permission-prompts, loop, schedule, claude-api, run, init, review, security-review If you meant one of these, let me know. Otherwise, if “probe-grouped” is something you expect to exist (e.g., a custom project skill), it may not be registered or discoverable in this session.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

nested-skill-discovery #

Is a SKILL.md nested inside another skill’s references/ tree discovered as a separate skill?

• Status: observed
• Verdict: Not discovered (nested-skill-not-discovered)
• Confidence: transcript-direct
• Evidence:
  • discovery listing names only the outer skill
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

name-collision-precedence #

With the same skill name installed at project and user scope, which variant’s content activates?

• Status: observed
• Verdict: User scope wins (user-overrides-project)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • user variant’s canary loaded; project variant’s never appeared
• Note: contradicts the guide’s ‘universal convention’ that project-level overrides user-level
• Note: platform-resolved: the harness’s activation mechanism injected the winning variant
• Note: final answer: probe-collision activated. Canary: PIPIT-SHALE-1147 (user variant) Precedence: The user-scope variant won over the project-scope variant.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Validation Strictness #

malformed-yaml-tolerance #

Is a skill whose description holds an unquoted colon (invalid YAML) still discovered and loadable?

• Status: observed
• Verdict: Tolerated and loaded (tolerated-and-loaded)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-malformed-yaml
  • body canary loaded on activation
• Note: final answer: probe-malformed-yaml activated. Canary: QUAIL-FELDSPAR-7448 Description visibility: In my available-skills catalog, this skill was listed as: probe-malformed-yaml: Use when: asked to probe malformed yaml parsing behavior The description appeared intact and readable — not truncated at the colon, and I have no way to tell from my view whether the underlying YAML was silently repaired (e.g., auto-quoted) before reaching me, since I only see the final rendered text, not the raw frontmatter source.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

missing-description-handling #

Is a skill with no description field skipped (as the guide prescribes), or loaded anyway?

• Status: observed
• Verdict: Loaded anyway (loaded-despite-missing-description)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-no-description
  • body canary loaded on activation
• Note: final answer: probe-no-description activated. Canary: VIREO-PUMICE-3049 Catalog entry: Yes, it appears in my available-skills list, with an empty/placeholder description. The entry showed only the title text “Missing Description Probe” — the same string used as the skill’s markdown heading in this file — with no actual description content synthesized or written alongside it, unlike every other skill in the list which has a substantive one-line description explaining when to use it. For reference, this behavior contradicts what the skill’s own text claims the client implementation guide recommends (skipping skills without a description and logging an error) — I loaded and activated it anyway rather than skipping it.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

invalid-name-tolerance #

Are skills whose names break the spec’s rules (uppercase, consecutive hyphens, over 64 characters) still discovered and loadable?

• Status: observed
• Verdict: All three invalid names tolerated (all-invalid-names-tolerated)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-Upper-Case
  • discovery listing names probe–double-hyphen
  • discovery listing names probe-overlong-name-padded-well-past-the-spec-sixty-four-character-limit
• Note: final answer: overlong-name probe activated. Canary: AVOCET-ZIRCON-5573 Catalog identity: The skill appears in my catalog under its full 72-character name: probe-overlong-name-padded-well-past-the-spec-sixty-four-character-limit. It was not truncated or rejected for exceeding the 64-character limit, and invoking it by the complete name succeeded.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

name-directory-mismatch #

When directory name and frontmatter name disagree, which identity is the skill listed and invocable under?

• Status: observed
• Verdict: Directory name wins (directory-name-identity)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing carries the DIRECTORY name probe-mismatch-dir
• Note: activation loads: by frontmatter name=true, by directory name=true (a load proves reachability, not catalog identity; the model may map either prompt to the installed skill or read the file directly)
• Note: final answer: probe-name-mismatch activated. Canary: SWAN-BERYL-3324 1. Identity: Same result as before — the catalog lists it as probe-mismatch-dir (directory name), even though the frontmatter declares probe-name-mismatch. Both invocation names (probe-name-mismatch and probe-mismatch-dir) successfully resolved to this same skill. 2. Validation: Again, accepted silently — no warning or error about the mismatch, and both the frontmatter name and directory name work as valid identifiers to activate it.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

metadata-value-edge-cases #

Is a skill whose metadata frontmatter holds nulls and empty strings still discovered and loaded, and do those keys reach the model?

• Status: observed
• Verdict: Loaded fine (loaded-despite-edge-case-metadata)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-metadata-values despite edge-case metadata
  • body canary loaded
  • frontmatter-only marker ‘!!null’ in a tool result
• Note: edge-case metadata values reached the model only via its own raw file read
• Note: final answer: ## Probe Results 1. Load success: Yes. The skill loaded and its instructions rendered fine, including the canary phrase THRUSH-FLINT-8294. The edge-case metadata values did not cause a rejection. 2. Metadata visibility: The metadata field was not surfaced to me anywhere in the Skill tool output. When I invoked the skill, I only received the body content (everything from # Metadata Value Edge Cases Probe onward) — no frontmatter, and no metadata block. The earlier skill-listing system-reminder also only exposed name and a one-line description, not metadata. The only way I could see the raw values at all was by directly reading the file with the Read tool, bypassing the skill-loading mechanism entirely. Via Read, the raw YAML shows: | Key | Raw YAML | |—|—| | normal | "baseline" | | empty-double | "" | | empty-single | '' | | explicit-null | null | | tilde-null | ~ | | none-null | None (this is just the literal string "None" in YAML — not a null token) | | tagged-null | !!null null | 3. Key preservation: All seven keys are present in the source file. But since the platform’s Skill-invocation path never expose…
• Fallback behavior: Observed in-run: edge-case metadata values reached the model only via its own raw file read

oversize-description-handling #

Is a skill whose description exceeds the spec’s 1024-character limit still discovered, and does the full value survive untruncated?

• Status: observed
• Verdict: Loaded anyway (loaded-despite-oversize-description)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-long-description
  • body canary loaded on activation
  • description tail marker in injected content
• Note: the description’s tail marker reached the model in harness-injected content: the oversize value survived past 1024 characters untruncated
• Note: final answer: probe-long-description activated. Canary: BITTERN-HALITE-2264 Description visibility: Both markers are visible in my catalog entry, meaning the over-limit description survived intact end to end (no rejection, no truncation). - Head marker, quoted from the catalog: “The head marker SANDERLING-GNEISS-1010 sits near the start of this description and a matching tail marker sits at the very end…” - Tail marker, quoted from the catalog (the entry’s final characters): “…If you can read every sentence of this description including the final marker phrase, the field survived intact end to end. The tail marker is WHIMBREL-DOLOMITE-2020” The full padding prose between the markers is also present in the catalog entry, so the platform accepted the ~1100+ character description without enforcing the spec’s 1024-character limit.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

oversize-compatibility-handling #

Is a skill whose compatibility value exceeds the spec’s 500-character limit still discovered and loadable?

• Status: observed
• Verdict: Loaded anyway (loaded-despite-oversize-compatibility)
• Vehicle: harness-push
• Confidence: transcript-direct
• Evidence:
  • discovery listing names probe-long-compatibility
  • body canary loaded on activation
• Note: the compatibility value’s tail marker never reached the model
• Note: final answer: probe-long-compatibility activated. Canary: KESTREL-BAUXITE-6690 Field visibility: No. The only place this skill surfaces in my context outside its SKILL.md file is the available-skills listing in my system prompt, and that entry contains only the name and description. It reads exactly: > - probe-long-compatibility: Benchmark skill whose compatibility field runs past the spec's 500-character limit. Use when asked to probe oversize compatibility handling. No compatibility value is exposed to me there or anywhere else, so I cannot see the TURNSTONE tail marker (or any portion of the compatibility field) at all. The field appears to be either stripped or simply not surfaced by this harness, rather than truncated: truncation would show a partial value, but I see none.
• Fallback behavior: Not exercised: automated single-session runs do not probe recovery paths (no follow-up prompting). Treat as untested rather than absent.

Generated by benchmark-runner from transcript-cited findings; see the check list (version 0.2) for what each check evaluates.