https://github.com/appsechq/skill-scanner-test
Skill Scanner - An automated security scanning pipeline for AI agent SKILL.md and Agent plugins in popular public skills directories.
https://github.com/appsechq/skill-scanner-test
appsec security security-tools skills
Last synced: 29 days ago
JSON representation
Skill Scanner - An automated security scanning pipeline for AI agent SKILL.md and Agent plugins in popular public skills directories.
- Host: GitHub
- URL: https://github.com/appsechq/skill-scanner-test
- Owner: AppSecHQ
- License: mit
- Created: 2026-01-30T21:35:44.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2026-05-10T08:16:53.000Z (about 2 months ago)
- Last Synced: 2026-05-10T10:24:52.907Z (about 2 months ago)
- Topics: appsec, security, security-tools, skills
- Language: Shell
- Homepage: https://skillscan.io
- Size: 1.99 MB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Skill Scanner Test
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
[](https://skillscan.io)
[](https://skillscan.io)
An automated security scanning pipeline for AI agent skills and plugins in public skill directories.
AI agent skills -- installable packages that extend what coding assistants and AI agents can do -- are a growing attack surface. Skills can contain prompt injection, data exfiltration, command injection, and other vulnerabilities, whether introduced intentionally or by accident. This project systematically scans public skills using Cisco's open-source [skill-scanner](https://github.com/cisco-ai-defense/skill-scanner) and publishes the results.
## Status
This is early-stage and evolving. The current implementation:
- **Scanner:** [Cisco AI Defense skill-scanner](https://github.com/cisco-ai-defense/skill-scanner) (static + behavioral + trigger + LLM + meta-analysis)
- **Skill registries:** [skills.sh](https://skills.sh/) and [clawhub.ai](https://clawhub.ai/)
- **Coverage:** 614 skills scanned so far
- **Test suite:** 77 tests passing
This could be expanded along both axes -- adding more scanners and targeting more skill directories.
## Scan Results
| Metric | Count |
|--------|-------|
| Total Skills Scanned | 650 |
| Safe Skills | 508 (78%) |
| Skills with Issues | 142 (22%) |
| Total Findings | 1381 |
| Severity | Count |
|----------|-------|
| CRITICAL | 209 |
| HIGH | 251 |
| MEDIUM | 714 |
| LOW | 180 |
| Category | Count |
|----------|-------|
| data_exfiltration | 423 |
| command_injection | 211 |
| social_engineering | 168 |
| transitive_trust_abuse | 151 |
| unauthorized_tool_use | 133 |
| policy_violation | 91 |
| resource_abuse | 88 |
| prompt_injection | 53 |
| tool_chaining_abuse | 30 |
| hardcoded_secrets | 23 |
| skill_discovery_abuse | 6 |
| obfuscation | 3 |
| autonomy_abuse | 1 |
- These results are visualized at [skillscan.appsechq.com](https://skillscan.appsechq.com/). See [summary-report.md](results/summary-report.md) for detailed findings by skill, severity breakdowns, and top risks. Per-skill scan results (JSON + Markdown) are in the [`results/`](results/) directory.
## Getting Started
Requires Python 3.10+ and Git.
```bash
git clone https://github.com/AppSecHQ/skill-scanner-test.git
cd skill-scanner-test
python -m venv .venv
source .venv/bin/activate
pip install -r scripts/requirements.txt
```
If you're in a container or environment where you don't need a venv, you can skip the venv steps and install directly with `pip install -r scripts/requirements.txt --break-system-packages`.
### Scan top skills from a registry
```bash
# Scan top 25 skills from skills.sh (default)
python scripts/scan-skills.py -n 25
# Scan top 10 from clawhub.ai
python scripts/scan-skills.py --source clawhub -n 10
# Scan skills 11-25 (pagination)
python scripts/scan-skills.py -n 15 --offset 10
# Enable LLM semantic analysis (requires SKILL_SCANNER_LLM_API_KEY)
python scripts/scan-skills.py -n 25 --use-llm
# Enable LLM + meta-analysis for false positive filtering
python scripts/scan-skills.py -n 25 --use-llm --enable-meta
```
### Scan a specific repo
```bash
# Scan a single GitHub repo
python scripts/scan-skills.py --repo owner/repo --repo-only
# Scan a repo alongside top skills
python scripts/scan-skills.py -n 10 --repo owner/repo
```
### Other options
```bash
# List skills without scanning
python scripts/scan-skills.py --list-only -n 25
# Generate report from existing results (skip fetch + scan)
python scripts/scan-skills.py --skip-scan -o results
# Custom report name
python scripts/scan-skills.py -n 25 --report-name top-25
# Verbose logging
python scripts/scan-skills.py -n 10 -v
```
Run `python scripts/scan-skills.py --help` for the full set of options.
### Running tests
```bash
python -m pytest tests/ -v
```
## Project Structure
```
scripts/
scan-skills.py # Main orchestrator and CLI
fetch_skills.py # API fetching from skills.sh and clawhub.ai
run_scans.py # Clone, download, and scan logic
generate_report.py # Aggregate results into markdown/JSON reports
pipeline_utils.py # Shared utilities: logging, HTTP session management
requirements.txt # Dependencies
tests/ # 77 tests across 4 modules
conftest.py # Shared fixtures
test_run_scans.py # Clone, download, scan, ZIP security tests
test_generate_report.py # Report aggregation tests
test_fetch_skills.py # API fetching tests
test_pipeline_utils.py # Session management and retry tests
results/
summary-report.md # Consolidated findings across all skills
-scan.json # Raw scan output per skill
-scan.md # Readable scan report per skill
```
Cloned skill repositories are kept in `skills/` locally but excluded from version control via `.gitignore`.
## Known Limitations
- No runtime verification -- analysis is static and semantic only
- LLM and meta-analysis require an API key (`SKILL_SCANNER_LLM_API_KEY`); without one, scans fall back to static/behavioral/trigger only
- Limited to skills with public source repos
- Scanner coverage depends on the rule sets of the underlying tools
## Links
- [skillscan.io](https://skillscan.io/) -- interactive dashboard of scan results
- [Cisco AI Defense skill-scanner](https://github.com/cisco-ai-defense/skill-scanner)
- [skills.sh](https://skills.sh/)
- [clawhub.ai](https://clawhub.ai/)
- [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://arxiv.org/abs/2601.10338) -- large-scale security analysis of 31k+ skills from skills.rest and skillsmp.com
## License
MIT