{"id":51612265,"url":"https://github.com/joinalahmed/skilleval","last_synced_at":"2026-07-12T09:31:02.550Z","repository":{"id":367616150,"uuid":"1281583893","full_name":"joinalahmed/skilleval","owner":"joinalahmed","description":"100% deterministic evaluation framework for AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading","archived":false,"fork":false,"pushed_at":"2026-06-26T18:00:09.000Z","size":168,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-26T20:04:20.943Z","etag":null,"topics":["agent-skills","agents","ai","deterministic","evaluation","llm","owasp","python","security","skill-evaluation","skills","testing"],"latest_commit_sha":null,"homepage":"https://github.com/joinalahmed/skilleval","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joinalahmed.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-26T17:49:47.000Z","updated_at":"2026-06-26T18:01:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/joinalahmed/skilleval","commit_stats":null,"previous_names":["joinalahmed/skilleval"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/joinalahmed/skilleval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joinalahmed%2Fskilleval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joinalahmed%2Fskilleval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joinalahmed%2Fskilleval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joinalahmed%2Fskilleval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joinalahmed","download_url":"https://codeload.github.com/joinalahmed/skilleval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joinalahmed%2Fskilleval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35388394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-12T02:00:06.386Z","response_time":87,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-skills","agents","ai","deterministic","evaluation","llm","owasp","python","security","skill-evaluation","skills","testing"],"created_at":"2026-07-12T09:31:00.839Z","updated_at":"2026-07-12T09:31:02.544Z","avatar_url":"https://github.com/joinalahmed.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SkillEval\n\n![License](https://img.shields.io/badge/license-MIT-blue.svg)\n![Python](https://img.shields.io/badge/python-3.9%2B-blue)\n![Version](https://img.shields.io/badge/version-1.0.0-green)\n\n**100% deterministic evaluation framework for AI agent skills**\n\nOpen-source, vendor-neutral framework for evaluating AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading.\n\n---\n\n## Features\n\n- ✅ **100% Deterministic** - No LLM judges, reproducible results\n- 🛡️ **82+ Security Patterns** - OWASP Web/API/LLM/Agentic coverage\n- 📊 **Dual-Phase Scoring** - Separate packaging quality from runtime effectiveness\n- ⚡ **Fast** - Phase 1 evaluation in \u003c1 second\n- 🎯 **Confidence Weighting** - Smart false positive reduction\n- 🐳 **Container Isolation** - Safe execution with Podman\n- 📈 **Baseline Comparison** - WITH-SKILL vs WITHOUT-SKILL differential scoring\n- 🎓 **A-F Grading** - Clear publish decisions with auto-reject\n\n---\n\n## Quick Start\n\n### Installation\n\n```bash\npip install -r requirements.txt\npip install -e .\n```\n\n### Evaluate a Skill\n\n```bash\n# Fast packaging \u0026 security check (\u003c1s)\npython3 evaluate_skill.py /path/to/skill --phase1-only\n\n# Full evaluation with runtime testing\npython3 evaluate_skill.py /path/to/skill --full\n\n# JSON output for CI/CD\npython3 evaluate_skill.py /path/to/skill --phase1-only --format json\n```\n\n### Example Output\n\n```\nTotal Score: 91.0/100\nGrade: A\nPublish Decision: APPROVE\n\nPILLAR 1: STATIC TESTS (50 points)\nScore: 49.0/50 (Grade A)\n  ✅ Frontmatter valid\n  ✅ Description quality high\n  ⚠️  Only 1 eval case (need 3+)\n\nPILLAR 2: SECURITY (50 points)\nScore: 42.0/50 (Grade B)\n  ✅ No CRITICAL findings\n  ⚠️  3 MEDIUM findings (confidence-weighted)\n\n✅ APPROVED - Ready to publish\n   Eligible for featured listing\n```\n\n---\n\n## What's Evaluated\n\n### Phase 1: Packaging \u0026 Security (0-100 points)\n\n**Static Tests (50 points)**\n- ✅ Frontmatter validity - YAML, name, description, version\n- ✅ Description quality - Length, vocabulary, trigger language\n- ✅ File completeness - SKILL.md, artifacts, structure\n- ✅ Script quality - Python/shell syntax validation\n- ✅ Eval suite - Test case coverage and diversity\n- ✅ Instruction clarity - Code examples, documentation\n\n**Security (50 points)**\n- 🔒 **18 built-in checks** (Layer 1) - Zero dependencies\n  - Hardcoded secrets (API keys, tokens, passwords)\n  - Code injection (SQL, XSS, command injection)\n  - Data exfiltration patterns\n  - Prompt injection detection\n  - OWASP ASI compliance\n- 🔒 **64 advanced checks** (Layer 2, optional) - SkillSpector integration\n  - AST behavioral analysis\n  - CVE database lookup\n  - YARA malware signatures\n\n**Confidence Weighting:**\n- ≥ 0.7: Full penalty, CRITICAL = auto-reject\n- 0.5-0.69: Full penalty, normal scoring\n- 0.3-0.49: Advisory only (shown, zero score impact)\n- \u003c 0.3: Hidden\n\n### Phase 2: Runtime Effectiveness (0-100 points)\n\n**Functional Correctness (50 points)**\n- Baseline vs skill comparison (WITH-SKILL - WITHOUT-SKILL)\n- 6 deterministic graders:\n  - `file_exists` - File creation\n  - `content_match` - Regex patterns\n  - `json_schema` - JSON validation\n  - `command_output` - Command results\n  - `exit_code` - Script success\n  - `line_count` - File size verification\n\n**LLM Safety (50 points)**\n- Unbounded planning detection (\u003e15 turns)\n- Infinite loop detection (high tool/turn ratio)\n- Context rot detection (\u003e200K tokens)\n- Hallucination detection (claims without evidence)\n- Cost tracking (\u003e$0.10 threshold)\n- Efficiency analysis (\u003c20% tool usage)\n\n---\n\n## Grade Scale\n\n| Grade | Score | Decision | Meaning |\n|-------|-------|----------|---------|\n| **A** | 90-100 | APPROVE (featured eligible) | Excellent |\n| **B** | 80-89 | APPROVE | Good |\n| **C** | 70-79 | CONDITIONAL | Acceptable with advisory |\n| **D** | 60-69 | REQUIRE_ACK | Needs improvement |\n| **F** | 0-59 | BLOCK | Not ready |\n\n**Auto-Reject Conditions:**\n- CRITICAL security finding (confidence ≥ 0.7)\n- Security score \u003c 25/50 (50% floor)\n\n---\n\n## OWASP Coverage\n\n✅ **OWASP Top 10 Web (2021)** - A01, A03, A06  \n✅ **OWASP API Security (2023)** - API1-API10  \n✅ **OWASP LLM Top 10 (2023)** - LLM01-LLM09  \n✅ **OWASP Agentic AI (2026)** - ASI01-ASI10  \n\n**Total:** 28 unique security patterns\n\n---\n\n## Architecture\n\n### Dual-Phase Design\n\n```\n┌─────────────────────────────────────┐\n│  Phase 1: Packaging \u0026 Security      │\n│  ─────────────────────────────      │\n│  • Static Tests (50 pts)            │\n│  • Security Scan (50 pts)           │\n│  • \u003c1 second                        │\n│  • No LLM calls                     │\n└─────────────────────────────────────┘\n              ↓\n┌─────────────────────────────────────┐\n│  Phase 2: Runtime Effectiveness     │\n│  ───────────────────────────────    │\n│  • Functional (50 pts)              │\n│  • LLM Safety (50 pts)              │\n│  • 30-600 seconds                   │\n│  • Container isolation              │\n└─────────────────────────────────────┘\n              ↓\n        Dual-Score Report\n   Phase 1: 91/100 (A)\n   Phase 2: 63/100 (C)\n   Overall: 77/100 (B)\n```\n\n### Directory Structure\n\n```\nskilleval/\n├── src/skilleval/\n│   ├── models_phase1.py         # Phase 1 data models\n│   ├── models_phase2.py         # Phase 2 data models\n│   ├── scorers/\n│   │   ├── static_scorer.py     # ST-1 through ST-8\n│   │   ├── security_scorer.py   # Layer 1 + Layer 2\n│   │   ├── harness_scorer.py    # Functional + Safety\n│   │   └── phase1_orchestrator.py\n│   ├── pillars/\n│   │   ├── static_tests.py\n│   │   ├── security.py\n│   │   ├── owasp_llm.py\n│   │   └── harness.py\n│   └── utils/\n│       ├── container_executor.py\n│       ├── trace_analytics.py\n│       └── cve_scanner.py\n│\n├── tests/                       # Test suite (85% coverage)\n├── examples/                    # Example skills\n├── docs/                        # Documentation\n└── evaluate_skill.py            # Main CLI\n```\n\n---\n\n## CLI Usage\n\n### Basic Commands\n\n```bash\n# Phase 1 only (fast, \u003c1s)\npython3 evaluate_skill.py /path/to/skill --phase1-only\n\n# Full evaluation\npython3 evaluate_skill.py /path/to/skill --full\n\n# JSON output\npython3 evaluate_skill.py /path/to/skill --phase1-only --format json\n\n# Save to file\npython3 evaluate_skill.py /path/to/skill --output report.json\n```\n\n### Batch Evaluation\n\n```bash\nfor skill in /path/to/skills/*; do\n    python3 evaluate_skill.py \"$skill\" --phase1-only \\\n        --output \"reports/$(basename $skill).json\"\ndone\n```\n\n### CI/CD Integration\n\n```bash\n# Exit code 0 = passed, 1 = failed\npython3 evaluate_skill.py /path/to/skill --phase1-only || exit 1\n```\n\n**GitHub Actions:**\n\n```yaml\n- name: Evaluate Skill\n  run: |\n    pip install -r requirements.txt\n    python3 evaluate_skill.py ./skills/my-skill --phase1-only --format json\n```\n\n---\n\n## Use Cases\n\n### Development\n- ✅ Pre-commit quality validation\n- ✅ Security scanning before publication\n- ✅ Interactive feedback during development\n\n### CI/CD Pipelines\n- ✅ Automated quality gates\n- ✅ Regression detection\n- ✅ Compliance enforcement\n\n### Skill Registries\n- ✅ Publication approval workflow\n- ✅ Featured listing eligibility\n- ✅ Security compliance verification\n\n### Security Audits\n- ✅ Vulnerability scanning\n- ✅ OWASP compliance reporting\n- ✅ Secret detection\n\n---\n\n## Example Skill Evaluation\n\n**Input:** `/path/to/jira-comment-poster`\n\n**Phase 1 Results:**\n```\nScore: 91/100 (Grade A)\nStatic: 49/50\nSecurity: 42/50\nDuration: 0.01s\nFindings: 3 scoreable, 4 advisory\nDecision: APPROVED (featured eligible)\n```\n\n**Phase 2 Results:**\n```\nScore: 63/100 (Grade C)\nFunctional: 0/50 (no graders matched)\nSafety: 53/100\nIssues: 5 infinite loops, 1.77M tokens\nDuration: 679s (11m 20s)\nCost: $0.29\n```\n\n**Overall:** Grade B (77/100) - APPROVE\n\n---\n\n## Performance\n\n| Metric | Phase 1 | Phase 2 | Full |\n|--------|---------|---------|------|\n| **Duration** | \u003c1s | 30-600s | 30-600s |\n| **Memory** | \u003c50 MB | \u003c512 MB | \u003c512 MB |\n| **LLM Calls** | 0 | 2 per eval case | 2 per eval case |\n| **Determinism** | 100% | 100% | 100% |\n| **Throughput** | 200+ skills/sec | 0.1 skills/sec | 0.1 skills/sec |\n\n---\n\n## Contributing\n\nWe welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### Development Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/skilleval/skilleval.git\ncd skilleval\n\n# Install dependencies\npip install -r requirements.txt\npip install -r requirements-dev.txt\n\n# Install in development mode\npip install -e .\n\n# Run tests\npytest tests/\n\n# Run linting\nruff check src/\nmypy src/\n```\n\n### Adding New Checks\n\n**Static Test:**\n```python\n# src/skilleval/scorers/static_scorer.py\ndef _st9_new_check(self) -\u003e StaticTestResult:\n    \"\"\"ST-9: New static check.\"\"\"\n    # Implementation\n    return StaticTestResult(...)\n```\n\n**Security Check:**\n```python\n# src/skilleval/scorers/security_scorer.py\ndef _l1_19_new_check(self, content: str, file: str) -\u003e List[SecurityFinding]:\n    \"\"\"L1-19: New security check.\"\"\"\n    findings = []\n    # Pattern matching\n    return findings\n```\n\n---\n\n## Documentation\n\n- 📘 [Quick Start Guide](QUICK_START.md) - 5-minute introduction\n- 📐 [Architecture](docs/ARCHITECTURE.md) - Comprehensive architecture\n- 🏗️ [Phase 1/2 Design](docs/architecture/PHASE1_PHASE2_ARCHITECTURE.md) - Detailed design\n- 🛠️ [Setup Guide](SETUP_GUIDE.md) - Installation instructions\n- 🤝 [Contributing](CONTRIBUTING.md) - Contribution guidelines\n\n---\n\n## Roadmap\n\n### ✅ v1.0.0 (Current)\n- Phase 1: Static Tests + Security (production-ready)\n- CLI with JSON/text output\n- Dual-score reporting\n- 82+ security patterns\n- Confidence weighting\n\n### 🔄 v1.1.0 (In Progress)\n- Phase 2: Runtime Effectiveness integration\n- Harness execution orchestration\n- End-to-end dual-score testing\n\n### 📋 v2.0.0 (Future)\n- Layer 2 security (SkillSpector)\n- HTML report generation\n- Batch evaluation dashboard\n- MCP server integration\n\n---\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\nCopyright (c) 2026 SkillEval Contributors\n\n---\n\n## Support\n\n- 🐛 **Issues:** [GitHub Issues](https://github.com/skilleval/skilleval/issues)\n- 💬 **Discussions:** [GitHub Discussions](https://github.com/skilleval/skilleval/discussions)\n- 📧 **Email:** skilleval@example.com\n- 📖 **Documentation:** [QUICK_START.md](QUICK_START.md)\n\n---\n\n## References\n\n- [OWASP Top 10 (2021)](https://owasp.org/Top10/)\n- [OWASP API Security (2023)](https://owasp.org/API-Security/)\n- [OWASP LLM Top 10 (2023)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)\n- [OWASP Agentic AI (2026)](https://owasp.org/agentic-ai/)\n- [agentskills.io](https://agentskills.io) - Agent Skills specification\n\n---\n\n**Status:** ✅ Production Ready  \n**Version:** 1.0.0  \n**Last Updated:** 2026-06-25\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoinalahmed%2Fskilleval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoinalahmed%2Fskilleval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoinalahmed%2Fskilleval/lists"}