https://github.com/joinalahmed/skilleval

100% deterministic evaluation framework for AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading
https://github.com/joinalahmed/skilleval

agent-skills agents ai deterministic evaluation llm owasp python security skill-evaluation skills testing

Last synced: 6 days ago
JSON representation

100% deterministic evaluation framework for AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading

• Host: GitHub
• URL: https://github.com/joinalahmed/skilleval
• Owner: joinalahmed
• License: mit
• Created: 2026-06-26T17:49:47.000Z (22 days ago)
• Default Branch: main
• Last Pushed: 2026-06-26T18:00:09.000Z (22 days ago)
• Last Synced: 2026-06-26T20:04:20.943Z (22 days ago)
• Topics: agent-skills, agents, ai, deterministic, evaluation, llm, owasp, python, security, skill-evaluation, skills, testing
• Language: Python
• Homepage: https://github.com/joinalahmed/skilleval
• Size: 164 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# SkillEval

![License](https://img.shields.io/badge/license-MIT-blue.svg)

![Python](https://img.shields.io/badge/python-3.9%2B-blue)

![Version](https://img.shields.io/badge/version-1.0.0-green)

**100% deterministic evaluation framework for AI agent skills**

Open-source, vendor-neutral framework for evaluating AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading.

---

## Features

- ✅ **100% Deterministic** - No LLM judges, reproducible results

- 🛡️ **82+ Security Patterns** - OWASP Web/API/LLM/Agentic coverage

- 📊 **Dual-Phase Scoring** - Separate packaging quality from runtime effectiveness

- ⚡ **Fast** - Phase 1 evaluation in <1 second

- 🎯 **Confidence Weighting** - Smart false positive reduction

- 🐳 **Container Isolation** - Safe execution with Podman

- 📈 **Baseline Comparison** - WITH-SKILL vs WITHOUT-SKILL differential scoring

- 🎓 **A-F Grading** - Clear publish decisions with auto-reject

---

## Quick Start

### Installation

```bash

pip install -r requirements.txt

pip install -e .

```

### Evaluate a Skill

```bash

# Fast packaging & security check (<1s)

python3 evaluate_skill.py /path/to/skill --phase1-only

# Full evaluation with runtime testing

python3 evaluate_skill.py /path/to/skill --full

# JSON output for CI/CD

python3 evaluate_skill.py /path/to/skill --phase1-only --format json

```

### Example Output

```

Total Score: 91.0/100

Grade: A

Publish Decision: APPROVE

PILLAR 1: STATIC TESTS (50 points)

Score: 49.0/50 (Grade A)

  ✅ Frontmatter valid

  ✅ Description quality high

  ⚠️  Only 1 eval case (need 3+)

PILLAR 2: SECURITY (50 points)

Score: 42.0/50 (Grade B)

  ✅ No CRITICAL findings

  ⚠️  3 MEDIUM findings (confidence-weighted)

✅ APPROVED - Ready to publish

   Eligible for featured listing

```

---

## What's Evaluated

### Phase 1: Packaging & Security (0-100 points)

**Static Tests (50 points)**

- ✅ Frontmatter validity - YAML, name, description, version

- ✅ Description quality - Length, vocabulary, trigger language

- ✅ File completeness - SKILL.md, artifacts, structure

- ✅ Script quality - Python/shell syntax validation

- ✅ Eval suite - Test case coverage and diversity

- ✅ Instruction clarity - Code examples, documentation

**Security (50 points)**

- 🔒 **18 built-in checks** (Layer 1) - Zero dependencies

  - Hardcoded secrets (API keys, tokens, passwords)

  - Code injection (SQL, XSS, command injection)

  - Data exfiltration patterns

  - Prompt injection detection

  - OWASP ASI compliance

- 🔒 **64 advanced checks** (Layer 2, optional) - SkillSpector integration

  - AST behavioral analysis

  - CVE database lookup

  - YARA malware signatures

**Confidence Weighting:**

- ≥ 0.7: Full penalty, CRITICAL = auto-reject

- 0.5-0.69: Full penalty, normal scoring

- 0.3-0.49: Advisory only (shown, zero score impact)

- < 0.3: Hidden

### Phase 2: Runtime Effectiveness (0-100 points)

**Functional Correctness (50 points)**

- Baseline vs skill comparison (WITH-SKILL - WITHOUT-SKILL)

- 6 deterministic graders:

  - `file_exists` - File creation

  - `content_match` - Regex patterns

  - `json_schema` - JSON validation

  - `command_output` - Command results

  - `exit_code` - Script success

  - `line_count` - File size verification

**LLM Safety (50 points)**

- Unbounded planning detection (>15 turns)

- Infinite loop detection (high tool/turn ratio)

- Context rot detection (>200K tokens)

- Hallucination detection (claims without evidence)

- Cost tracking (>$0.10 threshold)

- Efficiency analysis (<20% tool usage)

---

## Grade Scale

| Grade | Score | Decision | Meaning |

|-------|-------|----------|---------|

| **A** | 90-100 | APPROVE (featured eligible) | Excellent |

| **B** | 80-89 | APPROVE | Good |

| **C** | 70-79 | CONDITIONAL | Acceptable with advisory |

| **D** | 60-69 | REQUIRE_ACK | Needs improvement |

| **F** | 0-59 | BLOCK | Not ready |

**Auto-Reject Conditions:**

- CRITICAL security finding (confidence ≥ 0.7)

- Security score < 25/50 (50% floor)

---

## OWASP Coverage

✅ **OWASP Top 10 Web (2021)** - A01, A03, A06  

✅ **OWASP API Security (2023)** - API1-API10  

✅ **OWASP LLM Top 10 (2023)** - LLM01-LLM09  

✅ **OWASP Agentic AI (2026)** - ASI01-ASI10  

**Total:** 28 unique security patterns

---

## Architecture

### Dual-Phase Design

```

┌─────────────────────────────────────┐

│  Phase 1: Packaging & Security      │

│  ─────────────────────────────      │

│  • Static Tests (50 pts)            │

│  • Security Scan (50 pts)           │

│  • <1 second                        │

│  • No LLM calls                     │

└─────────────────────────────────────┘

              ↓

┌─────────────────────────────────────┐

│  Phase 2: Runtime Effectiveness     │

│  ───────────────────────────────    │

│  • Functional (50 pts)              │

│  • LLM Safety (50 pts)              │

│  • 30-600 seconds                   │

│  • Container isolation              │

└─────────────────────────────────────┘

              ↓

        Dual-Score Report

   Phase 1: 91/100 (A)

   Phase 2: 63/100 (C)

   Overall: 77/100 (B)

```

### Directory Structure

```

skilleval/

├── src/skilleval/

│   ├── models_phase1.py         # Phase 1 data models

│   ├── models_phase2.py         # Phase 2 data models

│   ├── scorers/

│   │   ├── static_scorer.py     # ST-1 through ST-8

│   │   ├── security_scorer.py   # Layer 1 + Layer 2

│   │   ├── harness_scorer.py    # Functional + Safety

│   │   └── phase1_orchestrator.py

│   ├── pillars/

│   │   ├── static_tests.py

│   │   ├── security.py

│   │   ├── owasp_llm.py

│   │   └── harness.py

│   └── utils/

│       ├── container_executor.py

│       ├── trace_analytics.py

│       └── cve_scanner.py

│

├── tests/                       # Test suite (85% coverage)

├── examples/                    # Example skills

├── docs/                        # Documentation

└── evaluate_skill.py            # Main CLI

```

---

## CLI Usage

### Basic Commands

```bash

# Phase 1 only (fast, <1s)

python3 evaluate_skill.py /path/to/skill --phase1-only

# Full evaluation

python3 evaluate_skill.py /path/to/skill --full

# JSON output

python3 evaluate_skill.py /path/to/skill --phase1-only --format json

# Save to file

python3 evaluate_skill.py /path/to/skill --output report.json

```

### Batch Evaluation

```bash

for skill in /path/to/skills/*; do

    python3 evaluate_skill.py "$skill" --phase1-only \

        --output "reports/$(basename $skill).json"

done

```

### CI/CD Integration

```bash

# Exit code 0 = passed, 1 = failed

python3 evaluate_skill.py /path/to/skill --phase1-only || exit 1

```

**GitHub Actions:**

```yaml

- name: Evaluate Skill

  run: |

    pip install -r requirements.txt

    python3 evaluate_skill.py ./skills/my-skill --phase1-only --format json

```

---

## Use Cases

### Development

- ✅ Pre-commit quality validation

- ✅ Security scanning before publication

- ✅ Interactive feedback during development

### CI/CD Pipelines

- ✅ Automated quality gates

- ✅ Regression detection

- ✅ Compliance enforcement

### Skill Registries

- ✅ Publication approval workflow

- ✅ Featured listing eligibility

- ✅ Security compliance verification

### Security Audits

- ✅ Vulnerability scanning

- ✅ OWASP compliance reporting

- ✅ Secret detection

---

## Example Skill Evaluation

**Input:** `/path/to/jira-comment-poster`

**Phase 1 Results:**

```

Score: 91/100 (Grade A)

Static: 49/50

Security: 42/50

Duration: 0.01s

Findings: 3 scoreable, 4 advisory

Decision: APPROVED (featured eligible)

```

**Phase 2 Results:**

```

Score: 63/100 (Grade C)

Functional: 0/50 (no graders matched)

Safety: 53/100

Issues: 5 infinite loops, 1.77M tokens

Duration: 679s (11m 20s)

Cost: $0.29

```

**Overall:** Grade B (77/100) - APPROVE

---

## Performance

| Metric | Phase 1 | Phase 2 | Full |

|--------|---------|---------|------|

| **Duration** | <1s | 30-600s | 30-600s |

| **Memory** | <50 MB | <512 MB | <512 MB |

| **LLM Calls** | 0 | 2 per eval case | 2 per eval case |

| **Determinism** | 100% | 100% | 100% |

| **Throughput** | 200+ skills/sec | 0.1 skills/sec | 0.1 skills/sec |

---

## Contributing

We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

### Development Setup

```bash

# Clone repository

git clone https://github.com/skilleval/skilleval.git

cd skilleval

# Install dependencies

pip install -r requirements.txt

pip install -r requirements-dev.txt

# Install in development mode

pip install -e .

# Run tests

pytest tests/

# Run linting

ruff check src/

mypy src/

```

### Adding New Checks

**Static Test:**

```python

# src/skilleval/scorers/static_scorer.py

def _st9_new_check(self) -> StaticTestResult:

    """ST-9: New static check."""

    # Implementation

    return StaticTestResult(...)

```

**Security Check:**

```python

# src/skilleval/scorers/security_scorer.py

def _l1_19_new_check(self, content: str, file: str) -> List[SecurityFinding]:

    """L1-19: New security check."""

    findings = []

    # Pattern matching

    return findings

```

---

## Documentation

- 📘 [Quick Start Guide](QUICK_START.md) - 5-minute introduction

- 📐 [Architecture](docs/ARCHITECTURE.md) - Comprehensive architecture

- 🏗️ [Phase 1/2 Design](docs/architecture/PHASE1_PHASE2_ARCHITECTURE.md) - Detailed design

- 🛠️ [Setup Guide](SETUP_GUIDE.md) - Installation instructions

- 🤝 [Contributing](CONTRIBUTING.md) - Contribution guidelines

---

## Roadmap

### ✅ v1.0.0 (Current)

- Phase 1: Static Tests + Security (production-ready)

- CLI with JSON/text output

- Dual-score reporting

- 82+ security patterns

- Confidence weighting

### 🔄 v1.1.0 (In Progress)

- Phase 2: Runtime Effectiveness integration

- Harness execution orchestration

- End-to-end dual-score testing

### 📋 v2.0.0 (Future)

- Layer 2 security (SkillSpector)

- HTML report generation

- Batch evaluation dashboard

- MCP server integration

---

## License

MIT License - see [LICENSE](LICENSE) for details.

Copyright (c) 2026 SkillEval Contributors

---

## Support

- 🐛 **Issues:** [GitHub Issues](https://github.com/skilleval/skilleval/issues)

- 💬 **Discussions:** [GitHub Discussions](https://github.com/skilleval/skilleval/discussions)

- 📧 **Email:** skilleval@example.com

- 📖 **Documentation:** [QUICK_START.md](QUICK_START.md)

---

## References

- [OWASP Top 10 (2021)](https://owasp.org/Top10/)

- [OWASP API Security (2023)](https://owasp.org/API-Security/)

- [OWASP LLM Top 10 (2023)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)

- [OWASP Agentic AI (2026)](https://owasp.org/agentic-ai/)

- [agentskills.io](https://agentskills.io) - Agent Skills specification

---

**Status:** ✅ Production Ready  

**Version:** 1.0.0  

**Last Updated:** 2026-06-25