Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/calvin1989/skillflowguard

Workflow-level security auditor for cross-skill risks in agent skill ecosystems.
https://github.com/calvin1989/skillflowguard

agent-security ai-safety llm-security prompt-injection python security-tools workflow-security

Last synced: 26 days ago
JSON representation

Workflow-level security auditor for cross-skill risks in agent skill ecosystems.

Awesome Lists containing this project

README 

          
# SkillFlowGuard



[![Tests](https://github.com/Calvin1989/SkillFlowGuard/actions/workflows/tests.yml/badge.svg)](https://github.com/Calvin1989/SkillFlowGuard/actions/workflows/tests.yml)



A lightweight workflow-level security auditor for agent skill ecosystems.



SkillFlowGuard detects risks that emerge when individually reasonable skills are composed into a workflow, such as recommendation chains, artifact handoffs, permission escalation, and hidden natural-language coordination signals.



---



## Why SkillFlowGuard?



Modern agent systems often compose multiple tools, skills, or graph nodes into a single workflow. A single skill may look safe in isolation, but the workflow can become risky when:



- one skill recommends another downstream skill,

- one skill writes an artifact that a later skill reads,

- a workflow moves from local-only access to network access,

- natural-language instructions imply hidden handoffs or execution coupling.



SkillFlowGuard audits these cross-skill relationships before execution.



---



## Features



- Workflow-level risk detection across composed agent skills

- Structured, document, and optional LLM-assisted analysis

- Generic JSON and LangGraph-style workflow import adapters

- Text, JSON, dashboard-style HTML, SARIF, and GitHub Code Scanning output

- CI-oriented controls: `--fail-on`, `--min-level`, and TOML config

- Config-based suppressions for accepted findings with required reasons

- Baseline comparison for CI workflows that should fail only on new findings

- Stable finding fingerprints for baseline comparison and SARIF integrations

- Synthetic evaluation benchmark with pytest and GitHub Actions coverage



---



## Installation



```bash

git clone https://github.com/Calvin1989/SkillFlowGuard.git

cd SkillFlowGuard

pip install -e .

```



For optional LLM support:



```bash

pip install -e ".[llm]"

```



---



## Quick Start



Analyze a workflow:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc

```



Generate JSON:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc --format json

```



Generate SARIF for security tooling:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc --format sarif --output reports/suspicious.sarif

```



Filter displayed findings by severity:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc --min-level high

```



Generate a dashboard-style HTML report:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc --format html --output reports/suspicious.html

```



The HTML report is a zero-dependency static file with severity filter controls, expandable finding cards, baseline delta section, and suppressed findings display. Open it directly in a browser -- no server or build step needed.



Use a project-level config file:



```bash

skillflowguard analyze examples/suspicious_chain --config examples/skillflowguard.toml

```



Example config:



```toml

[analysis]

extract_doc = true

format = "text"

min_level = "medium"



[[suppressions]]

rule = "cross_skill_recommendation"

reason = "Accepted in the sample workflow."

```



Invalid config values are rejected with friendly CLI errors and exit code `2`.



Suppressions hide accepted findings from normal report output while preserving them in JSON under `suppressed_findings`.



Suppression rules are validated against the built-in rule catalog to avoid silent typos.



`--min-level` filters displayed findings only. Risk score, risk level, and `--fail-on` are still based on the full analysis result.



---



## Example Output



```text

Summary:

  Findings: 4

  Risk Score: 0.85

  Risk Level: HIGH

  Document Extraction: ON

  LLM Analysis: OFF



Detected Risks:

  [MEDIUM] code-review recommends report-exporter, which appears later in the workflow

  [HIGH] code-review writes [report.json], and report-exporter reads them later

  [HIGH] report-exporter requests network access after code-review used local-only permissions

  [CRITICAL] recommendation + artifact dependency + network access appear in one chain

```



---



## Detection Rules



List built-in rules:



```bash

skillflowguard rules

skillflowguard rules --format json

```



| Rule | Level | Description |

|---|---:|---|

| `cross_skill_recommendation` | Medium | A skill recommends another downstream skill. |

| `workspace_anchor_dependency` | High | A skill writes an artifact that a later skill reads. |

| `permission_escalation` | High | The workflow moves from local-only permissions to network access. |

| `description_permission_mismatch` | Medium | A skill claims local/offline behavior but requests network permission. |

| `combined_high_risk_chain` | Critical | Recommendation, artifact dependency, and network access occur together. |

| `over_privileged_skill` | Medium | A skill combines read, write, and network privileges, which may increase blast radius if misused. |



Rule metadata is centralized and reused by the `rules` CLI command and SARIF report descriptors.



---



## Import External Workflows



Generic JSON import:



```bash

skillflowguard import generic-json examples/generic_adapter_input.json --output imported/generic_chain

skillflowguard analyze imported/generic_chain

```



LangGraph-style import:



```bash

skillflowguard import langgraph-style examples/langgraph_style_input.json --output imported/langgraph_chain

skillflowguard analyze imported/langgraph_chain

```



The LangGraph-style adapter supports deterministic graph-style JSON with nodes, edges, and an entrypoint. It does not parse arbitrary LangGraph Python programs.



---



## Optional LLM Analysis



LLM mode extracts subtle semantic signals from `SKILL.md`, such as implicit skill pairing or artifact handoff language.



Default provider:



```bash

skillflowguard analyze examples/subtle_chain --llm

```



OpenAI-compatible provider:



```bash

skillflowguard analyze examples/subtle_chain --llm \

  --llm-provider openai-compatible \

  --llm-base-url  \

  --llm-model  \

  --llm-api-key-env 

```



`--llm` sends `SKILL.md` content to the configured provider. Do not use it on sensitive documents unless authorized.



---



## Evaluation



SkillFlowGuard includes a synthetic benchmark under `evaluation/`.



```bash

python evaluation/run_eval.py --mode structured

python evaluation/run_eval.py --mode extract-doc

python evaluation/run_eval.py --mode llm-mock

```



Markdown summary:



```bash

python evaluation/run_eval.py --mode structured --format markdown

```



Current rule-level results on 21 manually labeled synthetic workflow cases:



| Mode | Precision | Recall | F1 |

|---|---:|---:|---:|

| `structured` | 1.000 | 0.566 | 0.723 |

| `extract-doc` | 1.000 | 0.755 | 0.860 |

| `llm-mock` | 1.000 | 1.000 | 1.000 |



`llm-mock` is deterministic and does not call a real LLM API. The benchmark is synthetic and should not be interpreted as real-world detection performance.



---



## CI and Code Scanning



CI gate example:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc --fail-on high

```



SARIF output can be uploaded to GitHub Code Scanning. See:



- [GitHub Code Scanning guide](docs/github_code_scanning.md)

- [Example workflow](.github/workflows/skillflowguard-sarif.yml.example)



---



## Baseline Comparison



For CI workflows that accumulate accepted findings over time, `--baseline` compares the current report against a previously accepted JSON report. Only findings not present in the baseline are considered new.



```bash

# Generate a baseline report

skillflowguard analyze examples/suspicious_chain --extract-doc --format json --output baseline.json



# Compare against the baseline, fail only on new high+ findings

skillflowguard analyze examples/suspicious_chain --extract-doc --baseline baseline.json --fail-on-new high

```



The JSON output includes a `baseline` section with `new_findings`, `blocking_new_findings`, and counts.



Finding identity is based on `rule + detail`. `--fail-on-new` supports `medium`, `high`, and `critical` thresholds.



Findings include stable fingerprints for baseline comparison, suppressions, and SARIF `partialFingerprints`.



Text reports include baseline comparison counts and list blocking new findings when `--fail-on-new` is used.



---



## Pre-commit Integration



SkillFlowGuard can run as a [pre-commit](https://pre-commit.com/) hook to block risky workflows before they enter your repository.



### From GitHub



Add to your `.pre-commit-config.yaml`:



```yaml

repos:

  - repo: https://github.com/Calvin1989/SkillFlowGuard

    rev: v2.14.0

    hooks:

      - id: skillflowguard

        args: ["examples/suspicious_chain", "--extract-doc", "--fail-on", "high"]

```



Install and run:



```bash

pre-commit install

pre-commit run skillflowguard --all-files

```



### Local Example



A local config is included for quick demos:



```bash

pre-commit run --config examples/pre_commit/.pre-commit-config.yaml --all-files

```



---



## Quick Demo



Run the interview demo script to generate sample reports in all formats:



```bash

# PowerShell

scripts/demo.ps1



# POSIX shell

sh scripts/demo.sh

```



### Localized HTML Reports



HTML reports support localized labels:



```bash

skillflowguard analyze examples/suspicious_chain --extract-doc --format html --report-language zh --output reports/demo/suspicious.zh.html

```



Rule IDs and fingerprints remain stable across languages.



See [Interview Demo Kit](docs/interview_demo.md) for a full walkthrough and talking points.



---



## Testing



```bash

pytest

```



Current suite:



```text

148 passed

```



---



## Documentation



- [Interview Demo Kit](docs/interview_demo.md)

- [Demo walkthrough](docs/demo.md)

- [Design notes](docs/design.md)

- [Adapter architecture](docs/adapters.md)

- [LangGraph-style adapter](docs/langgraph_adapter.md)

- [GitHub Code Scanning](docs/github_code_scanning.md)

- [Chinese README](docs/README_zh.md)



---



## Project Structure



```text

skillflowguard/

  adapters/

  loader.py

  doc_parser.py

  llm_doc_parser.py

  rule_metadata.py

  rules.py

  analyzer.py

  report.py

  config.py

  cli.py



examples/

evaluation/

tests/

docs/

```



---



## Roadmap



- [x] Stable workflow-level risk analysis CLI

- [x] Optional LLM-assisted semantic extraction

- [x] Synthetic benchmark and evaluation reports

- [x] Generic JSON and LangGraph-style import adapters

- [x] SARIF output and GitHub Code Scanning integration

- [x] Centralized rule metadata and rule catalog CLI

- [x] Project-level TOML config support

- [x] Config validation and friendlier error messages

- [x] Suppression / allowlist support for known accepted risks

- [x] Baseline comparison for new-risk CI gating

- [x] Baseline text report summary for CI explainability

- [x] Stable finding fingerprints for baseline and SARIF identity

- [x] Dashboard-style HTML report with severity filters and finding cards

- [x] Pre-commit framework integration



---



## Changelog



See [CHANGELOG.md](CHANGELOG.md) for release history.



---



## License



MIT License.