https://github.com/tranhieutt/ai_pentest_system

AI-Powered Autonomous Penetration Testing System
https://github.com/tranhieutt/ai_pentest_system

pentest security

Last synced: about 2 months ago
JSON representation

AI-Powered Autonomous Penetration Testing System

Host: GitHub
URL: https://github.com/tranhieutt/ai_pentest_system
Owner: tranhieutt
Created: 2026-05-04T06:04:23.000Z (3 months ago)
Default Branch: main
Last Pushed: 2026-05-07T05:03:51.000Z (3 months ago)
Last Synced: 2026-05-07T06:10:00.085Z (3 months ago)
Topics: pentest, security
Language: Python
Homepage:
Size: 1.05 MB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: Changelog.md
- Agents: AGENTS.md

Awesome Lists containing this project

README

# AI-Powered Autonomous Penetration Testing System

Automated web security testing platform for authorized targets.

> Use only against systems you own or have explicit written permission to test.

---

## Current Status

Latest local verification on 2026-05-07:

```bash
.\.venv\Scripts\python.exe -m pytest
# 350 passed, 1 warning
```

The MVP runtime is deterministic: no LLM API calls are required during scanning, validation, attack chaining, or report generation. LLM enrichment remains post-MVP and opt-in.

Runtime persistence currently uses SQLite through `ScanRepository` / `AssetGraphRepository`. PostgreSQL remains the target production architecture documented in PRD and deployment planning, but the active repository backend is SQLite today.

---

## What Is Built

| Area | Status | Notes |
|---|---|---|
| Core 9-stage pipeline | Done | Target validation, recon, aggregation, context, scanning, bypass, validation, chaining, reporting |
| Advanced engines | Done | SSRF, smuggling, GraphQL, race condition, takeover, cloud file exposure, upload/path issues |
| False-positive reduction | Done | Baseline comparison, deduplication, smart filtering, confidence scoring |
| FastAPI service | Done | Scan start/status/list/stop, planning, subscan, asset APIs, GitHub repo scan |
| Queue execution | Done | Celery/Redis scan tasks, stop/revoke, subscan runner |
| Scan profiles | Done | YAML profiles with validation and dry-run planning |
| Asset inventory | Done | Subdomains, endpoints, ports, screenshots, technologies, DNS records |
| External adapters | Done | Tool registry, Nuclei, Dalfox, CRLFuzz, scope filtering, dedup |
| Secret hunting | Done | 48-pattern scanner, streaming mode, cross-file dedup, read-only validators |
| Identity Fabric | MVP done | M365 OIDC, device-code posture, Okta, ADFS, Google Workspace; deeper M365 probes deferred |
| Asset Graph | Runtime wired | Recon output writes graph nodes/edges while legacy dict output remains compatible |
| Graph attack chaining | Runtime wired | ChainEngine can add `chain_type="graph_traversal"` chains from Asset Graph adjacency |
| Contextual severity | Runtime wired | 20 YAML rules in `configs/severity_rules.yaml`, report shows adjusted severity/rule |
| Cloud bucket enum | Runtime wired | S3/GCS/Azure read-only HEAD/GET probes, findings and graph cloud bucket nodes |
| Continuous monitoring | Backlog | Recurring schedules, change detection, notifications |
| SARIF / PDF | Backlog | JSON exists; SARIF and PDF still pending |
| Back-off ladder | Backlog | Detection-aware HTTP back-off middleware still pending |
| RBAC / Dashboard / CI/CD ops | Backlog | Future service hardening and UI work |

---

## Architecture

```text
User / API / CLI
|
+-- FastAPI: src/api/main.py
+-- CLI: run_pentest.py
|
v
Celery / Redis queue
|
v
PentestOrchestrator
|
+-- Stage 1: Target validation and scope enforcement
+-- Stage 2: Recon and OSINT
+-- Stage 3: Asset aggregation + Asset Graph write path
+-- Stage 4: Deterministic context analysis
+-- Stage 5: Vulnerability engines + external adapters
+-- Stage 6: WAF bypass payload generation
+-- Stage 7: Validation and false-positive reduction
+-- Stage 8: Pattern + relationship + graph traversal chains
+-- Stage 9: Markdown / JSON reporting with severity rules
|
v
SQLite runtime DB + reports/ + artifacts/
```

---

## Key Features

- **Scope-first scanning**: exact domains, wildcard domains, URL-prefix scope, out-of-scope deny rules.
- **Deterministic MVP**: repeatable scan logic, no required LLM calls.
- **6 internal engines**: injection, web/client, auth/access, API/logic, file/path, infra/advanced.
- **External tools**: Nuclei, Dalfox, CRLFuzz, subfinder, waybackurls, theHarvester wrappers.
- **Asset Graph**: `asset_nodes` / `asset_edges` with typed node and edge contracts.
- **Attack chaining**: pattern matching, relationship matching, and graph traversal.
- **Secret handling**: regex detection plus read-only validators; tests enforce no PUT/POST/DELETE/PATCH in validators.
- **Contextual severity**: YAML rules SR-001..SR-020 can adjust severity/confidence in reports.
- **Cloud bucket enum**: S3/GCS/Azure read-only probes; confirmed listings become High/CONFIRMED.
- **Asset APIs**: endpoints, ports, screenshots, subdomains, inventory.
- **GitHub repo scan**: repository metadata, branch protection, workflow permissions, Dependabot, secret scanning, sensitive files.
- **YesWeHack foundation**: program loader/cache, automation policy gate, scope enforcer, per-program rate profiles.

---

## Vulnerability Coverage

| Engine | Examples |
|---|---|
| Injection | SQLi, NoSQLi, OS command injection, SSTI, CRLF |
| Web and client | Reflected XSS, DOM XSS, CSP weakness, clickjacking, CORS |
| Auth and access | IDOR/BOLA, JWT issues, auth bypass, password reset poisoning, admin panels |
| API and logic | GraphQL introspection, mass assignment, data exposure, race conditions |
| File and path | Upload RCE, LFI, RFI, directory traversal |
| Infra and advanced | SSRF, request smuggling, cache poisoning, host header, takeover, cloud exposure, prototype pollution |
| Secrets and cloud | API keys/tokens, read-only validation, cloud bucket existence/listing |
| External templates | Nuclei CVEs, exposures, misconfigs, default-logins |

---

## Requirements

- Python 3.11+
- Docker + Docker Compose recommended
- Redis for Celery queue
- SQLite runtime DB by default
- PostgreSQL service exists in Docker planning, but repository backend is not PostgreSQL yet
- Optional binaries: `nuclei`, `dalfox`, `crlfuzz`, `subfinder`, `waybackurls`, `theHarvester`

---

## Install

### Docker

```bash
cd ai-pentest-system
docker-compose up -d
```

Services: API, worker, Redis, PostgreSQL service, mounted data/report/artifact folders.

### Local

```bash
cd ai-pentest-system
python -m venv .venv
.\.venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
```

Optional environment variables:

```bash
AI_PENTEST_DB_PATH=data/ai_pentest.db
AI_PENTEST_ARTIFACT_ROOT=artifacts
AI_PENTEST_CELERY_BROKER_URL=redis://localhost:6379/0
AI_PENTEST_CELERY_RESULT_BACKEND=redis://localhost:6379/0
SHODAN_API_KEY=...
CENSYS_API_ID=...
CENSYS_API_SECRET=...
HIBP_API_KEY=...
GITHUB_TOKEN=...
YESWEHACK_API_TOKEN=...
NUCLEI_CONCURRENCY=1
NUCLEI_TEMPLATES_DIR=...
```

---

## CLI Usage

Choose one mode explicitly when possible:

- `internal`: pre-release testing for targets you own or have written authorization to test.
- `bounty`: Bug bounty/VDP assist; scope, policy, and rate limits are enforced before traffic starts.

Internal pre-release mode:

```bash
python run_pentest.py --mode internal https://staging.example.com --engine full
python run_pentest.py https://example.com
python run_pentest.py https://example.com --rate-limit 2
python run_pentest.py https://example.com --engine full
python run_pentest.py https://example.com --engine quick
python run_pentest.py https://example.com --dry-run
```

Bug bounty mode:

```bash
python run_pentest.py --list-bounty
python run_pentest.py --list-bounty --platform hackerone
python run_pentest.py --list-bounty --platform yeswehack
python run_pentest.py --mode bounty --bounty harman-international-web-applications --platform yeswehack --rate-limit 1
python run_pentest.py --bounty shopify
python run_pentest.py --bounty github --platform hackerone --rate-limit 1
python run_pentest.py --bounty example-program --platform yeswehack --rate-limit 1
```

Supported platforms: `hackerone`, `bugcrowd`, `intigriti`, `yeswehack`, `all`.

YesWeHack mode adds a pre-scan automation policy gate:

- `ALLOWED`: normal scoped scan.
- `MANUAL_ONLY`: recon-only path; vulnerability engines are skipped.
- `BLOCKED`: scan is not started. Do not bypass this status.

`YESWEHACK_API_TOKEN` is optional. Without it, the CLI uses bounty-targets-data and caches generated data in `data/yeswehack_programs.json`.

---

## REST API

Run API:

```bash
python -m uvicorn src.api.main:app --port 8000
```

Main endpoints:

| Method | Path | Description |
|---|---|---|
| GET | `/health` | Health check |
| POST | `/api/v1/scan/start` | Start scan |
| POST | `/api/v1/scan/plan` | Dry-run scan plan |
| GET | `/api/v1/scan/{scan_id}/status` | Scan status and activity |
| GET | `/api/v1/scans` | List scans |
| POST | `/api/v1/scan/{scan_id}/stop` | Stop scan |
| POST | `/api/v1/scan/{scan_id}/subscan` | Start subscan |
| GET | `/api/v1/scan/{scan_id}/subscans` | List subscans |
| GET | `/api/v1/scans/{scan_id}/assets` | Asset inventory |
| GET | `/api/v1/scans/{scan_id}/assets/subdomains` | Subdomain assets |
| GET | `/api/v1/scans/{scan_id}/assets/endpoints` | Endpoint inventory |
| GET | `/api/v1/scans/{scan_id}/assets/ports` | Port inventory |
| GET | `/api/v1/scans/{scan_id}/screenshots` | Screenshot gallery |
| POST | `/api/v1/github/scan` | GitHub repository security scan |

Interactive docs: `http://localhost:8000/docs`

API scan bodies use the same two modes:

```json
{"scan_mode": "internal", "target_url": "https://staging.example.com", "engine_profile": "full", "rate_limit": 2}
```

```json
{"scan_mode": "bounty", "bounty_handle": "harman-international-web-applications", "platform": "yeswehack", "rate_limit": 1}
```

---

## Scan Profiles

Profiles live under `configs/scan_engines/`.

| Profile | Description |
|---|---|
| `passive` | Passive recon and low-noise checks |
| `full` | Broad profile with internal engines and external tools |
| `quick` | Faster profile for high-signal checks |
| `recommended` | Balanced default profile |
| `vuln-only` | Vulnerability stages without broad recon |
| `osint-only` | OSINT and subdomain discovery |
| `port-only` | Port/service inventory |
| `screenshot-only` | Screenshot capture |

---

## Project Layout

```text
src/
api/ FastAPI app
pentest/
bounty/ Scope and policy validation
engines/infra/ Secret scan and read-only validators
models/ Finding, target, report, chain, asset graph models
pipeline/ Orchestrator, recon, engines, validation, reporting
attack_chaining/ Pattern/relationship/graph chain engine
reporting/ Report generator and severity engine
vuln_engines/ Internal vulnerability engines
recon/ Identity Fabric and cloud bucket enumeration
scan_engines/ YAML profile models, validation, planning
storage/ SQLite repository and asset graph repository
tools/ External adapters/runners
configs/
scan_engines/ Scan profiles
severity_rules.yaml Contextual severity rules SR-001..SR-020
tests/ Unit and integration tests
reports/ Generated reports
docs/ PRD, sprint plans, ADRs, architecture reports
```

---

## Safety Notes

- Confirm written authorization before scanning.
- Keep rate limits conservative.
- Treat bug bounty policy as source of truth.
- `ScopeValidator` filters target URLs before scan/adapters run.
- External adapters scope-filter before subprocess execution.
- Secret validators are read-only by invariant test.
- Review findings manually before disclosure or remediation.
- Back-off ladder is not implemented yet; use conservative rate limits on WAF-protected targets.

---

## Known Gaps

- PostgreSQL backend not active in repository layer.
- SARIF and PDF report export not implemented yet.
- Detection-aware HTTP back-off ladder not implemented yet.
- `PipelineRunner.vulnerability_scan` remains subscan-limited; full vuln scans use `PentestOrchestrator`.
- Deeper Identity Fabric probes are deferred.
- Continuous monitoring, notifications, RBAC, dashboard, and CI/CD operations remain backlog.

---

## Roadmap

- **Sprint 5**: back-off ladder, SARIF, PDF/reporting improvements, remaining service API work.
- **Sprint 6**: CI/CD, operations docs, backup/restore, service smoke tests.
- **Future**: RBAC, dashboard, continuous monitoring, optional LLM enrichment.

---

## Changelog

See [Changelog.md](Changelog.md).

## License

MIT. See [LICENSE](LICENSE) if present.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/tranhieutt/ai_pentest_system

Awesome Lists containing this project

README