{"id":50257600,"url":"https://github.com/tranhieutt/ai_pentest_system","last_synced_at":"2026-05-27T07:05:18.633Z","repository":{"id":356215473,"uuid":"1228545483","full_name":"tranhieutt/ai_pentest_system","owner":"tranhieutt","description":"AI-Powered Autonomous Penetration Testing System","archived":false,"fork":false,"pushed_at":"2026-05-07T05:03:51.000Z","size":1098,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-07T06:10:00.085Z","etag":null,"topics":["pentest","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tranhieutt.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-05-04T06:04:23.000Z","updated_at":"2026-05-07T05:03:55.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/tranhieutt/ai_pentest_system","commit_stats":null,"previous_names":["tranhieutt/ai_pentest_system"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/tranhieutt/ai_pentest_system","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tranhieutt%2Fai_pentest_system","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tranhieutt%2Fai_pentest_system/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tranhieutt%2Fai_pentest_system/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tranhieutt%2Fai_pentest_system/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tranhieutt","download_url":"https://codeload.github.com/tranhieutt/ai_pentest_system/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tranhieutt%2Fai_pentest_system/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33554820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pentest","security"],"created_at":"2026-05-27T07:05:17.567Z","updated_at":"2026-05-27T07:05:18.620Z","avatar_url":"https://github.com/tranhieutt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI-Powered Autonomous Penetration Testing System\n\nAutomated web security testing platform for authorized targets.\n\n9-stage deterministic pipeline | Scope enforcement | Asset Graph | Attack chaining | Celery queue | External adapters | Markdown/JSON reports\n\n\u003e Use only against systems you own or have explicit written permission to test.\n\n---\n\n## Current Status\n\nLatest local verification on 2026-05-07:\n\n```bash\n.\\.venv\\Scripts\\python.exe -m pytest\n# 350 passed, 1 warning\n```\n\nThe MVP runtime is deterministic: no LLM API calls are required during scanning, validation, attack chaining, or report generation. LLM enrichment remains post-MVP and opt-in.\n\nRuntime persistence currently uses SQLite through `ScanRepository` / `AssetGraphRepository`. PostgreSQL remains the target production architecture documented in PRD and deployment planning, but the active repository backend is SQLite today.\n\n---\n\n## What Is Built\n\n| Area | Status | Notes |\n|---|---|---|\n| Core 9-stage pipeline | Done | Target validation, recon, aggregation, context, scanning, bypass, validation, chaining, reporting |\n| Advanced engines | Done | SSRF, smuggling, GraphQL, race condition, takeover, cloud file exposure, upload/path issues |\n| False-positive reduction | Done | Baseline comparison, deduplication, smart filtering, confidence scoring |\n| FastAPI service | Done | Scan start/status/list/stop, planning, subscan, asset APIs, GitHub repo scan |\n| Queue execution | Done | Celery/Redis scan tasks, stop/revoke, subscan runner |\n| Scan profiles | Done | YAML profiles with validation and dry-run planning |\n| Asset inventory | Done | Subdomains, endpoints, ports, screenshots, technologies, DNS records |\n| External adapters | Done | Tool registry, Nuclei, Dalfox, CRLFuzz, scope filtering, dedup |\n| Secret hunting | Done | 48-pattern scanner, streaming mode, cross-file dedup, read-only validators |\n| Identity Fabric | MVP done | M365 OIDC, device-code posture, Okta, ADFS, Google Workspace; deeper M365 probes deferred |\n| Asset Graph | Runtime wired | Recon output writes graph nodes/edges while legacy dict output remains compatible |\n| Graph attack chaining | Runtime wired | ChainEngine can add `chain_type=\"graph_traversal\"` chains from Asset Graph adjacency |\n| Contextual severity | Runtime wired | 20 YAML rules in `configs/severity_rules.yaml`, report shows adjusted severity/rule |\n| Cloud bucket enum | Runtime wired | S3/GCS/Azure read-only HEAD/GET probes, findings and graph cloud bucket nodes |\n| Continuous monitoring | Backlog | Recurring schedules, change detection, notifications |\n| SARIF / PDF | Backlog | JSON exists; SARIF and PDF still pending |\n| Back-off ladder | Backlog | Detection-aware HTTP back-off middleware still pending |\n| RBAC / Dashboard / CI/CD ops | Backlog | Future service hardening and UI work |\n\n---\n\n## Architecture\n\n```text\nUser / API / CLI\n  |\n  +-- FastAPI: src/api/main.py\n  +-- CLI: run_pentest.py\n  |\n  v\nCelery / Redis queue\n  |\n  v\nPentestOrchestrator\n  |\n  +-- Stage 1: Target validation and scope enforcement\n  +-- Stage 2: Recon and OSINT\n  +-- Stage 3: Asset aggregation + Asset Graph write path\n  +-- Stage 4: Deterministic context analysis\n  +-- Stage 5: Vulnerability engines + external adapters\n  +-- Stage 6: WAF bypass payload generation\n  +-- Stage 7: Validation and false-positive reduction\n  +-- Stage 8: Pattern + relationship + graph traversal chains\n  +-- Stage 9: Markdown / JSON reporting with severity rules\n  |\n  v\nSQLite runtime DB + reports/ + artifacts/\n```\n\n---\n\n## Key Features\n\n- **Scope-first scanning**: exact domains, wildcard domains, URL-prefix scope, out-of-scope deny rules.\n- **Deterministic MVP**: repeatable scan logic, no required LLM calls.\n- **6 internal engines**: injection, web/client, auth/access, API/logic, file/path, infra/advanced.\n- **External tools**: Nuclei, Dalfox, CRLFuzz, subfinder, waybackurls, theHarvester wrappers.\n- **Asset Graph**: `asset_nodes` / `asset_edges` with typed node and edge contracts.\n- **Attack chaining**: pattern matching, relationship matching, and graph traversal.\n- **Secret handling**: regex detection plus read-only validators; tests enforce no PUT/POST/DELETE/PATCH in validators.\n- **Contextual severity**: YAML rules SR-001..SR-020 can adjust severity/confidence in reports.\n- **Cloud bucket enum**: S3/GCS/Azure read-only probes; confirmed listings become High/CONFIRMED.\n- **Asset APIs**: endpoints, ports, screenshots, subdomains, inventory.\n- **GitHub repo scan**: repository metadata, branch protection, workflow permissions, Dependabot, secret scanning, sensitive files.\n- **YesWeHack foundation**: program loader/cache, automation policy gate, scope enforcer, per-program rate profiles.\n\n---\n\n## Vulnerability Coverage\n\n| Engine | Examples |\n|---|---|\n| Injection | SQLi, NoSQLi, OS command injection, SSTI, CRLF |\n| Web and client | Reflected XSS, DOM XSS, CSP weakness, clickjacking, CORS |\n| Auth and access | IDOR/BOLA, JWT issues, auth bypass, password reset poisoning, admin panels |\n| API and logic | GraphQL introspection, mass assignment, data exposure, race conditions |\n| File and path | Upload RCE, LFI, RFI, directory traversal |\n| Infra and advanced | SSRF, request smuggling, cache poisoning, host header, takeover, cloud exposure, prototype pollution |\n| Secrets and cloud | API keys/tokens, read-only validation, cloud bucket existence/listing |\n| External templates | Nuclei CVEs, exposures, misconfigs, default-logins |\n\n---\n\n## Requirements\n\n- Python 3.11+\n- Docker + Docker Compose recommended\n- Redis for Celery queue\n- SQLite runtime DB by default\n- PostgreSQL service exists in Docker planning, but repository backend is not PostgreSQL yet\n- Optional binaries: `nuclei`, `dalfox`, `crlfuzz`, `subfinder`, `waybackurls`, `theHarvester`\n\n---\n\n## Install\n\n### Docker\n\n```bash\ncd ai-pentest-system\ndocker-compose up -d\n```\n\nServices: API, worker, Redis, PostgreSQL service, mounted data/report/artifact folders.\n\n### Local\n\n```bash\ncd ai-pentest-system\npython -m venv .venv\n.\\.venv\\Scripts\\activate\npip install -r requirements.txt\ncp .env.example .env\n```\n\nOptional environment variables:\n\n```bash\nAI_PENTEST_DB_PATH=data/ai_pentest.db\nAI_PENTEST_ARTIFACT_ROOT=artifacts\nAI_PENTEST_CELERY_BROKER_URL=redis://localhost:6379/0\nAI_PENTEST_CELERY_RESULT_BACKEND=redis://localhost:6379/0\nSHODAN_API_KEY=...\nCENSYS_API_ID=...\nCENSYS_API_SECRET=...\nHIBP_API_KEY=...\nGITHUB_TOKEN=...\nYESWEHACK_API_TOKEN=...\nNUCLEI_CONCURRENCY=1\nNUCLEI_TEMPLATES_DIR=...\n```\n\n---\n\n## CLI Usage\n\nChoose one mode explicitly when possible:\n\n- `internal`: pre-release testing for targets you own or have written authorization to test.\n- `bounty`: Bug bounty/VDP assist; scope, policy, and rate limits are enforced before traffic starts.\n\nInternal pre-release mode:\n\n```bash\npython run_pentest.py --mode internal https://staging.example.com --engine full\npython run_pentest.py https://example.com\npython run_pentest.py https://example.com --rate-limit 2\npython run_pentest.py https://example.com --engine full\npython run_pentest.py https://example.com --engine quick\npython run_pentest.py https://example.com --dry-run\n```\n\nBug bounty mode:\n\n```bash\npython run_pentest.py --list-bounty\npython run_pentest.py --list-bounty --platform hackerone\npython run_pentest.py --list-bounty --platform yeswehack\npython run_pentest.py --mode bounty --bounty harman-international-web-applications --platform yeswehack --rate-limit 1\npython run_pentest.py --bounty shopify\npython run_pentest.py --bounty github --platform hackerone --rate-limit 1\npython run_pentest.py --bounty example-program --platform yeswehack --rate-limit 1\n```\n\nSupported platforms: `hackerone`, `bugcrowd`, `intigriti`, `yeswehack`, `all`.\n\nYesWeHack mode adds a pre-scan automation policy gate:\n\n- `ALLOWED`: normal scoped scan.\n- `MANUAL_ONLY`: recon-only path; vulnerability engines are skipped.\n- `BLOCKED`: scan is not started. Do not bypass this status.\n\n`YESWEHACK_API_TOKEN` is optional. Without it, the CLI uses bounty-targets-data and caches generated data in `data/yeswehack_programs.json`.\n\n---\n\n## REST API\n\nRun API:\n\n```bash\npython -m uvicorn src.api.main:app --port 8000\n```\n\nMain endpoints:\n\n| Method | Path | Description |\n|---|---|---|\n| GET | `/health` | Health check |\n| POST | `/api/v1/scan/start` | Start scan |\n| POST | `/api/v1/scan/plan` | Dry-run scan plan |\n| GET | `/api/v1/scan/{scan_id}/status` | Scan status and activity |\n| GET | `/api/v1/scans` | List scans |\n| POST | `/api/v1/scan/{scan_id}/stop` | Stop scan |\n| POST | `/api/v1/scan/{scan_id}/subscan` | Start subscan |\n| GET | `/api/v1/scan/{scan_id}/subscans` | List subscans |\n| GET | `/api/v1/scans/{scan_id}/assets` | Asset inventory |\n| GET | `/api/v1/scans/{scan_id}/assets/subdomains` | Subdomain assets |\n| GET | `/api/v1/scans/{scan_id}/assets/endpoints` | Endpoint inventory |\n| GET | `/api/v1/scans/{scan_id}/assets/ports` | Port inventory |\n| GET | `/api/v1/scans/{scan_id}/screenshots` | Screenshot gallery |\n| POST | `/api/v1/github/scan` | GitHub repository security scan |\n\nInteractive docs: `http://localhost:8000/docs`\n\nAPI scan bodies use the same two modes:\n\n```json\n{\"scan_mode\": \"internal\", \"target_url\": \"https://staging.example.com\", \"engine_profile\": \"full\", \"rate_limit\": 2}\n```\n\n```json\n{\"scan_mode\": \"bounty\", \"bounty_handle\": \"harman-international-web-applications\", \"platform\": \"yeswehack\", \"rate_limit\": 1}\n```\n\n---\n\n## Scan Profiles\n\nProfiles live under `configs/scan_engines/`.\n\n| Profile | Description |\n|---|---|\n| `passive` | Passive recon and low-noise checks |\n| `full` | Broad profile with internal engines and external tools |\n| `quick` | Faster profile for high-signal checks |\n| `recommended` | Balanced default profile |\n| `vuln-only` | Vulnerability stages without broad recon |\n| `osint-only` | OSINT and subdomain discovery |\n| `port-only` | Port/service inventory |\n| `screenshot-only` | Screenshot capture |\n\n---\n\n## Project Layout\n\n```text\nsrc/\n  api/                         FastAPI app\n  pentest/\n    bounty/                    Scope and policy validation\n    engines/infra/             Secret scan and read-only validators\n    models/                    Finding, target, report, chain, asset graph models\n    pipeline/                  Orchestrator, recon, engines, validation, reporting\n      attack_chaining/         Pattern/relationship/graph chain engine\n      reporting/               Report generator and severity engine\n      vuln_engines/            Internal vulnerability engines\n    recon/                     Identity Fabric and cloud bucket enumeration\n    scan_engines/              YAML profile models, validation, planning\n    storage/                   SQLite repository and asset graph repository\n    tools/                     External adapters/runners\nconfigs/\n  scan_engines/                Scan profiles\n  severity_rules.yaml          Contextual severity rules SR-001..SR-020\ntests/                         Unit and integration tests\nreports/                       Generated reports\ndocs/                          PRD, sprint plans, ADRs, architecture reports\n```\n\n---\n\n## Safety Notes\n\n- Confirm written authorization before scanning.\n- Keep rate limits conservative.\n- Treat bug bounty policy as source of truth.\n- `ScopeValidator` filters target URLs before scan/adapters run.\n- External adapters scope-filter before subprocess execution.\n- Secret validators are read-only by invariant test.\n- Review findings manually before disclosure or remediation.\n- Back-off ladder is not implemented yet; use conservative rate limits on WAF-protected targets.\n\n---\n\n## Known Gaps\n\n- PostgreSQL backend not active in repository layer.\n- SARIF and PDF report export not implemented yet.\n- Detection-aware HTTP back-off ladder not implemented yet.\n- `PipelineRunner.vulnerability_scan` remains subscan-limited; full vuln scans use `PentestOrchestrator`.\n- Deeper Identity Fabric probes are deferred.\n- Continuous monitoring, notifications, RBAC, dashboard, and CI/CD operations remain backlog.\n\n---\n\n## Roadmap\n\n- **Sprint 5**: back-off ladder, SARIF, PDF/reporting improvements, remaining service API work.\n- **Sprint 6**: CI/CD, operations docs, backup/restore, service smoke tests.\n- **Future**: RBAC, dashboard, continuous monitoring, optional LLM enrichment.\n\n---\n\n## Changelog\n\nSee [Changelog.md](Changelog.md).\n\n## License\n\nMIT. See [LICENSE](LICENSE) if present.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftranhieutt%2Fai_pentest_system","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftranhieutt%2Fai_pentest_system","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftranhieutt%2Fai_pentest_system/lists"}