{"id":47306201,"url":"https://github.com/rhamenator/ai-scraping-defense","last_synced_at":"2026-04-02T00:31:58.237Z","repository":{"id":288695954,"uuid":"968935050","full_name":"rhamenator/ai-scraping-defense","owner":"rhamenator","description":"**In development and testing** This multi-platform system combats content scraping by unauthorized AI bots targeting creative, FOSS, or documentation sites. It employs a multi-layered defense strategy including real-time detection, tarpitting, honeypots, and behavioral analysis with optional AI/LLM integration for sophisticated threat assessment","archived":false,"fork":false,"pushed_at":"2026-03-17T19:49:23.000Z","size":32366,"stargazers_count":11,"open_issues_count":13,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-03-17T21:27:02.854Z","etag":null,"topics":["anti-scraping","api-gateway","bot-detection","devops","docker","https","kubernetes","load-balancer","lua","nginx","rate-limiting","redis","reverse-proxy","security-tools","tls","waf","web-security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rhamenator.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"docs/code_of_conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":"docs/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null},"funding":{"github":"rhamenator"}},"created_at":"2025-04-19T02:51:14.000Z","updated_at":"2026-03-17T19:49:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"1e24c1eb-7851-4b41-9616-251c10668c03","html_url":"https://github.com/rhamenator/ai-scraping-defense","commit_stats":null,"previous_names":["rhamenator/ai-scraping-defense"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rhamenator/ai-scraping-defense","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhamenator%2Fai-scraping-defense","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhamenator%2Fai-scraping-defense/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhamenator%2Fai-scraping-defense/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhamenator%2Fai-scraping-defense/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rhamenator","download_url":"https://codeload.github.com/rhamenator/ai-scraping-defense/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhamenator%2Fai-scraping-defense/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31293383,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anti-scraping","api-gateway","bot-detection","devops","docker","https","kubernetes","load-balancer","lua","nginx","rate-limiting","redis","reverse-proxy","security-tools","tls","waf","web-security"],"created_at":"2026-03-17T08:13:33.383Z","updated_at":"2026-04-02T00:31:58.197Z","avatar_url":"https://github.com/rhamenator.png","language":"Python","funding_links":["https://github.com/sponsors/rhamenator"],"categories":[],"sub_categories":[],"readme":"# AI Scraping Defense\n\n\n[![codecov](https://codecov.io/gh/rhamenator/ai-scraping-defense/branch/main/graph/badge.svg)](https://codecov.io/gh/rhamenator/ai-scraping-defense)\n\nThis project provides a multi-layered, microservice-based defense system against sophisticated AI-powered web scrapers and malicious bots.\n\n## Key Features\n\n- **Layered Defense:** Uses a combination of Nginx, Lua, and a suite of Python microservices for defense in depth.\n- **Intelligent Analysis:** Employs heuristics, a machine learning model, and optional LLM integration to analyze suspicious traffic.\n- **Model Agnostic:** A flexible adapter pattern allows for easy integration with various ML models and LLM providers (OpenAI, Mistral, Cohere, etc.).\n- **Active Countermeasures:** Includes a \"Tarpit API\" to actively waste the resources of confirmed bots.\n- **End-User Verification:** Optional reCAPTCHA challenge service logs successful verifications for training data.\n- **Rate Limiting:** Adaptive per-IP limits updated by a small daemon writing to Nginx.\n- **Community Blocklist:** Optional daemon to sync IPs from a shared blocklist service.\n- **Public Community Blocklist Service:** Lightweight FastAPI app for contributors to share and fetch malicious IPs.\n- **Federated Threat Sharing:** Peer-to-peer sync exchanges blocklisted IPs between deployments.\n- **Containerized:** Fully containerized with Docker and ready for deployment on Kubernetes.\n- **Multi-Tenant Ready:** Namespace configuration and Redis keys with `TENANT_ID` for easy isolation.\n- **Optional Cloud Integrations:** Toggle CDN caching, DDoS mitigation, managed TLS, and a Web Application Firewall using environment variables.\n- **Plugin API:** Drop-in Python modules allow custom rules to extend detection logic.\n- **Anomaly Detection via AI:** Move beyond heuristics and integrate anomaly detection models for more adaptive security.\n- **API Sequence Anomaly Detection:** Markov-based scoring highlights unusual request patterns.\n- **Crawler Authentication \u0026 Pay-Per-Crawl:** Token registry and usage accounting enable monetization experiments. Whether you actually permit bots to pay for access or keep them blocked is entirely your choice.\n- **Pay-Per-Crawl Blockchain Logging:** Optional hash-chained audit log for crawler billing. See `docs/PAY_PER_CRAWL_BLOCKCHAIN_LOG.md`.\n- **Payment Gateway Integration:** Multi-provider gateways (`StripeGateway`,\n  `PayPalGateway`, `BraintreeGateway`, `SquareGateway`,\n  `AdyenGateway`, `AuthorizeNetGateway`, or a generic HTTP backend) handle crawler account creation,\n  balance lookups, charging and refunds against external billing APIs.\n- **AI Labyrinth Honeypots:** Optional endless maze pages trap persistent bots.\n- **Zero Trust Risk \u0026 Attack Scoring:** Placeholder modules provide risk analytics hooks.\n- **Automated Configuration Recommendations:** AI-driven service that analyzes traffic patterns and suggests firewall and tarpit tuning.\n- **Audit Logging:** Sensitive actions are written to a rotating `audit.log` for forensic review.\n- **Anomaly Alerting Actions:** Alert, blocklist, or escalate based on anomaly score. See `docs/ANOMALY_ALERTING.md`.\n- **RBAC Controls:** Admin endpoints verify an `ADMIN_UI_ROLE` environment variable and reject non-admin users.\n- **Admin UI SSO:** Optional OIDC/SAML authentication with MFA. See `docs/ADMIN_UI_SSO.md`.\n- **Model Version Metrics:** Prometheus gauge `model_version_info` exposes the running ML model version.\n- **CORS \u0026 CSP Headers:** The Admin UI sets CORS policies and a default Content-Security-Policy header. `ADMIN_UI_CORS_ORIGINS` defaults to `http://localhost` and must list explicit origins; wildcard `*` is rejected when credentials are allowed.\n- **Additional Security Headers:** Nginx now sends `Referrer-Policy`, `Permissions-Policy`, and `X-Permitted-Cross-Domain-Policies` headers by default. See `docs/SECURITY_HEADERS.md`.\n- **Structured Errors:** API errors include stable codes and request IDs. See `docs/ERROR_HANDLING.md`.\n- **Standardized HTTP Retries:** Shared retry/backoff and circuit breaker policy. See `docs/HTTP_CLIENT_POLICY.md`.\n\n## Repository Structure\n\n- `src/` – core Python microservices and shared modules.\n- `scripts/` – setup helpers and deployment utilities.\n- `rag/` – retrieval-augmented generation resources and training tools.\n- `docs/` – project documentation.\n\n## CI Automation: Audits + Autofix\n\n- Workflows provide category audits plus automated fixes with guardrails:\n  - `/.github/workflows/master-problem-detection.yml`: orchestrates all categories; input `autofix=true` opens PRs per category.\n  - `/.github/workflows/comprehensive-*-audit.yml`: run a single category with `autofix` (defaults to true).\n  - `/.github/workflows/autofix.yml`: generic autofix launcher without running audits.\n- Guardrails compare pre/post results (flake8, bandit, eslint, yamllint, shellcheck, hadolint, markdownlint, golangci-lint, tflint/tfsec) and run tests if present. If any metric regresses, the workflow opens an issue and does not enable automerge.\n- PRs are labeled with `autofix` and the category, and automerge is enabled only when guardrails are clean.\n- The legacy `security-autofix.yml` now delegates to the generic autofixer for compatibility.\n- The `security-controls` workflow acts as the security baseline gate. See `docs/SECURITY_BASELINE_GATE.md`.\n\n## Release Path\n\n- `ci-tests.yml` is the primary cross-platform validation workflow for pushes to `main` and pull requests.\n- `tests.yml` now serves as a dedicated Rust nightly smoke workflow instead of duplicating the main PR test path.\n- `security-attack-regression.yml` is the deterministic PR-time ingress regression gate; it boots the Compose stack locally and asserts expected security behaviour.\n- `kali-security-sweep.yml` is the broader external attack sweep intended for a self-hosted Kali runner against a preview or staging target.\n- tagged releases publish signed container images through `/.github/workflows/release-images.yml`.\n- tagged releases also publish versioned installer bundles through `/.github/workflows/release-bundles.yml`.\n\nSee [docs/release_checklist.md](docs/release_checklist.md) and [docs/release_artifacts.md](docs/release_artifacts.md) for the expected release process and artifact policy.\n\n## Architecture Overview\n\nThe following diagram provides a high-level view of how the major components interact. Note that the AI Service merely receives webhook data and enqueues it for the Escalation Engine, which performs the actual analysis. See [docs/architecture.md](docs/architecture.md) for a deeper explanation.\n\n```mermaid\ngraph TD\n    subgraph \"User / Bot Traffic\"\n        direction LR\n        User[\"👤 User\"]\n        Bot[\"🤖 Bot\"]\n    end\n\n    subgraph \"Defense System\"\n        direction TB\n        Nginx[\"🛡️ Nginx Proxy w/ Lua\"]\n\n        subgraph \"Analysis \u0026 Logic (Python Microservices)\"\n            direction LR\n            AIService[\"AI Service Webhook\"]\n            EscalationEngine[\"🧠 Escalation Engine\"]\n            AdminUI[\"📊 Admin UI\"]\n            CloudDashboard[\"☁️ Cloud Dashboard\"]\n            ConfigRecommender[\"🔧 Config Recommender\"]\n            BlocklistSync[\"🔄 Blocklist Sync\"]\n            PeerSync[\"🔄 Peer Sync\"]\n            RateLimitDaemon[\"⚙️ Rate Limit Daemon\"]\n        end\n\n        subgraph \"Countermeasures\"\n            TarpitAPI[\"🕸️ Tarpit API\"]\n        end\n\n        subgraph \"Data \u0026 State Stores\"\n            direction LR\n            Redis[\"⚡ Redis\\n(Blocklist, Cache)\"]\n            Postgres[\"🐘 PostgreSQL\\n(Markov Data)\"]\n        end\n    end\n\n    subgraph \"External Services\"\n        LLM[\"☁️ LLM APIs\\n(OpenAI, Mistral, etc.)\"]\n        CommunityBlocklist[\"☁️ Community Blocklist\"]\n        PeerDeployments[\"☁️ Peer Deployments\"]\n    end\n\n    User -- \"Legitimate Request\" --\u003e Nginx\n    Bot -- \"Suspicious Request\" --\u003e Nginx\n\n    Nginx -- \"Block Immediately\" --\u003e Bot\n    Nginx -- \"Forward for Analysis\" --\u003e AIService\n    Nginx -- \"Serve Content\" --\u003e User\n    Nginx -- \"Redirect to Tarpit\" --\u003e Bot\n\n    AIService -- \"Queues Request\" --\u003e EscalationEngine\n    EscalationEngine -- \"Reads/Writes\" --\u003e Redis\n    EscalationEngine -- \"Reads\" --\u003e Postgres\n    EscalationEngine -- \"Calls for Final Verdict\" --\u003e LLM\n    EscalationEngine -- \"Updates\" --\u003e AdminUI\n    AdminUI -- \"Streams Metrics\" --\u003e CloudDashboard\n    AdminUI -- \"Feeds\" --\u003e ConfigRecommender\n    ConfigRecommender -- \"Suggestions\" --\u003e AdminUI\n    BlocklistSync -- \"Update\" --\u003e Redis\n    PeerSync -- \"Share IPs\" --\u003e Redis\n    RateLimitDaemon -- \"Adjust Limits\" --\u003e Nginx\n    BlocklistSync -- \"Fetch IPs\" --\u003e CommunityBlocklist\n    PeerSync -- \"Exchange IPs\" --\u003e PeerDeployments\n\n    AdminUI -- \"Manages\" --\u003e Redis\n\n    TarpitAPI -- \"Reads\" --\u003e Postgres\n```\n## Threat Model\n\nSee [docs/threat_model.md](docs/threat_model.md) for the adversaries and attack vectors this project targets.\n\n## Public Blocklist API\n\nThe public blocklist service requires authentication. Set the\n`PUBLIC_BLOCKLIST_API_KEY` environment variable and supply its value in the\n`X-API-Key` header when calling the `/report` endpoint. Requests missing the\nheader or using the wrong key receive an HTTP 401 response.\n\n## Beginner Quickstart\n\nBelow are platform-specific steps to install the prerequisites and run the helper script. The\nscript copies `sample.env` (use `sample.env.min` for a minimal config), generates secrets,\ninstalls dependencies, and starts Docker Compose. See\n[docs/getting_started.md](docs/getting_started.md) for a deeper walkthrough.\n\n### Linux\n1. [Install Docker Engine](https://docs.docker.com/engine/install/) and ensure it is running.\n2. Install Python 3.10 or newer (`sudo apt install python3 python3-venv` on Debian-based distros).\n3. Clone the repository and execute the Linux installer:\n   ```bash\n   git clone https://github.com/your-username/ai-scraping-defense.git\n   cd ai-scraping-defense\n   sudo ./scripts/linux/install.sh\n   ```\n4. When the containers finish starting, open [http://localhost:5002](http://localhost:5002) to view the Admin UI dashboard.\n5. If you see \"Cannot connect to the Docker daemon,\" start the service with\n   `sudo systemctl start docker` and verify it with `docker info`.\n\n### macOS\n1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop) and start the application.\n2. Install Python 3.10+ using [Homebrew](https://brew.sh/): `brew install python`.\n3. Clone the repository and run:\n   ```bash\n   git clone https://github.com/your-username/ai-scraping-defense.git\n   cd ai-scraping-defense\n    ./scripts/macos/install.zsh\n   ```\n4. When the containers finish starting, visit [http://localhost:5002](http://localhost:5002) to open the Admin UI.\n5. If containers fail to start, confirm Docker Desktop is running by opening the Docker menu.\n6. For a step-by-step walkthrough, including security tooling, see the [macOS setup guide](docs/macos_setup.md).\n\n### Windows\n1. Install [Docker Desktop for Windows](https://www.docker.com/products/docker-desktop) and ensure it is running.\n2. Install Python 3.10+ from [python.org](https://www.python.org/downloads/windows/).\n3. In an **Administrator PowerShell** window run:\n   ```powershell\n   git clone https://github.com/your-username/ai-scraping-defense.git\n   cd ai-scraping-defense\n   .\\scripts\\windows\\install.ps1\n   ```\n4. After the containers start, browse to [http://localhost:5002](http://localhost:5002) to access the Admin UI.\n5. If Docker commands are not found, verify Docker Desktop is running and try `docker version`.\n\n## Quick Local Setup\n\nRun the automated installer after cloning the repository:\n\n```bash\ngit clone https://github.com/your-username/ai-scraping-defense.git\ncd ai-scraping-defense\n\ncp sample.env .env\npython scripts/validate_env.py\n\nsudo ./scripts/linux/install.sh          # Linux\n./scripts/macos/install.zsh          # macOS\n\n```\n\nFor the security testing environment, a helper `scripts/linux/security_setup.sh` script installs all Python requirements and security tools used by `scripts/linux/security_scan.sh`.\n\nOn Windows, open an **Administrator PowerShell** window and run `scripts\\windows\\install.ps1` instead.\n\nThe Linux installer generates local secrets, installs Python requirements with\n`pip install -r requirements.txt -c constraints.txt`, re-runs\n`python scripts/validate_env.py`, launches Docker Compose through the selected\nreverse-proxy helper, and runs the Linux smoke test for you. For Linux-specific\nrollback and uninstall steps, see [docs/linux_installer.md](docs/linux_installer.md).\nThe stack requires Rust 1.78.0. `mise` (or `rustup`) installs this toolchain automatically.\n\nIf you prefer downloading a tagged release bundle instead of cloning the\nrepository, use the release `.zip` bundle on Windows or the `.tar.gz` bundle on\nLinux and macOS, then run the same installer entrypoints from the extracted\nrepository root.\n\nIf you want local GGUF inference through the `llamacpp://` adapter, install the\noptional native dependency set after the base environment is ready:\n\n```bash\npip install -r requirements-local-llm.txt -c constraints.txt\n```\nIf you see a warning about `idiomatic_version_file_enable_tools`, silence it with:\n\n```bash\nmise settings add idiomatic_version_file_enable_tools rust\n```\n\nYou can ignore the message if Rust 1.78.0 is already installed.\n\n\nFor a step-by-step explanation of each setup script, see [docs/getting_started.md](docs/getting_started.md).\nIf you run into errors during setup, consult [docs/troubleshooting.md](docs/troubleshooting.md).\n\n## Manual Local Setup\n\nFollow these steps if you prefer to configure everything yourself.\n\n1. **Clone the Repository:**\n\n    ```bash\n    git clone https://github.com/your-username/ai-scraping-defense.git\n    cd ai-scraping-defense\n    ```\n\n2. **Create Environment File:**\n    Copy the example environment file or use the interactive helper to customise settings.\n\n    ```bash\n    cp sample.env .env\n    # optional guided setup\n    python scripts/interactive_setup.py\n    ```\n\n    The interactive helper can also launch Docker Compose or deploy to\n    Kubernetes when it finishes, if you choose to proceed automatically.\n    If you agree when prompted, your secrets are saved in a local SQLite\n    database at `secrets/local_secrets.db`. Delete this file or answer **n**\n    during the prompt to disable the database and clear stored values.\n\n    Open `.env` and review the defaults. Set `TENANT_ID` for isolated deployments and add any API keys you plan to use. The sample file now defaults the containerized Nginx proxy to `8088`/`8443` so it can coexist with host Apache or nginx on Ubuntu. For **production** or takeover deployments, update `NGINX_HTTP_PORT` to `80` and `NGINX_HTTPS_PORT` to `443` once the stack is the only web listener on the host. Use `REAL_BACKEND_HOSTS` to supply a comma-separated list of backend servers for load balancing or `REAL_BACKEND_HOST` for a single destination. See [docs/ubuntu_reverse_proxy.md](docs/ubuntu_reverse_proxy.md) for the recommended host reverse-proxy topology.\nFor a full walkthrough of bringing the stack live, review [docs/test_to_production.md](docs/test_to_production.md).\n\n3. **Set Up Python Virtual Environment:**\n    Run the setup script to create a virtual environment and install all Python dependencies.\n    After the environment is created, install the project requirements with pinned\n    constraints:\n\n    ```bash\n    pip install -r requirements.txt -c constraints.txt\n    ```\n\n    If you plan to use the Kubernetes integrations, install the optional\n    dependency set instead:\n\n    ```bash\n    pip install -r requirements-kubernetes.txt -c constraints.txt\n    ```\n\n    Note: the Kubernetes client currently requires `urllib3\u003c2.4.0`, which\n    has known CVEs. Use the Kubernetes requirements only if you accept that\n    trade-off until upstream loosens the constraint.\n\n    To audit the optional Kubernetes dependency set in CI, run the\n    `security-controls` workflow manually and set `audit_kubernetes=true`.\n\n    To run an optional Kubernetes smoke test in CI (with kind on GitHub\n    runners), trigger the `k8s-smoke` workflow. It creates a throwaway\n    cluster, verifies connectivity, and exercises the Kubernetes client.\n\n    *On Linux or macOS:*\n\n    ```bash\n    sudo bash ./reset_venv.sh\n    ```\n\n    *On Windows (PowerShell as Administrator):*\n\n    ```powershell\n    .\\reset_venv.ps1\n    ```\n\n4. **Generate Secrets:**\n    Run the secret generation script to create passwords for the database, Admin UI, and other services. It writes a `kubernetes/secrets.yaml` file and prints the credentials to your console. When run with `--update-env` (as in the interactive setup), the script also updates `.env` and writes the database and Redis passwords to `secrets/pg_password.txt` and `secrets/redis_password.txt` for Docker Compose. Those two Compose-mounted files are written with container-readable permissions for local and CI use.\n\n    The local setup scripts lock down the `secrets/` directory (Unix chmod or\n    Windows ACLs). If you store secrets elsewhere, you can leave these files\n    empty or remove them after testing.\n\n    Local secrets hygiene checklist:\n    - Use full-disk encryption (LUKS, FileVault, BitLocker) on machines that\n      store secrets files.\n    - Exclude `secrets/` from any cloud sync or backup tools.\n    - Prefer a secrets manager (Vault, Docker secrets, or Kubernetes secrets)\n      for production deployments.\n\n    *On Linux or macOS:*\n\n    ```bash\n     bash ./scripts/linux/generate_secrets.sh --update-env\n     # export credentials to a JSON file\n     bash ./scripts/linux/generate_secrets.sh --export-path my_secrets.json\n    ```\n\n    *On Windows:*\n\n    ```powershell\n     .\\scripts\\windows\\Generate-Secrets.ps1\n     # save credentials to a JSON file\n     .\\scripts\\windows\\Generate-Secrets.ps1 -ExportPath my_secrets.json\n    ```\n\n5. **Enable HTTPS (Optional):**\n    Edit `.env` and set `ENABLE_HTTPS=true` with paths to your certificate and key.\n    The setup scripts generate a self-signed certificate in `nginx/certs/` if\n    one does not exist, which is fine for local testing. Replace it with a\n    trusted certificate before production use.\n\n    ```bash\n    ENABLE_HTTPS=true\n    TLS_CERT_PATH=./nginx/certs/tls.crt\n    TLS_KEY_PATH=./nginx/certs/tls.key\n    ```\n\n6. **Launch the Stack:**\n    Build and start the services with Docker Compose.\n\n    ```bash\n    docker-compose up --build -d\n    ```\n\n    For an optional hardened profile (drops Linux capabilities and enforces\n    `no-new-privileges` on core services), layer in the security override file:\n\n    ```bash\n    docker compose -f docker-compose.yaml -f docker-compose.security.yml up -d\n    ```\n\n    To enable optional Nginx performance tuning directives, add the\n    performance override file:\n\n    ```bash\n    docker compose -f docker-compose.yaml -f docker-compose.performance.yml up -d\n    ```\n\n    If you'd like to try the proxy in front of a WordPress site, run `./setup_wordpress_website.sh` (or `./setup_wordpress_website.ps1` on Windows) instead. It launches WordPress and MariaDB containers and sets `REAL_BACKEND_HOST` automatically. For a smaller test, `./setup_fake_website.sh` creates a simple nginx site and updates the variable in the same way.\n\n7. **Access the Services:**\n    - **Admin UI:** `http://localhost:5002`\n    - **Cloud Dashboard:** `http://localhost:5006`\n    - **Cloud Proxy:** `http://localhost:8008`\n    - **Prompt Router:** `http://localhost:8009`\n    - **Nginx Proxy (recommended):** `http://localhost:8088`\n    - **Nginx Proxy HTTPS (if enabled):** `https://localhost:8443`\n    - **Apache Proxy (optional alternative):** `http://localhost:8080`\n\n## Optional Features\n\nSeveral integrations are disabled by default to keep the stack lightweight. You can enable them by editing `.env`:\n\n - **Web Application Firewall** (`ENABLE_WAF`) – Mounts ModSecurity rules from `WAF_RULES_PATH` for additional filtering. See [docs/waf_setup.md](docs/waf_setup.md) for setup steps.\n- **Global CDN** (`ENABLE_GLOBAL_CDN`) – Uses Cloudflare for edge cache purge integration. Requires `CLOUD_CDN_ZONE_ID` and `CLOUD_CDN_API_TOKEN` (or `CLOUD_CDN_API_TOKEN_FILE`).\n- **DDoS Mitigation** (`ENABLE_DDOS_PROTECTION`) – The optional `ddos_guard.py` log monitor detects flooding patterns, classifies them as HTTP floods or volumetric attacks, and reports offenders to the local escalation engine. Requests may also be forwarded to an external provider.\n- **Managed TLS** (`ENABLE_MANAGED_TLS`) – Automatically issues certificates via `TLS_PROVIDER` with contact email `TLS_EMAIL`.\n- **CAPTCHA Verification** – Populate `CAPTCHA_SECRET` to activate reCAPTCHA challenges.\n- **Fail2ban** – Start the `fail2ban` container to insert firewall rules based on blocked IPs. See [docs/fail2ban.md](docs/fail2ban.md) for details.\n- **LLM Tarpit Pages** (`ENABLE_TARPIT_LLM_GENERATOR`) – Use an LLM to generate fake pages when a model URI is provided.\n- **Admin UI Two-Factor Auth** – Set `ADMIN_UI_2FA_SECRET` (or `ADMIN_UI_2FA_SECRET_FILE`) and provide a TOTP in the `X-2FA-Code` header.\n\nIf your ISP blocks inbound hosting ports, expose the local stack through Cloudflare Tunnel:\n\n```bash\n# Linux quick temporary URL (trycloudflare.com)\n./scripts/linux/start_cloudflare_tunnel.sh\n\n# Linux named tunnel with your existing Cloudflare Zero Trust tunnel token\nCLOUDFLARE_TUNNEL_TOKEN=\u003cyour_token\u003e ./scripts/linux/start_cloudflare_tunnel.sh\n```\n\n```zsh\n# macOS\n./scripts/macos/start_cloudflare_tunnel.zsh\n```\n\n```powershell\n# Windows PowerShell\n.\\scripts\\windows\\start_cloudflare_tunnel.ps1\n```\n\n## Project Structure\n\n- `src/`: Contains all Python source code for the microservices.\n- `kubernetes/`: Contains all Kubernetes manifests for production deployment.\n- `nginx/`: Nginx and Lua configuration files.\n- `docs/`: Project documentation, including architecture and data flows. See\n  [docs/antivirus.md](docs/antivirus.md) if your antivirus flags any files.\n- `test/`: Unit tests for the Python services.\n- `sample.env`: Template for local development configuration. See the [Configuration Reference](docs/configuration.md) for a description of every variable.\n- `Dockerfile`: A single Dockerfile used to build the base image for all Python services.\n- `jszip-rs/`: Rust implementation of the fake JavaScript archive generator.\n- `markov-train-rs/`: Rust implementation of the Markov training utility.\n\nWhen running in the security testing environment, execute `./scripts/linux/security_setup.sh` first to install all dependencies required for the unit tests and security scans.\n\n### Running Multiple Tenants\n\nCreate an `.env` file per tenant with a unique `TENANT_ID` and port mappings.\nLaunch each stack with a distinct project name so Docker Compose keeps the\nservices isolated:\n\n```bash\ndocker compose --env-file .env.siteA -p siteA up -d\ndocker compose --env-file .env.siteB -p siteB up -d\n```\n\nRedis keys and SQLite records are automatically prefixed with the tenant ID.\n## Configuring AI Models\n\nThe detection services load a model specified by the `MODEL_URI` value in `.env`. Examples include a local scikit-learn file or an external API:\n\n```bash\nMODEL_URI=sklearn:///app/models/bot_detection_rf_model.joblib\nMODEL_URI=openai://gpt-4-turbo\nMODEL_URI=mistral://mistral-large-latest\n```\n\nFor remote providers, set the corresponding API key in `.env` (`OPENAI_API_KEY`, `MISTRAL_API_KEY`, etc.).\n\nWhen referencing a local file with the `sklearn://` scheme, the model must reside\nin a trusted directory. By default, the adapters only load models from the\n`models/` folder (override with the `TRUSTED_MODEL_DIR` environment variable).\nFiles outside this directory are ignored to avoid executing untrusted code.\n\nAll LLM requests from the Escalation Engine are sent to the **Prompt Router**. The\nrouter constructs the final target URL from `PROMPT_ROUTER_HOST` and\n`PROMPT_ROUTER_PORT` and decides whether to use a local model or forward the\nprompt to the cloud proxy. The default port values are shown in\n`sample.env`:\n\n```env\n# excerpt from sample.env\nPROMPT_ROUTER_PORT=8009\nPROMETHEUS_PORT=9090\nGRAFANA_PORT=3000\n```\n\n## Model Adapter Guide\n\nThe [Model Adapter Guide](docs/model_adapter_guide.md) explains all available schemes and how to extend the system with new providers.\n\n## Markov Training Utility (Rust)\n\n`markov-train-rs` contains a high-performance implementation of the corpus loader.\nIt exposes a `train_from_corpus_rs` function callable from Python via PyO3.\nThe repository is pinned to **Rust 1.78.0** via `rust-toolchain.toml`. Ensure\nthat toolchain is installed before building the Rust crates.\n\nBuild the extension with Cargo:\n\n```bash\ncd markov-train-rs\ncargo build --release\n```\n\nOnce built, call the function to populate PostgreSQL:\n\n```bash\npython -c \"import markov_train_rs, os; markov_train_rs.train_from_corpus_rs(os.environ['CORPUS_FILE_PATH'])\"\n```\n\nEnsure the usual `PG_HOST`, `PG_PORT`, `PG_DBNAME`, `PG_USER`, and `PG_PASSWORD_FILE` environment variables are set so the library can connect to PostgreSQL.\n\n## JS ZIP Generator (Rust)\n\n`jszip-rs` provides an optional Rust backend for generating the large fake JavaScript archives used by the tarpit. It can be built with Cargo:\n\n```bash\ncd jszip-rs\ncargo build --release\n```\nThe build requires Python development headers (e.g. `python3-dev` on Debian-based systems) so that PyO3 can link against `libpython`.\n\nThe resulting `jszip_rs` Python module will be used automatically if available.\n\n## Training the Detection Model\n\nThe `src/rag/training.py` script now accepts a `--model` flag to select which\nmachine learning algorithm to train. Supported values are `rf` (RandomForest,\ndefault), `xgb` (XGBoost), and `lr` (Logistic Regression). Example usage:\n\n```bash\npython src/rag/training.py --model xgb\n```\n\nThis flexibility makes it easy to experiment with different classifiers.\n\nThe fine-tuning path now expects provenance sidecars next to exported\n`*.jsonl` datasets. The normal `src/rag/training.py` export flow writes these\nmetadata files automatically; imported datasets should provide matching\n`*.metadata.json` files or `src/rag/finetune.py` will reject them by default.\nSee [docs/local_model_training.md](docs/local_model_training.md) for the trust\nboundary and review expectations.\n\n## Quick Kubernetes Deployment\n\nRun the helper script to deploy everything to Kubernetes in one step. Ensure the\n`kubernetes/secrets.yaml` file already exists (generate it with\n`scripts/linux/generate_secrets.sh` or the interactive setup):\n\n```bash\n./scripts/linux/quick_deploy.sh       # or .\\scripts\\windows\\quick_deploy.ps1 on Windows\n```\n\nIf you're on Windows, run `scripts\\windows\\quick_deploy.ps1` from an **Administrator PowerShell** window.\n\nThe script applies all manifests using `kubectl`; it does not generate secrets.\n\n## Manual Kubernetes Deployment\n\nFor a detailed, step-by-step guide see [docs/kubernetes_deployment.md](docs/kubernetes_deployment.md). The `scripts/linux/deploy.sh` and `scripts/windows/deploy.ps1` scripts provide a manual approach if you need more control.\n\n## Cloud Deployment (GKE Example)\n\nTo deploy the stack to a managed Kubernetes service such as Google Kubernetes Engine, follow the instructions in [docs/cloud_provider_deployment.md](docs/cloud_provider_deployment.md). Convenience scripts are provided for automation:\n\n```bash\n./scripts/linux/gke_deploy.sh       # or .\\scripts\\windows\\gke_deploy.ps1 on Windows\n```\n\n### GitHub Actions Runner Deployment\n\nA workflow named `linux-stack.yml` now demonstrates a deployment to the GitHub runner environment. Trigger it from the **Actions** tab to see a deployment artifact uploaded. See [docs/github_actions_deployment.md](docs/github_actions_deployment.md) for more information.\n\n\n## Load Testing Helpers\n\nTo experiment with the stack's performance under load, run the helper script:\n\n```bash\n./setup_load_test_suite.sh (or ./setup_load_test_suite.ps1 on Windows)\n```\n\nIt installs common open-source tools such as **wrk**, **siege**, **ab**, **k6**, and **locust**. Use them responsibly and only against environments you control.\n\nAfter installing the tools, you can run a basic stress test using the provided scripts:\n\n```powershell\n./stress_test.ps1 -Target http://your-linux-host:8088 -VUs 50 -DurationSeconds 30\n```\n\n```bash\n./stress_test.sh http://your-linux-host:8088\n```\n\n## Security Scan Helper\n\nThe optional scripts `scripts/linux/security_scan.sh` and `scripts/windows/security_scan.ps1` automate tools such as **Nmap**, **Nikto**, and **Trivy** to perform vulnerability checks. Install these dependencies and run them with the appropriate privileges so network scans can complete. See [docs/security_scan.md](docs/security_scan.md) for more details. **Use these scripts only on systems you own or have permission to test.**\n\n## Security Alert Management\n\n### Creating Issues from Security Alerts\n\nThe `scripts/create_issues_from_alerts.py` script automatically creates GitHub issues from code scanning and secret scanning alerts, making it easy to track and manage security findings.\n\n**Quick Start:**\n\n```bash\n# Install dependencies\npip install requests PyGithub\n\n# Run in dry-run mode (preview only, no issues created)\nexport GITHUB_TOKEN=\"your_github_token\"\n./scripts/run_create_issues.sh\n\n# Create issues (live mode)\n./scripts/run_create_issues.sh --live\n```\n\n**Features:**\n- ✅ Fetches all open code scanning and secret scanning alerts\n- ✅ Groups similar alerts to avoid creating duplicate issues\n- ✅ Creates detailed issues with remediation guidance\n- ✅ Checks for existing issues before creating new ones\n- ✅ Safe dry-run mode for testing\n\n**Documentation:** See [docs/creating_issues_from_alerts.md](docs/creating_issues_from_alerts.md) for detailed usage instructions.\n\n**Automation:** The workflow runs automatically every Monday via `.github/workflows/create-issues-from-alerts.yml` or can be triggered manually from the Actions tab.\n\n### Managing and Consolidating Alerts\n\nThe `scripts/manage_alerts_issues_prs.py` script helps manage security alerts, issues, and pull requests by identifying and consolidating duplicates, diagnosing error-state alerts, and keeping your repository organized.\n\n**Quick Start:**\n\n```bash\n# Install dependencies\npip install requests PyGithub\n\n# Run in dry-run mode (no changes)\nexport GITHUB_TOKEN=\"your_github_token\"\n./scripts/run_alert_management.sh\n\n# Or run directly with Python\npython scripts/manage_alerts_issues_prs.py \\\n  --owner rhamenator \\\n  --repo ai-scraping-defense \\\n  --dry-run\n```\n\n**What it does:**\n- ✅ Consolidates duplicate security alerts (code scanning, secret scanning, Dependabot)\n- ✅ Closes duplicate issues and PRs with superseding notes\n- ✅ Diagnoses and suggests fixes for error-state alerts\n- ✅ Generates comprehensive reports of all actions\n\nThe script intelligently groups items by their essential properties (problem description, not file paths or IDs), ensuring that duplicate alerts affecting different files are properly consolidated. All closed items receive notes referencing the primary item.\n\n**Documentation:**\n- **Quick Start:** [QUICK_START_ALERT_MANAGEMENT.md](QUICK_START_ALERT_MANAGEMENT.md)\n- **Detailed Guide:** [docs/alert_management_guide.md](docs/alert_management_guide.md)\n- **Technical Docs:** [scripts/ALERT_MANAGEMENT_README.md](scripts/ALERT_MANAGEMENT_README.md)\n\nYou can also run this automatically via GitHub Actions (see `.github/workflows/manage-alerts.yml`).\n\n## Monitoring Stack\n\nDocker Compose includes a small Prometheus and Grafana setup. Prometheus scrapes\nthe Python services every 15 seconds using `monitoring/prometheus.yml`, and\nGrafana exposes dashboards on `${GRAFANA_PORT:-3000}`.\n\n```env\n# excerpt from sample.env\nPROMETHEUS_PORT=9090\nGRAFANA_PORT=3000\n```\n\n- **Prometheus UI:** [http://localhost:${PROMETHEUS_PORT:-9090}](http://localhost:9090) shows raw metrics and scrape targets.\n- **Grafana UI:** [http://localhost:${GRAFANA_PORT:-3000}](http://localhost:3000) (default login `admin` / `admin`). You can import a dashboard from Grafana's library or create your own to monitor request rates and response times.\n\nThe stack also runs a `watchtower` container that checks for image updates every\nminute and restarts services automatically. Remove or comment out the\n`watchtower` section in `docker-compose.yaml` if you prefer manual updates.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhamenator%2Fai-scraping-defense","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frhamenator%2Fai-scraping-defense","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhamenator%2Fai-scraping-defense/lists"}