{"id":50768360,"url":"https://github.com/catsec/sitesweep","last_synced_at":"2026-06-11T16:01:04.346Z","repository":{"id":362042097,"uuid":"1257003997","full_name":"catsec/sitesweep","owner":"catsec","description":"Merchant site violation \u0026 SEO-injection checker (Hebrew UI) — scans live sites or HAR captures for injected gambling/pharma/adult/counterfeit content and parasite-SEO compromises.","archived":false,"fork":false,"pushed_at":"2026-06-02T11:02:29.000Z","size":154,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-02T11:18:48.309Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/catsec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-02T09:21:35.000Z","updated_at":"2026-06-02T11:02:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/catsec/sitesweep","commit_stats":null,"previous_names":["catsec/sitesweep"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/catsec/sitesweep","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitesweep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitesweep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitesweep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitesweep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/catsec","download_url":"https://codeload.github.com/catsec/sitesweep/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/catsec%2Fsitesweep/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34206492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-11T16:01:03.390Z","updated_at":"2026-06-11T16:01:04.340Z","avatar_url":"https://github.com/catsec.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sitesweep\n\nA merchant **site violation \u0026 SEO-injection checker** with a Hebrew web front end.\nScans a live site (1 level deep) or a saved **HAR** capture and reports — in Hebrew —\nwhether the page carries violation content (gambling / pharma / adult / counterfeit)\nor the mechanics of a parasite-SEO compromise: hidden/cloaked content, injected\ndofollow links to violation domains, foreign-script anomalies, suspicious JS,\nviolation slugs in URLs, and UA-based cloaking.\n\n**Design principle:** it judges the content actually served, with weighted signals and\ncorroboration — so a lone weak keyword (e.g. \"slot\" on a site called *asiatico*) never\ntrips a verdict. This avoids the index-collision false positives that naive\n\"domain + keyword\" detection produces.\n\n---\n\n## Run with Docker (recommended)\n\n```bash\ncp .env.example .env          # then paste your Cloudflare tunnel token into .env\ndocker compose up -d          # pulls ghcr.io/catsec/sitesweep:latest (amd64/arm64)\n```\n\nThe image is published by GitHub Actions to **`ghcr.io/catsec/sitesweep`** as a\nmulti-arch manifest, so the same tag runs on Intel/AMD64 and arm64 (Apple Silicon,\nGraviton, Pi 64-bit). To build locally instead of pulling, uncomment `build: .` in\n`docker-compose.yml` and run `docker compose build`.\n\nTwo services come up:\n\n- **sitesweep** — the app. It publishes **no host port**; nothing is exposed on the LAN.\n- **cloudflared** — the Cloudflare Tunnel. Authentication is handled by **Cloudflare Access**\n  in front of the tunnel. In the Zero Trust dashboard, point the tunnel's public hostname\n  (e.g. `sitesweep.catsec.com`) at the origin **`http://sitesweep:8000`** — that's the app's\n  service name on the internal Docker network.\n\nThe tunnel token lives in `.env` (`CF_TUNNEL_TOKEN`) and is passed to cloudflared as\n`TUNNEL_TOKEN`, so it never appears in the compose file or in the container's process\narguments. `.env` is gitignored; never commit it.\n\nThe compose file is hardened: non-root, `cap_drop: ALL`, `no-new-privileges`,\nread-only root FS with `tmpfs`, resource limits, health check, log rotation, pinned tags.\n\nThe image bundles **Chromium** so the \"רינדור עם דפדפן\" (headless Chrome) option works.\nFor a slim static/HAR-only image, delete the Chromium `RUN` block in the `Dockerfile`\nand set `SITESWEEP_ALLOW_RENDER=0`.\n\n### Environment variables\n\n| Var | Default | Meaning |\n|-----|---------|---------|\n| `SITESWEEP_ALLOW_RENDER` | `1` | allow headless-Chrome rendering |\n| `SITESWEEP_MAX_PAGES` | `50` | hard cap on pages per URL scan |\n| `SITESWEEP_MAX_HAR_MB` | `40` | max uploaded HAR size |\n| `SITESWEEP_SCAN_TIMEOUT` | `300` | hard wall-clock cap (seconds) on a single scan job |\n| `SERPAPI_KEY` | — | enable the search-index check via SerpAPI (see below) |\n\n---\n\n## Install the container, per platform\n\nThe runtime is the same everywhere — two containers (`sitesweep` + `cloudflared`)\ndefined in `docker-compose.yml`, pulling the multi-arch image from GHCR. Only the\nway you install Docker and start the project differs.\n\n**Common to every platform**\n\n1. Get the project files (you need `docker-compose.yml` and `.env.example`):\n   ```bash\n   git clone https://github.com/catsec/sitesweep.git\n   cd sitesweep\n   cp .env.example .env        # paste your Cloudflare tunnel token into CF_TUNNEL_TOKEN\n   ```\n2. If the GHCR package is **private**, authenticate once before pulling\n   (create a GitHub token with the `read:packages` scope):\n   ```bash\n   echo \"$GHCR_TOKEN\" | docker login ghcr.io -u \u003cyour-github-user\u003e --password-stdin\n   ```\n   If the package has been made public, you can skip this.\n3. No host port is published by default — the app is reached through the\n   Cloudflare Tunnel hostname you configure in the Zero Trust dashboard\n   (origin `http://sitesweep:8000`). For **local testing without the tunnel**,\n   uncomment the `ports:` block in `docker-compose.yml` to expose\n   `127.0.0.1:8000` and open \u003chttp://127.0.0.1:8000\u003e.\n\n### Linux\n\n```bash\n# Install Docker Engine + the Compose plugin (Debian/Ubuntu shown)\ncurl -fsSL https://get.docker.com | sh\nsudo usermod -aG docker \"$USER\"     # log out/in so you can run docker without sudo\n\n# from the cloned repo:\ndocker compose up -d\ndocker compose logs -f              # watch startup; Ctrl-C to stop watching\n```\n\n### macOS\n\n```bash\n# Install Docker Desktop (includes Compose v2)\nbrew install --cask docker          # then launch Docker.app once to start the engine\n# (Apple Silicon pulls the arm64 image automatically; Intel Macs pull amd64)\n\n# from the cloned repo:\ndocker compose up -d\n```\n\n### Windows\n\nInstall **Docker Desktop for Windows** (WSL 2 backend recommended), then from a\nPowerShell prompt in the cloned repo:\n\n```powershell\ndocker compose up -d\ndocker compose logs -f\n```\n\nIf you don't have `git`, download the repo Zip from GitHub (**Code → Download ZIP**),\nextract it, and run the same commands inside the extracted folder. Edit `.env`\nin any text editor (e.g. Notepad) and paste the tunnel token.\n\n### Synology NAS (DSM 7)\n\nUse **Container Manager** (Package Center → install *Container Manager*; on\nDSM 7.0–7.1 it's called *Docker*). Works on **64-bit** models — x86_64 (Intel/AMD,\npulls `amd64`) and arm64 (aarch64, pulls `arm64`). 32-bit ARM models are not\nsupported.\n\n1. Copy the project folder to the NAS (e.g. `/volume1/docker/sitesweep`) via\n   File Station or an SMB share — include `docker-compose.yml` and your\n   filled-in `.env`.\n2. In **Container Manager → Project → Create**:\n   - **Path:** the folder you copied.\n   - **Source:** *Use existing docker-compose.yml*.\n   - Container Manager reads `.env` from that folder for `CF_TUNNEL_TOKEN`.\n3. Build/Run the project. Container Manager pulls the image and starts both\n   services; the tunnel connects outbound, so **no router port-forwarding is\n   needed** and you should not map a NAS port to the app.\n4. If the GHCR package is private, add a registry login under\n   **Container Manager → Registry → Settings** (registry `ghcr.io`, your GitHub\n   user, a `read:packages` token) before creating the project.\n\n\u003e Note: the image bundles Chromium for the \"render with browser\" option, which is\n\u003e RAM-hungry. On NAS units with ≤2 GB RAM, set `SITESWEEP_ALLOW_RENDER=0` in the\n\u003e compose `environment:` block (HAR + static scans still work) or raise the\n\u003e memory limit to suit your model.\n\u003e\n\u003e The compose file caps CPU with `cpu_shares` (a relative weight) rather than a hard\n\u003e `cpus:` limit on purpose — Synology's kernel lacks CFS CPU-quota support, so a hard\n\u003e limit fails with *\"NanoCPUs can not be set.\"*\n\n---\n\n## Run locally (macOS, venv)\n\n```bash\npython3 -m venv .venv \u0026\u0026 source .venv/bin/activate\npip install -r requirements.txt\nplaywright install chromium      # only if you'll use rendering\n\n# web UI\nuvicorn app:app --host 127.0.0.1 --port 8000\n\n# or the CLI\npython sitesweep.py https://example.co.il --lang he\npython sitesweep.py --har capture.har --lang he\npython sitesweep.py https://example.co.il --render --format md -o report.md\n```\n\n---\n\n## CLI reference\n\n```\nsitesweep.py [URL] [options]\n  --har FILE         analyze a saved HAR instead of crawling\n  --html-file FILE   analyze one local HTML file (URL positional for link resolution)\n  --render           fetch with headless Chrome (executes JS)\n  --render-wait MS   settle time after load for injection JS (default 1500)\n  --depth N          crawl depth (default 1)\n  --max-pages N      page cap (default 25)\n  --no-cloak-check   skip the Googlebot cloaking diff\n  --index-check      query the search index for violation content on the domain\n  --history-check    query the Wayback archive for historical violation URLs\n  --format text|md|json\n  --lang en|he\n  -o FILE            write report to file\n```\n\n---\n\n## How the verdict works\n\nSignals are scored and summed; `\u003e=40` → INFECTED (נגוע), `\u003e=15` → SUSPICIOUS (חשוד),\nelse CLEAN (תקין). Bare hidden elements, lone weak keywords, and ordinary analytics JS\nnever reach a verdict on their own — they only count alongside real corroboration.\nUA cloaking (different content to Googlebot) or violation domains in network requests\nescalate on their own.\n\n## Search-index \u0026 history exposure\n\nTwo optional **site-level** checks catch what a live crawl alone can miss — a page\nthat was indexed or archived with violation content even though the live HTML now\nlooks clean (index lag, or a hack that was cleaned but still lingers in the search\nindex / the Wayback Machine):\n\n- **Search-index check** (`--index-check`, UI checkbox) — searches the index for\n  violation content **hosted on the domain itself**. The critical guard: a result\n  only counts if its host equals the target domain (or a subdomain). A site merely\n  *mentioning* a brand, or sharing a keyword, is never counted — so a restaurant\n  named *asiatico* is not flagged because gambling pages about \"slot a tema asiatico\"\n  exist elsewhere. Needs a search provider (below); without one it self-skips.\n- **Wayback history check** (`--history-check`, UI checkbox) — queries the Internet\n  Archive CDX API (free, no key) for archived violation URLs on the domain, with the\n  same host guard, and reports the first/last seen dates.\n\n**Verdict impact is deliberately conservative:** if the live pages are CLEAN but the\nindex/history shows violations on the domain, the verdict becomes **SUSPICIOUS**, not\nINFECTED — i.e. \"historical or index-lag, confirm manually.\" These checks never\ndeclare the live site infected on their own; they only corroborate a live finding.\n\n### Setting up the index check (SerpAPI)\n\nThe index check searches for violation content **hosted on the target domain**\n(`site:\u003cdomain\u003e …`) via **SerpAPI**, which runs the query through an official\nsearch API:\n\n1. Sign up at \u003chttps://serpapi.com/\u003e and copy your API key.\n2. Put it in `.env` (Docker reads it via `docker-compose.yml`) or export it for the CLI:\n   ```\n   SERPAPI_KEY=your-serpapi-key\n   ```\n3. The UI checkbox enables itself once the server sees the key (via `/api/capabilities`).\n\nSerpAPI's free tier is 250 searches/month; sitesweep sends ~1 query per violation\ncategory, so a scan costs a handful. If the quota is exhausted (or any other provider\nerror occurs) the scan still completes and the report says so. Without a key the index\ncheck simply self-skips —\nthe Wayback history check still runs keyless.\n\n## Technology fingerprinting\n\nEvery scan also reports the **stack behind the site** — CMS / page builder / e-commerce\n(WordPress, Elementor, WooCommerce, Shopify, Wix, Joomla, Drupal…), notable WP\nplugins/themes and their versions where exposed, JS libraries, server and CDN (from\nresponse headers). This is **informational only** and never affects the score or verdict;\nit's there to speed up remediation and risk profiling.\n\n## Limitations\n\n- Static fetch won't see JS-only injection — use `--render` / the browser toggle.\n- Class-based CSS cloaking is detected heuristically (via `\u003cstyle\u003e` scan), not full CSS resolution.\n- The URL scanner fetches arbitrary hosts by design; if exposing the UI beyond yourself,\n  put it behind auth and consider egress restrictions (SSRF).\n\n## Files\n\n```\nsitesweep.py        engine + CLI\napp.py              FastAPI web app\ntemplates/index.html  Hebrew RTL UI\nrequirements.txt\nDockerfile          hardened, multi-stage, Chromium included\ndocker-compose.yml  hardened runtime\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatsec%2Fsitesweep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcatsec%2Fsitesweep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcatsec%2Fsitesweep/lists"}