{"id":26831288,"url":"https://github.com/thesp0nge/nightcrawler-mitm","last_synced_at":"2025-04-30T09:08:15.154Z","repository":{"id":137434717,"uuid":"263627729","full_name":"thesp0nge/nightcrawler-mitm","owner":"thesp0nge","description":"A python program that crawls a website and tries to stress it, polluting forms with bogus data","archived":false,"fork":false,"pushed_at":"2025-03-27T16:25:08.000Z","size":169,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T16:42:02.722Z","etag":null,"topics":["crawler","offensive-scripts","offensive-security","stress-test","web-crawler","web-crawling"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thesp0nge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-13T12:47:11.000Z","updated_at":"2025-03-27T16:25:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"eff3c99d-0be2-449e-a502-f916b66e3fa5","html_url":"https://github.com/thesp0nge/nightcrawler-mitm","commit_stats":null,"previous_names":["thesp0nge/nightcrawler-mitm"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesp0nge%2Fnightcrawler-mitm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesp0nge%2Fnightcrawler-mitm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesp0nge%2Fnightcrawler-mitm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesp0nge%2Fnightcrawler-mitm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thesp0nge","download_url":"https://codeload.github.com/thesp0nge/nightcrawler-mitm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246332258,"owners_count":20760449,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","offensive-scripts","offensive-security","stress-test","web-crawler","web-crawling"],"created_at":"2025-03-30T14:32:03.905Z","updated_at":"2025-04-30T09:08:15.147Z","avatar_url":"https://github.com/thesp0nge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nightcrawler-mitm\n\nVersion: 0.6.0\n\nA mitmproxy addon for background passive analysis, crawling, and basic active\nscanning, designed as a security researcher's sidekick.\n\n**WARNING: BETA Stage - Use with caution, especially active scanning features**\n\n## FEATURES\n\n- Acts as an HTTP/HTTPS proxy.\n- Performs passive analysis:\n  - Security Headers (HSTS, CSP, XCTO, XFO, Referrer-Policy, Permissions-Policy,\n    COOP, COEP, CORP, basic weakness checks).\n  - Cookie Attributes (Secure, HttpOnly, SameSite).\n  - JWT Detection \u0026 Decoding (in Headers and JSON responses).\n  - Basic Info Disclosure checks (Comments, basic keyword context - Note:\n    API/Key/Secret checks temporarily disabled).\n- Crawls the target application to discover new endpoints.\n- Runs basic active scans for low-hanging fruit:\n  - Reflected XSS (basic reflection check).\n  - SQL Injection (basic error/time-based checks).\n  - Stored XSS (basic probe injection and revisit check).\n- Configurable target scope, concurrency, payloads, and output via command-line\n  options.\n- Logs findings to console and optionally to a JSONL file.\n\n## INSTALLATION\n\nYou can install `nightcrawler-mitm` directly from PyPI using pip (once\npublished):\n\n```\npip install nightcrawler-mitm`\n```\n\nIt's recommended to install it in a virtual environment. For development/local\ntesting:\n\n- Navigate to project root directory (containing pyproject.toml)\n- Activate your virtual environment (e.g., source .venv/bin/activate)\n\n```\npip install -e .\n```\n\n## USAGE\n\nOnce installed, a new command `nightcrawler` becomes available. This command\nwraps `mitmdump`, automatically loading the addon. You MUST specify the target\nscope using the `--set nc_scope=...` option.\n\nYou can pass any other valid `mitmproxy` arguments (like `--ssl-insecure`, `-p`,\n`-v`) AND Nightcrawler-specific options using the `--set name=value` syntax.\n\n1. Configure Browser/Client: Set proxy to 127.0.0.1:8080 (or specified port).\n2. Install Mitmproxy CA Certificate: Visit \u003chttp://mitm.it\u003e via proxy.\n3. Run Nightcrawler:\n\n   - Specify Target Scope (REQUIRED!): nightcrawler --set nc_scope=example.com\n\n   - Common Options (Combine as needed): nightcrawler -p 8081 --set\n     nc_scope=example.com nightcrawler --ssl-insecure --set\n     nc_scope=internal-site.local nightcrawler -v --set nc_scope=example.com #\n     Use -v or -vv for debug logs nightcrawler --set nc_max_concurrency=10 --set\n     nc_scope=secure.com nightcrawler --set nc_sqli_payload_file=sqli.txt --set\n     nc_output_file=findings.jsonl --set nc_scope=test.org\n\n   - Show Nightcrawler \u0026 Mitmproxy version: nightcrawler --version\n\n   - Show all Nightcrawler and Mitmproxy options (look for 'nc\\_' prefix):\n     nightcrawler --options\n\n   NOTE: If nc_scope is not set, Nightcrawler will run but remain idle.\n\n4. Browse: Browse the target application(s). Findings appear in the terminal and\n   optionally in the specified JSONL file.\n\n## CONFIGURATION\n\nNightcrawler configuration follows this precedence:\n\n1. Command-line --set options (highest precedence)\n2. Values in configuration file\n3. Built-in defaults (lowest precedence)\n\n**Configuration File:**\n\n- By default, Nightcrawler looks for a YAML configuration file at:\n  - `~/.config/nightcrawler-mitm/config.yaml` (on Linux/macOS, standard)\n  - `%APPDATA%/nightcrawler-mitm/config.yaml` (on Windows, needs check)\n  - _Fallback:_ `~/.nightcrawler-mitm/config.yaml` (if XDG path not\n    found/writable)\n- You can specify a different configuration file path using the `--nc-config`\n  option when running Nightcrawler (passed via `--set`):\n  `nightcrawler --set nc_config=/path/to/my_config.yaml ...`\n- The configuration file uses YAML format. Keys should match the addon option\n  names (without the `--set`).\n\n_Example `config.yaml`:_\n\n```yaml\n# ~/.config/nightcrawler-mitm/config.yaml\n# Nightcrawler Configuration Example\n\n# Target scope (REQUIRED if not using --set nc_scope)\nnc_scope: example.com,internal.dev\n\n# Worker concurrency\nnc_max_concurrency: 10\n\n# Custom User-Agent\nnc_user_agent: \"My Custom Scanner Bot/1.0\"\n\n# Custom payload files (paths relative to config file or absolute)\n# nc_sqli_payload_file: payloads/custom_sqli.txt\n# nc_xss_reflected_payload_file: /opt/payloads/xss.txt\n\n# Stored XSS settings\nnc_xss_stored_prefix: MyProbe\nnc_xss_stored_format: \"\u003cnc_probe data='{probe_id}'/\u003e\"\nnc_payload_max_age: 7200 # Track payloads for 2 hours\n\n# Output files (relative paths resolved against default data dir, absolute paths used as is)\n# nc_output_file: nightcrawler_results.jsonl # Saved in default data dir\n# nc_output_html: /var/www/reports/scan_report.html # Saved to absolute path\n\n# WebSocket inspection\nnc_inspect_websocket: false\n```\n\n### Command-Line Overrides (--set)\n\nYou can always override defaults or config file values using --set. This takes\nthe highest precedence.\n\n```\nnightcrawler --set nc_scope=specific-target.com --set nc_max_concurrency=3\n```\n\nTo see all available nc*options and their current effective values (after\nconsidering defaults, config file, and --set), run: nightcrawler --options |\ngrep nc*\n\n### Default Data Directory \u0026 Output Paths\n\n- If you specify relative paths for nc_output_file or nc_output_html (either in\n  the config file or via --set), Nightcrawler will attempt to save them relative\n  to a default data directory:\n  - Linux/macOS (XDG): ~/.local/share/nightcrawler-mitm/\n  - Windows (approx): %LOCALAPPDATA%/nightcrawler-mitm/\n- If you specify absolute paths (e.g., /tmp/report.html), they will be used\n  directly.\n- Nightcrawler will attempt to create these directories if they don't exist.\n\n## LIMITATIONS\n\n- Basic Active Scans: Scanners are basic, intended for low-hanging fruit. Cannot\n  detect complex vulnerabilities. DO NOT rely solely on this tool.\n- Stored XSS Detection: Basic implementation, may miss cases and have FPs.\n- Info Disclosure: Content checks for keys/secrets are basic and currently\n  disabled pending refactoring.\n- Resource Usage: Tune `--set nc_max_concurrency`.\n- False Positives/Negatives: Expected. Manual verification is required.\n\n## LICENSE\n\nThis project is licensed under the MIT License. See the LICENSE file for\ndetails.\n\n## CONTRIBUTING (Optional)\n\nContributions welcome! See the GitHub repository:\n\u003chttps://github.com/thesp0nge/nightcrawler-mitm\u003e\n\n```\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesp0nge%2Fnightcrawler-mitm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthesp0nge%2Fnightcrawler-mitm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesp0nge%2Fnightcrawler-mitm/lists"}