{"id":18004114,"url":"https://github.com/simon987/architeuthis","last_synced_at":"2025-10-08T06:55:58.905Z","repository":{"id":80438142,"uuid":"189039293","full_name":"simon987/Architeuthis","owner":"simon987","description":"  MITM HTTP(S) proxy with integrated load-balancing, rate-limiting and error handling. Built for automated web scraping.","archived":false,"fork":false,"pushed_at":"2020-01-03T17:29:21.000Z","size":315,"stargazers_count":41,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-17T21:16:03.324Z","etag":null,"topics":["influxdb","load-balancer","proxy","redis","scraping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simon987.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-28T13:59:09.000Z","updated_at":"2024-01-31T05:06:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"d63f1330-4f38-4462-806d-1b027d626f00","html_url":"https://github.com/simon987/Architeuthis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/simon987/Architeuthis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simon987%2FArchiteuthis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simon987%2FArchiteuthis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simon987%2FArchiteuthis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simon987%2FArchiteuthis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simon987","download_url":"https://codeload.github.com/simon987/Architeuthis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simon987%2FArchiteuthis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278903033,"owners_count":26065786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["influxdb","load-balancer","proxy","redis","scraping"],"created_at":"2024-10-30T00:13:26.646Z","updated_at":"2025-10-08T06:55:58.890Z","avatar_url":"https://github.com/simon987.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Architeuthis 🦑\n\n[![CodeFactor](https://www.codefactor.io/repository/github/simon987/architeuthis/badge)](https://www.codefactor.io/repository/github/simon987/architeuthis)\n![GitHub](https://img.shields.io/github/license/simon987/Architeuthis.svg)\n[![Build Status](https://ci.simon987.net/buildStatus/icon?job=architeuthis_builds)](https://ci.simon987.net/job/architeuthis_builds/)\n\nHTTP(S) proxy with integrated load-balancing, rate-limiting\nand error handling. Built for automated web scraping.\n\n* Strictly obeys configured rate-limiting for each IP \u0026 Host\n* Seamless exponential backoff retries on timeout or error HTTP codes\n* Requires no additional configuration for integration into existing programs\n* Configurable per-host behavior\n* Monitoring with InfluxDB\n\n![grafana](grafana.png)\n\n### Typical use case\n![user_case](use_case.png)\n\n### Usage\n\n```bash\ngit clone https://github.com/simon987/Architeuthis\nvim config.json # Configure settings here\n\ndocker-compose up\n```\n\nYou can add proxies using the `/add_proxy` API:\n\n```bash\ncurl http://\u003cArchiteuthis IP\u003e:5050/add_proxy?url=\u003curl\u003e\u0026name=\u003cname\u003e\n```\n\nOr automatically using Proxybroker:\n```bash\npython3 import_from_broker.py http://\u003cArchiteuthis IP\u003e:5050\n```\n\n### Example usage with wget\n```bash\nexport http_proxy=\"http://localhost:5050\"\n# --no-check-certificates is necessary for https mitm\n# You don't need to specify user-agent if it's already in your config.json\nwget -m -np -c --no-check-certificate -R index.html* http http://ca.releases.ubuntu.com/\n```\n\nWith `\"every\": \"500ms\"` and a single proxy, you should see\n```\n...\nlevel=trace msg=Sleeping wait=414.324437ms\nlevel=trace msg=\"Routing request\" conns=0 proxy=p0 url=\"http://ca.releases.ubuntu.com/12.04/SHA1SUMS.gpg\"\nlevel=trace msg=Sleeping wait=435.166127ms\nlevel=trace msg=\"Routing request\" conns=0 proxy=p0 url=\"http://ca.releases.ubuntu.com/12.04/SHA256SUMS\"\nlevel=trace msg=Sleeping wait=438.657784ms\nlevel=trace msg=\"Routing request\" conns=0 proxy=p0 url=\"http://ca.releases.ubuntu.com/12.04/SHA256SUMS.gpg\"\nlevel=trace msg=Sleeping wait=457.06543ms\nlevel=trace msg=\"Routing request\" conns=0 proxy=p0 url=\"http://ca.releases.ubuntu.com/12.04/ubuntu-12.04.5-alternate-amd64.iso\"\nlevel=trace msg=Sleeping wait=433.394361ms\n...\n```\n\n### Hot config reload\n\n```bash\n# Note: this will reset current rate limiters, if there are many active\n# connections, this might cause a small request spike and go over\n# the rate limits.\n./reload.sh\n```\n\n### Rules\n\n\nConditions\n\n| Left operand | Description | Allowed operators | Right operand\n| :--- | :--- | :--- | :---\n| body | Contents of the response | `=`, `!=` | String w/ wildcard\n| body | Contents of the response | `\u003c`, `\u003e` | float\n| status | HTTP response code | `=`, `!=` | String w/ wildcard\n| status | HTTP response code | `\u003c`, `\u003e` | float\n| response_time | HTTP response code | `\u003c`, `\u003e` | duration (e.g. `20s`)\n| header:`\u003cheader\u003e` | Response header | `=`, `!=` | String w/ wildcard\n| header:`\u003cheader\u003e` | Response header | `\u003c`, `\u003e` | float\n\nNote that `response_time` can never be higher than the configured `timeout` value.\n\nExamples:\n\n```json\n[\n  {\"condition\":  \"header:X-Test\u003e10\", \"action\":  \"...\"},\n  {\"condition\":  \"body=*Try again in a few minutes*\", \"action\":  \"...\"},\n  {\"condition\":  \"response_time\u003e10s\", \"action\":  \"...\"},\n  {\"condition\":  \"status\u003e500\", \"action\":  \"...\"},\n  {\"condition\":  \"status=404\", \"action\":  \"...\"},\n  {\"condition\":  \"status=40*\", \"action\":  \"...\"}\n]\n```\n\nActions\n\n| Action | Description\n| :--- | :--- |\n| should_retry | Override default retry behavior for http errors (by default it retries on 403,408,429,444,499,\u003e500)\n| force_retry | Always retry (Up to retries_hard times)\n| dont_retry | Immediately stop retrying\n\nIn the event of a temporary network error, `should_retry` is ignored (it will always retry unless `dont_retry` is set)\n\nNote that having too many rules for one host might negatively impact performance (especially the `body` condition for large requests)\n\n\n### Sample configuration\n\n```json\n{\n  \"addr\": \"localhost:5050\",\n  \"timeout\": \"15s\",\n  \"wait\": \"4s\",\n  \"multiplier\": 2.5,\n  \"retries\": 3,\n  \"hosts\": [\n    {\n      \"host\": \"*\",\n      \"every\": \"500ms\",\n      \"burst\": 25,\n      \"headers\": {\n        \"User-Agent\": \"Some user agent for all requests\",\n        \"X-Test\": \"Will be overwritten\"\n      }\n    },\n    {\n      \"host\": \"*.reddit.com\",\n      \"every\": \"2s\",\n      \"burst\": 2,\n      \"headers\": {\n        \"X-Test\": \"Will overwrite default\"\n      }\n    },\n    {\n      \"host\": \".s3.amazonaws.com\",\n      \"every\": \"2s\",\n      \"burst\": 30,\n      \"rules\": [\n        {\"condition\": \"status=403\", \"action\": \"dont_retry\"}\n      ]\n    }\n  ]\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimon987%2Farchiteuthis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimon987%2Farchiteuthis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimon987%2Farchiteuthis/lists"}