{"id":51438336,"url":"https://github.com/doug/polite-http","last_synced_at":"2026-07-05T09:01:45.213Z","repository":{"id":365599106,"uuid":"1272865192","full_name":"doug/polite-http","owner":"doug","description":"A courteous, dependency-free HTTP client with rate limiting, retries, and backoff.","archived":false,"fork":false,"pushed_at":"2026-06-18T02:49:19.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-18T04:24:12.351Z","etag":null,"topics":["backoff","http","http-client","python","rate-limiting","retry","urllib"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doug.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-18T02:16:07.000Z","updated_at":"2026-06-18T02:49:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/doug/polite-http","commit_stats":null,"previous_names":["doug/polite-http"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/doug/polite-http","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doug%2Fpolite-http","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doug%2Fpolite-http/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doug%2Fpolite-http/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doug%2Fpolite-http/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doug","download_url":"https://codeload.github.com/doug/polite-http/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doug%2Fpolite-http/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35148606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backoff","http","http-client","python","rate-limiting","retry","urllib"],"created_at":"2026-07-05T09:01:44.148Z","updated_at":"2026-07-05T09:01:45.091Z","avatar_url":"https://github.com/doug.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# polite-http\n\n[![CI](https://github.com/doug/polite-http/actions/workflows/ci.yml/badge.svg)](https://github.com/doug/polite-http/actions/workflows/ci.yml)\n[![PyPI version](https://img.shields.io/pypi/v/polite-http.svg)](https://pypi.org/project/polite-http/)\n[![Python versions](https://img.shields.io/pypi/pyversions/polite-http.svg)](https://pypi.org/project/polite-http/)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n\nA courteous, **dependency-free** HTTP client for Python. It plays nice with\nrate-limited APIs out of the box:\n\n- **Per-host rate limiting** — cross-process, via a shared file lock.\n- **Automatic retries** on transient errors (HTTP 429, 5xx) and network errors.\n- **Exponential backoff** with optional jitter.\n- **`Retry-After`** support (server-directed backoff, both seconds and\n  HTTP-date forms).\n- **`X-Throttling-Control`** proactive backpressure (as used by PubChem / NCBI).\n- **Streaming** helpers for large line-oriented and binary responses.\n- **Zero third-party dependencies** — built entirely on the standard library\n  (`urllib.request`).\n\n## Installation\n\n```bash\npip install polite-http\n```\n\nRequires Python 3.9+. Cross-process rate limiting works on Linux, macOS, and\nWindows — it uses `fcntl` on POSIX and `msvcrt` on Windows for the shared file\nlock (both standard library). On the rare platform that provides neither, it\nfalls back to a best-effort in-process timer.\n\n## Quick start\n\n```python\nfrom polite_http import HttpClient\n\n# Scope a client to a base URL and a steady-state rate of 3 requests/second.\nclient = HttpClient(\"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/\", qps=3)\n\n# GET + parse JSON (relative paths are resolved against the base URL).\ndata = client.fetch_json(\"esummary.fcgi?db=pubmed\u0026id=123456\")\n\n# POST a JSON body.\nresult = client.fetch_json(\n    \"esearch.fcgi\",\n    method=\"POST\",\n    json_body={\"db\": \"pubmed\", \"term\": \"cancer\"},\n)\n\n# Download raw bytes with a custom timeout.\npdf = client.fetch_bytes(\n    \"efetch.fcgi?db=pubmed\u0026id=123456\u0026rettype=abstract\",\n    timeout=60,\n)\n```\n\n## Streaming\n\n```python\n# Stream a large response line-by-line without buffering it all in memory.\nfor line in client.stream_lines(\"large-export.tsv\"):\n    process(line)\n\n# Stream binary content in chunks (e.g. to a file).\nwith open(\"paper.pdf\", \"wb\") as f:\n    for chunk in client.stream_bytes(\"paper.pdf\"):\n        f.write(chunk)\n```\n\n## Configuration\n\n`HttpClient` accepts the following keyword arguments:\n\n| Argument | Default | Description |\n| --- | --- | --- |\n| `qps` | _required_ | Maximum queries per second (steady state). |\n| `default_headers` | `None` | Headers added to every request. |\n| `max_retries` | `7` | Retry attempts for transient errors (total attempts = `max_retries + 1`). |\n| `timeout` | `60.0` | Per-request timeout in seconds. |\n| `backoff_base` | `3.0` | Base delay for exponential backoff. |\n| `backoff_max` | `180.0` | Cap on backoff delay. |\n| `jitter` | `0.5` | Max uniform random jitter added to each backoff. |\n| `user_agent` | env / `\"\"` | `User-Agent` header; falls back to `POLITE_HTTP_USER_AGENT`. |\n| `retryable_status_codes` | `{429, 500, 502, 503, 504}` | Status codes that trigger a retry. |\n| `referer` | `None` | Optional `Referer` header sent with every request. |\n\n### Environment variables\n\n- `POLITE_HTTP_USER_AGENT` — default `User-Agent` when one isn't passed\n  explicitly. Many APIs (e.g. NCBI) reject requests without a descriptive\n  User-Agent, so setting this is recommended.\n- `POLITE_HTTP_LOCK_DIR` — directory for the cross-process rate-limit lock\n  files (defaults to the system temp directory).\n\n## Error handling\n\nFailed requests raise `HttpError`, which carries the `status_code`, raw `body`\nbytes, and `url`:\n\n```python\nfrom polite_http import HttpClient, HttpError\n\nclient = HttpClient(\"https://api.example.com/\", qps=5)\ntry:\n    data = client.fetch_json(\"widgets/42\")\nexcept HttpError as exc:\n    print(exc.status_code, exc.url)\n    if exc.body:\n        print(exc.json())  # parse the error body as JSON, if applicable\n```\n\n## Acknowledgements\n\nThe HTTP client at the heart of this package is derived from the\n[`science-skills`](https://github.com/google-deepmind/science-skills) project by\nGoogle DeepMind, used under the Apache License 2.0. See [`NOTICE`](NOTICE) for\ndetails of the changes.\n\n## License\n\n[Apache License 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoug%2Fpolite-http","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoug%2Fpolite-http","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoug%2Fpolite-http/lists"}