{"id":25657469,"url":"https://github.com/cheginit/tiny-retriever","last_synced_at":"2025-10-24T22:03:41.935Z","repository":{"id":277082641,"uuid":"929958025","full_name":"cheginit/tiny-retriever","owner":"cheginit","description":"TinyRetriever: HTTP Requests Made Easy","archived":false,"fork":false,"pushed_at":"2025-04-15T15:30:21.000Z","size":1111,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T16:42:48.201Z","etag":null,"topics":["aiohttp","async","asynchronous","hydrology","python","web-service"],"latest_commit_sha":null,"homepage":"https://tiny-retriever.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cheginit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-09T19:34:34.000Z","updated_at":"2025-04-15T15:30:22.000Z","dependencies_parsed_at":"2025-04-01T02:27:38.048Z","dependency_job_id":"6b132305-17cd-4c82-8b36-c0324db9c57a","html_url":"https://github.com/cheginit/tiny-retriever","commit_stats":null,"previous_names":["cheginit/tiny-retriever"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Ftiny-retriever","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Ftiny-retriever/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Ftiny-retriever/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Ftiny-retriever/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cheginit","download_url":"https://codeload.github.com/cheginit/tiny-retriever/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249112749,"owners_count":21214755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","async","asynchronous","hydrology","python","web-service"],"created_at":"2025-02-23T23:19:30.207Z","updated_at":"2025-10-24T22:03:41.929Z","avatar_url":"https://github.com/cheginit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TinyRetriever: HTTP Requests Made Easy\n\n[![PyPi](https://img.shields.io/pypi/v/tiny-retriever.svg)](https://pypi.python.org/pypi/tiny-retriever)\n[![Conda Version](https://img.shields.io/conda/vn/conda-forge/tiny-retriever.svg)](https://anaconda.org/conda-forge/tiny-retriever)\n[![CodeCov](https://codecov.io/gh/cheginit/tiny-retriever/branch/main/graph/badge.svg)](https://codecov.io/gh/cheginit/tiny-retriever)\n[![Python Versions](https://img.shields.io/pypi/pyversions/tiny-retriever.svg)](https://pypi.python.org/pypi/tiny-retriever)\n\n[![Downloads](https://static.pepy.tech/badge/tiny-retriever)](https://pepy.tech/project/tiny-retriever)\n[![CodeFactor](https://www.codefactor.io/repository/github/cheginit/tiny-retriever/badge)](https://www.codefactor.io/repository/github/cheginit/tiny-retriever)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://github.com/pre-commit/pre-commit)\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/cheginit/tiny-retriever/HEAD?labpath=docs%2Fexamples)\n\nTinyRetriever is a lightweight synchronous wrapper for\n[AIOHTTP](https://docs.aiohttp.org/en/stable/) that abstracts away the complexities of\nmaking asynchronous HTTP requests. It is designed to be simple, easy to use, and\nefficient. TinyRetriever is built on top of AIOHTTP and\n[AIOFiles](https://github.com/Tinche/aiofiles), which are popular asynchronous HTTP\nclient and file management libraries for Python.\n\n📚 Full documentation is available [here](https://tiny-retriever.readthedocs.io).\n\n## Features\n\nTinyRetriever provides the following features:\n\n- **Concurrent Downloads**: Efficiently download multiple files simultaneously\n- **Flexible Response Types**: Get responses as text, JSON, or binary data\n- **Rate Limiting**: Built-in per-host connection limiting to respect server constraints\n- **Streaming Support**: Stream large files efficiently with customizable chunk sizes\n- **Unique Filenames**: Generate unique filenames based on query parameters\n- **Works in Jupyter Notebooks**: Easily use TinyRetriever in Jupyter notebooks without\n    any additional setup or dependencies\n- **Robust Error Handling**: Optional status raising and comprehensive error messages\n- **Performance Optimized**: Uses [`orjson`](https://github.com/ijl/orjson) when\n    available for up to 14x faster JSON parsing\n\nTinyRetriever does not use `nest-asyncio`, instead it creates and manages a dedicated\nthread for running the event loop. This allows you to use TinyRetriever in Jupyter\nnotebooks and other environments where the event loop is already running.\n\nThere are three main functions in TinyRetriever:\n\n- `download`: Download files concurrently;\n- `fetch`: Fetch queries concurrently and return responses as text, JSON, or binary;\n- `unique_filename`: Generate unique filenames based on query parameters.\n\n## Installation\n\nChoose your preferred installation method:\n\n### Using `pip`\n\n```console\npip install tiny-retriever\n```\n\n### Using `micromamba`\n\n```console\nmicromamba install -c conda-forge tiny-retriever\n```\n\nAlternatively, you can use `conda` or `mamba`.\n\n## Quick Start Guide\n\nPlease refer to the [documentation](https://tiny-retriever.readthedocs.io) for detailed\nusage instructions and more elaborate examples.\n\n### Downloading Files\n\n```python\nfrom pathlib import Path\nimport tiny_retriever as terry\n\nurls = [\"https://example.com/file1.pdf\", \"https://example.com/file2.pdf\"]\npaths = [Path(\"downloads/file1.pdf\"), Path(\"downloads/file2.pdf\")]\n# or generate unique filenames\npaths = (terry.unique_filename(u) for u in urls)\npaths = [Path(\"downloads\", p) for p in paths]\n\n# Download files concurrently\nterry.download(urls, paths)\n```\n\n### Fetching Data\n\n```python\nurls = [\"https://api.example.com/data1\", \"https://api.example.com/data2\"]\n\n# Get JSON responses\njson_responses = terry.fetch(urls, \"json\")\n\n# Get text responses\ntext_responses = terry.fetch(urls, \"text\")\n\n# Get binary responses\nbinary_responses = terry.fetch(urls, \"binary\")\n```\n\n### Generate Unique Filenames\n\n```python\nurl = \"https://api.example.com/data\"\nparams = {\"key\": \"value\"}\n\n# Generate unique filename based on URL and parameters\nfilename = terry.unique_filename(url, params=params, file_extension=\".json\")\n```\n\n## Advanced Usage\n\n### Custom Request Parameters\n\nNote that you can also pass a single url and a dictionary of request parameters to the\n`fetch` function. The default network related parameters are conservative and can be\nmodified as needed.\n\n```python\nurls = \"https://api.example.com/data\"\nkwargs = {\"headers\": {\"Authorization\": \"Bearer token\"}}\n\nresponses = terry.fetch(\n    urls,\n    return_type=\"json\",\n    request_method=\"post\",\n    request_kwargs=kwargs,\n    limit_per_host=2,\n    timeout=30,\n)\n```\n\n### Error Handling\n\n```python\nfrom tiny_retriever import fetch, ServiceError\n\ntry:\n    responses = fetch(urls, return_type=\"json\", raise_status=True)\nexcept ServiceError as e:\n    print(f\"Request failed: {e}\")\n```\n\n## Configuration\n\nTinyRetriever can be configured through environment variables:\n\n- `MAX_CONCURRENT_CALLS`: Maximum number of concurrent requests (default: 10)\n- Default chunk size for downloads: 1MB\n- Default timeout: 5 minutes\n- Default connections per host: 4\n\n## Contributing\n\nWe welcome contributions! Please see the\n[contributing](https://tiny-retriever.readthedocs.io/en/latest/CONTRIBUTING/) section\nfor guidelines and instructions.\n\n## License\n\nThis project is licensed under the terms of the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheginit%2Ftiny-retriever","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheginit%2Ftiny-retriever","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheginit%2Ftiny-retriever/lists"}