{"id":28894864,"url":"https://github.com/grll/redis-producer-consumer","last_synced_at":"2025-08-01T21:04:31.544Z","repository":{"id":291133839,"uuid":"976597580","full_name":"grll/redis-producer-consumer","owner":"grll","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-02T15:32:20.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-02T16:34:37.271Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-02T11:54:27.000Z","updated_at":"2025-05-02T15:32:23.000Z","dependencies_parsed_at":"2025-05-02T16:45:54.193Z","dependency_job_id":null,"html_url":"https://github.com/grll/redis-producer-consumer","commit_stats":null,"previous_names":["grll/redis-producer-consumer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/grll/redis-producer-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grll%2Fredis-producer-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grll%2Fredis-producer-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grll%2Fredis-producer-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grll%2Fredis-producer-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grll","download_url":"https://codeload.github.com/grll/redis-producer-consumer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grll%2Fredis-producer-consumer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261060194,"owners_count":23103990,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-21T04:14:31.911Z","updated_at":"2025-08-01T21:04:31.516Z","avatar_url":"https://github.com/grll.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HTTPX Consumer Producer with Redis\n\nThis module implements a robust, cross-process producer-consumer pattern for making HTTP requests using [httpx](https://www.python-httpx.org/) and Redis as a task queue. It allows you to submit HTTP requests from any process (producer) and have them executed by a dedicated consumer process at a controlled rate, with results returned via Redis.\n\n## Features\n\n- **Priority Queues:** Supports multiple queues (e.g., high, low) for prioritizing tasks.\n- **Async \u0026 Cross-Process:** Producers and consumers can run in separate processes or even on different machines.\n- **Result Handling:** Each task gets a unique result key; producers block until the result is available.\n- **Configurable:** Redis URL and queue priorities are configurable via environment variables.\n- **Rate Limiting:** The consumer processes tasks at a configurable tick interval.\n\n## Installation\n\n1. Clone the repository and install dependencies:\n    ```bash\n    git clone git@github.com:grll/redis-producer-consumer.git\n    cd redis-producer-consumer\n    uv sync\n    ```\n\n2. Install and start Redis (see [Redis documentation](https://redis.io/docs/getting-started/)).\n\n## Configuration\n\nYou can configure the consumer and producer via environment variables:\n\n- `HTTPX_CP_REDIS_URL`: Redis connection URL (default: `redis://localhost:6379`)\n- `HTTPX_CP_QUEUES`: Comma-separated list of queue names, ordered by priority (default: `high,low`)\n\nExample:\n```bash\nexport HTTPX_CP_REDIS_URL=redis://localhost:6379\nexport HTTPX_CP_QUEUES=high,low\n```\n\n## How It Works\n\n- **Producer:** Submits HTTP request tasks to a Redis queue and waits for the result.\n- **Consumer:** Continuously polls the queues (by priority), executes HTTP requests using httpx, and pushes results back to Redis.\n\n## Usage\n\n### 1. Start the Consumer\n\nThe consumer will process tasks from the queues at a fixed interval (`--tick` seconds):\n\n```bash\npython httpx_consumer_producer.py --tick 1\n```\n\n- `--tick`: Number of seconds to wait between processing tasks (default: 1).\n\n### 2. Submit Tasks from a Producer\n\nYou can submit tasks from any Python process:\n\n```python\nimport asyncio\nfrom httpx_consumer_producer import submit_request\n\nasync def main():\n    result = await submit_request(\n        \"GET\", \"https://www.google.com\",\n        queue=\"high\",  # optional, defaults to \"low\"\n        timeout=10     # passed to httpx.request\n    )\n    print(result)\n\nasyncio.run(main())\n```\n\n- `*args`: Arguments for `httpx.request` (e.g., method, URL).\n- `**kwargs`: Keyword arguments for `httpx.request` (e.g., `timeout`, `headers`). You can also specify `queue` to select the priority.\n\nThe result is a dictionary:\n```python\n{\n    \"id\": \"\u003ctask-uuid\u003e\",\n    \"status_code\": 200,\n    \"content\": \"...\",\n    \"headers\": {...},\n    \"error\": None\n}\n```\n\n### 3. Task Structure\n\nEach task is a dictionary with:\n- `id`: Unique identifier (auto-generated)\n- `httpx_request_args`: List of positional args for `httpx.request`\n- `httpx_request_kwargs`: Dict of keyword args for `httpx.request`\n\n### 4. Error Handling\n\nIf the HTTP request fails, the result will include an `\"error\"` field with the error message.\n\n## Example Workflow\n\n1. Start the consumer:\n    ```bash\n    python httpx_consumer_producer.py --tick 2\n    ```\n2. In another process, submit a request:\n    ```python\n    import asyncio\n    from httpx_consumer_producer import submit_request\n\n    async def main():\n        result = await submit_request(\"GET\", \"https://httpbin.org/get\", timeout=5)\n        print(result)\n\n    asyncio.run(main())\n    ```\n\n## Notes\n\n- The consumer and producers can run on different machines as long as they share the same Redis instance.\n- Results are stored in Redis with a 30-second expiry to avoid stale data.\n- The consumer logs all actions for easy debugging.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrll%2Fredis-producer-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrll%2Fredis-producer-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrll%2Fredis-producer-consumer/lists"}