{"id":28112036,"url":"https://github.com/abdulrahman-mh/unlimited-async","last_synced_at":"2026-02-07T19:02:03.791Z","repository":{"id":238151034,"uuid":"795982600","full_name":"abdulrahman-mh/unlimited-async","owner":"abdulrahman-mh","description":"Unlimited async operation with python","archived":false,"fork":false,"pushed_at":"2024-05-08T18:28:04.000Z","size":29,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-07T03:10:39.532Z","etag":null,"topics":["contai","microservices","optmization","prefo","py","reque"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abdulrahman-mh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-04T15:28:25.000Z","updated_at":"2025-03-31T21:13:23.000Z","dependencies_parsed_at":"2024-05-08T19:55:08.191Z","dependency_job_id":null,"html_url":"https://github.com/abdulrahman-mh/unlimited-async","commit_stats":null,"previous_names":["abdelrahman-mh/unlimited-async","abdulrahman-mh/unlimited-async"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/abdulrahman-mh/unlimited-async","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulrahman-mh%2Funlimited-async","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulrahman-mh%2Funlimited-async/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulrahman-mh%2Funlimited-async/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulrahman-mh%2Funlimited-async/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abdulrahman-mh","download_url":"https://codeload.github.com/abdulrahman-mh/unlimited-async/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abdulrahman-mh%2Funlimited-async/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29204952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T17:44:10.191Z","status":"ssl_error","status_checked_at":"2026-02-07T17:44:07.936Z","response_time":63,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["contai","microservices","optmization","prefo","py","reque"],"created_at":"2025-05-14T04:34:29.029Z","updated_at":"2026-02-07T19:02:03.747Z","avatar_url":"https://github.com/abdulrahman-mh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Unlimited Python Async Operations\n\nUnlimited async operation with Python, e.g., performing an unlimited number of HTTP requests.\n\n\u003e This method is inspired by the article [Making an Unlimited Number of Requests with Python aiohttp + pypeln](https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95).\n\n- [Overview](#overview)\n- [Enhanced Method:](#enhanced-method)\n- [Results](#results)\n- [Setup](#setup)\n\n# Overview\n\n### Original Method [Cristian Garcia](https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95):\n\n```python\nfrom aiohttp import ClientSession, TCPConnector\nimport asyncio\nimport sys\nimport pypeln as pl\n\n\nlimit = 1000\nurls = (\"http://007-server-1:8080/{}\".format(i) for i in range(int(sys.argv[1])))\n\n\nasync def main():\n    async with ClientSession(connector=TCPConnector(limit=0)) as session:\n\n        async def fetch(url):\n            async with session.get(url) as response:\n                print(url)\n                return await response.read()\n\n        await pl.task.each(\n            fetch,\n            urls,\n            workers=limit,\n        )\n\n\nasyncio.run(main())\n\n```\n\n- Method in `client/bench.py` file\n- This using `pypeln` for parallelizing and pipelining data processing tasks\n- `limit=1000` : maximum number of concurrent HTTP requests allowed, so if you increase the `limit` the Memory and CPU usage will increase to.\n\n### Enhanced Method:\n\n```python\nfrom aiohttp import ClientSession, TCPConnector\nimport asyncio\nimport sys\n\nlimit = 1000\nsem = asyncio.Semaphore(limit)\n\nasync def fetch(url, session):\n    async with sem:\n        async with session.get(url) as response:\n            print(url)\n            return await response.read()\n\nasync def main():\n    urls = [f\"http://007-server-1:8080/{i}\" for i in range(int(sys.argv[1]))]\n    async with ClientSession(connector=TCPConnector(limit=0)) as session:\n        tasks = [fetch(url, session) for url in urls]\n        await asyncio.gather(*tasks)\n\nasyncio.run(main())\n\n```\n\n- Method in `client/bench_v2.py` file.\n- It use `asyncio.Semaphore` to limit the number of concurrent tasks, thereby reducing Memory usage by approximately 50%.\n\n# Results\n\n#### System Information\n\n- **OS**: Ubuntu 22.04.4 LTS, jammy\n- 8GB Memory, CPU Intel(R) i3-10100F 3.60GHz - 8 Cors, 4 Core ber socket\n- **Docker version**: 26.1.0, build 9714adc\n\nMethod Version 2 that use `asyncio.Semaphore` in `client/bench_v2.py`, is better then origin [Cristian Garcia](https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95) method in `client/bench.py` file , just in memory usage, he lees then it almost by `50%`, but other thing like the time he takes and the CUP usage they almost same.\n\nThis chart explain some results for `bench.py` VS `bench_v2`, in Memory usage by diffracts request count:\n![bench_v1 VS bench_v2](./docs/image.png)\n\n\u003e Like we see V2 in `bench_v2.py` wins :muscle:\n\n```shell\n# For 100K Request\n\n# bench.py\n./timed.sh python bench.py 100_000\n# Memory usage: 402924KB  Time: 157.43s    CPU usage: 44%\n\n# bench_v2.py\n./timed.sh python bench_v2.py 100_000\n# Memory usage: 206204KB  Time: 154.36s    CPU usage: 31%\n```\n\n```shell\n# For 1M Request\n\n# bench.py\n./timed.sh python bench.py 1_000_000\n# Memory usage: 3639804KB Time: 1551.67s   CPU usage: 38%\n\n# bench_v2.py\n./timed.sh python bench_v2.py 1_000_000\n# Memory usage: 1607608KB Time: 1522.06s   CPU usage: 30%\n```\n\n# Setup\n\nEach service is separated in different container, `client` and `server`, you can use the server container if you wanted a tests local, or you can just run the `client`.\n\n- **Server service**: in `./server` it contain `./server/server.py` which functions as a simple `HTTP` server using [aiohttp](https://docs.aiohttp.org/en/stable/), we create a http://localhost:8080/{name} route, he simply have a delay with a random time between `1-3` second and return a response with the name you pass.\n\n- **Client service**: in `./client`, includes the original method(`./client/bench.py`) by [Cristian Garcia](https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95), and the enhanced version (`./client/bench_v2.py`) with `asyncio.Semaphore`, we use `timed.sh` script to calc the python app time taken and also the CPU and Memory usage\n\n### Setting Up Local Server \u0026 Client\n\nWe use `docker compose` to handle it:\n\n```shell\n# first you need to go in the project directory and simple run:\ndocker compose up\n\n# to run in background or detach mode add '-d' flag:\ndocker compose up -d\n```\n\nThis command starts both server and client services. The server runs on http://localhost:8080 on your system.\n\nThe client service does not execute any services but keeps the container alive by running `tail -f /dev/null`. Alternatively, you can modify the `./client/Dockerfile` to execute the Python script or benchmark test.\n\n### Setting Up Just the Client\n\n- Build the image:\n\n  ```shell\n  docker build -t py-bench ./client\n  # ./client is the directory containing the Dockerfile\n  ```\n\n- Run the container:\n\n  ```shell\n  docker run -d py-bench\n  ```\n\n### Development Client\n\nFor development, utilize the VSCode Dev Container. Run the container, then use \"Attach to Running Container\" in VSCode for a seamless development environment. Any changes made will be applied automatically.\n\nTo execute commands inside the container, open a shell in VSCode or use the following in your terminal:\n\n```shell\ndocker ps\n# List all running containers and note the client name.\n\ndocker exec -it \u003ccontainer name\u003e sh\n# Replace \u003ccontainer name\u003e with the actual container name.\n# Use a specific command instead of 'sh' for just running a command in the container and close the shell\n```\n\n### Running Tests\n\n- Use `./client/timed.sh` to measure the Python app's runtime, CPU, and Memory usage. For example:\n\n  ```shell\n  ./timed.sh python bench.py 10000\n  ```\n\n- \u003e `bench.py` and `bench_v2.py` takes the request count as a parameter\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdulrahman-mh%2Funlimited-async","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabdulrahman-mh%2Funlimited-async","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabdulrahman-mh%2Funlimited-async/lists"}