{"id":13468509,"url":"https://github.com/florimondmanca/aiometer","last_synced_at":"2025-04-09T03:13:44.699Z","repository":{"id":45295352,"uuid":"248952799","full_name":"florimondmanca/aiometer","owner":"florimondmanca","description":"A Python concurrency scheduling library, compatible with asyncio and trio.","archived":false,"fork":false,"pushed_at":"2025-03-26T08:25:17.000Z","size":73,"stargazers_count":387,"open_issues_count":6,"forks_count":13,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-01T18:21:25.019Z","etag":null,"topics":["async","asyncio","concurrency-management","flow-control","python","trio"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/aiometer/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/florimondmanca.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-21T10:26:37.000Z","updated_at":"2025-03-18T13:13:14.000Z","dependencies_parsed_at":"2024-06-11T19:09:31.334Z","dependency_job_id":null,"html_url":"https://github.com/florimondmanca/aiometer","commit_stats":{"total_commits":37,"total_committers":4,"mean_commits":9.25,"dds":"0.32432432432432434","last_synced_commit":"6fbc90850ee5b63acc8db25acf2f28f2a6d4b6a7"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/florimondmanca%2Faiometer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/florimondmanca%2Faiometer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/florimondmanca%2Faiometer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/florimondmanca%2Faiometer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/florimondmanca","download_url":"https://codeload.github.com/florimondmanca/aiometer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247968374,"owners_count":21025823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","asyncio","concurrency-management","flow-control","python","trio"],"created_at":"2024-07-31T15:01:12.534Z","updated_at":"2025-04-09T03:13:44.692Z","avatar_url":"https://github.com/florimondmanca.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# aiometer\n\n[![Build Status](https://dev.azure.com/florimondmanca/public/_apis/build/status/florimondmanca.aiometer?branchName=master)](https://dev.azure.com/florimondmanca/public/_build/latest?definitionId=4\u0026branchName=master)\n[![Coverage](https://codecov.io/gh/florimondmanca/aiometer/branch/master/graph/badge.svg)](https://codecov.io/gh/florimondmanca/aiometer)\n![Python versions](https://img.shields.io/pypi/pyversions/aiometer.svg)\n[![Package version](https://badge.fury.io/py/aiometer.svg)](https://pypi.org/project/aiometer)\n\n`aiometer` is a concurrency scheduling library compatible with `asyncio` and `trio` and inspired by [Trimeter](https://github.com/python-trio/trimeter). It makes it easier to execute lots of tasks concurrently while controlling concurrency limits (i.e. applying _[backpressure](https://lucumr.pocoo.org/2020/1/1/async-pressure/)_) and collecting results in a predictable manner.\n\n**Content**\n\n- [Example](#example)\n- [Features](#features)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Flow control](#flow-control)\n  - [Running tasks](#running-tasks)\n- [How To](#how-to)\n- [API Reference](#api-reference)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Example\n\nLet's use [HTTPX](https://github.com/encode/httpx) to make web requests concurrently...\n\n_Try this code interactively using [IPython](https://ipython.org/install.html)._\n\n```python\n\u003e\u003e\u003e import asyncio\n\u003e\u003e\u003e import functools\n\u003e\u003e\u003e import random\n\u003e\u003e\u003e import aiometer\n\u003e\u003e\u003e import httpx\n\u003e\u003e\u003e\n\u003e\u003e\u003e client = httpx.AsyncClient()\n\u003e\u003e\u003e\n\u003e\u003e\u003e async def fetch(client, request):\n...     response = await client.send(request)\n...     # Simulate extra processing...\n...     await asyncio.sleep(2 * random.random())\n...     return response.json()[\"json\"]\n...\n\u003e\u003e\u003e requests = [\n...     httpx.Request(\"POST\", \"https://httpbin.org/anything\", json={\"index\": index})\n...     for index in range(100)\n... ]\n...\n\u003e\u003e\u003e # Send requests, and process responses as they're made available:\n\u003e\u003e\u003e async with aiometer.amap(\n...     functools.partial(fetch, client),\n...     requests,\n...     max_at_once=10, # Limit maximum number of concurrently running tasks.\n...     max_per_second=5,  # Limit request rate to not overload the server.\n... ) as results:\n...     async for data in results:\n...         print(data)\n...\n{'index': 3}\n{'index': 4}\n{'index': 1}\n{'index': 2}\n{'index': 0}\n...\n\u003e\u003e\u003e # Alternatively, fetch and aggregate responses into an (ordered) list...\n\u003e\u003e\u003e jobs = [functools.partial(fetch, client, request) for request in requests]\n\u003e\u003e\u003e results = await aiometer.run_all(jobs, max_at_once=10, max_per_second=5)\n\u003e\u003e\u003e results\n[{'index': 0}, {'index': 1}, {'index': 2}, {'index': 3}, {'index': 4}, ...]\n```\n\n## Installation\n\n_Be sure to pin any dependencies to the latest major version._\n\n```bash\npip install \"aiometer==1.*\"\n```\n\n## Features\n\n- Concurrency management and throttling helpers.\n- `asyncio` and `trio` support.\n- Fully type annotated.\n- 100% test coverage.\n\n## Usage\n\n### Flow control\n\nThe key highlight of `aiometer` is allowing you to apply flow control strategies in order to limit the degree of concurrency of your programs.\n\nThere are two knobs you can play with to fine-tune concurrency:\n\n- `max_at_once`: this is used to limit the maximum number of concurrently running tasks at any given time. (If you have 100 tasks and set `max_at_once=10`, then `aiometer` will ensure that no more than 10 run at the same time.)\n- `max_per_second`: this option limits the number of tasks spawned per second. This is useful to not overload I/O resources, such as servers that may have a rate limiting policy in place.\n\nExample usage:\n\n```python\n\u003e\u003e\u003e import asyncio\n\u003e\u003e\u003e import aiometer\n\u003e\u003e\u003e async def make_query(query):\n...     await asyncio.sleep(0.05)  # Simulate a database request.\n...\n\u003e\u003e\u003e queries = ['SELECT * from authors'] * 1000\n\u003e\u003e\u003e # Allow at most 5 queries to run concurrently at any given time:\n\u003e\u003e\u003e await aiometer.run_on_each(make_query, queries, max_at_once=5)\n...\n\u003e\u003e\u003e # Make at most 10 queries per second:\n\u003e\u003e\u003e await aiometer.run_on_each(make_query, queries, max_per_second=10)\n...\n\u003e\u003e\u003e # Run at most 10 concurrent jobs, spawning new ones at least every 5 seconds:\n\u003e\u003e\u003e async def job(id):\n...     await asyncio.sleep(10)  # A very long task.\n...\n\u003e\u003e\u003e await aiometer.run_on_each(job, range(100),  max_at_once=10, max_per_second=0.2)\n```\n\n### Running tasks\n\n`aiometer` provides 4 different ways to run tasks concurrently in the form of 4 different run functions. Each function accepts all the options documented in [Flow control](#flow-control), and runs tasks in a slightly different way, allowing to address a variety of use cases. Here's a handy table for reference (see also the [API Reference](#api-reference)):\n\n| Entrypoint      | Use case                                       |\n| --------------- | ---------------------------------------------- |\n| `run_on_each()` | Execute async callbacks in any order.          |\n| `run_all()`     | Return results as an ordered list.             |\n| `amap()`        | Iterate over results as they become available. |\n| `run_any()`     | Return result of first completed function.     |\n\nTo illustrate the behavior of each run function, let's first setup a hello world async program:\n\n```python\n\u003e\u003e\u003e import asyncio\n\u003e\u003e\u003e import random\n\u003e\u003e\u003e from functools import partial\n\u003e\u003e\u003e import aiometer\n\u003e\u003e\u003e\n\u003e\u003e\u003e async def get_greeting(name):\n...     await asyncio.sleep(random.random())  # Simulate I/O\n...     return f\"Hello, {name}\"\n...\n\u003e\u003e\u003e async def greet(name):\n...     greeting = await get_greeting(name)\n...     print(greeting)\n...\n\u003e\u003e\u003e names = [\"Robert\", \"Carmen\", \"Lucas\"]\n```\n\nLet's start with `run_on_each()`. It executes an async function once for each item in a list passed as argument:\n\n```python\n\u003e\u003e\u003e await aiometer.run_on_each(greet, names)\n'Hello, Robert!'\n'Hello, Lucas!'\n'Hello, Carmen!'\n```\n\nIf we'd like to get the list of greetings in the same order as `names`, in a fashion similar to [`Promise.all()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all), we can use `run_all()`:\n\n```python\n\u003e\u003e\u003e await aiometer.run_all([partial(get_greeting, name) for name in names])\n['Hello, Robert', 'Hello, Carmen!', 'Hello, Lucas!']\n```\n\n`amap()` allows us to process each greeting as it becomes available (which means maintaining order is not guaranteed):\n\n```python\n\u003e\u003e\u003e async with aiometer.amap(get_greeting, names) as greetings:\n...     async for greeting in greetings:\n...         print(greeting)\n'Hello, Lucas!'\n'Hello, Robert!'\n'Hello, Carmen!'\n```\n\nLastly, `run_any()` can be used to run async functions until the first one completes, similarly to [`Promise.any()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/any):\n\n```python\n\u003e\u003e\u003e await aiometer.run_any([partial(get_greeting, name) for name in names])\n'Hello, Carmen!'\n```\n\nAs a last fun example, let's use `amap()` to implement a no-threads async version of [sleep sort](https://rosettacode.org/wiki/Sorting_algorithms/Sleep_sort):\n\n```python\n\u003e\u003e\u003e import asyncio\n\u003e\u003e\u003e from functools import partial\n\u003e\u003e\u003e import aiometer\n\u003e\u003e\u003e numbers = [0.3, 0.1, 0.6, 0.2, 0.7, 0.5, 0.5, 0.2]\n\u003e\u003e\u003e async def process(n):\n...     await asyncio.sleep(n)\n...     return n\n...\n\u003e\u003e\u003e async with aiometer.amap(process, numbers) as results:\n...     sorted_numbers = [n async for n in results]\n...\n\u003e\u003e\u003e sorted_numbers\n[0.1, 0.2, 0.2, 0.3, 0.5, 0.5, 0.6, 0.7]\n```\n\n## How To\n\n### Multiple parametrized values in `run_on_each` and `amap`\n\n`run_on_each` and `amap` only accept functions that accept a single positional argument (i.e. `(Any) -\u003e Awaitable`).\n\nSo if you have a function that is parametrized by multiple values, you should refactor it to match this form.\n\nThis can generally be achieved like this:\n\n1. Build a proxy container type (eg. a `namedtuple`), eg `T`.\n2. Refactor your function so that its signature is now `(T) -\u003e Awaitable`.\n3. Build a list of these proxy containers, and pass it to `aiometer`.\n\nFor example, assuming you have a function that processes X/Y coordinates...\n\n```python\nasync def process(x: float, y: float) -\u003e None:\n    pass\n\nxs = list(range(100))\nys = list(range(100))\n\nfor x, y in zip(xs, ys):\n    await process(x, y)\n```\n\nYou could use it with `amap` by refactoring it like this:\n\n```python\nfrom typing import NamedTuple\n\n# Proxy container type:\nclass Point(NamedTuple):\n    x: float\n    y: float\n\n# Rewrite to accept a proxy as a single positional argument:\nasync def process(point: Point) -\u003e None:\n    x = point.x\n    y = point.y\n    ...\n\nxs = list(range(100))\nys = list(range(100))\n\n# Build a list of proxy containers:\npoints = [Point(x, y) for x, y in zip(x, y)]\n\n# Use it:\nasync with aiometer.amap(process, points) as results:\n    ...\n```\n\n## API Reference\n\n### Common options\n\n* `max_at_once` (_Optional_, `int`): the maximum number of concurrently running tasks at any given time.\n* `max_per_second` (_Optional_, `int`): the maximum number of tasks spawned per second.\n\n### `aiometer.run_on_each()`\n\n**Signature**: _async_ aiometer.run_on_each(*async_fn*, *args*, *, *max_at_once=None*, *max_per_second=None*) -\u003e *None*\n\nConcurrently run the equivalent of `async_fn(arg) for arg in args`. Does not return any value. To get return values back, use [`aiometer.run_all()`](#aiometerrun_all).\n\n### `aiometer.run_all()`\n\n**Signature**: _async_ aiometer.run_all(*async_fns*, *max_at_once=None*, *max_per_second=None*) -\u003e *list*\n\nConcurrently run the `async_fns` functions, and return the list of results in the same order.\n\n### `aiometer.amap()`\n\n**Signature**: _async_ aiometer.amap(*async_fn*, *args*, *max_at_once=None*, *max_per_second=None*) -\u003e *async iterator*\n\nConcurrently run the equivalent of `async_fn(arg) for arg in args`, and return an async iterator that yields results as they become available.\n\n### `aiometer.run_any()`\n\n**Signature**: _async_ aiometer.run_any(*async_fns*, *max_at_once=None*, *max_per_second=None*) -\u003e *Any*\n\nConcurrently run the `async_fns` functions, and return the first available result.\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflorimondmanca%2Faiometer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflorimondmanca%2Faiometer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflorimondmanca%2Faiometer/lists"}