https://github.com/python-trio/trimeter
(not ready yet) A simple but powerful job scheduler for Trio programs
https://github.com/python-trio/trimeter
async async-await concurrency-management map python trio
Last synced: 3 months ago
JSON representation
(not ready yet) A simple but powerful job scheduler for Trio programs
- Host: GitHub
- URL: https://github.com/python-trio/trimeter
- Owner: python-trio
- License: other
- Created: 2018-10-07T14:49:11.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2020-09-03T05:52:21.000Z (about 5 years ago)
- Last Synced: 2025-06-28T08:32:25.391Z (3 months ago)
- Topics: async, async-await, concurrency-management, map, python, trio
- Language: Python
- Homepage: https://trimeter.readthedocs.io
- Size: 49.8 KB
- Stars: 67
- Watchers: 11
- Forks: 3
- Open Issues: 11
-
Metadata Files:
- Readme: README.rst
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
.. image:: https://img.shields.io/badge/chat-join%20now-blue.svg
:target: https://gitter.im/python-trio/general
:alt: Join chatroom
.. image:: https://img.shields.io/badge/docs-read%20now-blue.svg
:target: https://trimeter.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://img.shields.io/pypi/v/trimeter.svg
:target: https://pypi.org/project/trimeter
:alt: Latest PyPi version
.. image:: https://travis-ci.org/python-trio/trimeter.svg?branch=master
:target: https://travis-ci.org/python-trio/trimeter
:alt: Automated test status
.. image:: https://codecov.io/gh/python-trio/trimeter/branch/master/graph/badge.svg
:target: https://codecov.io/gh/python-trio/trimeter
:alt: Test coverage
Warning
=======
This library isn't ready for release yet. Feedback welcome!
Trimeter
========
Trio is a friendly Python library for async concurrency and
networking. Trimeter is a simple but powerful job scheduler for
programs using Trio, released under your choice of the MIT or Apache 2
licenses.
Trimeter's core purpose is to make it easy to execute lots tasks
concurrently, with rich options to **control the degree of
concurrency** and to **collect the task results**.
Say you have 1000 urls that you want to fetch and process somehow:
.. code-block:: python3
# Old slow way
for url in urls:
await fetch_and_process(url)
That's slow, so you want to do several at the same time... but to
avoid overloading the network, you want to limit it to at most 5 calls
at once. Oh, and there's a request quota, so we have to throttle it
down to 1 per second. No problem:
.. code-block:: python3
# New and fancy way
await trimeter.run_on_each(
fetch_and_process, urls, max_at_once=5, max_per_second=1
)
What if we don't know the whole list of urls up front? No worries,
just pass in an async iterable instead, and Trimeter will do the right
thing.
What if we want to get the result from each call as it finishes, so we
can do something further with it? Just use ``amap`` (= short for
"async `map
`__"):
.. code-block:: python3
async with trimeter.amap(fetch_and_process, urls, ...) as results:
# Then iterate over the return values, as they become available
# (i.e., not necessarily in the original order)
async for result in results:
...
Of course ``amap`` also accepts throttling options like
``max_at_once``, ``max_per_second``, etc.
What if we want to use the `outcome library
`__ to capture exceptions, so one
call crashing doesn't terminate the whole program? And also, we want
to pass through the original url alongside each result, so we know
which result goes with which url?
.. code-block:: python3
async with trimeter.amap(
fetch_and_process,
urls,
capture_outcome=True,
include_value=True,
) as outcomes:
# Then iterate over the return values, as they become available
# (i.e., not necessarily in the original order)
async for url, outcome in outcomes:
try:
return_value = outcome.unwrap()
except Exception as exc:
print(f"error while processing {url}: {exc!r}")
What if we just want to call a few functions in parallel and then get
the results as a list, like `asyncio.gather
`__
or `Promise.all
`__?
.. code-block:: python3
return_values = await trimeter.run_all([
async_fn1,
async_fn2,
functools.partial(async_fn3, extra_arg, kwarg="yeah"),
])
Of course, this takes all the same options as the other functions, so
you can control the degree of parallelism, use ``capture_outcome`` to
capture exceptions, and so forth.
For more details, see `the fine manual
`__.
Can you summarize that in iambic trimeter?
------------------------------------------
`Iambic trimeter `__?
No problem:
| Trimeter gives you tools
| for running lots of tasks
| to do your work real fast
| but not so fast you crash.
Code of conduct
---------------
Contributors are requested to follow our `code of conduct
`__ in all
project spaces.