{"id":18084873,"url":"https://github.com/jwodder/interleave","last_synced_at":"2025-04-12T20:10:09.836Z","repository":{"id":41491280,"uuid":"448099300","full_name":"jwodder/interleave","owner":"jwodder","description":"Yield from multiple iterators as values become available","archived":false,"fork":false,"pushed_at":"2025-01-28T19:48:01.000Z","size":88,"stargazers_count":7,"open_issues_count":4,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-12T20:10:00.214Z","etag":null,"topics":["available-on-pypi","interleaving","iterators","multithreading","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jwodder.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-14T20:30:23.000Z","updated_at":"2025-01-28T19:48:05.000Z","dependencies_parsed_at":"2023-09-25T22:50:01.188Z","dependency_job_id":"bdbb53b6-38b6-4ada-a4e0-872dff99d859","html_url":"https://github.com/jwodder/interleave","commit_stats":{"total_commits":78,"total_committers":1,"mean_commits":78.0,"dds":0.0,"last_synced_commit":"621f052a9fc96d912aab98070f5f4d81c3f2dee0"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Finterleave","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Finterleave/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Finterleave/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jwodder%2Finterleave/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jwodder","download_url":"https://codeload.github.com/jwodder/interleave/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625493,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["available-on-pypi","interleaving","iterators","multithreading","python"],"created_at":"2024-10-31T15:08:28.638Z","updated_at":"2025-04-12T20:10:09.809Z","avatar_url":"https://github.com/jwodder.png","language":"Python","readme":"|repostatus| |ci-status| |coverage| |pyversions| |conda| |license|\n\n.. |repostatus| image:: https://www.repostatus.org/badges/latest/active.svg\n    :target: https://www.repostatus.org/#active\n    :alt: Project Status: Active — The project has reached a stable, usable\n          state and is being actively developed.\n\n.. |ci-status| image:: https://github.com/jwodder/interleave/actions/workflows/test.yml/badge.svg\n    :target: https://github.com/jwodder/interleave/actions/workflows/test.yml\n    :alt: CI Status\n\n.. |coverage| image:: https://codecov.io/gh/jwodder/interleave/branch/master/graph/badge.svg\n    :target: https://codecov.io/gh/jwodder/interleave\n\n.. |pyversions| image:: https://img.shields.io/pypi/pyversions/interleave.svg\n    :target: https://pypi.org/project/interleave/\n\n.. |conda| image:: https://img.shields.io/conda/vn/conda-forge/interleave.svg\n    :target: https://anaconda.org/conda-forge/interleave\n    :alt: Conda Version\n\n.. |license| image:: https://img.shields.io/github/license/jwodder/interleave.svg\n    :target: https://opensource.org/licenses/MIT\n    :alt: MIT License\n\n`GitHub \u003chttps://github.com/jwodder/interleave\u003e`_\n| `PyPI \u003chttps://pypi.org/project/interleave/\u003e`_\n| `Issues \u003chttps://github.com/jwodder/interleave/issues\u003e`_\n| `Changelog \u003chttps://github.com/jwodder/interleave/blob/master/CHANGELOG.md\u003e`_\n\nThe ``interleave`` package provides a function of the same name that takes a\nnumber of iterators, runs them in separate threads, and yields the values\nproduced as soon as each one is available.\n\nInstallation\n============\n``interleave`` requires Python 3.8 or higher.  Just use `pip\n\u003chttps://pip.pypa.io\u003e`_ for Python 3 (You have pip, right?) to install\n``interleave`` and its dependencies::\n\n    python3 -m pip install interleave\n\n\nExample\n=======\n\n\n.. code:: python\n\n    \u003e\u003e\u003e from time import sleep, strftime\n    \u003e\u003e\u003e from interleave import interleave\n    \u003e\u003e\u003e\n    \u003e\u003e\u003e def sleeper(idno, delays):\n    ...     for i, d in enumerate(delays):\n    ...         sleep(d)\n    ...         yield (idno, i)\n    ...\n    \u003e\u003e\u003e with interleave(\n    ...     [\n    ...         sleeper(0, [0, 1, 2]),\n    ...         sleeper(1, [2, 2, 2]),\n    ...         sleeper(2, [5, 2, 1]),\n    ...     ]\n    ... ) as it:\n    ...     for x in it:\n    ...         print(strftime(\"%H:%M:%S\"), x)\n    ...\n    22:08:39 (0, 0)\n    22:08:40 (0, 1)\n    22:08:41 (1, 0)\n    22:08:42 (0, 2)\n    22:08:43 (1, 1)\n    22:08:44 (2, 0)\n    22:08:45 (1, 2)\n    22:08:46 (2, 1)\n    22:08:47 (2, 2)\n\n\nAPI\n===\n\n.. code:: python\n\n    interleave.interleave(\n        iterators: Iterable[Iterator[T]],\n        *,\n        max_workers: int | None = None,\n        thread_name_prefix: str = \"\",\n        queue_size: int | None = None,\n        onerror: interleave.OnError = interleave.STOP,\n    ) -\u003e interleave.Interleaver[T]\n\n``interleave()`` runs the given iterators in separate threads and returns an\niterator that yields the values yielded by them as they become available.  (See\nbelow for details on the ``Interleaver`` class.)\n\nNote that ``iterators`` (but not its elements) is fully evaluated \u0026 consumed\nbefore `interleave()` returns.  If you instead want ``iterators`` to be\nevaluated concurrently with the iterators themselves, see\n``lazy_interleave()``.\n\nThe ``max_workers`` and ``thread_name_prefix`` parameters are passed through to\nthe underlying |ThreadPoolExecutor|_ (q.v.).  ``max_workers`` determines the\nmaximum number of iterators to run at one time.\n\n.. |ThreadPoolExecutor| replace:: ``concurrent.futures.ThreadPoolExecutor``\n.. _ThreadPoolExecutor:\n   https://docs.python.org/3/library/concurrent.futures.html\n   #concurrent.futures.ThreadPoolExecutor\n\nThe ``queue_size`` parameter sets the maximum size of the queue used internally\nto pipe values yielded by the iterators; when the queue is full, any iterator\nwith a value to yield will block waiting for the next value to be dequeued by a\ncall to the interleaver's ``__next__``.  When ``queue_size`` is ``None`` (the\ndefault), ``interleave()`` uses a ``queue.SimpleQueue``, which has no maximum\nsize.  When ``queue_size`` is non-``None`` (including zero, signifying no\nmaximum size), ``interleave()`` uses a ``queue.Queue``, whose ``get()`` method\nis uninterruptible (including by ``KeyboardInterrupt``) on Windows.\n\nThe ``onerror`` parameter is an enum that determines how ``interleave()``\nshould behave if one of the iterators raises an exception.  The possible values\nare:\n\n``STOP``\n    *(default)* Stop iterating over all iterators, cancel any outstanding\n    iterators that haven't been started yet, wait for all running threads to\n    finish, and reraise the exception.  Note that, due to the inability to stop\n    an iterator between yields, the \"waiting\" step involves waiting for each\n    currently-running iterator to yield its next value before stopping.  This\n    can deadlock if the queue fills up in the interim.\n\n``DRAIN``\n    Like ``STOP``, but any remaining values yielded by the iterators before\n    they finish are yielded by the interleaver before raising the exception\n\n``FINISH_ALL``\n    Continue running as normal and reraise the exception once all iterators\n    have finished\n\n``FINISH_CURRENT``\n    Like ``FINISH_ALL``, except that only currently-running iterators are run\n    to completion; any iterators whose threads haven't yet been started when\n    the exception is raised will have their jobs cancelled\n\nRegardless of the value of ``onerror``, any later exceptions raised by\niterators after the initial exception are discarded.\n\n.. code:: python\n\n    interleave.lazy_interleave(\n        iterators: Iterable[Iterator[T]],\n        *,\n        max_workers: int | None = None,\n        thread_name_prefix: str = \"\",\n        queue_size: int | None = None,\n        onerror: interleave.OnError = interleave.STOP,\n    ) -\u003e interleave.Interleaver[T]\n\n*New in version 0.3.0*\n\nLike ``interleave()``, but instead of fully evaluating ``iterators``\nimmediately, it is iterated over in a thread in the thread pool, and as each\niterator is produced, it is submitted to the ``Interleaver`` concurrently with\nthe ``Interleaver`` iterating over the other iterators already produced.  This\nis useful if the creation of the iterators themselves is nontrivial and\ninvolves work that could be done currently with the iterators themselves.\n\nNote that the ``iterators``-evaluation thread is handled the same way as other\nthreads when it comes to error handling: an exception occurring in one of the\niterator threads may (depending on the value of ``onerror``) cause the\n``iterators`` thread to be stopped, and if ``next(iter(iterators))`` raises an\nexception, it may cause all other threads to be stopped.\n\n.. code:: python\n\n    class Interleaver(Generic[T]):\n        def __init__(\n            self,\n            max_workers: int | None = None,\n            thread_name_prefix: str = \"\",\n            queue_size: int | None = None,\n            onerror: OnError = STOP,\n        )\n\nAn iterator and context manager.  As an iterator, it yields the values\ngenerated by the iterators passed to the corresponding ``interleave()`` call as\nthey become available.  As a context manager, it returns itself on entry and,\non exit, cleans up any unfinished threads by calling the\n``shutdown(wait=True)`` method (see below).\n\nAn ``Interleaver`` can be instantiated either by calling ``interleave()`` or by\ncalling the constructor directly.  The constructor takes the same arguments as\n``interleave()``, minus ``iterators``, and produces a new ``Interleaver`` that\nis not yet running any iterators.  Iterators are submitted to a new\n``Interleaver`` via the ``submit()`` method; once all desired iterators have\nbeen submitted, the ``finalize()`` method **must** be called so that the\n``Interleaver`` can tell when everything's finished.\n\nAn ``Interleaver`` will shut down its ``ThreadPoolExecutor`` and wait for the\nthreads to finish after yielding its final value (specifically, when a call is\nmade to ``__next__``/``get()`` that would result in ``StopIteration`` or\nanother exception being raised).  In the event that an ``Interleaver`` is\nabandoned before iteration completes, the associated resources may not be\nproperly cleaned up, and threads may continue running indefinitely.  For this\nreason, it is strongly recommended that you wrap any iteration over an\n``Interleaver`` in the context manager in order to handle a premature end to\niteration (including from a ``KeyboardInterrupt``).\n\nBesides the iterator and context manager APIs, an ``Interleaver`` has the\nfollowing public methods:\n\n.. code:: python\n\n    Interleaver.submit(it: Iterator[T]) -\u003e None\n\n*New in version 0.2.0*\n\nAdd an iterator to the ``Interleaver``.\n\nIf the ``Interleaver`` was returned from ``interleave()`` or has already had\n``finalize()`` called on it, calling ``submit()`` will result in a\n``ValueError``.\n\n.. code:: python\n\n    Interleave.finalize() -\u003e None\n\n*New in version 0.2.0*\n\nNotify the ``Interleaver`` that all iterators have been registered.  This\nmethod must be called in order for the ``Interleaver`` to detect the end of\niteration; if this method has not been called and all submitted iterators have\nfinished \u0026 had their values retrieved, then a subsequent call to ``next(it)``\nwill end up hanging indefinitely.\n\n.. code:: python\n\n    Interleaver.get(block: bool = True, timeout: float | None = None) -\u003e T\n\n*New in version 0.2.0*\n\nFetch the next value generated by the iterators.  If all iterators have\nfinished and all values have been retrieved, raises\n``interleaver.EndOfInputError``.  If ``block`` is ``False`` and no values are\nimmediately available, raises ``queue.Empty``.  If ``block`` is ``True``, waits\nup to ``timeout`` seconds (or indefinitely, if ``timeout`` is ``None``) for the\nnext value to become available or for all iterators to end; if nothing happens\nbefore the timeout expires, raises ``queue.Empty``.\n\n``it.get(block=True, timeout=None)`` is equivalent to ``next(it)``, except that\nthe latter converts an ``EndOfInputError`` to ``StopIteration``.\n\n**Note:** When ``onerror=STOP`` and a timeout is set, if an iterator raises an\nexception, the timeout may be exceeded as the ``Interleaver`` waits for all\nremaining threads to shut down.\n\n.. code:: python\n\n    Interleaver.shutdown(wait: bool = True) -\u003e None\n\nCall ``finalize()`` if it hasn't been called yet, tell all running iterators to\nstop iterating, cancel any outstanding iterators that haven't been started yet,\nand shut down the ``ThreadPoolExecutor``.  The ``wait`` parameter is passed\nthrough to the call to ``ThreadPoolExecutor.shutdown()``.\n\nThe ``Interleaver`` can continue to be iterated over after calling\n``shutdown()`` and will yield any remaining values produced by the iterators\nbefore they stopped completely.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwodder%2Finterleave","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjwodder%2Finterleave","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjwodder%2Finterleave/lists"}