https://github.com/ftpsolutions/collapsing-thread-pool-executor

A take on the concurrent.futures.thread.ThreadPoolExecutor that self-manages the number of threads
https://github.com/ftpsolutions/collapsing-thread-pool-executor

mit-license python python3 threadpoolexecutor

Last synced: 5 months ago
JSON representation

A take on the concurrent.futures.thread.ThreadPoolExecutor that self-manages the number of threads

• Host: GitHub
• URL: https://github.com/ftpsolutions/collapsing-thread-pool-executor
• Owner: ftpsolutions
• License: mit
• Created: 2018-03-16T06:27:52.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2020-08-18T13:46:45.000Z (almost 6 years ago)
• Last Synced: 2026-01-06T00:52:53.088Z (6 months ago)
• Topics: mit-license, python, python3, threadpoolexecutor
• Language: Python
• Size: 24.4 KB
• Stars: 11
• Watchers: 9
• Forks: 4
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# CollapsingThreadPoolExecutor

The CollapsingThreadPoolExecutor is inspired by and compatible with the ThreadPoolExecutor from the

"futures" module, it operates differently in that worker threads are handled with a stack which results in the same worker or workers doing all the work (and idle workers being destroyed).

## How to install

    $ pip install collapsing-thread-pool-executor

## How to develop

**Prerequisites**

* python3 w/ pip

* python2 w/ pip

* virtualenvwrapper

* entr

**Set up the environments**

    $ mkvirtualenv -p `which python2.7` collapsing-thread-pool-executor-py2

    $ pip install .

    $ pip install -r requirements.txt

    $ mkvirtualenv -p `which python3` collapsing-thread-pool-executor-py3

    $ pip install .

    $ pip install -r requirements.txt

**Watch the tests**

    # watch python2 tests in one window

    $ workon collapsing-thread-pool-executor-py2

    $ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor

    

    # watch python3 tests in one window

    $ workon collapsing-thread-pool-executor-py3

    $ find ./ -name '*.py' | entr -c py.test -v --log-level=DEBUG collapsing_thread_pool_executor

## Examples

The example below will execute `some_task()` 100 times; as `some_task()` should take a second to execute and as we've allocated 10 workers, the whole thing should take about 10 seconds.

    import time

    from collapsing_thread_pool_executor import CollapsingThreadPoolExecutor

    def some_task():

        time.sleep(1)

    # all arguments are optional

    pool = CollapsingThreadPoolExecutor(

        workers=10,

        thread_name_prefix='SomePool',

        permitted_thread_age_in_seconds=60,

    )

    for i in range(0, 100):

        pool.submit(some_task)