https://github.com/isarandi/throttledpool

Parallel process pool that throttles the task producer thread to avoid out-of-memory issues
https://github.com/isarandi/throttledpool

blocking-queue bounded-buffer multiprocessing process-pool semaphore

Last synced: 3 months ago
JSON representation

Parallel process pool that throttles the task producer thread to avoid out-of-memory issues

• Host: GitHub
• URL: https://github.com/isarandi/throttledpool
• Owner: isarandi
• License: mit
• Created: 2023-06-05T13:02:01.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2023-06-06T00:00:20.000Z (almost 2 years ago)
• Last Synced: 2025-01-16T15:35:59.717Z (4 months ago)
• Topics: blocking-queue, bounded-buffer, multiprocessing, process-pool, semaphore
• Language: Python
• Homepage:
• Size: 5.86 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# ThrottledPool

Parallel process pool that throttles the task producer thread to avoid out-of-memory issues.

## Example usage

```python

results = []

with throttledpool.ThrottledPool() as pool:

    for item in items:

        pool.apply_async(process_fn, (item, other_arg), kwargs=dict(a=1), callback=results.append)