{"id":19100663,"url":"https://github.com/thomhurst/EnumerableAsyncProcessor","last_synced_at":"2025-04-18T18:30:44.454Z","repository":{"id":43490711,"uuid":"444218938","full_name":"thomhurst/EnumerableAsyncProcessor","owner":"thomhurst","description":"Process Multiple Asynchronous Tasks in Various Ways - One at a time / Batched / Rate limited / Concurrently","archived":false,"fork":false,"pushed_at":"2024-10-28T12:17:19.000Z","size":235,"stargazers_count":115,"open_issues_count":5,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-01T16:09:10.611Z","etag":null,"topics":["async","async-await","asynchronous","asynchronous-programming","rate-limiting","task","tasks"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thomhurst.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-03T22:39:35.000Z","updated_at":"2024-10-03T15:31:25.000Z","dependencies_parsed_at":"2023-10-05T05:23:51.123Z","dependency_job_id":"95479dc0-078d-4c81-9620-a904623fe201","html_url":"https://github.com/thomhurst/EnumerableAsyncProcessor","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomhurst%2FEnumerableAsyncProcessor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomhurst%2FEnumerableAsyncProcessor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomhurst%2FEnumerableAsyncProcessor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomhurst%2FEnumerableAsyncProcessor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thomhurst","download_url":"https://codeload.github.com/thomhurst/EnumerableAsyncProcessor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223783097,"owners_count":17201904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","async-await","asynchronous","asynchronous-programming","rate-limiting","task","tasks"],"created_at":"2024-11-09T03:52:56.708Z","updated_at":"2025-04-18T18:30:44.447Z","avatar_url":"https://github.com/thomhurst.png","language":"C#","funding_links":[],"categories":["others"],"sub_categories":[],"readme":"# EnumerableAsyncProcessor\n\nProcess Multiple Asynchronous Tasks in Various Ways - One at a time / Batched / Rate limited / Concurrently\n\n[![nuget](https://img.shields.io/nuget/v/EnumerableAsyncProcessor.svg)](https://www.nuget.org/packages/EnumerableAsyncProcessor/)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/9c57d16dc4a841629560707c5ab3019d)](https://www.codacy.com/gh/thomhurst/EnumerableAsyncProcessor/dashboard?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=thomhurst/EnumerableAsyncProcessor\u0026amp;utm_campaign=Badge_Grade)\n[![CodeFactor](https://www.codefactor.io/repository/github/thomhurst/enumerableAsyncProcessor/badge)](https://www.codefactor.io/repository/github/thomhurst/enumerableAsyncProcessor)\n\u003c!-- ![Nuget](https://img.shields.io/nuget/dt/EnumerableAsyncProcessor) --\u003e\n\n## Installation\n\nInstall via Nuget\n`Install-Package EnumerableAsyncProcessor`\n\n## Why I built this\n\nBecause I've come across situations where you need to fine tune the rate at which you do things.\nMaybe you want it fast.\nMaybe you want it slow.\nMaybe you want it at a safe balance.\nMaybe you just don't want to write all the boilerplate code that comes with managing asynchronous operations!\n\n### Rate Limited Parallel Processor\n\n**Types**  \n\n| Type                                                        | Source Object | Return Object | Method 1            | Method 2           |\n|--------------------------------------------------|---------------|---------------|--------------------| ------------------ |\n| `RateLimitedParallelAsyncProcessor`                         | ❌             | ❌             | `.WithExecutionCount(int)` | `.ForEachAsync(delegate)` |\n| `RateLimitedParallelAsyncProcessor\u003cTInput\u003e`                | ✔             | ❌             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.ForEachAsync(delegate)` |\n| `ResultRateLimitedParallelAsyncProcessor\u003cTOutput\u003e`          | ❌             | ✔             | `.WithExecutionCount(int)` | `.SelectAsync(delegate)`  |\n| `ResultRateLimitedParallelAsyncProcessor\u003cTInput, TOutput\u003e` | ✔             | ✔             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.SelectAsync(delegate)`  |\n\n**How it works**  \nProcesses your Asynchronous Tasks in Parallel, but honouring the limit that you set. As one finishes, another will start.\n\nE.g. If you set a limit of 100, only 100 should ever run at any one time\n\nThis is a hybrid between Parallel Processor and Batch Processor (see below) - Trying to address the caveats of both. Increasing the speed of batching, but not overwhelming the system by using full parallelisation.\n\n**Usage**  \n\n```csharp\nvar ids = Enumerable.Range(0, 5000).ToList();\n\n// SelectAsync for if you want to return something\nvar results = await ids\n        .SelectAsync(id =\u003e DoSomethingAndReturnSomethingAsync(id), CancellationToken.None)\n        .ProcessInParallel(levelOfParallelism: 100);\n\n// ForEachAsync for when you have nothing to return\nawait ids\n        .ForEachAsync(id =\u003e DoSomethingAsync(id), CancellationToken.None) \n        .ProcessInParallel(levelOfParallelism: 100);\n```\n\n### Timed Rate Limited Parallel Processor (e.g. Limit RPS)\n\n**Types**  \n\n| Type                                                        | Source Object | Return Object | Method 1            | Method 2           |\n|--------------------------------------------------|---------------|---------------|--------------------| ------------------ |\n| `TimedRateLimitedParallelAsyncProcessor`                         | ❌             | ❌             | `.WithExecutionCount(int)` | `.ForEachAsync(delegate)` |\n| `TimedRateLimitedParallelAsyncProcessor\u003cTInput\u003e`                | ✔             | ❌             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.ForEachAsync(delegate)` |\n| `ResultTimedRateLimitedParallelAsyncProcessor\u003cTOutput\u003e`          | ❌             | ✔             | `.WithExecutionCount(int)` | `.SelectAsync(delegate)`  |\n| `ResultTimedRateLimitedParallelAsyncProcessor\u003cTInput, TOutput\u003e` | ✔             | ✔             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.SelectAsync(delegate)`  |\n\n**How it works**  \nProcesses your Asynchronous Tasks in Parallel, but honouring the limit that you set over the timespan that you set. As one finishes, another will start, unless you've hit the maximum allowed for the current timespan duration.\n\nE.g. If you set a limit of 100, and a timespan of 1 second, only 100 operation should ever run at any one time over the course of a second. If the operation finishes sooner than a second (or your provided timespan), it'll wait and then start the next operation once that timespan has elapsed.\n\nThis is useful in scenarios where, for example, you have an API but it has a request per second limit\n\n**Usage**  \n\n```csharp\nvar ids = Enumerable.Range(0, 5000).ToList();\n\n// SelectAsync for if you want to return something\nvar results = await ids\n        .SelectAsync(id =\u003e DoSomethingAndReturnSomethingAsync(id), CancellationToken.None)\n        .ProcessInParallel(levelOfParallelism: 100, TimeSpan.FromSeconds(1));\n\n// ForEachAsync for when you have nothing to return\nawait ids\n        .ForEachAsync(id =\u003e DoSomethingAsync(id), CancellationToken.None) \n        .ProcessInParallel(levelOfParallelism: 100, TimeSpan.FromSeconds(1));\n```\n\n**Caveats**  \n\n- If your operations take longer than your provided TimeSpan, you probably won't get your desired throughput. This processor ensures you don't go over your rate limit, but will not increase parallel execution if you're below it.\n\n### One At A Time\n\n**Types**  \n\n| Type                                               | Source Object | Return Object | Method 1            | Method 2           |\n|--------------------------------------------------|---------------|---------------|--------------------| ------------------ |\n| `OneAtATimeAsyncProcessor`                         | ❌             | ❌             | `.WithExecutionCount(int)` | `.ForEachAsync(delegate)` |\n| `OneAtATimeAsyncProcessor\u003cTInput\u003e`                | ✔             | ❌             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.ForEachAsync(delegate)` |\n| `ResultOneAtATimeAsyncProcessor\u003cTOutput\u003e`          | ❌             | ✔             | `.WithExecutionCount(int)` | `.SelectAsync(delegate)`  |\n| `ResultOneAtATimeAsyncProcessor\u003cTInput, TOutput\u003e` | ✔             | ✔             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.SelectAsync(delegate)`  |\n\n**How it works**  \nProcesses your Asynchronous Tasks One at a Time. Only one will ever progress at a time. As one finishes, another will start\n\n**Usage**  \n\n```csharp\nvar ids = Enumerable.Range(0, 5000).ToList();\n\n// SelectAsync for if you want to return something\nvar results = await ids\n        .SelectAsync(id =\u003e DoSomethingAndReturnSomethingAsync(id), CancellationToken.None)\n        .ProcessOneAtATime();\n\n// ForEachAsync for when you have nothing to return\nawait ids\n        .ForEachAsync(id =\u003e DoSomethingAsync(id), CancellationToken.None) \n        .ProcessOneAtATime();\n```\n\n**Caveats**  \n\n- Slowest method\n\n### Batch\n\n**Types**  \n\n| Type                                          | Source Object | Return Object | Method 1           | Method 2           |\n|--------------------------------------------------|---------------|---------------|--------------------| ------------------ |\n| `BatchAsyncProcessor`                         | ❌             | ❌             | `.WithExecutionCount(int)` | `.ForEachAsync(delegate)` |\n| `BatchAsyncProcessor\u003cTInput\u003e`                | ✔             | ❌             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.ForEachAsync(delegate)` |\n| `ResultBatchAsyncProcessor\u003cTOutput\u003e`          | ❌             | ✔             | `.WithExecutionCount(int)` | `.SelectAsync(delegate)`  |\n| `ResultBatchAsyncProcessor\u003cTInput, TOutput\u003e` | ✔             | ✔             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.SelectAsync(delegate)`  |\n\n**How it works**  \nProcesses your Asynchronous Tasks in Batches. The next batch will not start until every Task in previous batch has finished\n\n**Usage**  \n\n```csharp\nvar ids = Enumerable.Range(0, 5000).ToList();\n\n// SelectAsync for if you want to return something\nvar results = await ids\n        .SelectAsync(id =\u003e DoSomethingAndReturnSomethingAsync(id), CancellationToken.None)\n        .ProcessInBatches(batchSize: 100);\n\n// ForEachAsync for when you have nothing to return\nawait ids\n        .ForEachAsync(id =\u003e DoSomethingAsync(id), CancellationToken.None) \n        .ProcessInBatches(batchSize: 100);\n```\n\n**Caveats**  \n\n- If even just 1 Task in a batch is slow or hangs, this will prevent the next batch from starting\n- If you set a batch of 100, and 70 have finished, you'll only have 30 left executing. This could slow things down\n\n### Parallel\n\n**Types**  \n\n| Type                                             | Source Object | Return Object | Method 1           | Method 2           |\n|--------------------------------------------------|---------------|---------------|--------------------| ------------------ |\n| `ParallelAsyncProcessor`                         | ❌             | ❌             | `.WithExecutionCount(int)` | `.ForEachAsync(delegate)` |\n| `ParallelAsyncProcessor\u003cTInput\u003e`                | ✔             | ❌             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.ForEachAsync(delegate)` |\n| `ResultParallelAsyncProcessor\u003cTOutput\u003e`          | ❌             | ✔             | `.WithExecutionCount(int)` | `.SelectAsync(delegate)`  |\n| `ResultParallelAsyncProcessor\u003cTInput, TOutput\u003e` | ✔             | ✔             | `.WithItems(IEnumerable\u003cTInput\u003e)` | `.SelectAsync(delegate)`  |\n\n**How it works**  \nProcesses your Asynchronous Tasks as fast as it can. All at the same time if it can\n\n**Usage**  \n\n```csharp\nvar ids = Enumerable.Range(0, 5000).ToList();\n\n// SelectAsync for if you want to return something\nvar results = await ids\n        .SelectAsync(id =\u003e DoSomethingAndReturnSomethingAsync(id), CancellationToken.None)\n        .ProcessInParallel();\n\n// ForEachAsync for when you have nothing to return\nawait ids\n        .ForEachAsync(id =\u003e DoSomethingAsync(id), CancellationToken.None) \n        .ProcessInParallel();\n```\n\n**Caveats**  \n\n- Depending on how many operations you have, you could overwhelm your system. Memory and CPU and Network usage could spike, and cause bottlenecks / crashes / exceptions\n\n### Processor Methods\n\nAs above, you can see that you can just `await` on the processor to get the results.\nBelow shows examples of using the processor object and the various methods available.\n\nThis is for when you need to Enumerate through some objects and use them in your operations. E.g. Sending notifications to certain ids\n\n```csharp\n    var httpClient = new HttpClient();\n\n    var ids = Enumerable.Range(0, 5000).ToList();\n\n    // This is for when you need to Enumerate through some objects and use them in your operations\n    \n    var itemProcessor = Enumerable.Range(0, 5000).ToAsyncProcessorBuilder()\n        .SelectAsync(NotifyAsync)\n        .ProcessInParallel(100);\n\n    // Or\n    // var itemProcessor = AsyncProcessorBuilder.WithItems(ids)\n    //     .SelectAsync(NotifyAsync, CancellationToken.None)\n    //     .ProcessInParallel(100);\n\n// GetEnumerableTasks() returns IEnumerable\u003cTask\u003cTOutput\u003e\u003e - These may have completed, or may still be waiting to finish.\n    var tasks = itemProcessor.GetEnumerableTasks();\n\n// Or call GetResultsAsyncEnumerable() to get an IAsyncEnumerable\u003cTOutput\u003e so you can process them in real-time as they finish.\n    await foreach (var httpResponseMessage in itemProcessor.GetResultsAsyncEnumerable())\n    {\n        // Do something\n    }\n\n// Or call GetResultsAsync() to get a Task\u003cTOutput[]\u003e that contains all of the finished results \n    var results = await itemProcessor.GetResultsAsync();\n\n// My dummy method\n    Task\u003cHttpResponseMessage\u003e NotifyAsync(int id)\n    {\n        return httpClient.GetAsync($\"https://localhost:8080/notify/{id}\");\n    }\n```\n\nThis is for when you need to don't need any objects - But just want to do something a certain amount of times. E.g. Pinging a site to warm up multiple instances\n\n```csharp\n    var httpClient = new HttpClient();\n\n    var itemProcessor = AsyncProcessorBuilder.WithExecutionCount(100)\n        .SelectAsync(PingAsync, CancellationToken.None)\n        .ProcessInParallel(10);\n\n// GetEnumerableTasks() returns IEnumerable\u003cTask\u003cTOutput\u003e\u003e - These may have completed, or may still be waiting to finish.\n    var tasks = itemProcessor.GetEnumerableTasks();\n\n// Or call GetResultsAsyncEnumerable() to get an IAsyncEnumerable\u003cTOutput\u003e so you can process them in real-time as they finish.\n    await foreach (var httpResponseMessage in itemProcessor.GetResultsAsyncEnumerable())\n    {\n        // Do something\n    }\n\n// Or call GetResultsAsync() to get a Task\u003cTOutput[]\u003e that contains all of the finished results \n    var results = await itemProcessor.GetResultsAsync();\n\n// My dummy method\n    Task\u003cHttpResponseMessage\u003e PingAsync()\n    {\n        return httpClient.GetAsync(\"https://localhost:8080/ping\");\n    }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomhurst%2FEnumerableAsyncProcessor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthomhurst%2FEnumerableAsyncProcessor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomhurst%2FEnumerableAsyncProcessor/lists"}