https://github.com/g-andrade/taskforce

On-demand worker pools for parallelizable tasks
https://github.com/g-andrade/taskforce

erlang parallelization worker-pool

Last synced: 4 months ago
JSON representation

On-demand worker pools for parallelizable tasks

• Host: GitHub
• URL: https://github.com/g-andrade/taskforce
• Owner: g-andrade
• License: mit
• Created: 2015-04-03T12:14:17.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2022-04-09T19:14:40.000Z (almost 3 years ago)
• Last Synced: 2024-04-27T07:42:33.396Z (9 months ago)
• Topics: erlang, parallelization, worker-pool
• Language: Erlang
• Homepage:
• Size: 66.4 KB
• Stars: 42
• Watchers: 4
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


# taskforce #

**This library is not under active maintenance; if you'd like to perform

maintenance yourself, feel free to open an issue requesting access.**

Copyright (c) 2015-2022 Guilherme Andrade

__Version:__ 1.2.3

__Authors:__ Guilherme Andrade ([`[email protected]`](mailto:[email protected])).

`taskforce` allows you to parallelise arbitrary tasks in a controlled way.

---------

### Creating tasks ###

```erlang

Tasks = #{ some_task => taskforce:task(fun work/0, Args, #{ timeout => 2000 }),

           similar_task => taskforce:task(fun work/0, Args123, #{ timeout => 2500 }),

           other_task => taskforce:task(fun other_work/1, OtherArgs, #{ timeout => 500 }) }.

```

### Executing tasks ###

```erlang

#{ completed := Completed } = taskforce:execute(Tasks),

SomeTaskResult = maps:get(some_task, Completed),

SimilarTaskResult = maps:get(similar_task, Completed),

OtherTaskResult = maps:get(other_task, Completed).

```

### Finely tuning execution ###

```erlang

ExecutionOptions = #{ max_workers => 8, timeout => 5000 },

#{ completed := Completed } = taskforce:execute(Tasks, ExecutionOptions),

% ...

```

### Individual task timeouts ###

```erlang

% ...

#{ individual_timeouts := IndividualTimeouts } = taskforce:execute(Tasks),

(length(IndividualTimeouts) > 0

 andalso io:format("oh noes! tasks with ids ~p timed-out",

                   [IndividualTimeouts]))

```

### Global execution timeouts ###

```erlang

% ...

#{ global_timeouts := GlobalTimeouts } = taskforce:execute(Tasks),

(length(GlobalTimeouts) > 0

 andalso io:format("execution ran out of time; tasks with ids ~p timed-out",

                   [GlobalTimeouts]))

```

### Full example ###

```erlang

%

% Calculate first 200 prime numbers using 4 processes,

% with a timeout of 2s per individual calculation and 10s

% for the whole batch.

%

NrOfPrimes = 200,

Tasks =

    maps:from_list(

        [{Nth, taskforce:task(fun fancy_lib:find_nth_prime/1,

                              [Nth], #{ timeout => 2000 })}

         || Nth <- lists:seq(1, NrOfPrimes)]),

ExecutionOptions =

    #{ % default is the sum of all individual task timeouts

       timeout => 10000,

       % default is number of active schedulers

       max_workers => 4 },

#{ completed := NthPrimes } = taskforce:execute(Tasks, ExecutionOptions),

io:format("200th prime is: ~p~n", [maps:get(200, NthPrimes)]).

```

Also in `examples/`.

## Modules ##

taskforce