Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fredwu/opq
Elixir queue! A simple, in-memory queue with worker pooling and rate limiting in Elixir.
https://github.com/fredwu/opq
back-pressure demand-control elixir fifo gen-stage genstage in-memory opq pool pooling queue rate-limit worker-pool
Last synced: 7 days ago
JSON representation
Elixir queue! A simple, in-memory queue with worker pooling and rate limiting in Elixir.
- Host: GitHub
- URL: https://github.com/fredwu/opq
- Owner: fredwu
- Created: 2017-08-09T13:38:18.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2023-09-29T13:49:51.000Z (about 1 year ago)
- Last Synced: 2024-09-01T02:38:44.810Z (2 months ago)
- Topics: back-pressure, demand-control, elixir, fifo, gen-stage, genstage, in-memory, opq, pool, pooling, queue, rate-limit, worker-pool
- Language: Elixir
- Homepage: https://hex.pm/packages/opq
- Size: 92.8 KB
- Stars: 260
- Watchers: 11
- Forks: 8
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
- freaking_awesome_elixir - Elixir - A simple, in-memory queue with worker pooling and rate limiting in Elixir. (Queue)
- fucking-awesome-elixir - opq - A simple, in-memory queue with worker pooling and rate limiting in Elixir. (Queue)
- awesome-elixir - opq - A simple, in-memory queue with worker pooling and rate limiting in Elixir. (Queue)
README
# OPQ: One Pooled Queue
[](https://github.com/fredwu/opq/actions)
[](https://codebeat.co/projects/github-com-fredwu-opq-master)
[](https://coveralls.io/github/fredwu/opq?branch=master)
[](https://hex.pm/packages/opq)
[](https://hexdocs.pm/opq/)
[](https://hex.pm/packages/opq)
[](https://github.com/fredwu/opq/blob/master/LICENSE.md)
[](https://github.com/fredwu/opq/commits/master)
## Elixir Queue!
A simple, in-memory queue with worker pooling and rate limiting in Elixir. OPQ leverages Erlang's [queue](http://erlang.org/doc/man/queue.html) module and Elixir's [GenStage](https://github.com/elixir-lang/gen_stage).
Originally built to support [Crawler](https://github.com/fredwu/crawler).
## Features
- A fast, in-memory FIFO queue.
- Worker pool.
- Rate limit.
- Timeouts.
- Pause / resume / stop the queue.
See [Hex documentation](https://hexdocs.pm/opq/).
## Installation
```Elixir
def deps do
[{:opq, "~> 4.0"}]
end
```
## Usage
### A simple example:
```elixir
{:ok, opq} = OPQ.init()
OPQ.enqueue(opq, fn -> IO.inspect("hello") end)
OPQ.enqueue(opq, fn -> IO.inspect("world") end)
```
### Specify module, function and arguments:
```elixir
{:ok, opq} = OPQ.init()
OPQ.enqueue(opq, IO, :inspect, ["hello"])
OPQ.enqueue(opq, IO, :inspect, ["world"])
```
### Specify a custom name for the queue:
```elixir
OPQ.init(name: :items)
OPQ.enqueue(:items, fn -> IO.inspect("hello") end)
OPQ.enqueue(:items, fn -> IO.inspect("world") end)
```
### Start as part of a supervision tree:
Note, when starting as part of a supervision tree, the `:name` option must be provided.
```elixir
children = [
{OPQ, name: :items}
]
```
### Specify a custom worker to process items in the queue:
```elixir
defmodule CustomWorker do
def start_link(item) do
Task.start_link(fn ->
Agent.update(:bucket, &[item | &1])
end)
end
end
Agent.start_link(fn -> [] end, name: :bucket)
{:ok, opq} = OPQ.init(worker: CustomWorker)
OPQ.enqueue(opq, "hello")
OPQ.enqueue(opq, "world")
Agent.get(:bucket, & &1) # => ["world", "hello"]
```
### Rate limit:
```elixir
{:ok, opq} = OPQ.init(workers: 1, interval: 1000)
Task.async(fn ->
OPQ.enqueue(opq, fn -> IO.inspect("hello") end)
OPQ.enqueue(opq, fn -> IO.inspect("world") end)
end)
```
If no interval is supplied, the ratelimiter will be bypassed.
### Check the queue and number of available workers:
```elixir
{:ok, opq} = OPQ.init()
OPQ.enqueue(opq, fn -> Process.sleep(1000) end)
{status, queue, available_workers} = OPQ.info(opq) # => {:normal, #OPQ.Queue<[]>, 9}
Process.sleep(1200)
{status, queue, available_workers} = OPQ.info(opq) # => {:normal, #OPQ.Queue<[]>, 10}
```
If you just need to get the queue itself:
```elixir
OPQ.queue(opq) # => #OPQ.Queue<[]>
```
### Queue
OPQ implements `Enumerable`, so you can perform enumerable functions on the queue:
```elixir
{:ok, opq} = OPQ.init()
queue = OPQ.queue(opq)
Enum.count(queue) # => 0
Enum.empty?(queue) # => true
```
### Stop the queue:
```elixir
{:ok, opq} = OPQ.init()
OPQ.enqueue(opq, fn -> IO.inspect("hello") end)
OPQ.stop(opq)
OPQ.enqueue(opq, fn -> IO.inspect("world") end) # => (EXIT) no process...
```
### Pause and resume the queue:
```elixir
{:ok, opq} = OPQ.init()
OPQ.enqueue(opq, fn -> IO.inspect("hello") end) # => "hello"
OPQ.pause(opq)
OPQ.info(opq) # => {:paused, {[], []}, 10}
OPQ.enqueue(opq, fn -> IO.inspect("world") end)
OPQ.resume(opq) # => "world"
OPQ.info(opq) # => {:normal, {[], []}, 10}
```
## Configurations
| Option | Type | Default Value | Description |
|--------------|-------------|----------------|-------------|
| `:name` | atom/module | pid | The name of the queue.
| `:worker` | module | `OPQ.Worker` | The worker that processes each item from the queue.
| `:workers` | integer | `10` | Maximum number of workers.
| `:interval` | integer | `0` | Rate limit control - number of milliseconds before asking for more items to process, defaults to `0` which is effectively no rate limit.
| `:timeout` | integer | `5000` | Number of milliseconds allowed to perform the work, it should always be set to higher than `:interval`.
## Changelog
Please see [CHANGELOG.md](CHANGELOG.md).
## License
Licensed under [MIT](http://fredwu.mit-license.org/).