{"id":13509026,"url":"https://github.com/fredwu/opq","last_synced_at":"2025-04-12T20:46:23.606Z","repository":{"id":48413268,"uuid":"99812433","full_name":"fredwu/opq","owner":"fredwu","description":"Elixir queue! A simple, in-memory queue with worker pooling and rate limiting in Elixir.","archived":false,"fork":false,"pushed_at":"2023-09-29T13:49:51.000Z","size":95,"stargazers_count":268,"open_issues_count":0,"forks_count":8,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-04T00:09:55.585Z","etag":null,"topics":["back-pressure","demand-control","elixir","fifo","gen-stage","genstage","in-memory","opq","pool","pooling","queue","rate-limit","worker-pool"],"latest_commit_sha":null,"homepage":"https://hex.pm/packages/opq","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fredwu.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-08-09T13:38:18.000Z","updated_at":"2025-02-28T16:12:50.000Z","dependencies_parsed_at":"2023-01-22T10:00:52.794Z","dependency_job_id":"5f4e0857-c1fb-4d4c-95f5-3b686fcbc1ff","html_url":"https://github.com/fredwu/opq","commit_stats":{"total_commits":72,"total_committers":7,"mean_commits":"10.285714285714286","dds":"0.11111111111111116","last_synced_commit":"0574def684c2b271749057a19fae07df192e2f69"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredwu%2Fopq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredwu%2Fopq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredwu%2Fopq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fredwu%2Fopq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fredwu","download_url":"https://codeload.github.com/fredwu/opq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631710,"owners_count":21136559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["back-pressure","demand-control","elixir","fifo","gen-stage","genstage","in-memory","opq","pool","pooling","queue","rate-limit","worker-pool"],"created_at":"2024-08-01T02:01:01.965Z","updated_at":"2025-04-12T20:46:23.551Z","avatar_url":"https://github.com/fredwu.png","language":"Elixir","funding_links":[],"categories":["Queue"],"sub_categories":[],"readme":"# OPQ: One Pooled Queue\n\n[![Build Status](https://github.com/fredwu/opq/actions/workflows/ci.yml/badge.svg)](https://github.com/fredwu/opq/actions)\n[![CodeBeat](https://codebeat.co/badges/76916047-5b66-466d-91d3-7131a269899a)](https://codebeat.co/projects/github-com-fredwu-opq-master)\n[![Coverage](https://img.shields.io/coveralls/fredwu/opq.svg)](https://coveralls.io/github/fredwu/opq?branch=master)\n[![Hex Version](https://img.shields.io/hexpm/v/opq.svg)](https://hex.pm/packages/opq)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/opq/)\n[![Total Download](https://img.shields.io/hexpm/dt/opq.svg)](https://hex.pm/packages/opq)\n[![License](https://img.shields.io/hexpm/l/opq.svg)](https://github.com/fredwu/opq/blob/master/LICENSE.md)\n[![Last Updated](https://img.shields.io/github/last-commit/fredwu/opq.svg)](https://github.com/fredwu/opq/commits/master)\n\n## Elixir Queue!\n\nA simple, in-memory queue with worker pooling and rate limiting in Elixir. OPQ leverages Erlang's [queue](http://erlang.org/doc/man/queue.html) module and Elixir's [GenStage](https://github.com/elixir-lang/gen_stage).\n\nOriginally built to support [Crawler](https://github.com/fredwu/crawler).\n\n## Features\n\n- A fast, in-memory FIFO queue.\n- Worker pool.\n- Rate limit.\n- Timeouts.\n- Pause / resume / stop the queue.\n\nSee [Hex documentation](https://hexdocs.pm/opq/).\n\n## Installation\n\n```Elixir\ndef deps do\n  [{:opq, \"~\u003e 4.0\"}]\nend\n```\n\n## Usage\n\n### A simple example:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -\u003e IO.inspect(\"hello\") end)\nOPQ.enqueue(opq, fn -\u003e IO.inspect(\"world\") end)\n```\n\n### Specify module, function and arguments:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, IO, :inspect, [\"hello\"])\nOPQ.enqueue(opq, IO, :inspect, [\"world\"])\n```\n\n### Specify a custom name for the queue:\n\n```elixir\nOPQ.init(name: :items)\n\nOPQ.enqueue(:items, fn -\u003e IO.inspect(\"hello\") end)\nOPQ.enqueue(:items, fn -\u003e IO.inspect(\"world\") end)\n```\n\n### Start as part of a supervision tree:\n\nNote, when starting as part of a supervision tree, the `:name` option must be provided.\n\n```elixir\nchildren = [\n  {OPQ, name: :items}\n]\n```\n\n### Specify a custom worker to process items in the queue:\n\n```elixir\ndefmodule CustomWorker do\n  def start_link(item) do\n    Task.start_link(fn -\u003e\n      Agent.update(:bucket, \u0026[item | \u00261])\n    end)\n  end\nend\n\nAgent.start_link(fn -\u003e [] end, name: :bucket)\n\n{:ok, opq} = OPQ.init(worker: CustomWorker)\n\nOPQ.enqueue(opq, \"hello\")\nOPQ.enqueue(opq, \"world\")\n\nAgent.get(:bucket, \u0026 \u00261) # =\u003e [\"world\", \"hello\"]\n```\n\n### Rate limit:\n\n```elixir\n{:ok, opq} = OPQ.init(workers: 1, interval: 1000)\n\nTask.async(fn -\u003e\n  OPQ.enqueue(opq, fn -\u003e IO.inspect(\"hello\") end)\n  OPQ.enqueue(opq, fn -\u003e IO.inspect(\"world\") end)\nend)\n```\n\nIf no interval is supplied, the ratelimiter will be bypassed.\n\n### Check the queue and number of available workers:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -\u003e Process.sleep(1000) end)\n\n{status, queue, available_workers} = OPQ.info(opq) # =\u003e {:normal, #OPQ.Queue\u003c[]\u003e, 9}\n\nProcess.sleep(1200)\n\n{status, queue, available_workers} = OPQ.info(opq) # =\u003e {:normal, #OPQ.Queue\u003c[]\u003e, 10}\n```\n\nIf you just need to get the queue itself:\n\n```elixir\nOPQ.queue(opq) # =\u003e #OPQ.Queue\u003c[]\u003e\n```\n\n### Queue\n\nOPQ implements `Enumerable`, so you can perform enumerable functions on the queue:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nqueue = OPQ.queue(opq)\n\nEnum.count(queue) # =\u003e 0\nEnum.empty?(queue) # =\u003e true\n```\n\n### Stop the queue:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -\u003e IO.inspect(\"hello\") end)\nOPQ.stop(opq)\nOPQ.enqueue(opq, fn -\u003e IO.inspect(\"world\") end) # =\u003e (EXIT) no process...\n```\n\n### Pause and resume the queue:\n\n```elixir\n{:ok, opq} = OPQ.init()\n\nOPQ.enqueue(opq, fn -\u003e IO.inspect(\"hello\") end) # =\u003e \"hello\"\nOPQ.pause(opq)\nOPQ.info(opq) # =\u003e {:paused, {[], []}, 10}\nOPQ.enqueue(opq, fn -\u003e IO.inspect(\"world\") end)\nOPQ.resume(opq) # =\u003e \"world\"\nOPQ.info(opq) # =\u003e {:normal, {[], []}, 10}\n```\n\n## Configurations\n\n| Option       | Type        | Default Value  | Description |\n|--------------|-------------|----------------|-------------|\n| `:name`      | atom/module | pid            | The name of the queue.\n| `:worker`    | module      | `OPQ.Worker`   | The worker that processes each item from the queue.\n| `:workers`   | integer     | `10`           | Maximum number of workers.\n| `:interval`  | integer     | `0`            | Rate limit control - number of milliseconds before asking for more items to process, defaults to `0` which is effectively no rate limit.\n| `:timeout`   | integer     | `5000`         | Number of milliseconds allowed to perform the work, it should always be set to higher than `:interval`.\n\n## Changelog\n\nPlease see [CHANGELOG.md](CHANGELOG.md).\n\n## License\n\nLicensed under [MIT](http://fredwu.mit-license.org/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffredwu%2Fopq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffredwu%2Fopq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffredwu%2Fopq/lists"}