{"id":13509008,"url":"https://github.com/scripbox/flume","last_synced_at":"2026-01-26T21:48:50.810Z","repository":{"id":37735464,"uuid":"218271813","full_name":"scripbox/flume","owner":"scripbox","description":"A blazing fast job processing system backed by GenStage \u0026 Redis.","archived":false,"fork":false,"pushed_at":"2024-05-07T05:03:53.000Z","size":385,"stargazers_count":78,"open_issues_count":9,"forks_count":7,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-30T13:34:50.657Z","etag":null,"topics":["background-jobs","batch-processing","concurrent","elixir-lang","genstage","rate-limiting","redis","scheduled-jobs"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scripbox.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-29T11:32:16.000Z","updated_at":"2024-08-08T01:11:00.000Z","dependencies_parsed_at":"2024-06-21T16:40:23.526Z","dependency_job_id":"282d53a4-b9a0-4fa7-8e8c-53c8b093e2c9","html_url":"https://github.com/scripbox/flume","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/scripbox/flume","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scripbox%2Fflume","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scripbox%2Fflume/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scripbox%2Fflume/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scripbox%2Fflume/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scripbox","download_url":"https://codeload.github.com/scripbox/flume/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scripbox%2Fflume/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28789048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T21:13:08.818Z","status":"ssl_error","status_checked_at":"2026-01-26T21:13:08.448Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","batch-processing","concurrent","elixir-lang","genstage","rate-limiting","redis","scheduled-jobs"],"created_at":"2024-08-01T02:01:01.701Z","updated_at":"2026-01-26T21:48:50.793Z","avatar_url":"https://github.com/scripbox.png","language":"Elixir","readme":"# Flume\n\n![Test](https://github.com/scripbox/flume/workflows/CI/badge.svg?branch=master\u0026event=push)\n\nFlume is a job processing system backed by [GenStage](https://github.com/elixir-lang/gen_stage) \u0026 [Redis](https://redis.io/)\n\n## Table of Contents\n\n- [Features](#features)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Pipelines](#pipelines)\n  - [Enqueuing Jobs](#enqueuing-jobs)\n  - [Creating Workers](#creating-workers)\n  - [Scheduled Jobs](#scheduled-jobs)\n  - [Rate Limiting](#rate-limiting)\n  - [Batch Processing](#batch-processing)\n  - [Pipeline Control](#pipeline-control)\n  - [Instrumentation](#instrumentation)\n- [Writing Tests](#writing-tests)\n- [Roadmap](#roadmap)\n- [References](#references)\n- [Contributing](#contributing)\n\n## Features\n\n- **Durability** - Jobs are backed up before processing. Incase of crashes, these\n  jobs are restored.\n- **Back-pressure** - Uses [gen_stage](https://github.com/elixir-lang/gen_stage) to support this.\n- **Scheduled Jobs** - Jobs can be scheduled to run at any point in future.\n- **Rate Limiting** - Uses redis to maintain rate-limit on pipelines.\n- **Batch Processing** - Jobs are grouped based on size.\n- **Logging** - Provides a behaviour `Flume.Logger` to define your own logger module.\n- **Pipeline Control** - Queues can be pause/resume at runtime.\n- **Instrumentation** - Metrics like worker duration and latency to fetch jobs from redis are emitted via [telemetry](https://github.com/beam-telemetry/telemetry).\n- **Exponential Back-off** - On failure, jobs are retried with exponential back-off. Minimum and maximum can be set via configuration.\n\n## Requirements\n\n- Elixir 1.6.6+\n- Erlang/OTP 21.1+\n- Redis 4.0+\n\n## Installation\n\nAdd Flume to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:flume, github: \"scripbox/flume\"}\n  ]\nend\n```\n\nThen run `mix deps.get` to install Flume and its dependencies.\n\n## Usage\n\nAdd Flume supervisor to your application's supervision tree:\n\n```elixir\ndefmodule MyApplication.Application do\n  use Application\n\n  import Supervisor.Spec\n\n  def start(_type, _args) do\n    children = [\n      # Start Flume supervisor\n      supervisor(Flume, [])\n    ]\n\n    opts = [strategy: :one_for_one, name: MyApplication.Supervisor]\n    Supervisor.start_link(children, opts)\n  end\nend\n```\n\nAdd `config/flume.exs`:\n\n```elixir\nconfig :flume,\n  name: Flume,\n  # Redis host\n  host: \"127.0.0.1\",\n  # Redis port\n  port: \"6379\",\n  # Redis keys namespace\n  namespace: \"my-app\",\n  # Redis database\n  database: 0,\n  # Redis pool size\n  redis_pool_size: 10,\n  # Redis connection timeout in ms (Default 5000 ms)\n  redis_timeout: 10_000,\n  # Retry backoff intial in ms (Default 500 ms)\n  backoff_initial: 30_000,\n  # Retry backoff maximum in ms (Default 10_000 ms)\n  backoff_max: 36_00_000,\n  # Maximum number of retries (Default 5)\n  max_retries: 15,\n  # Scheduled jobs poll interval in ms (Default 10_000 ms)\n  scheduler_poll_interval: 10_000,\n  # Time to move jobs from processing queue to retry queue in seconds (Default 600 sec)\n  visibility_timeout: 600,\n  # ttl of the acquired lock to fetch jobs for bulk pipelines in ms (Default 30_000 ms)\n  dequeue_lock_ttl: 30_000,\n  # process timeout to fetch jobs for bulk pipelines in ms (Default 10_000 ms)\n  dequeue_process_timeout: 10_000,\n  # time to poll the queue again if it was locked by another process in ms (Default 500 ms)\n  dequeue_lock_poll_interval: 500\n```\n\nImport flume config in `config/config.exs` as given below:\n\n```elixir\n...\nimport_config \"#{Mix.env()}.exs\"\n+import_config \"flume.exs\"\n```\n\n### Pipelines\n\nEach pipeline is a GenStage pipeline having these parameters -\n\n* `name` - Name of the pipeline\n* `queue` - Name of the Redis queue to pull jobs from\n* `max_demand` - Maximum number of jobs to pull from the queue\n\n**Configuration**\n\n```elixir\nconfig :flume,\n  pipelines: [\n    %{name: \"default_pipeline\", queue: \"default\", max_demand: 1000},\n  ]\n```\n\nFlume supervisor will start these processes:\n\n```asciidoc\n                  [Flume.Supervisor]   \u003c- (Supervisor)\n                         |\n                         |\n                         |\n              [default_pipeline_producer]   \u003c- (Producer)\n                         |\n                         |\n                         |\n          [default_pipeline_producer_consumer]   \u003c- (ProducerConsumer)\n                         |\n                         |\n                         |\n         [default_pipeline_consumer_supervisor]   \u003c- (ConsumerSupervisor)\n                        / \\\n                       /   \\\n                      /     \\\n             [worker_1]     [worker_2]   \u003c- (Worker Processes)\n```\n\n### Enqueuing Jobs\n\nEnqueuing jobs into flume requires these things -\n\n* Specify a `queue-name` (like `priority`)\n* Specify the worker module (`MyApp.FancyWorker`)\n* Specify the worker module's function name (default `:perform`)\n* Specify the arguments as per the worker module's function arity\n\n**With default function**\n\n```elixir\nFlume.enqueue(:queue_name, MyApp.FancyWorker, [arg_1, arg_2])\n```\n\n**With custom function**\n\n```elixir\nFlume.enqueue(:queue_name, MyApp.FancyWorker, :myfunc, [arg_1, arg_2])\n```\n\n### Creating Workers\n\nWorker modules are responsible for processing a job.\nA worker module should define the `function-name` with the exact arity used while queuing the job.\n\n```elixir\ndefmodule MyApp.FancyWorker do\n  def perform(arg_1, arg_2) do\n    # your job processing logic\n  end\nend\n```\n\n### Scheduled Jobs\n\n**With default function**\n\n```elixir\n# 10 seconds\nschedule_time = 10_000\n\nFlume.enqueue_in(:queue_name, schedule_time, MyApp.FancyWorker, [arg_1, arg_2])\n```\n\n**With custom function**\n\n```elixir\n# 10 seconds\nschedule_time = 10_000\n\nFlume.enqueue_in(:queue_name, schedule_time, MyApp.FancyWorker, :myfunc, [arg_1, arg_2])\n```\n\n### Rate Limiting\n\nFlume supports rate-limiting for each configured pipeline.\n\nRate-Limiting has two key parameters -\n\n- `rate_limit_scale` - Time scale in `milliseconds` for the pipeline\n- `rate_limit_count` - Total number of jobs to be processed within the time scale\n- `rate_limit_key`(optional) - Using this option, rate limit can be set across pipelines.\n\n       **Note**: When this option is not set, rate limit will be maintained for a pipeline.\n\n```elixir\nrate_limit_count = 1000\nrate_limit_scale = 6 * 1000\n\nconfig :flume,\n  pipelines: [\n    # This pipeline will process 1000 jobs every 6 seconds\n    %{\n      name: \"promotional_email_pipeline\",\n      queue: \"promotional_email\",\n      rate_limit_count: rate_limit_count,\n      rate_limit_scale: rate_limit_scale,\n      rate_limit_key: \"email\"\n    },\n    %{\n      name: \"transactional_email_pipeline\",\n      queue: \"transactional_email\",\n      rate_limit_count: rate_limit_count,\n      rate_limit_scale: rate_limit_scale,\n      rate_limit_key: \"email\"\n    }\n  ]\n\nOR\n\nconfig :flume\n  pipelines: [\n    %{\n      name: \"webhooks_pipeline\",\n      queue: \"webhooks\",\n      rate_limit_count: 1000,\n      rate_limit_scale: 5000\n    }\n  ]\n```\n\nFlume will process the configured number of jobs (`rate_limit_count`) for each rate-limited pipeline,\neven if we are running multiple instances of our application.\n\n### Batch Processing\n\nFlume supports batch-processing for each configured pipeline.\nIt groups individual jobs by the configured `batch_size` option and\neach worker process will receive a group of jobs.\n\n```elixir\nconfig :flume,\n  pipelines: [\n    # This pipeline will pull (100 * 10) jobs from the queue\n    # and group them in batches of 10.\n    %{\n      name: \"batch_pipeline\",\n      queue: \"batch-queue\",\n      max_demand: 100,\n      batch_size: 10\n    }\n  ]\n```\n\n```elixir\ndefmodule MyApp.BatchWorker do\n  def perform(args) do\n    # args will be a list of arguments\n    # E.g - [[job_1_args], [job_2_args], ...]\n    # your job processing logic\n  end\nend\n```\n\n### Pipeline Control\n\nFlume has support to pause/resume each pipeline.\nOnce a pipeline is paused, the producer process will stop pulling jobs from the queue.\nIt will process the jobs which are already pulled from the queue.\n\nRefer to \"Options\" section for supported options and default values.\n\n**Pause all pipelines**\n```elixir\n# Pause all pipelines permanently (in Redis) and asynchronously\nFlume.pause_all(temporary: false, async: true)\n```\n\n**Pause a pipeline**\n\n```elixir\n# Pause a pipeline temporarily (in current node) and asynchronously\nFlume.pause(:default_pipeline, temporary: true, async: true)\n```\n\n**Resume all pipelines**\n```elixir\n# Resume all pipelines temporarily (in current node) and synchronously with infinite timeout\nFlume.resume_all(temporary: true, async: false, timeout: :infinity)\n```\n\n**Resume a pipeline**\n\n```elixir\n# Resume a pipeline permanently (in Redis) and synchronously with a 10000 milli-second timeout\nFlume.resume(:default_pipeline, temporary: false, async: false, timeout: 10000)\n```\n\n#### Options\nThe following options can be used to pause/resume a pipeline\n  * `:async` - (boolean) Defaults to `false`.\n      * `true` - The caller will not wait for the operation to complete.\n      * `false` - The caller will wait for the operation to complete, this can lead to timeout if the operation takes too long to succeed. See https://hexdocs.pm/elixir/GenServer.html#call/3 for more details.\n  * `:temporary` - (boolean) Defaults to `true`.\n      * `true` - The pause/resume operation will be applied only on the current node.\n      * `false` - Will update the value in persistent-store (Redis) and will apply the operation on all nodes.\n  * `:timeout` - (timeout) Defaults to `5000`. Timeout(in milliseconds) for synchronous pause/resume calls. See https://hexdocs.pm/elixir/GenServer.html#call/3-timeouts for more details.\n\n### Instrumentation\n\nWe use [telemetry](https://github.com/beam-telemetry/telemetry) to emit metrics.\nFollowing metrics are emitted:\n\n- duration of a job/worker\n- count, latency and payload_size of dequeued jobs\n\n## Writing Tests\n\n**To enable mock in the test environment**\n\n**config/test.exs**\n\n```elixir\nconfig :flume, mock: true\n```\n\n**To mock individual test**\n\n```elixir\nimport Flume.Mock\n...\ndescribe \"enqueue/4\" do\n  test \"mock works\" do\n    with_flume_mock do\n      Flume.enqueue(:test, List, :last, [[1]])\n\n      assert_receive %{\n        queue: :test,\n        worker: List,\n        function_name: :last,\n        args: [[1]]\n      }\n    end\n  end\nend\n```\n\n**To enable mock for all tests in a module**\n\n```elixir\ndefmodule ListTest do\n  use ExUnit.Case, async: true\n  use Flume.Mock\n\n  describe \"enqueue/4\" do\n    test \"mock works\" do\n      Flume.enqueue(:test, List, :last, [[1]])\n\n      assert_receive %{\n        queue: :test,\n        worker: List,\n        function_name: :last,\n        args: [[1]]\n      }\n    end\n  end\nend\n```\n\n## Roadmap\n\n* Support multiple queue backends (right now only Redis is supported)\n\n## References\n\n* Background Processing in Elixir with GenStage (https://medium.com/@scripbox_tech/background-processing-in-elixir-with-genstage-efb6cb8ca94a)\n\n## Contributing\n\n* Check formatting (`mix format --check-formatted`)\n* Run all tests (`mix test`)\n","funding_links":[],"categories":["Queue","Elixir"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscripbox%2Fflume","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscripbox%2Fflume","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscripbox%2Fflume/lists"}