{"id":32168159,"url":"https://github.com/dnlserrano/firefighter","last_synced_at":"2026-03-01T04:37:29.894Z","repository":{"id":43070126,"uuid":"315096109","full_name":"dnlserrano/firefighter","owner":"dnlserrano","description":"Amazon Kinesis Data Firehose configurable queue supporting arbitrary adapters","archived":false,"fork":false,"pushed_at":"2022-03-20T19:08:35.000Z","size":82,"stargazers_count":5,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-12-26T11:56:21.645Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dnlserrano.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-22T17:40:02.000Z","updated_at":"2024-12-10T09:13:21.000Z","dependencies_parsed_at":"2022-08-24T08:01:33.590Z","dependency_job_id":null,"html_url":"https://github.com/dnlserrano/firefighter","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/dnlserrano/firefighter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnlserrano%2Ffirefighter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnlserrano%2Ffirefighter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnlserrano%2Ffirefighter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnlserrano%2Ffirefighter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dnlserrano","download_url":"https://codeload.github.com/dnlserrano/firefighter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnlserrano%2Ffirefighter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29701970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T23:35:04.139Z","status":"ssl_error","status_checked_at":"2026-02-21T23:35:03.832Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-21T15:47:46.882Z","updated_at":"2026-03-01T04:37:29.867Z","avatar_url":"https://github.com/dnlserrano.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 👨‍🚒 firefighter\n\n![Build Status](https://github.com/dnlserrano/firefighter/actions/workflows/main.yml/badge.svg)\n[![Hex pm](http://img.shields.io/hexpm/v/firefighter.svg?style=flat)](https://hex.pm/packages/firefighter)\n\nAmazon Kinesis Data Firehose configurable queue supporting arbitrary adapters.\n\n## Motivation\n\nWhen you want to integrate with Amazon Kinesis Data Firehose, you will most likely want to batch the requests you do in order to not hit Amazon limits. Hence, you'd ideally have an abstraction that allows you to push data, automatically buffering it and pumping data to any given stream from time to time. This is what `firefighter` does.\n\nYou can configure different options (e.g., [`:batch_size`, `:interval`, `:delimiter`, `:flush_grace_period`](https://github.com/dnlserrano/firefighter/blob/main/lib/firefighter.ex#L9-L11)) which should be tuned to your specific usage. Defaults are as follows:\n\n* `:batch_size`: `40`\n* `:interval`: `2_000` (milliseconds, i.e., 2 seconds)\n* `:delimiter`: `\"\"` (i.e., the empty string)\n* `:flush_grace_period`: `30_000` (milliseconds, i.e., 30 seconds)\n\n## Installation\n\nThe package can be installed by adding `firefighter` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:firefighter, \"~\u003e 0.1.2\"} # check most recent version in this project's mix.exs\n  ]\nend\n```\n\n## Usage\n\nYou should configure `firefighter` in, e.g., `config/config.exs` and select your specific adapter.\n\n```elixir\n# config/config.exs\n\nconfig :firefighter, :adapter, Firefighter.Adapters.ExAws\n```\n\nAdapters provide implementations for the underlying libraries you may use to pump data to Firehose. By default, we provide a logger adapter that just logs data. We also provide an adapter for [`ex_aws`](https://github.com/ex-aws/ex_aws) out of the box. It should be easy enough to expand on this to provide more adapters (e.g., a new adapter for [`aws-elixir`](https://github.com/aws-beam/aws-elixir)).\n\n### Example\n\n```elixir\n# config/config.exs\n\nconfig :firefighter, :adapter, Firefighter.Adapters.Logger\n```\n\n```elixir\n# config/prod.exs\n\nconfig :firefighter, :adapter, Firefighter.Adapters.ExAws\n```\n\n```elixir\n# lib/example/application.ex\n\ndefmodule Example.Application do\n  use Application\n\n  @impl true\n  def start(_type, _args) do\n    children = [\n      {Firefighter, [name: :my_firefighter, delivery_stream_name: \"s3-firehose\", batch_size: 10]}\n    ]\n\n    opts = [strategy: :one_for_one, name: Example.Supervisor]\n    Supervisor.start_link(children, opts)\n  end\nend\n```\n\n```elixir\n# lib/example.ex\n\ndefmodule Example do\n  def run do\n    pid = Process.whereis(:my_firefighter)\n    for i \u003c- 0..30, do: Firefighter.push(pid, \"sample-data-#{i}\")\n    pid\n  end\nend\n```\n\n#### Alternative Usage\n\nAs an alternative usage method, you might also decide to go with the `Firefighter.Execution` abstraction, provided out of the box:\n\n```elixir\nExecution.start(%{user_id: \"user-1\", post_id: \"post-123\"})\n|\u003e Execution.record(%{age: 29})\n|\u003e Execution.push(:my_firefighter)\n```\n\n\nYou can also just hose it directly, without ceremony, if you're pumping a simple record that needs no composition:\n\n```elixir\nExecution.hose(:my_firefighter, %{user_id: \"user-1\", post_id: \"post-123\", age: 29})\n```\n\nYou can also use pids instead of the process name (`:my_firefighter` in the example above).\n\nFor a detailed example project using `firefighter`, check the [`example/`](./example) directory.\n\n## License\n\n    Copyright © 2020-present Daniel Serrano \u003cdanieljdserrano at protonmail\u003e\n\n    This work is free. You can redistribute it and/or modify it under the\n    terms of the MIT License. See the LICENSE file for more details.\n\nMade in Portugal :portugal: by [dnlserrano](https://dnlserrano.dev)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdnlserrano%2Ffirefighter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdnlserrano%2Ffirefighter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdnlserrano%2Ffirefighter/lists"}