{"id":15056427,"url":"https://github.com/fatum/firehose","last_synced_at":"2025-04-10T04:10:51.944Z","repository":{"id":33162024,"uuid":"140674801","full_name":"fatum/firehose","owner":"fatum","description":"Events aggregator for Amazon Firehose service written in elixir","archived":false,"fork":false,"pushed_at":"2021-11-27T19:00:59.000Z","size":18,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"feature/v2","last_synced_at":"2025-03-24T05:26:31.025Z","etag":null,"topics":["aws","elixir","erlang","firehose","kinesis"],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fatum.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-12T07:04:57.000Z","updated_at":"2024-02-09T13:47:13.000Z","dependencies_parsed_at":"2022-08-07T20:01:34.109Z","dependency_job_id":null,"html_url":"https://github.com/fatum/firehose","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fatum%2Ffirehose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fatum%2Ffirehose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fatum%2Ffirehose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fatum%2Ffirehose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fatum","download_url":"https://codeload.github.com/fatum/firehose/tar.gz/refs/heads/feature/v2","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154987,"owners_count":21056543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","elixir","erlang","firehose","kinesis"],"created_at":"2024-09-24T21:51:17.740Z","updated_at":"2025-04-10T04:10:51.923Z","avatar_url":"https://github.com/fatum.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Firehose\n\nThis library helps to manage data sending to AWS Firehose dealing with its limitations:\n\n\u003e The maximum size of a record sent to Kinesis Data Firehose, before base64-encoding, is 1,000 KiB.\n\u003e The PutRecordBatch operation can take up to 500 records per call or 4 MiB per call, whichever is smaller...the size of each record rounded up to the nearest 5KB\n\n## Installation\n\nIf [available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `firehose` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:firehose, \"~\u003e 1.0.0\"}\n  ]\nend\n```\n\n## Usage\n\nCreate module which will handle callbacks:\n\n```elixir\ndefmodule Event.Handler do\n  def subscribe(:failed, %Firehose.Batch{} = batch, aws_response) do\n    # This callback is triggered when retry count exhausted and batch not delivered.\n    # You can add specific logic for this case (write data to queue or database etc) or\n    # send metrics to monitoring service\n  end\n\n  def subscribe(:completed, %Firehose.Batch{stream: stream, size: batch_size, records: records}) do\n    # This callback is triggered after each successfull delivery to Firehose.\n    # You can use this callback to send metrics to monitoring service\n  end\nend\n```\n\nAdd appropriate configurations for `ex_aws` and `firehose`:\n\n```elixir\nconfig :ex_aws,\n  access_key_id: [{:system, \"AWS_ACCESS_KEY_ID\"}, :instance_role],\n  secret_access_key: [{:system, \"AWS_SECRET_ACCESS_KEY\"}, :instance_role]\n\nconfig :firehose, Firehose.Manager,\n  handlers: [Event.Handler],\n  flush_interval: 1_000, # Flush batches every seconds or on batch size limit\n  retries: 5,            # Retry count for failed requests\n  serializer: Poison,    # Event serializer: should respond to `encode` and `decode` methods\n  delimiter: \"\\n\",       # Add delimiter after each event or add nothing if `false` or `nil` set\n```\n\nAdd `Firehose.Supervisor` to your supervisor tree and thats it. You can emit events to AWS Firehose like this:\n\n```elixir\nFirehose.emit(\"logs\", %{\"name\" =\u003e \"pageview\", \"ua\" =\u003e \"...\"})\n```\n\n\nDocumentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)\nand published on [HexDocs](https://hexdocs.pm). Once published, the docs can\nbe found at [https://hexdocs.pm/firehose](https://hexdocs.pm/firehose).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffatum%2Ffirehose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffatum%2Ffirehose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffatum%2Ffirehose/lists"}