{"id":13508984,"url":"https://github.com/dashbitco/broadway","last_synced_at":"2025-05-14T22:05:17.801Z","repository":{"id":37709361,"uuid":"156210099","full_name":"dashbitco/broadway","owner":"dashbitco","description":"Concurrent and multi-stage data ingestion and data processing with Elixir","archived":false,"fork":false,"pushed_at":"2025-04-30T20:17:24.000Z","size":735,"stargazers_count":2513,"open_issues_count":6,"forks_count":166,"subscribers_count":44,"default_branch":"main","last_synced_at":"2025-05-05T19:14:14.107Z","etag":null,"topics":["broadway","concurrent","data-ingestion","data-processing","elixir","genstage"],"latest_commit_sha":null,"homepage":"https://elixir-broadway.org","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dashbitco.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-11-05T11:52:33.000Z","updated_at":"2025-04-30T20:17:29.000Z","dependencies_parsed_at":"2024-11-09T20:17:58.713Z","dependency_job_id":"4d7ff5e2-3ce0-4a74-8953-bb82199e8448","html_url":"https://github.com/dashbitco/broadway","commit_stats":{"total_commits":393,"total_committers":83,"mean_commits":4.734939759036145,"dds":0.7964376590330788,"last_synced_commit":"9eaf2140b6c1f36b25f990a7c74052bece26aadc"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashbitco%2Fbroadway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashbitco%2Fbroadway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashbitco%2Fbroadway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dashbitco%2Fbroadway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dashbitco","download_url":"https://codeload.github.com/dashbitco/broadway/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252769398,"owners_count":21801376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broadway","concurrent","data-ingestion","data-processing","elixir","genstage"],"created_at":"2024-08-01T02:01:01.375Z","updated_at":"2025-05-07T08:31:33.814Z","avatar_url":"https://github.com/dashbitco.png","language":"Elixir","readme":"# Broadway\n\n[![CI](https://github.com/dashbitco/broadway/actions/workflows/ci.yml/badge.svg)](https://github.com/dashbitco/broadway/actions/workflows/ci.yml)\n\nBuild concurrent and multi-stage data ingestion and data processing pipelines with Elixir. Broadway allows developers to consume data efficiently from different sources, known as producers, such as Amazon SQS, Apache Kafka, Google Cloud PubSub, RabbitMQ, and others. Broadway pipelines are long-lived, concurrent, and robust, thanks to the Erlang VM and its actors.\n\nBroadway takes its name from the famous [Broadway street](https://en.wikipedia.org/wiki/Broadway_theatre) in New York City, renowned for its stages, actors, and producers.\n\nTo learn more and get started, check out [our official website](https://elixir-broadway.org) and [our guides and docs](https://hexdocs.pm/broadway).\n\n![Broadway Logo](https://user-images.githubusercontent.com/9582/117824616-ed298500-b26e-11eb-8ded-0fb7e608bf70.png)\n\n## Built-in features\n\nBroadway takes the burden of defining concurrent GenStage topologies and provides a simple configuration API that automatically defines concurrent producers, concurrent processing, batch handling, and more, leading to both time and cost efficient ingestion and processing of data. It features:\n\n  * Back-pressure\n  * Automatic acknowledgements at the end of the pipeline\n  * Batching\n  * Fault tolerance\n  * Graceful shutdown\n  * Built-in testing\n  * Custom failure handling\n  * Ordering and partitioning\n  * Rate-limiting\n  * Metrics\n\n### Producers\n\nThere are several producers that you can use to integrate with existing services and technologies. [See the docs for detailed how-tos and supported producers](https://hexdocs.pm/broadway/introduction.html#official-producers).\n\n## Installation\n\nAdd `:broadway` to the list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:broadway, \"~\u003e 1.0\"}\n  ]\nend\n```\n\n## A quick example: SQS integration\n\nAssuming you have added [`broadway_sqs`](https://github.com/dashbitco/broadway_sqs) as a dependency and configured your SQS credentials accordingly, you can consume Amazon SQS events in only 20 LOCs:\n\n```elixir\ndefmodule MyBroadway do\n  use Broadway\n\n  alias Broadway.Message\n\n  def start_link(_opts) do\n    Broadway.start_link(__MODULE__,\n      name: __MODULE__,\n      producer: [\n        module: {BroadwaySQS.Producer, queue_url: \"https://us-east-2.queue.amazonaws.com/100000000001/my_queue\"}\n      ],\n      processors: [\n        default: [concurrency: 50]\n      ],\n      batchers: [\n        s3: [concurrency: 5, batch_size: 10, batch_timeout: 1000]\n      ]\n    )\n  end\n\n  def handle_message(_processor_name, message, _context) do\n    message\n    |\u003e Message.update_data(\u0026process_data/1)\n    |\u003e Message.put_batcher(:s3)\n  end\n\n  def handle_batch(:s3, messages, _batch_info, _context) do\n    # Send batch of messages to S3\n  end\n\n  defp process_data(data) do\n    # Do some calculations, generate a JSON representation, process images.\n  end\nend\n```\n\nOnce your Broadway module is defined, you just need to add it as a child of your application supervision tree as `{MyBroadway, []}`.\n\n## Comparison to Flow\n\nYou may also be interested in [Flow by Dashbit](https://github.com/dashbitco/flow). Both Broadway and Flow are built on top of GenStage. Flow is a more general abstraction than Broadway that focuses on data as a whole, providing features like aggregation, joins, windows, etc. Broadway focuses on events and on operational features, such as metrics, automatic acknowledgements, failure handling, and so on. Broadway is recommended for continuous, long-running pipelines. Flow works with short- and long-lived data processing.\n\n## License\n\nCopyright 2019 Plataformatec\\\nCopyright 2020 Dashbit\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","funding_links":[],"categories":["Queue","Elixir"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdashbitco%2Fbroadway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdashbitco%2Fbroadway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdashbitco%2Fbroadway/lists"}