{"id":16726854,"url":"https://github.com/tpitale/convoy","last_synced_at":"2025-06-21T06:05:14.234Z","repository":{"id":66660126,"uuid":"183075081","full_name":"tpitale/convoy","owner":"tpitale","description":"Elixir library to handle common Kinesis behaviors","archived":false,"fork":false,"pushed_at":"2021-05-11T18:42:50.000Z","size":38,"stargazers_count":6,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-14T19:08:54.829Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tpitale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-23T18:36:43.000Z","updated_at":"2023-05-14T14:40:19.000Z","dependencies_parsed_at":"2023-02-21T12:16:02.441Z","dependency_job_id":null,"html_url":"https://github.com/tpitale/convoy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tpitale/convoy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpitale%2Fconvoy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpitale%2Fconvoy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpitale%2Fconvoy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpitale%2Fconvoy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tpitale","download_url":"https://codeload.github.com/tpitale/convoy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tpitale%2Fconvoy/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261073340,"owners_count":23105638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T22:54:35.596Z","updated_at":"2025-06-21T06:05:09.224Z","avatar_url":"https://github.com/tpitale.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Convoy\n\nConvoy wraps and adapts different queue/stream systems, starting with AWS Kinesis.\n\n## Installation\n\nIf [available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `convoy` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:convoy, \"~\u003e 0.1.0\"}\n  ]\nend\n```\n\n## Starting a Queue ##\n\nAny `Queue` should be supervised, so add it to your supervision tree. Or, if you\nneed to start one after startup, you can make use of `DynamicSupervisor`.\n\n```ex\n{Convoy.Queue, [stream: :your_stream_name]}\n```\n\n## Attaching a Handler ##\n\n```ex\nConvoy.Queue.attach(\n  stream_id :: binary | atom,\n  handler_id :: binary,\n  handler_function/2,\n  info :: map\n)\n```\n\nWhere `handler_function` has an arity of 2 and accepts an individual `record`\nalong with the `info` map you pass in (useful for later matching your function).\n\nAny time a call to `Convoy.Queue.get` will notify these handlers with records.\n\n## Queue Configuration ##\n\nThe only required option is `stream`. The `stream` option will be used as the\n`stream_id`, if none is provided.\n\nIf you wish to connect to the same stream multiple times, you have to pass\na `stream_id`. It is not possible to use the same `stream` as the identifier.\n\n```ex\n# Defaults\n%{\n  stream: nil,\n  stream_id: nil,\n  service: Convoy.Services.Kinesis,\n  batch_timeout: 5000,\n  poll_interval: nil,\n  iterator_type: :latest\n}\n```\n\n### Batch Timeout ###\n\nIf you wish to disable batching, or increase the frequency of batches transmitting\nto the service, you can change the `batch_timeout` option from the default of 5\nseconds. This value is configured in `ms`, so 5000 is 5 seconds.\n\nSetting this value to `0` will disable batching.\n\n### Poll Interval ###\n\nDisabled by default, setting the `poll_interval` value will configure the `Queue`\nto poll for new records at that interval. You'll want to combine this with calls\nto `attach_handler/4`. Polling will begin only after there is at least one handler\nattached for the `stream_id`. If all handlers are detached, polling will stop.\n\n### Service ###\n\nAt this time, Convoy only supports Kinesis, but you can write your own `service`\nthat fulfills the `Convoy.Service` behaviour.\n\n**TODO:** Planned support for services like SQS and Kafka.\n\n### Kinesis Configuration ###\n\nIf using Kinesis, configure the underlying `ex_aws_kinesis` library in your `dev.exs`:\n\n```\nconfig :ex_aws,\n  debug_requests: true,\n  kinesis: %{\n    host: \"localhost\",\n    port: 4567,\n    scheme: \"http://\",\n    access_key_id: [{:system, \"KINESIS_AWS_ACCESS_KEY_ID\"}, :instance_role],\n    secret_access_key: [{:system, \"KINESIS_SECRET_ACCESS_KEY\"}, :instance_role]\n  }\n```\n\n### Local Development ###\n\nWe recommend running Kinesalite locally for development.\n\n\n\n\n\n```\n{:ok, response} = ExAws.Kinesis.describe_stream(\"device_service_dev\") |\u003e ExAws.request\nshard_id = response[\"StreamDescription\"][\"Shards\"] |\u003e hd() |\u003e Map.get(\"ShardId\")\n\n{:ok, iterator} = ExAws.Kinesis.get_shard_iterator(\"device_service_dev\", shard_id, :latest) |\u003e ExAws.request\n\n# get the shard id and iterator at gen_server/gen_stage init\n\n{:ok, response} = ExAws.Kinesis.get_records(iterator[\"ShardIterator\"], %{limit: 10}) |\u003e ExAws.request\n\n# includes the next shard iterator … store in database in case we crash\n\n# route the work to the next gen_stage based on the partition key\n\n# is there any way to filter records on partition key?\n\nConvoy.Queue.put(\"device_service_dev\", \"devices\", \"{}\")\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftpitale%2Fconvoy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftpitale%2Fconvoy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftpitale%2Fconvoy/lists"}