{"id":17321410,"url":"https://github.com/tompave/magic_pipe","last_synced_at":"2025-04-14T15:40:56.966Z","repository":{"id":56882524,"uuid":"114780466","full_name":"tompave/magic_pipe","owner":"tompave","description":"MagicPipe is a Ruby library to push data to remote destinations on multiple topics.","archived":false,"fork":false,"pushed_at":"2020-05-15T15:24:52.000Z","size":112,"stargazers_count":5,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-28T04:34:53.372Z","etag":null,"topics":["kafka","message-bus","ruby","sqs","thrift"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tompave.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-19T15:19:33.000Z","updated_at":"2024-07-05T06:42:00.000Z","dependencies_parsed_at":"2022-08-20T13:10:47.699Z","dependency_job_id":null,"html_url":"https://github.com/tompave/magic_pipe","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tompave%2Fmagic_pipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tompave%2Fmagic_pipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tompave%2Fmagic_pipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tompave%2Fmagic_pipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tompave","download_url":"https://codeload.github.com/tompave/magic_pipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248907853,"owners_count":21181415,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","message-bus","ruby","sqs","thrift"],"created_at":"2024-10-15T13:37:21.252Z","updated_at":"2025-04-14T15:40:56.945Z","avatar_url":"https://github.com/tompave.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MagicPipe\n\n[![Build Status](https://travis-ci.org/tompave/magic_pipe.svg?branch=master)](https://travis-ci.org/tompave/magic_pipe) [![Gem Version](https://badge.fury.io/rb/magic_pipe.svg)](https://badge.fury.io/rb/magic_pipe)\n\nMagicPipe is a Ruby library to push data to remote destinations on multiple topics.\n\nIt provides client adapters for several popular message busses, and it's meant to facilitate publishing messages and streaming data, in different formats and to different backends.\n\n## Content\n\n* [Design, concepts and internals](#design-concepts-and-internals)\n  - [The moving parts](#the-moving-parts)\n    + [Codecs](#codecs)\n    + [Transports](#transports)\n    + [Senders](#senders)\n    + [Loaders](#loaders)\n  - [Gluing everything together](#gluing-everything-together)\n  - [Multiple pipes](#multiple-pipes)\n  - [The message payloads](#the-message-payloads)\n* [Usage](#usage)\n  - [Configuration](#configuration)\n* [Dependencies](#dependencies)\n* [Use cases](#use-cases)\n* [Installation](#installation)\n\n## Design, concepts and internals\n\nIts design principles are:\n\n* It should be plug and play, with minimal configuration -- it's an opinionated library.\n* It should support different message formats and backends with a consistent interface.\n* It should allow for multiple backends to be targeted at the same time.\n* It should be extendable and customizable.\n\nTo achieve these goals, MagicPipe adopts a modular design with interchangeable parts.\n\n### The moving parts\n\nThe four main units of work are codecs, transports, senders and loaders. MagicPipe provides a set of classes out of the box, but users of the library can configure their own custom classes that implement the correct interface.\n\n#### Codecs\n\nCodecs accept a Ruby object and produce an encoded output. The input must respond to `as_json` and return a Hash. The provided codecs are:\n\n* Yaml\n* JSON\n* MessagePack\n* Thrift (work in progress)\n\n#### Transports\n\nTransports are adapters for the different backends, and take care of establishing connections and submitting the payloads. The provided transports are:\n\n* Debug\n* Log\n* HTTPS\n* SQS\n* Kafka (work in progress)\n* DynamoDB (work in progress)\n* Multi (allows to target different transports at the same time)\n\n#### Senders\n\nSenders glue things together and implement a processing strategy. The provided senders are:\n\n* Sync\n* Async (Sidekiq)\n\n#### Loaders\n\nLoaders are used with the Async sender to serialize Ruby objects into something that can be passed to Sidekiq, and then to rebuild the original Objects inside the Sidekiq workers. The provided loaders are:\n\n* SimpleActiveRecord: it takes `ActiveRecord::Base` instances and an optional wrapper (e.g. an `ActiveModel::Serializer`) and turns them into class references (Strings) and a record ID. Then, when the Sidekiq jobs execute, it loads the record from the DB and wraps it in the serializer, if present.\n\n### Gluing everything together\n\nA MagicPipe client encapsulates the configured parts. For example, clients with a single and multiple transports can be represented like this:\n\n```\nClient\n├ Configuration\n└ Sender\n  ├ Codec\n  ├ (Loader)\n  └ Transport\n\nClient\n├ Configuration\n└ Sender\n  ├ Codec\n  ├ (Loader)\n  └ Transports\n    ├ Transport A\n    ├ Transport B\n    └ Transport C\n```\n\nA client can only have a single codec and sender, but can have multiple transports to submit data to multiple backends. This is particularly efficient because it allows to reduce the number of DB queries to re-load the data, when using the async sender.\n\n### Multiple pipes\n\nMultiple clients with different configurations can be used together in the same process.\n\nThe main use case is to support different codecs (message formats). Some applications may in fact need to emit messages with different formats to different backends, for example JSON to both SQS and a remote HTTPS endpoint, and Thrift to Kafka.\n\nAnother use case is to use different Sidekiq queues (and worker pools) for different topics, which can be accomplished by using different MagicPipe clients for different types of objects.\n\n### The message payloads\n\nMagicPipe wraps the payloads in message envelopes with extra metadata. These extra attributes are:\n\n* the message topic (string)\n* the producer name (string)\n* the submission timestamp, captured when `client#send_data` is invoked (integer)\n* the payload mime type, e.g. `application/json` (string)\n\nSome transports will additionally provide this metadata as message meta attributes. For example, the HTTPS transport will set them as custom HTTP request headers, and the SQS transport will set them as the SQS message custom attributes.\n\n## Usage\n\nCreate and configure a MagicPipe client:\n(Temporary API! Still a work in progress)\n\n```ruby\nrequire \"magic_pipe/senders/async\"\nrequire \"magic_pipe/transports/https\"\nrequire \"magic_pipe/transports/sqs\"\n\n$magic_pipe = MagicPipe.build do |mp|\n  mp.client_name = :my_magic_pipe\n  mp.producer_name = \"My Awesome Service (Production)\"\n\n  mp.sender = :async\n  mp.loader = :simple_active_record\n\n  mp.codec = :json\n  mp.transports = [:https, :sqs]\n\n  mp.sidekiq_options = {\n    queue: \"magic_pipe\"\n  }\n  mp.https_transport_options = {\n    url: \"https://my.receiver.service/foo\",\n    basic_auth: \"bar:foo\",\n  }\n  mp.sqs_transport_options = {\n    queue: \"my_data_stream\"\n  }\n\n  mp.logger = Rails.logger\n  mp.metrics_client = $statsd_client\nend\n```\n\nThen, to submit a message:\n\n```ruby\n$magic_pipe.send_data(\n  object: object,\n  topic: \"my_topic\",\n  wrapper: nil, # default\n  time: Time.now.utc # default\n)\n```\n\nA more concrete example, with an active record object:\n\n```ruby\nclass Article \u003c ActiveRecord::Base\n  after_commit :send_on_magic_pipe\n\n  private\n\n  def send_on_magic_pipe\n    $magic_pipe.send_data(\n      object: self,\n      topic: \"articles\",\n      wrapper: Serializers::InventoryArticleSerializer,\n      time: updated_at\n    )\n  end\nend\n```\n\n### Configuration\n\nThe [`MagicPipe::Config`](https://github.com/tompave/magic_pipe/blob/master/lib/magic_pipe/config.rb) class lists all supported configuration options and their default values.\n\nUsers of the library are strongly encouraged to set explicit values for these settings:\n\n* `client_name`: used internally to identify a specific magic pipe client instance, for example when handing over the message publication to Sidekiq.\n* `producer_name`: the name of your application, as it should appear in the metadata of the published messages.\n* `logger`: when used in Rails, this should be set to `Rails.logger`.\n* `metrics_client`: to collect operational metrics. Wrapped by [`MagicPipe::Metrics`](https://github.com/tompave/magic_pipe/blob/master/lib/magic_pipe/metrics.rb). It's mainly designed to work with [DataDog's StatsD](https://rubygems.org/gems/dogstatsd-ruby) out of the box, but any stats collector library would work, if you provide a thin adapter wrapper. If not configured, the default implementation will simply send the metrics to the logger.\n\nMore specific configuration is then required for the different modules of the library, when used.\n\n#### Transport: SQS\n\nThis transport requires credentials for the AWS API. The credentials need to be associated to an IAM user with full access to SQS, and need to be present in the system env:\n\n```\nexport AWS_ACCESS_KEY_ID='foo'\nexport AWS_SECRET_ACCESS_KEY='bar'\nexport AWS_REGION='us-east-1'\n```\n\n#### Transport: HTTPS\n\nThis transport builds a [Faraday](https://github.com/lostisland/faraday) connection. A number of options can be configured:\n\n```ruby\n$magic_pipe = MagicPipe.build do |mp|\n  mp.transport = :https\n\n  mp.https_transport_options = {\n    url: \"https://my.receiver.service/messages\",\n    dynamic_path_builder: -\u003e (topic) { topic }\n\n    basic_auth: \"foo:bar\",\n\n    timeout: 2,\n    open_timeout: 3,\n  }\nend\n```\n\nThe `dynamic_path_builder` setting should be a callable that will receive the topic name. It defaults to `nil`, in which case the base URL will be used as is. If present, its return value will be used in Faraday as:\n\n```ruby\nfaraday_connection.post do |r|\n  r.url dynamic_path_builder.call(current_topic)\nend\n```\n\nWhich will result in requests as:\n\n```\nHTTP POST https://my.receiver.service/messages/a-topic-name\n```\n\n\n## Dependencies\n\nBecasuse of MagicPipe's modular design, and in order to keep a small installation footprint, all of its dependencies are optional. Users of the library need to manually install the required dependencies in their projects.\n\nThe Ruby gems MagicPipe's modules depend on are:\n\n* Senders:\n  - Async: `sidekiq`\n* Transports:\n  - SQS: `aws-sdk-sqs`\n  - HTTPS: `faraday`, `typhoeus`\n* Codecs:\n  - JSON: `oj` (optional, will fallback to `json` from the stdlib if `oj` is missing)\n  - MessagePack: `msgpack`\n\n## Use cases\n\nTODO\n\n* event driven architectures\n* streaming domain data to replicate it somewhere else\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'magic_pipe'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install magic_pipe\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftompave%2Fmagic_pipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftompave%2Fmagic_pipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftompave%2Fmagic_pipe/lists"}