{"id":13878708,"url":"https://github.com/nxt-insurance/nxt_pipeline","last_synced_at":"2025-08-01T20:31:15.345Z","repository":{"id":34239721,"uuid":"172033247","full_name":"nxt-insurance/nxt_pipeline","owner":"nxt-insurance","description":"A simple orchestration framework to reduce over your (service) objects like a pro. ","archived":false,"fork":false,"pushed_at":"2024-04-25T07:23:05.000Z","size":130,"stargazers_count":23,"open_issues_count":13,"forks_count":1,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-04-25T08:29:31.647Z","etag":null,"topics":["nxt-pipeline","orchestrator","ruby","ruby-on-rails","service","service-objects"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/nxt_pipeline","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nxt-insurance.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-22T09:10:36.000Z","updated_at":"2024-06-02T18:36:40.194Z","dependencies_parsed_at":"2024-01-13T20:38:15.433Z","dependency_job_id":"24973d0d-8abe-45d1-bf7a-3676814d06af","html_url":"https://github.com/nxt-insurance/nxt_pipeline","commit_stats":{"total_commits":122,"total_committers":8,"mean_commits":15.25,"dds":0.5983606557377049,"last_synced_commit":"e86070d8e5fd322a6efe90ec32179aad608e2c2e"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nxt-insurance%2Fnxt_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nxt-insurance%2Fnxt_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nxt-insurance%2Fnxt_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nxt-insurance%2Fnxt_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nxt-insurance","download_url":"https://codeload.github.com/nxt-insurance/nxt_pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228402239,"owners_count":17914230,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nxt-pipeline","orchestrator","ruby","ruby-on-rails","service","service-objects"],"created_at":"2024-08-06T08:01:57.346Z","updated_at":"2024-12-06T03:17:02.685Z","avatar_url":"https://github.com/nxt-insurance.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/nxt-insurance/nxt_pipeline.svg?style=svg)](https://circleci.com/gh/nxt-insurance/nxt_pipeline)\n\n# NxtPipeline\n\nNxtPipeline is an orchestration framework for your service objects or function objects, how I like to call them.\nService objects are a very wide spread way of organizing code in the Ruby and Rails communities. Since it's little classes\ndoing one thing you can think of them as function objects and thus they often share a common interface in a project. \nThere are also many frameworks out there that normalize the usage of service objects and provide a specific way\nof writing service objects and often also allow to orchestrate (reduce) these service objects.\nCompare [light-service](https://github.com/adomokos/light-service) for instance.\n\nThe idea of NxtPipeline was to build a flexible orchestration framework for service objects without them having to conform\nto a specific interface. Instead NxtPipeline expects you to specify how to execute different kinds of service objects\nthrough so called constructors and thereby does not dictate you how to write your service objects. Nevertheless this still\nmostly makes sense if your service objects share common interfaces to keep the necessary configuration to a minimum.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'nxt_pipeline'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install nxt_pipeline\n\n## Usage\n\n### Example\n\nLet's look at an example. Here validator service objects are orchestrated with NxtPipeline to build a validation \npipeline. We inject the accumulator `{ value: 'aki', errors: [] }` that is then passed through all validation steps. \nIf an validator returns an error it's added to the array of errors of the accumulator to collect all errors of all steps.\n\n```ruby\nclass Validator\n  attr_accessor :error\nend\n\nclass TypeChecker \u003c Validator\n  def initialize(value, type:)\n    @value = value\n    @type = type\n  end\n\n  attr_reader :value, :type\n\n  def call\n    return if value.is_a?(type)\n    self.error = \"Value does not match type #{type}\"\n  end\nend\n\nclass MinSize \u003c Validator\n  def initialize(value, size:)\n    @value = value\n    @size = size\n  end\n\n  attr_reader :value, :size\n\n  def call\n    return if value.size \u003e= size\n    self.error = \"Value size must be greater than #{size-1}\"\n  end\nend\n\nclass MaxSize \u003c Validator\n  def initialize(value, size:)\n    @value = value\n    @size = size\n  end\n\n  attr_reader :value, :size\n\n  def call\n    return if value.size \u003c= size\n    self.error = \"Value size must be less than #{size+1}\"\n  end\nend\n\nclass Uniqueness \u003c Validator\n  def initialize(value, scope:)\n    @value = value\n    @scope = scope\n  end\n\n  attr_reader :value, :scope\n\n  def call\n    return if scope.count { |item| item == value }\n    self.error = \"Value is not unique in: #{scope}\"\n  end\nend\n\nresult = NxtPipeline.call({ value: 'aki', errors: [] }) do |p|\n  p.constructor(:validator, default: true) do |acc, step|\n    validator = step.argument.new(acc.fetch(:value), **step.options)\n    validator.call\n    acc[:errors] \u003c\u003c validator.error if validator.error.present?\n\n    acc\n  end\n\n  p.step TypeChecker, options: { type: String }\n  p.step MinSize, options: { size: 4 }\n  p.step MaxSize, options: { size: 10 }\n  p.step Uniqueness, options: { scope: ['andy', 'aki', 'lütfi', 'rapha'] }\nend\n\nresult # =\u003e { value: 'aki', errors: ['Value size must be greater than 3'] } \n```\n\n### Constructors\n\nIn order to reduce over your service objects you have to define constructors so that the pipeline knows how to execute\na specific step. You can define constructors globally and specific to a pipeline.\n\nMake a constructor available for all pipelines of your project by defining it globally with:\n\n```ruby\nNxtPipeline.constructor(:service) do |acc, step|\n  validator = step.argument.new(acc.fetch(:value), **step.options)\n  validator.call\n  acc[:errors] \u003c\u003c validator.error if validator.error.present?\n\n  acc\nend\n```\n\nOr define a constructor only locally for a specific pipeline.\n\n```ruby\nNxtPipeline.new({ value: 'aki', errors: [] }) do |p|\n  p.constructor(:validator, default: true) do |acc, step|\n    validator = step.argument.new(acc.fetch(:value), **step.options)\n    validator.call\n    acc[:errors] \u003c\u003c validator.error if validator.error.present?\n\n    acc\n  end\n\n  p.step TypeChecker, options: { type: String }\n  # ...\nend\n```\n\nConstructor Hierarchy\n\nIn order to execute a specific step the pipeline firstly checks whether a constructor was specified for a step: \n`pipeline.step MyServiceClass, constructor: :service`. If this is not the case it checks whether there is a resolver \nregistered that applies. If that's not the case the pipeline checks if there is a constructor registered for the \nargument that was passed in. This means if you register constructors directly for the arguments you pass in you don't\nhave to specify this constructor option. Therefore the following would work without the need to provide a constructor \nfor the steps.\n\n```ruby\nNxtPipeline.new({}) do |p|\n  p.constructor(:service) do |acc, step|\n    step.service_class.new(acc).call\n  end\n\n  p.step :service, service_class: MyServiceClass\n  p.step :service, service_class: MyOtherServiceClass\n  # ...\nend\n```\n\nLastly if no constructor could be resolved directly from the step argument, the pipelines falls back to the locally\nand then to the globally defined default constructors.\n\n### Defining steps\n\nOnce your pipeline knows how to execute your steps you can add those. The `pipeline.step` method expects at least one\nargument which you can access in the constructor through `step.argument`. You can also pass in additional options\nthat you can access through readers of a step. The `constructor:` option defines which constructor to use for a step\nwhere as you can name a step with the `to_s:` option.\n\n```ruby\n# explicitly define which constructor to use \npipeline.step MyServiceClass, constructor: :service\n# use a block as inline constructor\npipeline.step SpecialService, constructor: -\u003e(step, arg:) { step.argument.call(arg: arg) }\n# Rely on the default constructor\npipeline.step MyOtherServiceClass\n# Define a step name\npipeline.step MyOtherServiceClass, to_s: 'First Step'\n# Or simply execute a (named) block - NO NEED TO DEFINE A CONSTRUCTOR HERE  \npipeline.step :step_name_for_better_log do |acc, step|\n  # ...\nend\n```\n\nDefining multiple steps at once. This is especially useful to dynamically configure a pipeline for execution and\ncan potentially even come from a yaml configuration or from the database.\n\n```ruby\npipeline.steps([\n  [MyServiceClass, constructor: :service],\n  [MyOtherServiceClass, constructor: :service],\n  [MyJobClass, constructor: :job]\n])\n\n# You can also overwrite the steps of a pipeline through explicitly setting them. This will remove any previously \n# defined steps.\npipeline.steps = [\n  [MyServiceClass, constructor: :service],\n  [MyOtherServiceClass, constructor: :service]\n]\n```\n\n### Execution\n\nOnce a pipeline contains steps you can call it with `call(accumulator)` whereas it expects you to inject the accumulator\nas argument that is then passed through all steps.\n\n```ruby\npipeline.call(arg: 'initial argument')\n\n# Or directly pass the steps you want to execute:\npipeline.call(arg: 'initial argument') do |p|\n  p.step MyServiceClass, to_s: 'First step'\n  p.step MyOtherServiceClass, to_s: 'Second step'\n  p.step MyJobClass, constructor: :job\n  p.step MyOtherJobClass, constructor: :job\nend\n```\n\nYou can also create a new instance of a pipeline and directly run it with `call`:\n\n```ruby\nNxtPipeline.call(arg: 'initial argument') do |p|\n  p.steps # ...\nend\n```\n\nYou can query the steps of your pipeline simply by calling `pipeline.steps`. A NxtPipeline::Step will provide you with\nan interface for options, status, execution_finished_at execution_started_at,\nexecution_duration, result, error and the index in the pipeline.\n\n```\npipeline.steps.first\n# will give you a step object\n#\u003cNxtPipeline::Step:0x00007f83eb399448...\u003e\n```\n\n### Guard clauses\n\nYou can also define guard clauses that take a proc to prevent the execution of a step.\nA guard can accept the change set and the step as arguments.\n\n ```ruby\n pipeline.call('initial argument') do |p|\n  p.step MyServiceClass, if: -\u003e (acc, step) { acc == 'initial argument' }\n  p.step MyOtherServiceClass, unless: -\u003e { false }\nend\n\n ```\n\n### Error callbacks\n\nApart from defining constructors and steps you can also define error callbacks. Error callbacks can accept up to  \nthree arguments: `error, acc, step`.\n\n```ruby\nNxtPipeline.new do |p|\n  p.step # ... \n\n  p.on_error MyCustomError do |error|\n    # First matching error callback will be executed!\n  end\n\n  p.on_errors ArgumentError, KeyError do |error, acc|\n    # First matching error callback will be executed!\n  end\n\n  p.on_errors YetAnotherError, halt_on_error: false do |error, acc, step|\n    # After executing the callback the pipeline will not halt but continue to\n    # execute the next steps.\n  end\n\n  p.on_errors do |error, acc, step|\n    # This will match all errors inheriting from StandardError\n  end\nend\n```\n\n### Before, around and after callbacks\n\nYou can also define callbacks :before, :around and :after each step and or the `#execute` method. You can also register\nmultiple callbacks, but probably you want to keep them to a minimum to not end up in hell. Also note that before and\nafter callbacks will run even if a step was skipped through a guard clause.\n\n#### Step callbacks\n\n```ruby\nNxtPipeline.new do |p|\n  p.before_step do |_, change_set|\n    change_set[:acc] \u003c\u003c 'before step 1'\n    change_set\n  end\n\n  p.around_step do |_, change_set, execution|\n    change_set[:acc] \u003c\u003c 'around step 1'\n    execution.call # you have to specify where in your callback you want to call the inner block\n    change_set[:acc] \u003c\u003c 'around step 1'\n    change_set\n  end\n\n  p.after_step do |_, change_set|\n    change_set[:acc] \u003c\u003c 'after step 1'\n    change_set\n  end\nend\n```\n\n#### Execution callbacks\n\n```ruby\nNxtPipeline.new do |p|\n  p.before_execution do |_, change_set|\n    change_set[:acc] \u003c\u003c 'before execution 1'\n    change_set\n  end\n\n  p.around_execution do |_, change_set, execution|\n    change_set[:acc] \u003c\u003c 'around execution 1'\n    execution.call # you have to specify where in your callback you want to call the inner block\n    change_set[:acc] \u003c\u003c 'around execution 1'\n    change_set\n  end\n\n  p.after_execution do |_, change_set|\n    change_set[:acc] \u003c\u003c 'after execution 1'\n    change_set\n  end\nend\n```\n\nNote that the `after_execute` callback will not be called in case a step raises an error.\nSee the previous section (_Error callbacks_) for how to define callbacks that run in case of errors.\n\n### Constructor resolvers\n\nYou can also define constructor resolvers for a pipeline to dynamically define which previously registered constructor\nto use for a step based on the argument and options passed to the step.\n\n```ruby\nclass Transform\n  def initialize(word, operation)\n    @word = word\n    @operation = operation\n  end\n\n  attr_reader :word, :operation\n\n  def call\n    word.send(operation)\n  end\nend\n\nNxtPipeline.new do |pipeline|\n  # dynamically resolve to use a proc as constructor\n  pipeline.constructor_resolver do |argument, **opts|\n    argument.is_a?(Class) \u0026\u0026\n      -\u003e(step, arg:) {\n        result = step.argument.new(arg, opts.fetch(:operation)).call\n        # OR result = step.argument.new(arg, step.operation).call\n        { arg: result }\n      }\n  end\n\n  # dynamically resolve to a defined constructor\n  pipeline.constructor_resolver do |argument|\n    argument.is_a?(String) \u0026\u0026 :dynamic\n  end\n\n  pipeline.constructor(:dynamic) do |step, arg:|\n    if step.argument == 'multiply'\n      { arg: arg * step.multiplier }\n    elsif step.argument == 'symbolize'\n      { arg: arg.to_sym }\n    else\n      raise ArgumentError, \"Don't know how to deal with argument: #{step.argument}\"\n    end\n  end\n\n  pipeline.step Transform, operation: 'upcase'\n  pipeline.step 'multiply', multiplier: 2\n  pipeline.step 'symbolize'\n  pipeline.step :extract_value do |arg|\n    arg\n  end\nend\n```\n\n### Configurations\n\nYou probably do not have that many different kinds of steps that you execute within your pipelines. Otherwise the whole\nconcept does not make much sense. To make constructing a pipeline simpler you can therefore define configurations on\na global level simply by providing a name for a configuration along with a configuration block.\nThen you then create a preconfigure pipeline by passing in the name of the configuration when creating a new pipeline.\n\n```ruby\n# Define configurations in your initializer or somewhere upfront \nNxtPipeline.configuration(:test_processor) do |pipeline|\n  pipeline.constructor(:processor) do |arg, step|\n    { arg: step.argument.call(arg: arg) }\n  end\nend\n\nNxtPipeline.configure(:validator) do |pipeline|\n  pipeline.constructor(:validator) do |arg, step|\n    # ..\n  end\nend\n\n# ...\n\n# Later create a pipeline with a previously defined configuration\nNxtPipeline.new(configuration: :test_processor) do |p|\n  p.step -\u003e(arg) { arg + 'first ' }, constructor: :processor\n  p.step -\u003e(arg) { arg + 'second ' }, constructor: :processor\n  p.step -\u003e(arg) { arg + 'third' }, constructor: :processor\nend\n```\n\n### Step status and meta_data\nWhen executing your steps you can also log the status of a step by setting it in your constructors or callbacks in \nwhich you have access to the steps.\n\n```ruby\npipeline = NxtPipeline.new do |pipeline|\n  pipeline.constructor(:step, default: true) do |acc, step|\n    result = step.proc.call(acc)\n    step.status = result.present? # Set the status here\n    step.meta_data = 'additional info' # or some meta data\n    acc\n  end\n\n  pipeline.step :first_step do |acc, step|\n    step.status = 'it worked'\n    step.meta_data = { extra: 'info' }\n    acc\n  end\n\n  pipeline.step :second, proc: -\u003e(acc) { acc }\nend\n\npipeline.logger.log # =\u003e { \"first_step\" =\u003e 'it worked', \"second\" =\u003e true } \npipeline.steps.map(\u0026:meta_data) # =\u003e [{:extra=\u003e\"info\"}, \"additional info\"]\n```\n\n## Topics\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `bin/rspec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\nYou can also run `bin/guard` to automatically run specs when files are saved.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/nxt-insurance/nxt_pipeline.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnxt-insurance%2Fnxt_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnxt-insurance%2Fnxt_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnxt-insurance%2Fnxt_pipeline/lists"}