{"id":47812603,"url":"https://github.com/milkstrawai/good_pipeline","last_synced_at":"2026-04-03T18:16:55.665Z","repository":{"id":347074763,"uuid":"1187456581","full_name":"milkstrawai/good_pipeline","owner":"milkstrawai","description":"DAG-based job pipeline orchestration for Rails, built on GoodJob.","archived":false,"fork":false,"pushed_at":"2026-03-26T17:01:26.000Z","size":1371,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-27T05:59:30.649Z","etag":null,"topics":["activejob","postgresql","rails","ruby","ruby-on-rails"],"latest_commit_sha":null,"homepage":"https://milkstrawai.github.io/good_pipeline/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/milkstrawai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-20T18:42:50.000Z","updated_at":"2026-03-26T17:00:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/milkstrawai/good_pipeline","commit_stats":null,"previous_names":["milkstrawai/good_pipeline"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/milkstrawai/good_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milkstrawai%2Fgood_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milkstrawai%2Fgood_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milkstrawai%2Fgood_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milkstrawai%2Fgood_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/milkstrawai","download_url":"https://codeload.github.com/milkstrawai/good_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/milkstrawai%2Fgood_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31368162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T17:53:18.093Z","status":"ssl_error","status_checked_at":"2026-04-03T17:53:17.617Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activejob","postgresql","rails","ruby","ruby-on-rails"],"created_at":"2026-04-03T18:16:54.759Z","updated_at":"2026-04-03T18:16:55.639Z","avatar_url":"https://github.com/milkstrawai.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoodPipeline\n\nDAG-based job pipeline orchestration for Rails, built on [GoodJob](https://github.com/bensheldon/good_job).\n\nDefine multi-step workflows as directed acyclic graphs — not linear chains. Steps run in parallel when they can and wait for dependencies when they must. GoodPipeline handles dependency resolution, parallel execution, failure strategies, conditional branching, pipeline chaining, and lifecycle callbacks. It also ships with a web dashboard.\n\n## Requirements\n\n- Ruby \u003e= 3.2\n- Rails \u003e= 7.1\n- PostgreSQL\n- GoodJob \u003e= 3.10 with `preserve_job_records = true`\n\n## Installation\n\nAdd to your Gemfile:\n\n```ruby\ngem \"good_pipeline\"\n```\n\nThen install the migrations:\n\n```bash\nbin/rails generate good_pipeline:install\nbin/rails db:migrate\n```\n\nGoodPipeline requires GoodJob to preserve job records. Add this to your GoodJob configuration:\n\n```ruby\n# config/initializers/good_job.rb\nGoodJob.preserve_job_records = true\n```\n\nGoodPipeline will raise `GoodPipeline::ConfigurationError` at boot if this is not set.\n\n## Usage\n\n### Defining a pipeline\n\nSubclass `GoodPipeline::Pipeline` and implement `configure`. Use `run` to declare steps and `after:` to express dependencies:\n\n```ruby\nclass VideoProcessingPipeline \u003c GoodPipeline::Pipeline\n  description \"Downloads, transcodes and publishes a video\"\n  failure_strategy :halt\n\n  on_complete :notify\n  on_success :celebrate\n  on_failure :alert\n\n  def configure(video_id:)\n    run :download,  DownloadJob,  with: { video_id: video_id }\n    run :transcode, TranscodeJob, after: :download\n    run :thumbnail, ThumbnailJob, after: :download\n    run :publish,   PublishJob,   after: %i[transcode thumbnail]\n    run :cleanup,   CleanupJob,   after: :publish\n  end\n\n  private\n\n  def notify = Rails.logger.info(\"Pipeline complete\")\n  def celebrate = Rails.logger.info(\"All steps succeeded!\")\n  def alert = Rails.logger.warn(\"Pipeline had failures\")\nend\n```\n\nThis produces the following DAG:\n\n```mermaid\ngraph TD\n  download --\u003e transcode\n  download --\u003e thumbnail\n  transcode --\u003e publish\n  thumbnail --\u003e publish\n  publish --\u003e cleanup\n```\n\n### Running a pipeline\n\n```ruby\nVideoProcessingPipeline.run(video_id: 123)\n```\n\n### Step options\n\n```ruby\nrun :step_key, JobClass,\n  with:       { key: \"value\" },                # keyword args passed to the job\n  after:      :other_step,                     # dependency (symbol or array of symbols)\n  on_failure: :ignore,                         # step-level failure strategy override\n  enqueue:    { queue: :media, priority: 10 }  # options passed to job.enqueue()\n```\n\n### Failure strategies\n\nSet at the pipeline level with `failure_strategy`:\n\n| Strategy | Behaviour |\n|---|---|\n| `:halt` (default) | Stop all pending steps when any step fails |\n| `:continue` | Let independent branches continue; skip only blocked downstream steps |\n| `:ignore` | Treat failures as successes for dependency resolution |\n\nPer-step overrides via `on_failure:` in `run` apply to that step's outgoing edges only.\n\n### Conditional branching\n\nUse `branch` to take different paths at runtime based on application state:\n\n```ruby\nclass MediaPipeline \u003c GoodPipeline::Pipeline\n  def configure(media_id:)\n    run :analyze, AnalyzeJob, with: { media_id: media_id }\n\n    branch :format_check, after: :analyze, by: :detect_format do\n      on :hd do\n        run :transcode_hd, TranscodeHDJob, with: { media_id: media_id }\n        run :upscale, UpscaleJob, with: { media_id: media_id }, after: :transcode_hd\n      end\n\n      on :sd do\n        run :transcode_sd, TranscodeSDJob, with: { media_id: media_id }\n      end\n    end\n\n    run :publish, PublishJob, after: :format_check\n  end\n\n  private\n\n  def detect_format\n    Media.find(params[:media_id]).hd? ? :hd : :sd\n  end\nend\n```\n\nThe `by:` method is evaluated at runtime when the branch step is reached. The matching arm runs; other arms are skipped. `after: :format_check` waits for whichever arm was chosen to complete.\n\nArms can also be empty for an if-without-else pattern:\n\n```ruby\nbranch :quality_check, after: :analyze, by: :needs_processing do\n  on :yes do\n    run :process, ProcessJob\n  end\n  on :no  # skip — pipeline continues to next step\nend\n```\n\nThe dashboard renders branches as diamond decision nodes with labeled edges.\n\n### Pipeline chaining\n\nChain pipelines together with `.then()`:\n\n```ruby\n# Serial chain\nVideoProcessingPipeline\n  .run(video_id: 123)\n  .then(NotificationPipeline, with: { video_id: 123 })\n\n# Fan-out\nVideoProcessingPipeline\n  .run(video_id: 123)\n  .then(\n    [NotificationPipeline, with: { video_id: 123 }],\n    [AnalyticsPipeline,    with: { video_id: 123 }]\n  )\n\n# Parallel start with fan-in\nGoodPipeline.run(\n  [VideoProcessingPipeline, with: { video_id: 123 }],\n  [AudioProcessingPipeline, with: { audio_id: 456 }]\n).then(MergeMediaPipeline, with: { video_id: 123, audio_id: 456 })\n```\n\nIf an upstream pipeline fails or halts, downstream pipelines are automatically skipped.\n\n### Monitoring\n\n```ruby\npipeline = VideoProcessingPipeline.run(video_id: 123)\n\npipeline.status     # =\u003e \"running\"\npipeline.terminal?  # =\u003e false\npipeline.steps      # =\u003e all step records\npipeline.params     # =\u003e { \"video_id\" =\u003e 123 }\n\n# Query across pipelines\nGoodPipeline::PipelineRecord.where(status: \"failed\")\nGoodPipeline::PipelineRecord.where(type: \"VideoProcessingPipeline\")\n```\n\n### Lifecycle callbacks\n\n```ruby\nclass MyPipeline \u003c GoodPipeline::Pipeline\n  on_complete :always_runs    # any terminal state\n  on_success  :only_success   # pipeline succeeded\n  on_failure  :only_failure   # pipeline failed or halted\nend\n```\n\nCallbacks are dispatched asynchronously via a separate GoodJob job. They never block the coordinator or affect pipeline state.\n\n## Dashboard\n\nGoodPipeline includes a mountable web dashboard for inspecting pipeline executions:\n\n```ruby\n# config/routes.rb\nmount GoodPipeline::Engine =\u003e \"/good_pipeline\"\n```\n\nThe dashboard provides:\n\n- Pipeline Executions: filterable list with status tabs and pipeline type dropdown\n- Pipeline Details: steps table, DAG visualization, chain links, error info\n- Pipeline Definitions: catalog of all pipeline types with their DAG structure\n\n### Pipeline Executions\n\n![Pipeline Executions](docs/screenshots/index.png)\n\n### Pipeline Details\n\n![Pipeline Details](docs/screenshots/show.png)\n\n### Pipeline Definitions\n\n![Pipeline Definitions](docs/screenshots/definitions.png)\n\nNo build step. Uses Pico CSS and Mermaid.js from CDN.\n\n## Cleanup\n\nGoodPipeline automatically cleans up old terminal pipelines when GoodJob runs its own cleanup cycle. No configuration needed, it uses GoodJob's retention period (default 14 days).\n\nTo configure the retention period, set GoodJob's option:\n\n```ruby\n# config/application.rb\nconfig.good_job.cleanup_preserved_jobs_before_seconds_ago = 30.days.to_i\n```\n\n## Development\n\n```bash\nbin/setup\nmise docker:start  # PostgreSQL\nrake test\n```\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/milkstrawai/good_pipeline.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmilkstrawai%2Fgood_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmilkstrawai%2Fgood_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmilkstrawai%2Fgood_pipeline/lists"}