{"id":51082989,"url":"https://github.com/kanutocd/cdc-sidekiq","last_synced_at":"2026-06-23T20:00:53.663Z","repository":{"id":364029420,"uuid":"1263115095","full_name":"kanutocd/cdc-sidekiq","owner":"kanutocd","description":"Runs Sidekiq job payloads through cdc-core processor contracts using cdc-parallel or cdc-concurrent runtime primitives.","archived":false,"fork":false,"pushed_at":"2026-06-11T10:03:18.000Z","size":41,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T11:22:16.347Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kanutocd.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-08T16:28:27.000Z","updated_at":"2026-06-11T10:01:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kanutocd/cdc-sidekiq","commit_stats":null,"previous_names":["kanutocd/cdc-sidekiq"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/kanutocd/cdc-sidekiq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fcdc-sidekiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fcdc-sidekiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fcdc-sidekiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fcdc-sidekiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kanutocd","download_url":"https://codeload.github.com/kanutocd/cdc-sidekiq/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kanutocd%2Fcdc-sidekiq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34704748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-23T02:00:07.161Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-23T20:00:52.639Z","updated_at":"2026-06-23T20:00:53.649Z","avatar_url":"https://github.com/kanutocd.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cdc-sidekiq\n\n[![Gem Version](https://badge.fury.io/rb/cdc-sidekiq.svg)](https://badge.fury.io/rb/cdc-sidekiq)\n[![CI](https://github.com/kanutocd/cdc-sidekiq/workflows/CI/badge.svg)](https://github.com/kanutocd/cdc-sidekiq/actions)\n[![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.4-ruby.svg)](https://www.ruby-lang.org/en/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n`cdc-sidekiq` integrates Sidekiq with CDC execution primitives.\n\nSidekiq remains the durable job system. It owns scheduling, retries, queues, Redis persistence, and operational behavior.\n\n`cdc-sidekiq` owns what happens inside selected jobs: processor contracts, runtime selection, fan-out/fan-in execution, and normalized processor results.\n\n```text\nSidekiq Job\n      |\n      v\nCDC::Sidekiq::ProcessorJob\n      |\n      +--\u003e :direct\n      |       sequential processor calls\n      |\n      +--\u003e cdc-parallel\n      |       Ractor fan-out / fan-in\n      |\n      +--\u003e cdc-concurrent\n              Async task fan-out / fan-in\n```\n\n## Why cdc-sidekiq Exists\n\nSidekiq is excellent at durable background job execution.\n\nIt provides:\n\n- scheduling\n- retries\n- queues\n- Redis-backed persistence\n- operational familiarity\n\nHowever, Sidekiq intentionally leaves the internal execution strategy of a job to the application.\n\n`cdc-sidekiq` extends Sidekiq with CDC execution primitives:\n\n- processor contracts\n- parallel execution with `cdc-parallel`\n- concurrent execution with `cdc-concurrent`\n- fan-out / fan-in processing\n- ordered result collection\n- normalized processor results\n\nThe goal is not to replace Sidekiq.\n\nThe goal is to make execution topology an explicit choice inside Sidekiq jobs.\n\n```text\nSidekiq schedules the work.\nCDC primitives execute the work.\n```\n\n---\n\n## Roadmap\n\n### Community Edition\n\nThe open-source edition focuses on execution primitives.\n\nCurrent OSS capabilities:\n\n- `:direct` runtime\n- `:concurrent` runtime\n- `:parallel` runtime\n- ProcessorJob abstraction\n- `process`\n- `process_many`\n- ordered result collection\n- normalized ProcessorResult handling\n\n### Commercial Orchestration Boundary\n\nThe open-source edition focuses on Sidekiq integration and explicit execution primitives.\n\nAdvanced orchestration belongs above this gem. Commercial orchestration features such as hybrid runtimes, nested worker topologies, worker-local resource pools, adaptive sizing, capacity guards, telemetry, and advanced failure policies live in the commercial orchestrator layer.\n\n```text\ncdc-sidekiq\n    Sidekiq integration\n    ProcessorJob abstraction\n    :direct / :concurrent / :parallel runtime selection\n\ncdc-orchestrator-pro\n    Hybrid runtime\n    Nested runtime / worker-local resource pools\n    Orchestration, backpressure, telemetry, tuning\n```\n\nThis keeps the OSS gem small and useful while leaving operational orchestration to the commercial layer.\n\n## Runtime Selection Guide\n\nChoose the smallest runtime that matches the work.\n\n| Runtime | Best fit | Avoid when |\n| --- | --- | --- |\n| `:direct` | Tiny, cheap, already-batched work | You need internal fan-out |\n| `:parallel` | CPU-heavy, Ractor-shareable processing | Payloads/processors are not shareable or work is too tiny |\n| `:concurrent` | I/O-heavy processing with scheduler-friendly waits | Work is CPU-bound or effectively no-op |\n\nA useful rule of thumb:\n\n```text\nSidekiq controls how many jobs run.\ncdc-sidekiq controls how one selected job executes its internal payload.\n```\n\nDo not increase both Sidekiq concurrency and CDC runtime concurrency blindly. That can multiply downstream pressure on Redis, PostgreSQL, HTTP APIs, and other shared resources.\n\n## Correctness Boundary\n\n`cdc-sidekiq` controls execution topology. It does not make shared downstream state automatically safe.\n\nWhen multiple jobs or multiple internal work items update the same database rows, files, search documents, Redis keys, or external API resources, the processor or sink must provide the correctness policy. Common strategies include:\n\n- database transactions;\n- row-level locks such as `SELECT ... FOR UPDATE`;\n- optimistic locking;\n- idempotency keys;\n- conflict-safe writes;\n- single-writer sink patterns.\n\nThe runtime can provide controlled execution. Application-specific correctness remains the responsibility of the processor and sink implementation.\n\n## Benchmarks\n\nSee [benchmark/README.md](./benchmark/README.md) for the `bin/cdc-sidekiq-load` benchmark, recent 500,000-item snapshots, interpretation, and runtime tuning guidance.\n\n## Requirements\n\nRuby 3.4 or newer.\n\nRuntime support depends on the selected CDC execution substrate:\n\n| Runtime | Ruby | Required gems |\n| --- | --- | --- |\n| `:direct` | 3.4+ | `sidekiq`, `cdc-core` |\n| `:concurrent` | 3.4+ | `sidekiq`, `cdc-core`, `cdc-concurrent` |\n| `:parallel` | 4.0+ | `sidekiq`, `cdc-core`, `cdc-parallel` |\n\n`cdc-parallel` remains optional because it requires Ruby 4+. Ruby 3.4 users can still use `:direct` and `:concurrent`.\n\n## Installation\n\n```ruby\ngem \"cdc-sidekiq\"\n```\n\nRuntime gems are installed by the application according to the execution model it uses:\n\n```ruby\ngem \"cdc-concurrent\" # for Async-backed execution\ngem \"cdc-parallel\"   # for Ractor-backed execution\n```\n\nApplications that only use `:direct` do not need either optional runtime gem.\n\n## Configuration\n\n```ruby\nSidekiq.configure_server do |_config|\n  CDC::Sidekiq.configure do |cdc|\n    cdc.default_runtime = :concurrent\n    cdc.parallel_size = Etc.nprocessors - 1\n    cdc.concurrency = 100\n    cdc.timeout = nil\n    cdc.preserve_order = true\n    cdc.raise_on_failure = true\n    cdc.batch_payloads = true\n  end\nend\n```\n\nConfiguration defaults:\n\n| Option | Default | Meaning |\n| --- | --- | --- |\n| `default_runtime` | `:concurrent` | Runtime used when a job does not declare `cdc_runtime`. |\n| `parallel_size` | `Etc.nprocessors - 1`, minimum `1` | Ractor worker count for `:parallel` jobs. |\n| `concurrency` | `100` | Async task limit for `:concurrent` jobs. |\n| `timeout` | `nil` | Optional per-item timeout passed to CDC processor pools. |\n| `preserve_order` | `true` | Preserve input order for `:concurrent` result arrays. |\n| `raise_on_failure` | `true` | Raise failed `ProcessorResult` objects so Sidekiq can retry. |\n| `batch_payloads` | `true` | Process array payloads with `process_many` instead of treating the array as one item. |\n\nSidekiq concurrency and CDC runtime concurrency are intentionally separate.\n\n```text\nSidekiq concurrency\n  = how many Sidekiq jobs run at once\n\nCDC parallel size\n  = how many Ractors one CDC-enabled job may use internally\n\nCDC concurrency\n  = how many Async tasks one CDC-enabled job may use internally\n```\n\n## Usage\n\n### Parallel processor job\n\n```ruby\nclass UserIndexer \u003c CDC::Core::Processor\n  ractor_safe!\n\n  def process(user_id)\n    # CPU-heavy or shareable work\n    CDC::Core::ProcessorResult.success(user_id)\n  end\nend\n\nclass ReindexUsersJob\n  include Sidekiq::Job\n  include CDC::Sidekiq::ProcessorJob\n\n  cdc_processor UserIndexer\n  cdc_runtime :parallel\n  cdc_parallel_size Etc.nprocessors - 1\nend\n\nReindexUsersJob.perform_async([1, 2, 3, 4])\n```\n\nArray payloads are processed with `process_many` by default.\n\n```text\nSidekiq job payload\n      |\n      v\nprocess_many\n      |\n      v\ncdc-parallel ProcessorPool\n      |\n      v\nordered ProcessorResult array\n```\n\n### Concurrent processor job\n\n```ruby\nclass WebhookDeliverer \u003c CDC::Core::Processor\n  def concurrent_safe? = true\n\n  def process(webhook_payload)\n    # I/O-heavy scheduler-friendly work\n    CDC::Core::ProcessorResult.success(webhook_payload)\n  end\nend\n\nclass DeliverWebhooksJob\n  include Sidekiq::Job\n  include CDC::Sidekiq::ProcessorJob\n\n  cdc_processor WebhookDeliverer\n  cdc_runtime :concurrent\n  cdc_concurrency 250\n  cdc_timeout 5.0\nend\n```\n\n### Per-job runtime override\n\nGlobal configuration provides defaults. Each job can override the runtime.\n\n```ruby\nclass ProjectionJob\n  include Sidekiq::Job\n  include CDC::Sidekiq::ProcessorJob\n\n  cdc_processor ProjectionProcessor\n  cdc_runtime :parallel\nend\n```\n\n## Failure behavior\n\nBy default, failed `ProcessorResult` objects raise `CDC::Sidekiq::ProcessorFailureError` so Sidekiq can apply its normal retry behavior.\n\n```ruby\nclass BestEffortJob\n  include Sidekiq::Job\n  include CDC::Sidekiq::ProcessorJob\n\n  cdc_processor BestEffortProcessor\n  cdc_runtime :concurrent\n  cdc_raise_on_failure false\nend\n```\n\n## Benchmarking\n\n`cdc-sidekiq` includes `bin/cdc-sidekiq-load`, a benchmark aligned with Sidekiq's `bin/sidekiq-load` style.\n\nSidekiq's benchmark creates many no-op jobs and drains them as fast as possible. `cdc-sidekiq-load` keeps the no-op workload shape but measures the downstream CDC runtime path inside one CDC-aware Sidekiq job.\n\n```bash\nCOUNT=500000 RUNTIME=concurrent CDC_CONCURRENCY=100 \\\n  bundle exec bin/cdc-sidekiq-load\n```\n\n```bash\nCOUNT=500000 RUNTIME=parallel CDC_PARALLEL_SIZE=7 \\\n  bundle exec bin/cdc-sidekiq-load\n```\n\nSee [`benchmark/README.md`](https://github.com/kanutocd/cdc-sidekiq/tree/main/benchmark) for interpretation notes and all benchmark knobs.\n\n## Current scope\n\nThis gem currently implements the downstream/runtime integration only:\n\n```text\nSidekiq Job\n    ↓\nCDC execution primitives\n\nFuture:\n\nPostgreSQL WAL\n    ↓\npgoutput*\n    ↓\nSidekiq Job\n    ↓\nCDC execution primitives\n```\n\nA future upstream/source integration may map PostgreSQL logical replication events into Sidekiq work through the `pgoutput*` family and `pgoutput-source-adapter`, but that is intentionally out of scope for the initial release.\n\n## License\n\n[MIT](./LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanutocd%2Fcdc-sidekiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkanutocd%2Fcdc-sidekiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkanutocd%2Fcdc-sidekiq/lists"}