{"id":21307169,"url":"https://github.com/aliaksandr-master/unipipeline","last_synced_at":"2025-07-11T21:32:08.836Z","repository":{"id":39714435,"uuid":"376612110","full_name":"aliaksandr-master/unipipeline","owner":"aliaksandr-master","description":"simple way to build the declarative and destributed data pipelines with python","archived":false,"fork":false,"pushed_at":"2023-09-19T07:50:31.000Z","size":733,"stargazers_count":0,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-28T05:19:18.590Z","etag":null,"topics":["celery","data","pipeline","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aliaksandr-master.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-13T18:19:04.000Z","updated_at":"2022-01-26T11:53:02.000Z","dependencies_parsed_at":"2024-11-21T16:37:05.809Z","dependency_job_id":"3b1d3dcd-c641-4dde-b017-9b5e56d52a01","html_url":"https://github.com/aliaksandr-master/unipipeline","commit_stats":{"total_commits":203,"total_committers":3,"mean_commits":67.66666666666667,"dds":0.009852216748768461,"last_synced_commit":"d5882a524fb60f4250a84e829508b37f8372dc42"},"previous_names":[],"tags_count":61,"template":false,"template_full_name":null,"purl":"pkg:github/aliaksandr-master/unipipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliaksandr-master%2Funipipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliaksandr-master%2Funipipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliaksandr-master%2Funipipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliaksandr-master%2Funipipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aliaksandr-master","download_url":"https://codeload.github.com/aliaksandr-master/unipipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliaksandr-master%2Funipipeline/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264902629,"owners_count":23681099,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","data","pipeline","python"],"created_at":"2024-11-21T16:30:08.101Z","updated_at":"2025-07-11T21:32:08.343Z","avatar_url":"https://github.com/aliaksandr-master.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# unipipeline\nsimple way to build the declarative and distributed data pipelines. \n\n## Why you should use it\n- Declarative strict config\n- Scaffolding\n- Fully typed\n- Python support 3.6+ \n- Brokers support\n    - kafka\n    - rabbitmq\n    - inmemory simple pubsub\n- Interruption handling = safe user code transactions\n- CLI\n\n## How to Install\n```bash\n$ pip3 install unipipeline\n```\n\n## Example\n```yml\n# dag.yml\n---\n\nservice:\n  name: \"example\"\n  echo_colors: true\n  echo_level: error\n\n\nexternal:\n  service_name: {}\n\n\nbrokers:\n  default_broker:\n    import_template: \"unipipeline.brokers.uni_memory_broker:UniMemoryBroker\"\n\n  ender_broker:\n    import_template: \"example.brokers.uni_log_broker:LogBroker\"\n\n\nmessages:\n  __default__:\n    import_template: \"example.messages.{{name}}:{{name|camel}}\"\n\n  input_message: {}\n\n  inetermediate_message: {}\n\n  ender_message: {}\n\n\ncron:\n  my_super_task:\n    worker: my_super_cron_worker\n    when: 0/1 * * * *\n\n  my_mega_task:\n    worker: my_super_cron_worker\n    when: 0/2 * * * *\n\n  my_puper_task:\n    worker: my_super_cron_worker\n    when: 0/3 * * * *\n\n\nwaitings:\n  __default__:\n    import_template: example.waitings.{{name}}_wating:{{name|camel}}Waiting\n\n  common_db: {}\n\n\nworkers:\n  __default__:\n    import_template: \"example.workers.{{name}}:{{name|camel}}\"\n\n  my_super_cron_worker:\n    input_message: uni_cron_message\n\n  input_worker:\n    input_message: input_message\n    waiting_for:\n      - common_db\n\n  intermediate_first_worker:\n    input_message: inetermediate_message\n    output_workers:\n      - ender_second_worker\n    waiting_for:\n      - common_db\n\n  intermediate_second_worker:\n    input_message: inetermediate_message\n    external: service_name\n    output_workers:\n      - ender_frist_worker\n\n  ender_frist_worker:\n    input_message: ender_message\n\n  ender_second_worker:\n    input_message: ender_message\n    broker: ender_broker\n    waiting_for:\n      - common_db\n```\n\n## Get Started\n\n1) create `./unipipeline.yml` such as example above\n\n2) run cli command\n```shell\nunipipeline -f ./unipipeline.yml scaffold\n```\nIt should create all structure of your workers, brokers and so on\n\n3) remove error raising from workers\n\n4) correct message structure for make more usefull\n\n5) correct broker connection (if need)\n\n6) run cli command to run your consumer\n```shell\nunipipeline -f ./unipipeline.yml consume input_worker\n```\nor with python\n```python\nfrom unipipeline import Uni\nu = Uni(f'./unipipeline.yml')\nu.init_consumer_worker(f'input_worker')\nu.initialize()\nu.start_consuming()\n```\n\n7) produce some message to the message broker by your self or with tools\n```shell\nunipipeline -f ./unipipeline.yml produce --worker input_worker --data='{\"some\": \"prop\"}'\n```\nor with python\n```python\n# main.py\nfrom unipipeline import Uni\n\nu = Uni(f'./unipipeline.yml')\nu.init_producer_worker(f'input_worker')\nu.initialize()\nu.send_to(f'input_worker', dict(some='prop'))\n```\n\n## Definition\n\n### Service\n```yaml\nservice:\n  name: some_name       # need for health-check file name\n  echo_level: warning   # level of uni console logs (debug, info, warning, error)\n  echo_colors: true     # show colors in console\n```\n\n### External\n```yml\nexternal:\n  some_name_of_external_service: {}\n```\n- no props\n\n- it needs for declarative grouping the external workers with service\n\n### Worker\n```yaml\nworkers:\n  __default__:                                        # each worker get this default props if defined\n    retry_max_count: 10\n    \n  some_worker_name:\n    retry_max_count: 3                                # just counter. message move to /dev/null if limit has reached \n    retry_delay_s: 1                                  # delay before retry\n    topic: \"{{name}}\"                                 # template string\n    error_payload_topic: \"{{topic}}__error__payload\"  # template string\n    error_topic: \"{{topic}}__error\"                   # template string\n    broker: \"default_broker\"                          # broker name. reference to message transport \n    external: null                                    # name of external service. reference in this config file \n    ack_after_success: true                           # automatic ack after process message\n    waiting_for:                                      # list of references\n      - some_waiting_name                             # name of block. this worker must wait for connection of this external service if need\n    output_workers:                                   # list of references\n      - some_other_worker_name                        # allow worker sending messages to this worker\n    \n    inport_template: \"some.module.hierarchy.to.worker.{{name}}:{{name|camel}}OfClass\"   # required module and classname for import\n\n    input_message: \"name_of_message\"                  # required reference of input message type \n```\n\n### Waiting\n```yaml\nwaitings:\n  some_blocked_service_name:\n    retry_max_count: 3                         # the same semantic as worker.retry_max_count\n    retry_delay_s: 10                          # the same semantic as worker.retry_delay_s\n    import_template: \"some.module:SomeClass\"   # required. the same semantic as worker.import_template\n```\n\n### Broker\n```yaml\nbrokers:\n  some_name_of_broker:\n    retry_max_count: 3                         # the same semantic as worker.retry_max_count\n    retry_delay_s: 10                          # the same semantic as worker.retry_delay_s\n    content_type: application/json             # content type\n    compression: null                          # compression (null, application/x-gzip, application/x-bz2, application/x-lzma)\n    import_template: \"some.module:SomeClass\"   # required. the same semantic as worker.import_template\n```\n\n### Message\n```yaml\nmessages:\n  name_of_message:\n    import_template: \"some.module:SomeClass\"   # required. the same semantic as worker.import_template\n```\n\nbuild in messages:\n```yaml\nmessages:\n  uni_cron_message:\n    import_template: unipipeline.messages.uni_cron_message:UniCronMessage\n```\n\n## CLI\n\n### unipipeline\n```shell \nusage: unipipeline --help\n\nUNIPIPELINE: simple way to build the declarative and distributed data pipelines. this is cli tool for unipipeline\n\npositional arguments:\n  {check,scaffold,init,consume,cron,produce}\n                        sub-commands\n    check               check loading of all modules\n    scaffold            create all modules and classes if it is absent. no args\n    init                initialize broker topics for workers\n    consume             start consuming workers. connect to brokers and waiting for messages\n    cron                start cron jobs, That defined in config file\n    produce             publish message to broker. send it to worker\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --config-file CONFIG_FILE, -f CONFIG_FILE\n                        path to unipipeline config file (default: ./unipipeline.yml)\n  --verbose [VERBOSE]   verbose output (default: false)\n\n```\n\n### unipipeline check\n```\nusage: \n    unipipeline -f ./unipipeline.yml check\n    unipipeline -f ./unipipeline.yml --verbose=yes check\n\ncheck loading of all modules\n\noptional arguments:\n  -h, --help  show this help message and exit\n```\n\n### unipipeline init\n```\nusage: \n    unipipeline -f ./unipipeline.yml init\n    unipipeline -f ./unipipeline.yml --verbose=yes init\n    unipipeline -f ./unipipeline.yml --verbose=yes init --workers some_worker_name_01 some_worker_name_02\n\ninitialize broker topics for workers\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --workers INIT_WORKERS [INIT_WORKERS ...], -w INIT_WORKERS [INIT_WORKERS ...]\n                        workers list for initialization (default: [])\n```\n\n\n### unipipeline scaffold\n```\nusage: \n    unipipeline -f ./unipipeline.yml scaffold\n    unipipeline -f ./unipipeline.yml --verbose=yes scaffold\n\ncreate all modules and classes if it is absent. no args\n\noptional arguments:\n  -h, --help  show this help message and exit\n```\n\n### unipipeline consume\n\n```\nusage: \n    unipipeline -f ./unipipeline.yml consume\n    unipipeline -f ./unipipeline.yml --verbose=yes consume\n    unipipeline -f ./unipipeline.yml consume --workers some_worker_name_01 some_worker_name_02\n    unipipeline -f ./unipipeline.yml --verbose=yes consume --workers some_worker_name_01 some_worker_name_02\n\nstart consuming workers. connect to brokers and waiting for messages\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --workers CONSUME_WORKERS [CONSUME_WORKERS ...], -w CONSUME_WORKERS [CONSUME_WORKERS ...]\n                        worker list for consuming\n```\n\n### unipipeline produce\n```\nusage: \n    unipipeline -f ./unipipeline.yml produce --worker some_worker_name_01 --data {\"some\": \"json\", \"value\": \"for worker\"}\n    unipipeline -f ./unipipeline.yml --verbose=yes produce --worker some_worker_name_01 --data {\"some\": \"json\", \"value\": \"for worker\"}\n    unipipeline -f ./unipipeline.yml produce --alone --worker some_worker_name_01 --data {\"some\": \"json\", \"value\": \"for worker\"}\n    unipipeline -f ./unipipeline.yml --verbose=yes produce --alone --worker some_worker_name_01 --data {\"some\": \"json\", \"value\": \"for worker\"}\n\npublish message to broker. send it to worker\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --alone [PRODUCE_ALONE], -a [PRODUCE_ALONE]\n                        message will be sent only if topic is empty\n  --worker PRODUCE_WORKER, -w PRODUCE_WORKER\n                        worker recipient\n  --data PRODUCE_DATA, -d PRODUCE_DATA\n                        data for sending\n```\n\n### unipipeline cron\n```\nusage: \n    unipipeline -f ./unipipeline.yml cron\n    unipipeline -f ./unipipeline.yml --verbose=yes cron\n\nstart cron jobs, That defined in config file\n\noptional arguments:\n  -h, --help  show this help message and exit\n```\n\n\n## Contributing\n\n### TODO LIST\n0. [ ] RPC Gateways: http, tcp, udp\n1. [ ] Close/Exit uni by call method\n2. [ ] Async producer\n3. [ ] Common Error Handling\n4. [ ] Async get_answer\n5. [ ] Server of Message layout\n6. [ ] Prometheus api\n7. [ ] req/res Sdk\n8. [ ] request tasks result registry\n9. [ ] Async consumer\n10. [ ] Async by default\n11. [ ] Multi-threading start with run-groups\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faliaksandr-master%2Funipipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faliaksandr-master%2Funipipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faliaksandr-master%2Funipipeline/lists"}