{"id":51768656,"url":"https://github.com/datayoga-io/datayoga","last_synced_at":"2026-07-19T21:46:15.651Z","repository":{"id":47987590,"uuid":"503227821","full_name":"datayoga-io/datayoga","owner":"datayoga-io","description":"streaming data pipeline platform","archived":false,"fork":false,"pushed_at":"2026-05-28T16:10:19.000Z","size":2883,"stargazers_count":30,"open_issues_count":76,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-05-28T16:21:40.746Z","etag":null,"topics":["cassandra","cdc","data","database","elt","etl","kafka","pipeline","python","redis","singer","sql","sqlalchemy","transformation"],"latest_commit_sha":null,"homepage":"https://datayoga-io.github.io/datayoga/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datayoga-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-06-14T05:59:05.000Z","updated_at":"2026-04-07T14:05:15.000Z","dependencies_parsed_at":"2024-02-18T09:30:48.251Z","dependency_job_id":"4621e880-edd1-4601-9d14-c81849d33ef6","html_url":"https://github.com/datayoga-io/datayoga","commit_stats":{"total_commits":966,"total_committers":7,"mean_commits":138.0,"dds":0.55175983436853,"last_synced_commit":"73f07e7d4e6f1419ca4c2d31edb46365dfcc5bf3"},"previous_names":[],"tags_count":189,"template":false,"template_full_name":null,"purl":"pkg:github/datayoga-io/datayoga","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datayoga-io%2Fdatayoga","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datayoga-io%2Fdatayoga/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datayoga-io%2Fdatayoga/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datayoga-io%2Fdatayoga/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datayoga-io","download_url":"https://codeload.github.com/datayoga-io/datayoga/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datayoga-io%2Fdatayoga/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35666469,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-19T02:00:06.923Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","cdc","data","database","elt","etl","kafka","pipeline","python","redis","singer","sql","sqlalchemy","transformation"],"created_at":"2026-07-19T21:46:15.036Z","updated_at":"2026-07-19T21:46:15.645Z","avatar_url":"https://github.com/datayoga-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nDataYoga is a framework for building and running streaming or batch data pipelines. DataYoga uses low-code to easily define data pipelines using a declarative markup language using YAML files.\n\n![PyPI - License](https://img.shields.io/pypi/l/datayoga)\n![PyPI](https://img.shields.io/pypi/v/datayoga)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/datayoga)\n\n![DataYoga overview](docs/images/datayoga-overview.png \"DataYoga Overview\")\n\n## Concepts\n\n\u003cimg align=\"right\" src=\"docs/images/datayoga-job.png\" /\u003e\n\n`Job` - A Job is composed of a series of `Steps` that reads information from a source, performs transformations, and write to a target. Many sources and targets are supported, including relational databases, non relational databases, file formats, cloud storage, and HTTP servers.\n\n`Step` - Each Step runs a `Block` that uses specific business logic. The output of each Step is fed into the next Step, creating a chain of transformations.\n\n`Blocks` - The Block defines the business logic. Blocks can:\n\n- Read and write from relational and non relational databases\n- Read, write, and parse data from local storage and cloud storage\n- Perform transformations, modify structure, add computed fields, rename fields, or remove fields\n- Enrich data from external sources and APIs\n\n## DataYoga Runtime\n\nDataYoga provides a standalone stream processing engine, the DataYoga Runtime that validates and run Transformation Jobs. The Runtime provides:\n\n- Validation\n- Error handling\n- Metrics and observability\n- Credentials management\n\nThe Runtime supports multiple stream [processing strategies](docs/processing-strategies.md) including buffering and rate limiting.\nIt supports both async processing, multi-threading, and multi-processing to enable maximum throughput with a low footprint.\n\n## Quickstart\n\n```bash\npip install datayoga\n```\n\nVerify that the installation completed successfully by running this command:\n\n```bash\ndatayoga --version\n```\n\n## Create New DataYoga Project\n\nTo create a new DataYoga project, use the `init` command:\n\n```bash\ndatayoga init hello_world\ncd hello_world\n```\n\n\u003e [Directory structure](https://datayoga-io.github.io/datayoga/directory-structure.html)\n\n## Run Your First Job\n\nLet's run our first job. It is pre-defined in the samples folder as part of the `init` command:\n\n```bash\ndatayoga run sample.hello\n```\n\nIf all goes well, you should see some startup logs, and eventually:\n\n```yaml\n{\"id\": \"1\", \"fname\": \"john\", \"lname\": \"doe\", \"credit_card\": \"1234-1234-1234-1234\", \"country_code\": \"972\", \"country_name\": \"israel\", \"gender\": \"M\", \"full_name\": \"John Doe\", \"greeting\": \"Hello Mr. John Doe\"}\n{\"id\": \"2\", \"fname\": \"jane\", \"lname\": \"doe\", \"credit_card\": \"1000-2000-3000-4000\", \"country_code\": \"972\", \"country_name\": \"israel\", \"gender\": \"F\", \"full_name\": \"Jane Doe\", \"greeting\": \"Hello Ms. Jane Doe\"}\n{\"id\": \"3\", \"fname\": \"bill\", \"lname\": \"adams\", \"credit_card\": \"9999-8888-7777-666\", \"country_code\": \"1\", \"country_name\": \"usa\", \"gender\": \"M\", \"full_name\": \"Bill Adams\", \"greeting\": \"Hello Mr. Bill Adams\"}\n```\n\nThat's it! You've created your first job that loads data from CSV, runs it through a series of transformation steps, and shows the data to the standard output. A good start. Read on for a more detailed tutorial or check out the [reference](https://datayoga-io.github.io/datayoga/reference/blocks.html) to see the different block types currently available.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatayoga-io%2Fdatayoga","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatayoga-io%2Fdatayoga","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatayoga-io%2Fdatayoga/lists"}