{"id":28604925,"url":"https://github.com/supabase/etl","last_synced_at":"2025-06-11T18:03:37.649Z","repository":{"id":221354633,"uuid":"752940677","full_name":"supabase/etl","owner":"supabase","description":"Build Postgres replication apps in Rust","archived":false,"fork":false,"pushed_at":"2025-06-02T14:23:23.000Z","size":1160,"stargazers_count":688,"open_issues_count":44,"forks_count":35,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-06-02T16:37:52.710Z","etag":null,"topics":["logical-replication","postgres","postresql","replication","rust","rust-lang"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/supabase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["supabase"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2024-02-05T06:34:48.000Z","updated_at":"2025-06-01T02:32:02.000Z","dependencies_parsed_at":"2024-03-20T11:54:22.531Z","dependency_job_id":"5f931ae1-a880-4ef7-ae48-b11697523827","html_url":"https://github.com/supabase/etl","commit_stats":{"total_commits":352,"total_committers":3,"mean_commits":"117.33333333333333","dds":0.008522727272727293,"last_synced_commit":"76516a588c481be7d6693711b06ea80dd11cdb5f"},"previous_names":["imor/pg_replicate","supabase/pg_replicate","supabase/supabase_etl","supabase/etl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/supabase","download_url":"https://codeload.github.com/supabase/etl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/supabase%2Fetl/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":257877597,"owners_count":22616713,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["logical-replication","postgres","postresql","replication","rust","rust-lang"],"created_at":"2025-06-11T18:01:45.281Z","updated_at":"2025-06-11T18:03:37.643Z","avatar_url":"https://github.com/supabase.png","language":"Rust","funding_links":["https://github.com/sponsors/supabase"],"categories":["Rust","\u003ca name=\"Rust\"\u003e\u003c/a\u003eRust"],"sub_categories":[],"readme":"# ETL\n\n\u003e **Note:** Version 2 is currently under development under `/v2`, which includes a complete rework of the pipeline \n\u003e architecture for improved performance and scalability.\n\nA Rust crate to quickly build replication solutions for Postgres. It provides building blocks to construct data pipelines which can continually copy data from Postgres to other systems. It builds abstractions on top of Postgres's [logical streaming replication protocol](https://www.postgresql.org/docs/current/protocol-logical-replication.html) and pushes users towards the pit of success without letting them worry about low level details of the protocol.\n\n## Table of Contents\n\n- [Features](#features)\n- [Installation](#installation)\n- [Quickstart](#quickstart)\n- [Examples](#examples)\n- [Database Setup](#database-setup)\n- [Running Tests](#running-tests)\n- [Docker](#docker)\n- [Architecture](#architecture)\n- [Roadmap](#roadmap)\n- [Troubleshooting](#troubleshooting)\n- [License](#license)\n\n## Features\n\nThe `etl` crate supports the following destinations:\n\n- [x] BigQuery\n- [x] DuckDB\n- [x] MotherDuck\n- [ ] Snowflake (planned)\n- [ ] ClickHouse (planned)\n- [ ] Many more to come...\n\nNote: DuckDB and MotherDuck destinations do not use the batched pipeline, hence they currently perform poorly. A batched pipeline version of these destinations is planned.\n\n## Installation\n\nTo use `etl` in your Rust project, add it via a git dependency in `Cargo.toml`:\n\n```toml\n[dependencies]\netl = { git = \"https://github.com/supabase/etl\", features = [\"stdout\"] }\n```\n\nEach destination is behind a feature of the same name, so remember to enable the right feature. The git dependency is needed for now because `etl` is not yet published on crates.io.\n\n## Quickstart\n\nTo quickly try out `etl`, you can run the `stdout` example, which will replicate the data to standard output. First, create a publication in Postgres which includes the tables you want to replicate:\n\n```sql\ncreate publication my_publication\nfor table table1, table2;\n```\n\nThen run the `stdout` example:\n\n```bash\ncargo run -p etl --example stdout --features=\"stdout\" -- --db-host localhost --db-port 5432 --db-name postgres --db-username postgres --db-password password cdc my_publication stdout_slot\n```\n\nIn the above example, `etl` connects to a Postgres database named `postgres` running on `localhost:5432` with a username `postgres` and password `password`. The slot name `stdout_slot` will be created by `etl` automatically.\n\n## Examples\n\nFor code examples on how to use `etl`, please refer to the [examples](https://github.com/supabase/etl/tree/main/etl/examples) folder in the source.\n\n## Database Setup\n\nBefore running the examples, tests, or the API and replicator components, you'll need to set up a PostgreSQL database.\nWe provide a convenient script to help you with this setup. For detailed instructions on how to use the database setup script, please refer to our [Database Setup Guide](docs/database-setup.md).\n\n## Running Tests\n\nTo run the test suite:\n\n```bash\ncargo test\n```\n\n## Docker\n\nThe repository includes Docker support for both the `replicator` and `api` components:\n\n```bash\n# Build replicator image\ndocker build -f ./replicator/Dockerfile .\n\n# Build api image\ndocker build -f ./api/Dockerfile .\n```\n\n## Architecture\n\nFor a detailed explanation of the ETL architecture and design decisions, please refer to our [Design Document](docs/etl-design.md).\n\n## Troubleshooting\n\n### Too Many Open Files Error\n\nIf you see the following error when running tests on macOS:\n\n```\ncalled `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }\n```\n\nRaise the limit of open files per process with:\n\n```bash\nulimit -n 10000\n```\n\n### Performance Considerations\n\nCurrently, the data source and destinations copy table row and CDC events one at a time. This is expected to be slow. Batching and other strategies will likely improve the performance drastically. But at this early stage, the focus is on correctness rather than performance. There are also zero benchmarks at this stage, so commentary about performance is closer to speculation than reality.\n\n## License\n\nDistributed under the Apache-2.0 License. See `LICENSE` for more information.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupabase%2Fetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsupabase%2Fetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsupabase%2Fetl/lists"}