{"id":13561905,"url":"https://github.com/artie-labs/transfer","last_synced_at":"2026-04-30T22:01:15.304Z","repository":{"id":65681702,"uuid":"562368887","full_name":"artie-labs/transfer","owner":"artie-labs","description":"Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time. ","archived":false,"fork":false,"pushed_at":"2026-04-28T00:08:14.000Z","size":5991,"stargazers_count":840,"open_issues_count":23,"forks_count":53,"subscribers_count":8,"default_branch":"master","last_synced_at":"2026-04-28T01:24:54.956Z","etag":null,"topics":["apache-kafka","bigquery","cdc","change-data-capture","data-integration","data-pipelines","database","debezium","elt","golang","kafka","redshift","snowflake"],"latest_commit_sha":null,"homepage":"https://artie.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artie-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-11-06T05:50:02.000Z","updated_at":"2026-04-24T23:17:17.000Z","dependencies_parsed_at":"2026-04-17T21:01:25.960Z","dependency_job_id":null,"html_url":"https://github.com/artie-labs/transfer","commit_stats":{"total_commits":926,"total_committers":6,"mean_commits":"154.33333333333334","dds":"0.39848812095032393","last_synced_commit":"0a635d1d5e346dbac3dde964389ec315d41cd46b"},"previous_names":[],"tags_count":1127,"template":false,"template_full_name":null,"purl":"pkg:github/artie-labs/transfer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artie-labs%2Ftransfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artie-labs%2Ftransfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artie-labs%2Ftransfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artie-labs%2Ftransfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artie-labs","download_url":"https://codeload.github.com/artie-labs/transfer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artie-labs%2Ftransfer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32478162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","bigquery","cdc","change-data-capture","data-integration","data-pipelines","database","debezium","elt","golang","kafka","redshift","snowflake"],"created_at":"2024-08-01T13:01:02.434Z","updated_at":"2026-04-30T22:01:15.270Z","avatar_url":"https://github.com/artie-labs.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003ch1\n align=\"center\"\u003e\n \u003cimg\n      align=\"center\"\n      alt=\"Artie Transfer\"\n      src=\"https://github.com/user-attachments/assets/9aa54907-af10-433b-8659-c868d4479f79\"\n      style=\"width:100%;\"\n    /\u003e\n\u003c/h1\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ch3\u003eArtie Transfer\u003c/h3\u003e\n  \u003cp\u003e⚡️ Blazing fast data replication between OLTP and OLAP databases ⚡️\u003c/p\u003e\n  \u003ca href=\"https://artie.com/slack\"\u003e\u003cimg src=\"https://img.shields.io/badge/slack-@artie-blue.svg?logo=slack\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://artie.com/docs/open-source/running-artie/overview\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/4412200/226736695-6b8b9abd-c227-41c7-89a1-805a04c90d08.png\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/artie-labs/transfer/blob/master/LICENSE.txt\"\u003e\u003cimg src=\"https://user-images.githubusercontent.com/4412200/201544613-a7197bc4-8b61-4fc5-bf09-68ee10133fd7.svg\"/\u003e\u003c/a\u003e\n  \u003cimg src=\"https://github.com/artie-labs/transfer/actions/workflows/gha-go-test.yml/badge.svg\"/\u003e\n  \u003cbr/\u003e\n  \u003cb\u003e\u003ca target=\"_blank\" href=\"https://artie.com\" \u003eLearn more »\u003c/a\u003e\u003c/b\u003e\n\u003c/div\u003e\n\u003cbr/\u003e\n\nArtie Transfer is a real-time data replication solution for databases and data warehouses/lakes.\n\nTypical ETL solutions rely on batched processes or schedulers (i.e. DAGs, Airflow), which means the data in the downstream data warehouse is often several hours to days old. This problem is exacerbated as data volumes grow, as batched processes take increasingly longer to run.\n\nArtie leverages change data capture (CDC) and stream processing to perform data syncs in a more efficient way, which enables sub-minute latency.\n\nBenefits of Artie Transfer:\n\n- Sub-minute data latency: always have access to live production data.\n- Ease of use: just set up a simple configuration file, and you're good to go!\n- Automatic table creation and schema detection: Artie infers schemas and automatically merges changes to downstream destinations.\n- Reliability: Artie has automatic retries and processing is idempotent.\n- Scalability: handle anywhere from 1GB to 100+ TB of data.\n- Monitoring: built-in error reporting along with rich telemetry statistics.\n\nTake a look at this [guide](#getting-started) to get started!\n\n## Architecture\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/artie-labs/transfer/assets/4412200/a30a2ee1-7bdd-437c-9acb-ce6591654d18\"/\u003e\n\u003c/div\u003e\n\n## Examples\n\nTo run Artie Transfer's stack locally, please refer to the [examples folder](https://github.com/artie-labs/transfer/tree/master/examples).\n\n## Getting started\n\n[Getting started guide](https://artie.com/docs/open-source/running-artie/overview)\n\n## What is currently supported?\n\nTransfer is aiming to provide coverage across all OLTPs and OLAPs databases. Currently Transfer supports:\n\n- Message Queues\n  - Kafka (default)\n\n- [Destinations](https://artie.com/docs/destinations):\n    - BigQuery\n    - ClickHouse\n    - Databricks\n    - Iceberg\n        * S3Tables\n        * REST catalog\n    - Microsoft SQL Server\n    - MotherDuck\n    - PostgreSQL\n    - Redshift\n    - S3\n    - Snowflake\n    - Amazon SQS\n\n- [Sources](https://artie.com/docs/sources):\n    - DocumentDB\n    - DynamoDB\n    - Microsoft SQL Server\n    - MongoDB\n    - MySQL\n    - Oracle\n    - PostgreSQL\n\n\n_If the database you are using is not on the list, feel free to file for a [feature request](https://github.com/artie-labs/transfer/issues/new)._\n\n## Configuration File\n\n* [Artie Transfer configuration file guide](https://artie.com/docs/open-source/running-artie/options)\n* [Examples of configuration files](https://artie.com/docs/open-source/running-artie/examples)\n\n## Telemetry\n\n[Artie Transfer's telemetry guide](https://www.artie.com/docs/monitoring/available-metrics)\n\n## Tests\nTransfer is written in Go and uses [counterfeiter](https://github.com/maxbrunsfeld/counterfeiter) to mock.\nTo run the tests, run the following commands:\n\n```sh\nmake generate\nmake test\n```\n\n## Release\n\nArtie Transfer is released through [GoReleaser](https://goreleaser.com/), and we use it to cross-compile our binaries on the [releases](https://github.com/artie-labs/transfer/releases) as well as our Dockerhub. If your operating system or architecture is not supported, please file a feature request!\n\n## License\n\nArtie Transfer is licensed under ELv2. Please see the [LICENSE](https://github.com/artie-labs/transfer/blob/master/LICENSE.txt) file for additional information. If you have any licensing questions please email hi@artie.com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartie-labs%2Ftransfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartie-labs%2Ftransfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartie-labs%2Ftransfer/lists"}