{"id":33179311,"url":"https://github.com/tapdata/tapdata","last_synced_at":"2026-01-14T07:40:36.671Z","repository":{"id":41176289,"uuid":"508558917","full_name":"tapdata/tapdata","owner":"tapdata","description":"Tapdata Live Data Platform Project","archived":false,"fork":false,"pushed_at":"2026-01-12T07:31:13.000Z","size":219174,"stargazers_count":624,"open_issues_count":142,"forks_count":125,"subscribers_count":22,"default_branch":"main","last_synced_at":"2026-01-12T08:05:52.280Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tapdata.png","metadata":{"files":{"readme":"README.md","changelog":"changes/feat-engine-restart-time-check/README.md","contributing":"contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-06-29T05:31:50.000Z","updated_at":"2026-01-12T06:30:44.000Z","dependencies_parsed_at":"2025-12-29T04:06:41.935Z","dependency_job_id":null,"html_url":"https://github.com/tapdata/tapdata","commit_stats":{"total_commits":3996,"total_committers":35,"mean_commits":"114.17142857142858","dds":0.8583583583583584,"last_synced_commit":"16298d43682a564844393db9ee03e781ea51e6f9"},"previous_names":[],"tags_count":97,"template":false,"template_full_name":null,"purl":"pkg:github/tapdata/tapdata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tapdata%2Ftapdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tapdata%2Ftapdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tapdata%2Ftapdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tapdata%2Ftapdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tapdata","download_url":"https://codeload.github.com/tapdata/tapdata/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tapdata%2Ftapdata/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-16T03:00:36.829Z","updated_at":"2026-01-14T07:40:36.659Z","avatar_url":"https://github.com/tapdata.png","language":"Java","funding_links":[],"categories":["数据库中间件"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://tapdata.io/\"\u003e\n\u003cimg src=\"./assets/tapdata-banner.jpg\" width=\"900px\"/\u003e\n\u003c/a\u003e\n\u003cbr/\u003e\u003cbr/\u003e\n\n[![LICENSE](https://img.shields.io/github/license/tapdata/tapdata.svg)](https://github.com/tapdata/tapdata/blob/main/LICENSE)\n[![Contributors](https://img.shields.io/github/contributors/tapdata/tapdata)](https://github.com/tapdata/tapdata/graphs/contributors)\n[![Activity](https://img.shields.io/github/commit-activity/m/tapdata/tapdata)](https://github.com/tapdata/tapdata/pulse)\n[![Release](https://img.shields.io/github/v/tag/tapdata/tapdata.svg?sort=semver)](https://github.com/tapdata/tapdata/releases)\n\n\u003c/div\u003e\n\n---\n\n[![Try Cloud Service](\u003chttps://img.shields.io/badge/-Try%20Cloud%20Service%20%E2%86%92-rgb(255,140,0)?style=for-the-badge\u003e)](https://cloud.tapdata.net)\n[![Official Website](\u003chttps://img.shields.io/badge/-Official%20Website%20%E2%86%92-rgb(59,71,229)?style=for-the-badge\u003e)](https://tapdata.io)\n[![Docs](\u003chttps://img.shields.io/badge/-Online%20Documentation%20%E2%86%92-rgb(0,205,102)?style=for-the-badge\u003e)](https://docs.tapdata.io)\n\n## What is Tapdata ?\nTapdata is a Change Data Capture(CDC) based, Real-Time data integration platform that enables data to be synchronized in real-time among various systems such as databases, SaaS services, applications, and files.\nThe synchronization tasks can be easily built through drag-and-drop operations, from table creation to full and incremental synchronization, all processes are fully automated.\n\n1. [Supported Connectors](https://docs.tapdata.io/introduction/supported-databases)\n2. [FAQ](https://docs.tapdata.io/faq/)\n\nFor more details, please read [docs](https://docs.tapdata.io/)\n\nBased on different distribution strategies and functionalities, Tapdata has three distribution versions: Community Edition, Cloud Service, and Enterprise Edition. All content under this open-source project corresponds to the Community Edition.\n\nFor the differences between the three versions, you can refer to: https://tapdata.io/\n\n## Table of contents\n\u003cp align=\"left\"\u003e\n  1. \u003ca href=\"#key-features\" target=\"_blank\"\u003e\u003cb\u003eKey Features\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  2. \u003ca href=\"#primary-use-cases\"\u003e\u003cb\u003eUse Cases\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  3. \u003ca href=\"#quick-start\" target=\"_blank\"\u003e\u003cb\u003eQuick Start\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  4. \u003ca href=\"#examples\"\u003e\u003cb\u003eExamples\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  5. \u003ca href=\"#license\"\u003e\u003cb\u003eLicense\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  6. \u003ca href=\"#contact-us\"\u003e\u003cb\u003eContact Us\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  7. \u003ca href=\"./building.md\"\u003e\u003cb\u003eBuild From Source\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n  8. \u003ca href=\"./contributing.md\"\u003e\u003cb\u003eContributing\u003c/b\u003e\u003c/a\u003e\u003cbr/\u003e\n\u003c/p\u003e\n\n## Key Features\n### End-To-End Visual UI\n\n|                                                                                                                                                                                                                                                                                                                                                       |                                       |\n| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |\n| \u003cb\u003eFull Process Automation\u003c/b\u003e\u003cbr/\u003eAutomatic table creation, and Automatic switching between Full and Incremental synchronization.\u003cbr/\u003e\u003cbr/\u003e\u003cb\u003eDrag-And-Drop\u003c/b\u003e\u003cbr/\u003eOperate through a visual interface, Easy to use\u003cbr/\u003e\u003cbr/\u003e\u003cb\u003eMonitor\u003c/b\u003e\u003cbr/\u003esync performance, task progress, key events, and logs.                                                                    | \u003cimg src=\"./assets/monitor.jpg\"/\u003e     |\n\n\u003cbr /\u003e\n\n### Lightweight ETL Solution\n\n|                                                                                                                                                                                                                                                                                                                                                       |                                       |\n| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |\n| \u003cb\u003eTable Filter and Mapping\u003c/b\u003e\u003cbr/\u003eRich visual processors that allow table selection, renaming, adding or deleting fields, and renaming.\u003cbr/\u003e\u003cbr/\u003e\u003cb\u003eRecord Transformation\u003c/b\u003e\u003cbr/\u003eEasily use JS/Python code to do transformation\u003cbr/\u003e\u003cbr/\u003e\u003cb\u003eUnion\u003c/b\u003e\u003cbr/\u003eUnion multi tables to one target.                                                   | \u003cimg src=\"./assets/tapdata-etl.jpg\"/\u003e |\n\n\u003cbr /\u003e\n\n### Build materialized views in MongoDB\n\n|                                                                                                                                                                                                                                                                                                                                                       |                                        |\n| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |\n| \u003cb\u003eDocument-style view\u003c/b\u003e\u003cbr/\u003eSupport Embedded documents and arrays, deeply integrated with MongoDB.\u003cbr/\u003e\u003cbr/\u003e\u003cb\u003eMulti-table association\u003c/b\u003e\u003cbr/\u003ehierarchical nesting, and convenient construction of 1:1 and 1:N models.\u003cbr /\u003e\u003cbr/\u003e\u003cb\u003eMulti-table stream merging\u003c/b\u003e\u003cbr/\u003eUnified batch and stream processing, multi-table updates in any order. | \u003cimg src=\"./assets/tapdata-view.jpg\"/\u003e |\n\n\u003cbr /\u003e\n\n## Primary Use Cases\n1. Synchronize data from traditional RDBMS to modern databases such as MongoDB, Elasticsearch (ES), or Redis to support new business uses.\n2. Consolidate data from various databases into a unified data warehouse.\n3. Stream database changes to Kafka.\n4. Perform data synchronization between heterogeneous databases.\n5. Use MongoDB to build your unified data hub.\n \n## Quick Start\n### Start with docker\nTo run the complete service, the basic resource requirements are:\n1. At least 5GB of available memory\n2. At least 20GB of available disk space\n3. At least 1 free CPU core\n\nRUN `docker run -d -p 3030:3030 ghcr.io/tapdata/tapdata:latest`, wait for 3 minutes, then you can get it from http://localhost:3030/, if everything is ok, you can see login page like this:\n\u003cimg src=\"./assets/tapdata-login.jpg\" width=\"900px\"/\u003e\ndefault username is: admin@admin.com, default password is admin\n\n### Start with cloud service\nTapdata service is available in cloud service, you can use fully-managed service, or deploy engine to your private network\n\nTry on https://cloud.tapdata.io/, support google and github account login, free trial, NO credit card needed, start your real-time data journey immediately.\n\n## Examples\n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch4\u003e🗂️ Create Datasource and Test it\u003c/h4\u003e\u003c/summary\u003e\n\n1. Login tapdata platform\n\n2. In the left navigation panel, click Connections\n\n3. On the right side of the page, click Create\n\n4. In the pop-up dialog, search and select MySQL\n\n5. On the page that you are redirected to, follow the instructions below to fill in the connection information for MySQL\n\n\u003cimg src=\"./assets/example-1-create-mysql-connection.jpg\"\u003e\u003c/img\u003e\n\n6. Click Test, make sure all test pass, then click Save\n\n\u003cimg src=\"./assets/example-1-test.jpg\"\u003e\u003c/img\u003e\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch4\u003e🗂️ Sync Data From MySQL To MongoDB\u003c/h4\u003e\u003c/summary\u003e\n\n1. Create MySQL and MongoDB data source\n\n2. In the left navigation panel, click Data Pipelines -\u003e Data Replications\n\n3. On the right side of the page, click Create\n\n4. Drag and drop MySQL and MongoDB data sources onto the canvas\n\n5. Drag a line from the MySQL data source to MongoDB\n\n6. Configure the MySQL data source and select the data tables you want to synchronize\n\n\u003cimg src='./assets/example-2-config-mysql.jpg'\u003e\u003c/img\u003e\n\n7. Click the Save button in the upper right corner, then click the Start button\n\n8. Observe the indicators and events on the task page until data is in sync\n\n\u003cimg src='./assets/example-2-metrics.jpg'\u003e\u003c/img\u003e\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch4\u003e🗂️ MySQL To PostgreSQL with Simple ETL\u003c/h4\u003e\u003c/summary\u003e\n\n1. Create MySQL and PostgreSQL data source\n\n2. In the left navigation panel, click Data Pipelines -\u003e Data Transformation\n\n3. On the right side of the page, click Create\n\n4. Drag and drop MySQL and PostgreSQL data sources onto the canvas\n\n5. Drag a line from the MySQL data source to PostgreSQL\n\n6. Click the plus sign on the connection line and select Field Rename\n\n\u003cimg src='./assets/example-3-field-rename-1.jpg'\u003e\u003c/img\u003e\n\n7. Click Field Rename node, change i_price to price, i_data to data in config form\n\n\u003cimg src='./assets/example-3-field-rename-2.jpg'\u003e\u003c/img\u003e\n\n8. Click the Save button in the upper right corner, then click the Start button\n\n9. Observe the indicators and events on the task page until data is in sync\n\n\u003cimg src='./assets/example-3-metrics.jpg'\u003e\u003c/img\u003e\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch4\u003e🗂️ Making materialized views in MongoDB\u003c/h4\u003e\u003c/summary\u003e\n\nMaterialized view is a special feature of tapdata, You can give full play to the characteristics of MongoDB document database and create the data model you need, try enjoy it !\n\nIn this example, I will make a view using 2 tables in MySQL: order and product, make product as a embedded document of order, here is the step:\n\n1. Create MySQL and MongoDB data source\n\n2. In the left navigation panel, click Data Pipelines -\u003e Data Transformation\n\n3. On the right side of the page, click Create\n\n4. Click mysql data source in left up side, then drag and drop order table and product table onto the canvas\n\n5. Drag and drop \"Master-slave merge\" node in left bottom side onto the canvas\n\n6. Drag a line from the order table to Master-slave merge\n\n7. Drag a line from the product table to Master-slave merge\n\n8. Drag and drop MongoDB data source onto the canvas, and drag a line from the \"Master-slave merge\" node to MongoDB node\n\n\u003cimg src='./assets/example-4-1.jpg'\u003e\u003c/img\u003e\n\n9. Click \"Master-slave merge\" node, then drag product table into order table in the right side in \"Table Name\"\n\n\u003cimg src='./assets/example-4-2.jpg'\u003e\u003c/img\u003e\n\n10. Click \"Master-slave merge\" node, then click product table, config Data write model to \"Match and Merge\", Field write path to \"product\", Association Conditions to \"order_id\" =\u003e \"order_id\", then you can see Schema in bottom changed\n\n11. Click MongoDB node, and config target table name as order_with_product, update condition field config as \"order_id\"\n\n\u003cimg src='./assets/example-4-2.jpg'\u003e\u003c/img\u003e\n\n12. Click the Save button in the upper right corner, then click the Start button\n\n13. Observe the indicators and events on the task page until data is in sync\n\n14. Check collection order_with_product in MongoDB, and you will see the data model\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n    \u003csummary\u003e\u003ch4\u003e🗂️ Data consistency check\u003c/h4\u003e\u003c/summary\u003e\n\nUsing the data verification feature, you can quickly check whether the synchronized data is consistent and accurate\n\n1. In the left navigation panel, click Data Pipelines -\u003e Data Validation\n\n2. On the right side of the page, click Task Consistency Validation\n\n3. Choose 1 task, and valid type choose \"All Fields Validation\", it means system will check all fields for all record\n\n\u003cimg src='./assets/example-5-config.jpg'\u003e\u003c/img\u003e\n\n4. Click Save, then click Execute in the task list\n\n5. Wait validation task finished, click Result in the task list, and check the validation result\n\n\u003cimg src='./assets/example-5-result.jpg'\u003e\u003c/img\u003e\n\n\u003c/details\u003e\n\n## Architecture\n![Alt Text](./assets/architecture.jpg)\n\n## License\nTapdata Community is under the Apache 2.0 license. See the [LICENSE](https://github.com/tapdata/tapdata/blob/main/LICENSE) file for details.\n\n## Contact Us\n- [Send Email](mailto:team@tapdata.io)\n- [Slack channel](https://join.slack.com/t/tapdatacommunity/shared_invite/zt-1biraoxpf-NRTsap0YLlAp99PHIVC9eA)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftapdata%2Ftapdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftapdata%2Ftapdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftapdata%2Ftapdata/lists"}