{"id":24671884,"url":"https://github.com/Breeze0806/go-etl","last_synced_at":"2025-10-08T07:30:36.732Z","repository":{"id":40329862,"uuid":"317124708","full_name":"Breeze0806/go-etl","owner":"Breeze0806","description":"go-etl is a toolset for data extraction, transformation and loading.","archived":false,"fork":false,"pushed_at":"2025-09-30T08:51:51.000Z","size":1284,"stargazers_count":171,"open_issues_count":4,"forks_count":51,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-09-30T10:29:51.796Z","etag":null,"topics":["csv","db2","etl","excel","go","golang","greenplum","ibmdb2","mariadb","microsoft-sql-server","migration","mysql","oracle","postgresql","sql","sql-server","sqlite3","tidb","xlsx"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Breeze0806.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"Authors","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-11-30T05:55:33.000Z","updated_at":"2025-09-30T09:16:16.000Z","dependencies_parsed_at":"2024-02-20T13:48:55.392Z","dependency_job_id":"73abc651-5b19-4d12-a2de-430cf6de52e5","html_url":"https://github.com/Breeze0806/go-etl","commit_stats":{"total_commits":143,"total_committers":5,"mean_commits":28.6,"dds":0.3706293706293706,"last_synced_commit":"009d884f80e75acecaf3aac50b4e5fba99cd6885"},"previous_names":["breeze0806/go-datax"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/Breeze0806/go-etl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Breeze0806%2Fgo-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Breeze0806%2Fgo-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Breeze0806%2Fgo-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Breeze0806%2Fgo-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Breeze0806","download_url":"https://codeload.github.com/Breeze0806/go-etl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Breeze0806%2Fgo-etl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278908564,"owners_count":26066882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","db2","etl","excel","go","golang","greenplum","ibmdb2","mariadb","microsoft-sql-server","migration","mysql","oracle","postgresql","sql","sql-server","sqlite3","tidb","xlsx"],"created_at":"2025-01-26T10:12:05.426Z","updated_at":"2025-10-08T07:30:36.727Z","avatar_url":"https://github.com/Breeze0806.png","language":"Go","funding_links":[],"categories":["流处理","Recently Updated","Stream Processing"],"sub_categories":["HTTP客户端","[Feb 10, 2025](/content/2025/02/10/README.md)","HTTP Clients","Other Software"],"readme":"# go-etl\n[![LICENSE][license-img]][license]\n[![Language][lang-img]][lang]\n[![Build][ci-img]][ci]\n[![Go Report Card][report-img]][report]\n[![GitHub release][release-img]][release]\n[![GitHub release date][release-date-img]][release-date]\n[![Coverage Status][cov-img]][cov]\n[![GoDoc][doc-img]][doc]\n[![Mentioned in Awesome Go][awesome-img]][awesome]\n[![Docker Version][docker-img]][docker]\n\nEnglish | [简体中文](README_zh-CN.md)\n\ngo-etl is a toolset for extracting, transforming, and loading data sources, providing powerful data synchronization capabilities.\n\ngo-etl will provide the following ETL capabilities:\n\n- The ability to extract and load data from mainstream databases is implemented in the storage package\n- The ability to extract and load data from data streams in a two-dimensional table-like format is implemented in the stream package\n- Similar data synchronization capabilities to datax, implemented in the datax package\n\nSince I have limited energy, everyone is welcome to submit issues to discuss go-etl, let's make progress together!\n\n\n## Data Synchronization Tool\n\nThis data synchronization tool has the synchronization capability for the following data sources.\n\n| Type         | Data Source             | Reader | Writer| Document                                                        |\n| ------------ | ------------------ | ------------ | ---------- | ------------------------------------------------------------ |\n| Relational Database | MySQL/Mariadb/Tidb | √            | √          | [Read](datax/plugin/reader/mysql/README.md)、[Write](datax/plugin/writer/mysql/README.md) |\n|              | Postgres/Greenplum | √            | √          | [Read](datax/plugin/reader/postgres/README.md)、[Write](datax/plugin/writer/postgres/README.md) |\n|              | DB2 LUW            | √            | √          | [Read](datax/plugin/reader/db2/README.md)、[Write](datax/plugin/writer/db2/README.md) |\n|              | SQL Server            | √            | √          | [Read](datax/plugin/reader/sqlserver/README.md)、[Write](datax/plugin/writer/sqlserver/README.md) |\n|              | Oracle            | √            | √          | [Read](datax/plugin/reader/oracle/README.md)、[Write](datax/plugin/writer/oracle/README.md) |\n|              | Sqlite3            | √            | √          | [Read](datax/plugin/reader/sqlite3/README.md)、[Write](datax/plugin/writer/sqlite3/README.md) |\n| Unstructured Data Stream    | CSV                | √            | √          | [Read](datax/plugin/reader/csv/README.md)、[Write](datax/plugin/writer/csv/README.md) |\n|              | XLSX（excel）      | √            | √          | [Read](datax/plugin/reader/xlsx/README.md)、[Write](datax/plugin/writer/xlsx/README.md) |\n\n### Getting Started\n\n#### Run by testing performance\n\nIf you want to directly obtain performance-related information, you can deploy according to the [Prometheus Monitoring Deployment Manual](docker/README.md) to acquire relevant performance metrics and even performance visualization charts.\n\n#### Run by obtaining the binary program\n\nYou can download the 64-bit binary executable for Windows or Linux operating systems from the \n[latest releases](https://github.com/Breeze0806/go-etl/releases)\n\nStart data synchronization with the [go-etl Data Synchronization User Manual](README_USER.md)\n\n#### Run by obtaining the Docker image\n\n**Pull Docker Image**\n```bash\ndocker pull go-etl:v0.2.2\n```\n\n**Start Container**\n```bash\ndocker run -d -p 6080:6080 --name etl -v /data:/usr/local/go-etl/data go-etl:v0.2.2\n```\n\n**Execute Command in Container**\n```bash\ndocker exec -it etl bash\n```\n\nImportant Note:  \nCurrent version doesn't support direct usage of SQLite3 or Oracle databases. To enable these databases, you need to:  \n1. Download the corresponding ODBC drivers  \n2. Configure environment variables for database connections  \n\n#### Run by source code\n\n##### Linux\n\n###### Compilation dependencies\n\n1. golang 1.20 and later versions\n2. gcc 4.8 and later versions\n\n###### build\n\n```bash\ncd ${GO_PATH}/src\ngit clone https://github.com/Breeze0806/go-etl.git \"github.com/Breeze0806/go-etl\"\ncd github.com/Breeze0806/go-etl\nmake dependencies\nmake release\n```\n\n###### Removing DB2 dependency\n\nBefore compilation, it is necessary to use `export IGNORE_PACKAGES=db2`\n\n```bash\ncd ${GO_PATH}/src\ngit clone https://github.com/Breeze0806/go-etl.git \"github.com/Breeze0806/go-etl\"\ncd github.com/Breeze0806/go-etl\nexport IGNORE_PACKAGES=db2\nmake dependencies\nmake release\n```\n\n##### windows\n\n###### Compilation Dependencies:\n\n1. Mingw-w64 with gcc 7.2.0 or higher is required for compilation.\n2. Golang version 1.20 or above is necessary.\n3. The minimum compilation environment is Windows 7.\n\n###### build\n\n```bash\ncd ${GO_PATH}\\src\ngit clone https://github.com/Breeze0806/go-etl.git \"github.com/Breeze0806/go-etl\"\ncd github.com/Breeze0806/go-etl\nrelease.bat\n```\n\n###### Removing DB2 dependency\n\nBefore compilation, it is necessary to use `set IGNORE_PACKAGES=db2`\n\n```bash\ncd ${GO_PATH}\\src\ngit clone https://github.com/Breeze0806/go-etl.git \"github.com/Breeze0806/go-etl\"\ncd github.com/Breeze0806/go-etl\nset IGNORE_PACKAGES=db2\nrelease.bat\n```\n\n#### Compilation output\n\n```\n    +---go-etl---|---plugin---+---reader--mysql---|--README.md\n    |                        | .......\n    |                        |\n    |                        |---writer--mysql---|--README.md\n    |                        | .......\n    |\n    +---bin----go-etl\n    +---exampales---+---csvpostgres----config.json\n    |               |---db2------------config.json\n    |               | .......\n    |\n    +---README_USER.md\n\n```\n\n+ The datax/plugin directory contains the documentation for various plugins.\n+ The bin directory houses the data synchronization program, named go-etl.\n+ The examples directory includes configuration files for data synchronization in different scenarios.\n+ README_USER is the user manual or guide in English.\n\n#### Run by docker\n\nTo retrieve the `go-etl` project (version v0.2.1), follow these steps:\n\n```bash\n# Clone the repository\ngit clone https://github.com/Breeze0806/go-etl.git\ncd go-etl\n\n# Verify the tag version\ngit describe --abbrev=0 --tags\n```\n\nFor Docker image compilation:\n\n```bash\n# Build Docker image\ndocker build . -t go-etl:v0.2.2\n```\n\nTo start the container:\n\n```bash\n# Run container in detached mode\ndocker run -d -p 6080:6080 --name etl -v /data:/usr/local/go-etl/data go-etl:v0.2.2\n```\n\nTo access the container shell:\n\n```bash\n# Enter the running container\ndocker exec -it etl bash\n```\n\n### Data Synchronization Development Handbook\n\nRefer to the [go-etl Data Synchronization Developer Documentation](datax/README.md) to assist with your development.\n\n## Module Introduction\n### datax\n\nThis package will provide an interface similar to Alibaba's [DataX](https://github.com/alibaba/DataX) to implement an offline data synchronization framework in the Go programming language. The framework will enable users to perform data synchronization tasks efficiently and reliably, leveraging the power and flexibility of the Go language. It may include features such as pluggable data sources and destinations, support for various data formats, and robust error handling mechanisms.\n\n```\nreaderPlugin(reader)—\u003e Framework(Exchanger+Transformer) -\u003ewriterPlugin(riter)  \n```\n\nThe system is built using a Framework + plugin architecture. In this design, the reading and writing of data sources are abstracted into Reader/Writer plugins, which are integrated into the overall synchronization framework.\n\n+ Reader: The Reader module is responsible for data acquisition. It collects data from the source and sends it to the Framework.\n+ Writer: The Writer module handles data writing. It continuously retrieves data from the Framework and writes it to the destination.\n+ Framework: The Framework serves as the connection between the Reader and Writer. It functions as a data transmission channel, handling core technical aspects such as buffering, flow control, concurrency, and data transformation.\nThis architecture allows for flexibility and scalability, as new data sources and destinations can be easily added by developing new Reader and Writer plugins, respectively.\n\nFor detailed information, please refer to the [go-etl Data Synchronization Developer Documentation](datax/README.md). This documentation provides guidance on how to use the go-etl framework for data synchronization, including information on its architecture, plugin system, and how to develop custom Reader and Writer plugins.\n\n### element\nCurrently, the data types and data type conversions in go-etl have been implemented. For more information, please refer to the [go-etl Data Type Descriptions](element/README.md). This documentation provides details on the supported data types, their usage, and how to perform conversions between different types within the go-etl framework.\n\n### storage\n\n#### database\n\nWe have now implemented basic integration for databases, abstracting the database dialect (Dialect) interface. For specific implementation details, please refer to the [Database Storage Developer Guide](storage/database/README.md). This guide provides information on how to work with different database dialects within the framework, allowing for flexible and extensible database support.\n\n#### Stream\n\nPrimarily used for parsing byte streams, such as files, message queues, Elasticsearch, etc. The byte stream format can be CSV, JSON, XML, etc.\n\n##### File\n\nFocused on file parsing, including CSV, Excel, etc. It abstracts the InputStream and OutputStream interfaces. For specific implementation details, refer to the [Developer Guide for Tabular File Storage](storage/stream/file/README.md).\n\n### Tools\n\nA collection of utilities for compilation, adding licenses, etc.\n\n#### DataX\n\n##### Build\n\n```bash\ngo generate ./...\n```\nThis is the build command used to register developer-created reader and writer plugins into the program's code. Additionally, this command inserts compilation information, such as software version, Git version, Go compiler version, and compilation time, into the command line tool.\n\n##### Plugin\n\nA plugin template creation tool for data sources. It's used to create a new reader or writer template, in conjunction with the build command, to reduce the developer's workload.\n\n##### Release\n\nA packaging tool for the data synchronization program and user documentation.\n\n#### License\n\nAutomatically adds a license to Go code files and formats the code using `gofmt -s -w`.\n\n```bash\ngo run tools/license/main.go\n```\n\n[lang-img]:https://img.shields.io/badge/Language-Go-blue.svg\n[lang]:https://golang.org/\n[report-img]:https://goreportcard.com/badge/github.com/Breeze0806/go-etl\n[report]:https://goreportcard.com/report/github.com/Breeze0806/go-etl\n[doc-img]:https://godoc.org/github.com/Breeze0806/go-etl?status.svg\n[doc]:https://godoc.org/github.com/Breeze0806/go-etl\n[license-img]: https://img.shields.io/badge/License-Apache%202.0-blue.svg\n[license]: https://github.com/Breeze0806/go-etl/blob/main/LICENSE\n[ci-img]: https://github.com/Breeze0806/go-etl/actions/workflows/Build.yml/badge.svg\n[ci]: https://github.com/Breeze0806/go-etl/actions/workflows/Build.yml\n[release-img]: https://img.shields.io/github/tag/Breeze0806/go-etl.svg?label=release\n[release]: https://github.com/Breeze0806/go-etl/releases\n[release-date-img]: https://img.shields.io/github/release-date/Breeze0806/go-etl.svg\n[release-date]: https://github.com/Breeze0806/go-etl/releases\n[cov-img]: https://codecov.io/gh/Breeze0806/go-etl/branch/main/graph/badge.svg?token=UGb27Nysga\n[cov]: https://codecov.io/gh/Breeze0806/go-etl\n[awesome-img]:https://awesome.re/mentioned-badge.svg\n[awesome]:https://github.com/avelino/awesome-go\n[docker-img]:https://img.shields.io/docker/v/breeze0806/go-etl?sort=semver\u0026logo=docker\u0026logoColor=white\u0026label=Docker\u0026color=blue\n[docker]:https://hub.docker.com/r/breeze0806/go-etl","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBreeze0806%2Fgo-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBreeze0806%2Fgo-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBreeze0806%2Fgo-etl/lists"}