https://github.com/wgzhao/Addax
A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL seamlessly
https://github.com/wgzhao/Addax
clickhouse database etl excel hadoop hdfs hive impala influxdb kudu mysql oracle postgresql sqlserver trino
Last synced: 7 days ago
JSON representation
A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL seamlessly
- Host: GitHub
- URL: https://github.com/wgzhao/Addax
- Owner: wgzhao
- License: apache-2.0
- Created: 2019-07-17T13:58:01.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2025-04-11T07:28:41.000Z (10 days ago)
- Last Synced: 2025-04-12T00:02:54.270Z (9 days ago)
- Topics: clickhouse, database, etl, excel, hadoop, hdfs, hive, impala, influxdb, kudu, mysql, oracle, postgresql, sqlserver, trino
- Language: Java
- Homepage: https://wgzhao.github.io/Addax/
- Size: 43.9 MB
- Stars: 1,257
- Watchers: 34
- Forks: 309
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Security: SECURITY.md
- Support: support_data_sources.md
Awesome Lists containing this project
- my-awesome - wgzhao/Addax - 04 star:1.3k fork:0.3k A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL seamlessly (Java)
- awesome-clickhouse - wgzhao/Addax - Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration. (Integrations / ETL and Data Processing)
README
Addax
A versatile open-source ETL tool
Addax is an extensible ETL (Extract, Transform, Load) tool supporting over 20 SQL/NoSQL data sources, developed as a fork and evolution of Alibaba's DataX. It provides a growing ecosystem of plugins and offers easy-to-follow configuration for data integrations.
[简体中文](README_zh.md)
---
## 🚀 Features
- Supports 20+ SQL and NoSQL data sources, and easily extendable for more.
- Configurable via simple JSON-based job descriptions.
- Actively maintained with improved architecture and added functionality compared to [DataX](https://github.com/alibaba/datax).
- Docker images for quick deployment.---
## 📚 Documentation
Detailed instructions on installation, configuration, and usage are available:
- [Online Documentation](https://wgzhao.github.io/Addax/)
- [GitHub Docs](https://wgzhao.github.io/Addax)
- [Plugin Comparison](difference.md)---
## 📦 Supported Data Sources
Addax supports a wide range of database systems and file sources. Below is a selection of supported platforms:
> **See the [full list](support_data_sources.md) of supported data sources**.
---
## 🛠️ Getting Started
Addax can be quickly installed and used via Docker, installation scripts, or compiled from source.
### 1. Use docker image
Pull the prebuilt Docker image and run a test job:
```shell
docker pull quay.io/wgzhao/addax:latest
docker run -ti --rm --name addax \
quay.io/wgzhao/addax:latest \
/opt/addax/bin/addax.sh /opt/addax/job/job.json
```### 2. Use installation script
Install Addax with a single command:
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/wgzhao/Addax/master/install.sh)"
```> Installation paths: /usr/local (macOS), /opt/addax/ (Linux).
### 3. Compile and Package from Source
For developers targeting custom-builds, compile and package Addax locally:
```shell
git clone https://github.com/wgzhao/addax.git addax
cd addax
export MAVEN_OPTS="-DskipTests -Dmaven.javadoc.skip=true -Dmaven.source.skip=true -Dgpg.skip=true"
mvn clean package
mvn package -Pdistribution
```The compiled binary will be in `target/addax-`.
### 4. Run Your First Task
Load sample job configuration and test the setup:
```bash
bin/addax.sh job/job.json
```Explore more [example jobs](docs/assets/jobs)
---
## 📖 Runtime Requirements
- **Java Runtime**: JDK 1.8+
- **Python Version**: Python 2.7+ / 3.7+ (Windows only)---
## 🧩 Developing Addax
### Building Documentation
Generate or preview online documentation locally using [mkdocs](https://www.mkdocs.org):
```shell
python3 -m pip install mkdocs-material
mkdocs build
mkdocs serve -a 0.0.0.0:8888
```For publishing:
```shell
export version=4.1.5
git checkout $version
mike deploy -p $version
```---
## Code Style Guidelines
Follow general Java conventions and patterns:
1. Use IntelliJ IDE with [Airlift's Code Style](https://github.com/airlift/codestyle)
2. Categorize exceptions clearly with AddaxException (e.g., `AddaxException(REQUIRE_VALUE, "missing required parameter")`).
3. Use the Java 8 Stream API cautiously (avoid in performance-sensitive areas).
4. Avoid ternary operators for non-trivial expressions.
5. Include proper Apache license headers in every file.> Refer to our [Programming Guidelines](https://cbea.ms/git-commit/) for commit message formats.
---
## 🗓️ Versioning Scheme
This project adheres to the [Semantic Versioning (SemVer)](https://semver.org/) standard with the format `x.y.z`. The meanings of each segment are as follows:
- **z（Patch Version）**:
- Bug fixes and performance improvements that do not affect compatibility with existing features.
- Example: `1.2.3 → 1.2.4`- **y（Minor Version）**:
- Introducing new features or module adjustments that could break backward compatibility.
- Example: `1.2.3 → 1.3.0`- **x（Major Version）**:
- Significant changes or new features that are often incompatible with previous versions.
- Example: `1.3.0 → 2.0.0`---
## 🌟 Star History
## Star History
---
## ⚖️ License
This software is free to use under the [Apache License 2.0](/LICENSE).
---
## 💌 Special Thanks
Special thanks to [JetBrains](https://jb.gg/OpenSource) for providing open-source support to this project.
