{"id":19747641,"url":"https://github.com/codingmiao/datax-simple","last_synced_at":"2025-04-30T08:32:40.457Z","repository":{"id":37135899,"uuid":"243218579","full_name":"codingmiao/datax-simple","owner":"codingmiao","description":"一个datax的简化工具，帮助您轻量级的完成datax的二次开发和同步任务","archived":false,"fork":false,"pushed_at":"2022-09-22T19:05:30.000Z","size":289,"stargazers_count":10,"open_issues_count":12,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-04T07:17:39.138Z","etag":null,"topics":["datax"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codingmiao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-26T09:07:03.000Z","updated_at":"2022-10-05T08:39:04.000Z","dependencies_parsed_at":"2023-01-17T14:01:59.147Z","dependency_job_id":null,"html_url":"https://github.com/codingmiao/datax-simple","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingmiao%2Fdatax-simple","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingmiao%2Fdatax-simple/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingmiao%2Fdatax-simple/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codingmiao%2Fdatax-simple/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codingmiao","download_url":"https://codeload.github.com/codingmiao/datax-simple/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224203064,"owners_count":17272874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datax"],"created_at":"2024-11-12T02:18:24.947Z","updated_at":"2024-11-12T02:18:25.439Z","avatar_url":"https://github.com/codingmiao.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Datax-logo](https://github.com/alibaba/DataX/blob/master/images/DataX-logo.jpg)\n\n# 项目简介\n\nDataX是一款十分优秀的数据同步工具，以插件的方式集成了大量数据源的读写。\n\n但大部分情况下，我们不需要如此多种数据源的支持，例如，我目前的工作仅需要oracle、postgresql两种数据源的互转。\n也就是说，数据源的类型总是确定的，不需要以插件的形式动态加入，太多的插件反而显得有些重了。\n\n除了插件动态引入数据源的问题，datax还需要assembly打包支持，这给调试及二次开发带来了一定难度。\n\n另一方面，datax通过py脚本来运行的方式，也不太好和其它纯java任务直接配合。\n\n所以，本项目(datax-simple)对datax进行了精简，去掉了assembly,并将插件直接放在pom的dependencies下，得到了一个固定数据源的datax模板。\n\n本项目支持oracle、postgresql的读写，您可根据自己的需要，参考说明文档对支持的数据源类型进行替换、增减。\n\n# 适用场景\n\n如果你有如下需求，这个项目应该适合你：\n\n1、需要扩展、改造datax，或者单纯的想跑一下源码熟悉效果，但是又对assembly不熟悉(比如我o(╯□╰)o），那这个项目可以帮助你以无插件的maven工程来编写、调试代码；\n\n2、数据源种类确定，需要一个只包含指定数据源的精简包，那你可以在这个项目上随意增减数据源插件并直接调试；\n\n3、希望以一个纯java任务中搞定同步的全部逻辑，比如同步完后用javamail发一封邮件通知，那你可以方便地在Engine类里添加发邮件等代码。\n\n\n\n# 使用步骤\n\n## 0、下载本项目源码\n\no(*￣︶￣*)o\n\n## 1、更换需要的数据源\n本项目支持oracle、postgresql的读写，同时为了方便测试，也保留了steam数据源。\n所以如果您的同步需求刚好也是oracle和postgresql，您可跳过此步骤。\n\n从[DataX](https://github.com/alibaba/DataX)库下载源码，选择需要的module并加入datax-simple。\n\n#### 示例\n例如，需要从mysql中抽取数据。\n\n1）复制mysqlreader工程到项目中，复制完后，您的datax-simple结构如下:\n\n```\ndatax\n    - common\n    - core\n    - mysqlreader\n    - oraclereader\n    - .....\n    - pom.xml\n```\n\n2）在根目录的pom文件中添加mysqlreader\n\n```\n    \u003cmodules\u003e\n        \u003cmodule\u003ecommon\u003c/module\u003e\n        \u003cmodule\u003ecore\u003c/module\u003e\n        \u003cmodule\u003etransformer\u003c/module\u003e\n\n        \u003c!-- reader --\u003e\n        \u003cmodule\u003emysqlreader\u003c/module\u003e\n        \u003cmodule\u003epostgresqlreader\u003c/module\u003e\n        \u003cmodule\u003estreamreader\u003c/module\u003e\n        \u003cmodule\u003eoraclereader\u003c/module\u003e\n        \u003c!-- writer --\u003e\n        \u003cmodule\u003estreamwriter\u003c/module\u003e\n        \u003cmodule\u003epostgresqlwriter\u003c/module\u003e\n        \u003cmodule\u003eoraclewriter\u003c/module\u003e\n        \u003c!-- common support module --\u003e\n        \u003cmodule\u003eplugin-rdbms-util\u003c/module\u003e\n        \u003cmodule\u003eplugin-unstructured-storage-util\u003c/module\u003e\n    \u003c/modules\u003e\n\n```\n\n3)在core模块的pom文件中加入mysqlreader的依赖\n```\n \u003cdependencies\u003e\n        ....\n        \u003c!-- 需要用到的读写插件、可自行增删 --\u003e\n        \u003cdependency\u003e\n            \u003cartifactId\u003emysqlreader\u003c/artifactId\u003e\n            \u003cgroupId\u003ecom.alibaba.datax\u003c/groupId\u003e\n            \u003cversion\u003e${datax-project-version}\u003c/version\u003e\n            \u003cexclusions\u003e\n                \u003cexclusion\u003e\n                    \u003cartifactId\u003eslf4j-log4j12\u003c/artifactId\u003e\n                    \u003cgroupId\u003eorg.slf4j\u003c/groupId\u003e\n                \u003c/exclusion\u003e\n            \u003c/exclusions\u003e\n        \u003c/dependency\u003e\n        ....\n\n    \u003c/dependencies\u003e\n```\n4)打开目录 core/src/main/resources/datax/plugin，添加对应的插件配置信息到reader/writer文件夹\n\n本示例我们新建文件夹``core/src/main/resources/datax/plugin/reader/mysqlreader``,\n将``\\DataX源码\\mysqlreader\\src\\main\\resources``\n下的两个json到此文件夹，此时，文件结构为\n```\ncore/src/main/resources/datax/plugin\n    - reader\n        - mysqlreader\n            - pluhin.jdon\n            - plugin_job_template.json\n        - oraclereader\n        ...\n    - writer\n        - ...\n```\n\n## 2、运行\n先按照datax的编写规范，编写几个job json，然后把json的路径分行写入\n``\\core\\src\\main\\resources\\datax\\tasks.md``\n\n运行com.alibaba.datax.core.Engine类即可~\no(*￣︶￣*)o\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingmiao%2Fdatax-simple","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodingmiao%2Fdatax-simple","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodingmiao%2Fdatax-simple/lists"}