{"id":15041102,"url":"https://github.com/taogeyt/pyetl","last_synced_at":"2025-10-30T19:12:14.284Z","repository":{"id":62580201,"uuid":"102465949","full_name":"taogeYT/pyetl","owner":"taogeYT","description":"python ETL framework","archived":false,"fork":false,"pushed_at":"2021-09-08T15:52:02.000Z","size":134,"stargazers_count":98,"open_issues_count":0,"forks_count":36,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-12-19T09:06:43.708Z","etag":null,"topics":["csv","data-analytics","data-pipeline","data-platform","db","es","etl","etl-process","excel","export","hive","mysql","oracle","python","sql","sqlserver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taogeYT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-05T10:04:34.000Z","updated_at":"2024-11-25T05:39:03.000Z","dependencies_parsed_at":"2022-11-03T21:01:18.896Z","dependency_job_id":null,"html_url":"https://github.com/taogeYT/pyetl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taogeYT%2Fpyetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taogeYT%2Fpyetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taogeYT%2Fpyetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taogeYT%2Fpyetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taogeYT","download_url":"https://codeload.github.com/taogeYT/pyetl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230408170,"owners_count":18220974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","data-analytics","data-pipeline","data-platform","db","es","etl","etl-process","excel","export","hive","mysql","oracle","python","sql","sqlserver"],"created_at":"2024-09-24T20:45:33.054Z","updated_at":"2025-10-30T19:12:14.211Z","avatar_url":"https://github.com/taogeYT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pyetl\n\nPyetl is a **Python 3.6+** ETL framework\n\n## Installation:\n```shell script\npip3 install pyetl\n```\n\n## Example\n\n```python\nimport sqlite3\nimport pymysql\nfrom pyetl import Task, DatabaseReader, DatabaseWriter, ElasticsearchWriter, FileWriter\nsrc = sqlite3.connect(\"file.db\")\nreader = DatabaseReader(src, table_name=\"source_table\")\n# 数据库之间数据同步，表到表传输\ndst = pymysql.connect(host=\"localhost\", user=\"your_user\", password=\"your_password\", db=\"test\")\nwriter = DatabaseWriter(dst, table_name=\"target_table\")\nTask(reader, writer).start()\n# 数据库表导出到文件\nwriter = FileWriter(file_path=\"./\", file_name=\"file.csv\")\nTask(reader, writer).start()\n# 数据库表同步es\nwriter = ElasticsearchWriter(index_name=\"target_index\")\nTask(reader, writer).start()\n```\n\n#### 原始表目标表字段名称不同\n\n```python\nimport sqlite3\nfrom pyetl import Task, DatabaseReader, DatabaseWriter\ncon = sqlite3.connect(\"file.db\")\n# 原始表source_table包含uuid，full_name字段\nreader = DatabaseReader(con, table_name=\"source_table\")\n# 目标表target_table包含id，name字段\nwriter = DatabaseWriter(con, table_name=\"target_table\")\n# columns配置目标表和原始表的字段映射\ncolumns = {\"id\": \"uuid\", \"name\": \"full_name\"}\nTask(reader, writer, columns=columns).start()\n```\n\n#### 添加字段的udf映射，对字段进行规则校验、数据标准化、数据清洗等\n```python\n# functions配置字段的udf映射，如下id转字符串，name去除前后空格\nfunctions={\"id\": str, \"name\": lambda x: x.strip()}\nTask(reader, writer, columns=columns, functions=functions).start()\n```\n\n#### 继承Task，灵活扩展\n\n```python\nimport json\nfrom pyetl import Task, DatabaseReader, DatabaseWriter\nclass NewTask(Task):\n    reader = DatabaseReader(\"sqlite:///db.sqlite3\", table_name=\"source\")\n    writer = DatabaseWriter(\"sqlite:///db.sqlite3\", table_name=\"target\")\n    \n    def get_columns(self):\n        \"\"\"通过函数的方式生成字段映射配置，使用更灵活\"\"\"\n        # 以下示例将数据库中的字段映射配置取出后转字典类型返回\n        sql = \"select columns from task where name='new_task'\"\n        columns = self.writer.db.read_one(sql)[\"columns\"]\n        return json.loads(columns)\n      \n    def get_functions(self):\n        \"\"\"通过函数的方式生成字段的udf映射\"\"\"\n        # 以下示例将每个字段类型都转换为字符串\n        return {col: str for col in self.columns}\n      \n    def apply_function(self, record):\n        \"\"\"数据流中对一整条数据的udf\"\"\"\n        record[\"flag\"] = int(record[\"id\"]) % 2\n        return record\n\n    def before(self):\n        \"\"\"任务开始前要执行的操作, 如初始化任务表，创建目标表等\"\"\"\n        sql = \"create table destination_table(id int, name varchar(100))\"\n        self.writer.db.execute(sql)\n    \n    def after(self):\n        \"\"\"任务完成后要执行的操作，如更新任务状态等\"\"\"\n        sql = \"update task set status='done' where name='new_task'\"\n        self.writer.db.execute(sql)\n\nNewTask().start()\n```\n\n## Reader和Writer\n\n| Reader              | 介绍                       |\n| ------------------- | -------------------------- |\n| DatabaseReader      | 支持所有关系型数据库的读取    |\n| FileReader          | 结构化文本数据读取，如csv文件 |\n| ExcelReader         | Excel表文件读取             |\n| ElasticsearchReader | 读取es索引数据    |\n\n| Writer              | 介绍                       |\n| ------------------- | -------------------------- |\n| DatabaseWriter      | 支持所有关系型数据库的写入    |\n| ElasticsearchWriter | 批量写入数据到es索引         |\n| HiveWriter          | 批量插入hive表              |\n| HiveWriter2         | Load data方式导入hive表（推荐) |\n| FileWriter          | 写入数据到文本文件           |\n\n \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaogeyt%2Fpyetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaogeyt%2Fpyetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaogeyt%2Fpyetl/lists"}