{"id":26896968,"url":"https://github.com/xuwei95/ezdata","last_synced_at":"2025-04-01T04:02:35.219Z","repository":{"id":188408925,"uuid":"678649909","full_name":"xuwei95/ezdata","owner":"xuwei95","description":"基于python和llm大模型开发的数据处理和任务调度系统。 支持数据源管理，数据模型管理，数据集成，数据查询API接口，低代码自定义数据处理任务模版，单任务及dag任务工作流调度等功能。集成了llm模块实现rag知识库问答，链接各数据源数据进行数据对话问答，交互式数据分析功能。","archived":false,"fork":false,"pushed_at":"2025-03-28T17:25:33.000Z","size":890,"stargazers_count":176,"open_issues_count":1,"forks_count":33,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-28T18:30:05.578Z","etag":null,"topics":["chatdata","etl","llm","python","rag","scheduler","text2sql"],"latest_commit_sha":null,"homepage":"http://www.ezdata.cloud","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xuwei95.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-15T03:22:59.000Z","updated_at":"2025-03-28T17:25:38.000Z","dependencies_parsed_at":"2024-01-08T11:24:54.478Z","dependency_job_id":"f45946e8-d06b-4922-a662-feaed996c14e","html_url":"https://github.com/xuwei95/ezdata","commit_stats":null,"previous_names":["xuwei95/naivedata","xuwei95/ezdata"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuwei95%2Fezdata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuwei95%2Fezdata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuwei95%2Fezdata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xuwei95%2Fezdata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xuwei95","download_url":"https://codeload.github.com/xuwei95/ezdata/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246580468,"owners_count":20800111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatdata","etl","llm","python","rag","scheduler","text2sql"],"created_at":"2025-04-01T04:02:25.716Z","updated_at":"2025-04-01T04:02:35.213Z","avatar_url":"https://github.com/xuwei95.png","language":"Python","readme":"ezdata\n===============\n\n项目介绍\n-----------------------------------\n\nezdata 是基于python后端和vue3前端开发的数据处理分析和任务调度系统。  \n\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/ezdata.gif?raw=true \"在这里输入图片标题\")\n\n其主要功能如下\n- 多数据源管理，支持连接文件，关系型数据库，nosql数据库，时序数据库，图数据库等多种数据源。\n- 各数据源抽象为统一数据模型，支持创建，删除，字段管理，自定义查询取数，封装数据查询api接口等各种功能。\n- 集成chatgpt等llm，支持数据问答功能，使用数据对话方式实现交互式数据分析，ai自动输出数据结论，数据表格，统计报表等内容。\n- 低代码数据集成，可视化处理流中每一步结果，可使用分布式pandas引擎拓展至tb级大型数据集，使用多种内置转换算法或自定义代码快速实现数据传输管道。\n- 单任务和dag任务工作流调度，内置python，shell，数据集成等多种任务模版，也支持使用内置表单引擎和编写动态执行代码自定义任务模版，支持分布式worker执行，任务队列管理，任务失败重试，任务失败告警，任务运行日志及执行历史查看等调度系统功能。\n- 集成低代码数据可视化大屏系统，拖拽设计及快速对接数据api接口。\n\n项目链接\n-----------------------------------\n- 项目官网：  [http://www.ezdata.cloud](http://www.ezdata.cloud)\n\n- 在线演示 ： [在线演示](http://124.220.57.72)\n\n[comment]: \u003c\u003e (- 开发文档：  [主项目文档]\u0026#40;http://www.ezdata.cloud/docs/hello.html\u0026#41;)\n\n项目源码\n-----------------------------------\n| 仓库  | 后端  |前端 | 数据大屏前端   |\n|--------------------|--------------------|--------------------|--------------------|\n| Github | [ezdata](https://github.com/xuwei95/ezdata) | [ezdata_frontend](https://github.com/xuwei95/ezdata_frontend)  | [ezdata_bigscreen](https://github.com/xuwei95/ezdata_bigscreen)  |\n| 码云  | [ezdata](https://gitee.com/xuwei95/ezdata) | [ezdata_frontend](https://gitee.com/xuwei95/ezdata_frontend)  | [ezdata_bigscreen](https://gitee.com/xuwei95/ezdata_bigscreen)  |\n| gitcode  | [ezdata](https://gitcode.com/xuwei95/ezdata) | [ezdata_frontend](https://gitcode.com/xuwei95/ezdata_frontend)  | [ezdata_bigscreen](https://gitcode.com/xuwei95/ezdata_bigscreen)  |\n\n\n系统效果\n----\n##### 主页\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/dashboard.png?raw=true \"在这里输入图片标题\")\n##### 数据源管理\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/datasource.png?raw=true \"在这里输入图片标题\")\n##### 数据模型管理\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/datamodel.png?raw=true \"在这里输入图片标题\")\n##### 数据自定义查询及接口封装\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/data_query.png?raw=true \"在这里输入图片标题\")\n##### 数据对话，交互式数据分析\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/datachat_msg.png?raw=true \"在这里输入图片标题\")\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/datachat_table.png?raw=true \"在这里输入图片标题\")\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/datachat_chart.png?raw=true \"在这里输入图片标题\")\n##### 数据集成\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/etl.png?raw=true \"在这里输入图片标题\")\n##### 任务模版管理\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/task_template.png?raw=true \"在这里输入图片标题\")\n##### 任务调度\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/task_scheduler.png?raw=true \"在这里输入图片标题\")\n##### dag任务工作流\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/dag_detail.png?raw=true \"在这里输入图片标题\")\n##### worker执行节点管理\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/worker_ops.png?raw=true \"在这里输入图片标题\")\n##### 数据可视化大屏\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/bigscreen1.png?raw=true \"在这里输入图片标题\")\n![输入图片说明](https://raw.githubusercontent.com/xuwei95/ezdata_press/master/images/bigscreen2.png?raw=true \"在这里输入图片标题\")\n\n\n\n后端启动\n----\n### 依赖安装\n```\npip install -r requirements.txt -i https://pypi.doubanio.com/simple\n```\n### 系统web接口服务\n```\npython web_api.py\n```\n### 系统任务调度接口服务\n```\npython scheduler_api.py\n```\n## celery相关\n启动worker\n- windows\n```\ncelery -A tasks worker -P eventlet\n```\n- linux\n```\ncelery -A tasks worker\n```\n启动flower\n```\ncelery -A tasks flower\n```\n\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuwei95%2Fezdata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxuwei95%2Fezdata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuwei95%2Fezdata/lists"}