{"id":18524944,"url":"https://github.com/paddlepaddle/paddleflow","last_synced_at":"2025-04-05T11:09:53.674Z","repository":{"id":36966101,"uuid":"436903861","full_name":"PaddlePaddle/PaddleFlow","owner":"PaddlePaddle","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-20T06:56:29.000Z","size":28011,"stargazers_count":120,"open_issues_count":4,"forks_count":40,"subscribers_count":10,"default_branch":"develop","last_synced_at":"2025-03-29T10:07:44.765Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PaddlePaddle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-10T08:24:19.000Z","updated_at":"2025-02-20T06:56:35.000Z","dependencies_parsed_at":"2023-10-11T10:28:59.601Z","dependency_job_id":"d1f6e121-4cc9-4667-ac06-981b45ee5268","html_url":"https://github.com/PaddlePaddle/PaddleFlow","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPaddleFlow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPaddleFlow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPaddleFlow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PaddlePaddle%2FPaddleFlow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PaddlePaddle","download_url":"https://codeload.github.com/PaddlePaddle/PaddleFlow/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325693,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T17:43:52.189Z","updated_at":"2025-04-05T11:09:53.650Z","avatar_url":"https://github.com/PaddlePaddle.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"**PaddleFlow**简称PF，AI资源管理与调度工具。基于云原生Kubernetes或K3s，提供面向AI开发的批量作业执行系统，并且提供易用的共享文件系统，在apache license2.0 开源协议下发布。\nPaddleFlow作为机器学习平台的资源核心，适用于机器学习和深度学习的单机和分布式作业，针对作业依赖的算力和存储资源进行多租户管理和调度；提供缓存编排和加速的能力，降低访问远程存储的频率，提升训练效率；提供工作流用于编排AI作业，模板化AI作业的训练流程，提升实验效率。\n总的来说，PaddleFlow为用户屏蔽各种各样的资源对接和管理细节，让用户编排和执行AI作业更简单，更高效，上手成本更低。\n# 核心特性\n## 1.存储\n- 兼容posix协议，提供高性能fuse文件系统（PaddleFlowFS），内置两层缓存能力，提供API、Fuse、CSI三种使用方式\n- 同时，结合缓存亲和性调度策略提供缓存位置感知能力，将数据本地化，极大提升训练效率\n## 2.调度\n- 基于kubernetes的计算资源池化管理\n- 基于华为开源的volcano的队列调度\n- 内置主流深度学习计算框架引擎（Paddle、Tensorflow等）\n- 支持机器学习和数据分析框架（Spark等）\n## 3.工作流\n- 通过工作流抽象复杂的命令，将其模板化，可被多次运行（支持断点运行和产出Artifact管理）和分享。\n- 提供python客户端，同时支持通过静态Yaml的方式进行作业编排\n- 支持运行在主流的DAG执行引擎上（未来版本），如Argo、Airflow等\n# 架构\nPaddleFlow由四个部分组成：\n- 1.PaddleFlow 客户端（包含PaddleFlowFS）: 命令行工具方便用户在开发机安装和使用，PaddleFlow 客户端（包含PaddleFlowFS）: 命令行工具方便用户在开发机安装和使用，其中PaddleFlowFS基于fuse实现，兼容posix语义，支持AI作业常用的命令，内置缓存能力，加速远端数据读写，同时支持多种数据源的对接，比如BOS等类S3系统，HDFS，本地文件系统等。\n- 2.PaddleFlow server: PaddleFlow核心服务，主要包含队列、存储、工作流等核心功能的管理。\n- 3.volcano（基于开源volcano改造）: 主要增加elastic quota更灵活管理资源的能力，未来会逐步提交社区review。\n- 4.paddleflow-csi-plugin: 基于kubernetes csi插件机制实现了PaddleFlowFS接入并提供fuse客户端的管理能力。\n\n![PaddleFlow 功能架构](docs/zh_cn/images/pf-arch.png) \n\nPaddleFlow的部署主要分为客户端和服务端，其中客户端主要用于准备和打包作业，服务端主要用于作业解析和作业管理，其中执行作业如图中示例主要为kubernetes和k3s。其中，他们会共用一个共享的文件系统，这样会更加方便用户更加直观的查看作业状态和日志等。\n\n![PaddleFlow 部署架构](docs/zh_cn/images/pf-deploy-arch.png)\n\n# 快速部署\n点击[PaddleFlow安装部署](docs/zh_cn/deployment/how_to_install_paddleflow.md)\n里面包含PaddleFlow客户端和服务的部署方式\n# 快速上手\n## 命令行参考\n点击[命令行操作说明](docs/zh_cn/reference/client_command_reference.md) 获取所有操作命令和示例。\n## python sdk参考\n点击[sdk使用说明](docs/zh_cn/reference/sdk_reference/sdk_reference.md) 获取sdk的使用说明。\n## 其他详细参考\n工作流功能使用详解[工作流](docs/zh_cn/reference/pipeline/overview.md) \u003cbr\u003e\n作业功能实用详解[作业](docs/zh_cn/reference/job_reference.md) \u003cbr\u003e\n存储功能实用详解[存储](docs/zh_cn/reference/filesystem/filesystem_overview.md) \u003cbr\u003e\nCSI下存储挂载点恢复[挂载点恢复](docs/zh_cn/reference/filesystem/csi_mountpoint_recovery.md) \u003cbr\u003e\n# 开源协议\n使用 apache license 2.0开源，详见 LICENSE。\n# PaddlePaddle相关能力使用\n待补充。\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaddlepaddle%2Fpaddleflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaddlepaddle%2Fpaddleflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaddlepaddle%2Fpaddleflow/lists"}