{"id":13414602,"url":"https://github.com/fluid-cloudnative/fluid","last_synced_at":"2025-05-13T19:13:22.310Z","repository":{"id":36960579,"uuid":"278954673","full_name":"fluid-cloudnative/fluid","owner":"fluid-cloudnative","description":"Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF) ","archived":false,"fork":false,"pushed_at":"2025-04-27T16:11:13.000Z","size":56997,"stargazers_count":1754,"open_issues_count":347,"forks_count":1070,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-04-27T17:24:48.984Z","etag":null,"topics":["ai-framework","alluxio","big-data","data-abstraction","distributed-cache","kubernetes"],"latest_commit_sha":null,"homepage":"https://fluid-cloudnative.github.io/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluid-cloudnative.png","metadata":{"files":{"readme":"README-zh_CN.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":"GOVERNANCE.md","roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-11T22:57:18.000Z","updated_at":"2025-04-27T16:11:18.000Z","dependencies_parsed_at":"2023-01-17T08:31:34.165Z","dependency_job_id":"3ad1f238-fb32-4089-8f37-426b05fe53c1","html_url":"https://github.com/fluid-cloudnative/fluid","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":"kubernetes/kubernetes-template-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluid-cloudnative%2Ffluid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluid-cloudnative%2Ffluid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluid-cloudnative%2Ffluid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluid-cloudnative%2Ffluid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluid-cloudnative","download_url":"https://codeload.github.com/fluid-cloudnative/fluid/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251298817,"owners_count":21567088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-framework","alluxio","big-data","data-abstraction","distributed-cache","kubernetes"],"created_at":"2024-07-30T21:00:30.124Z","updated_at":"2025-04-28T10:48:29.576Z","avatar_url":"https://github.com/fluid-cloudnative.png","language":"Go","funding_links":[],"categories":["Go","Uncategorized","File Storage"],"sub_categories":["Uncategorized"],"readme":"[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![CircleCI](https://circleci.com/gh/fluid-cloudnative/fluid.svg?style=svg)](https://circleci.com/gh/fluid-cloudnative/fluid)\n[![Build Status](https://travis-ci.org/fluid-cloudnative/fluid.svg?branch=master)](https://travis-ci.org/fluid-cloudnative/fluid)\n[![codecov](https://codecov.io/gh/fluid-cloudnative/fluid/branch/master/graph/badge.svg)](https://codecov.io/gh/fluid-cloudnative/fluid)\n[![Go Report Card](https://goreportcard.com/badge/github.com/fluid-cloudnative/fluid)](https://goreportcard.com/report/github.com/fluid-cloudnative/fluid)\n[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/fluid)](https://artifacthub.io/packages/helm/fluid/fluid)\n[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/fluid-cloudnative/fluid/badge)](https://scorecard.dev/viewer/?uri=github.com/fluid-cloudnative/fluid)\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4886/badge)](https://bestpractices.coreinfrastructure.org/projects/4886)\n[![Leaderboard](https://img.shields.io/badge/Fluid-%E6%9F%A5%E7%9C%8B%E8%B4%A1%E7%8C%AE%E6%8E%92%E8%A1%8C%E6%A6%9C-orange)](https://opensource.alibaba.com/contribution_leaderboard/details?projectValue=fluid)\n\n# Fluid\n\n[English](./README.md) | 简体中文\n\n|![更新](static/bell-outline-badge.svg) 最新进展：|\n|------------------|\n|**最新版发布**：2024年4月17日, Fluid v1.0.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|v0.9.0版发布：2023年5月26日, Fluid v0.9.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|v0.8.0版发布：2022年9月3日, Fluid v0.8.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|v0.7.0版发布：2022年3月2日, Fluid v0.7.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|v0.6.0版发布：2021年8月11日, Fluid v0.6.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|v0.5.0版发布：2021年5月16日, Fluid v0.6.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|进入CNCF：2021年4月27日, Fluid通过CNCF Technical Oversight Committee (TOC)投票决定被接受进入CNCF，成为[CNCF Sandbox Project](https://lists.cncf.io/g/cncf-toc/message/5822)。|\n|v0.4.0版发布：2020年11月6日, Fluid v0.6.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n|v0.3.0版发布：2020年10月1日, Fluid v0.6.0  发布! 版本更新介绍详情情参见 [CHANGELOG](CHANGELOG.md)。|\n\n## 什么是Fluid\n\nFluid是一个开源的Kubernetes原生的分布式数据集编排和加速引擎，主要服务于云原生场景下的数据密集型应用，例如大数据应用、AI应用等。\n\nFluid现在是[Cloud Native Computing Foundation](https://cncf.io) (CNCF) 开源基金会旗下的一个沙箱项目。关于Fluid更多的原理性介绍, 可以参见我们的论文: \n\n1. **Rong Gu, Kai Zhang, Zhihao Xu, et al. [Fluid: Dataset Abstraction and Elastic Acceleration for Cloud-native Deep Learning Training Jobs](https://ieeexplore.ieee.org/abstract/document/9835158). IEEE ICDE, pp. 2183-2196, May, 2022. (Conference Version)**\n\n2. **Rong Gu, Zhihao Xu, Yang Che, et al. [High-level Data Abstraction and Elastic Data Caching for Data-intensive AI Applications on Cloud-native Platforms](https://ieeexplore.ieee.org/document/10249214). IEEE TPDS, pp. 2946-2964, Vol 34(11), 2023. (Journal Version)**\n\n\n通过定义数据集资源的抽象，实现如下功能：\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"static/architecture.png\" title=\"architecture\" width=\"60%\" height=\"60%\" alt=\"\"\u003e\n\u003c/div\u003e\n\n## 核心功能\n\n- __数据集抽象原生支持__\n\n  将数据密集型应用所需基础支撑能力功能化，实现数据高效访问并降低多维管理成本\n\n- __可扩展的数据引擎插件__\n\n\t提供统一的访问接口，方便接入第三方存储，通过不同的Runtime实现数据操作\n\n- __自动化的数据操作__\n\n  提供多种操作模式，与自动化运维体系相结合\n\n- __数据弹性与调度__\n\n\t将数据缓存技术和弹性扩缩容、数据亲和性调度能力相结合，提高数据访问性能\n\n- __运行时平台无关__\n\n\t支持原生、边缘、Serverless Kubernetes集群、Kubernetes多集群等多样化环境，适用于混合云场景\n\n## 重要概念\n\n**Dataset**: 数据集是逻辑上相关的一组数据的集合，会被运算引擎使用，比如大数据的Spark，AI场景的TensorFlow。而这些数据智能的应用会创造工业界的核心价值。Dataset的管理实际上也有多个维度，比如安全性，版本管理和数据加速。我们希望从数据加速出发，对于数据集的管理提供支持。\n\n**Runtime**: 实现数据集安全性，版本管理和数据加速等能力的执行引擎，定义了一系列生命周期的接口。可以通过实现这些接口，支持数据集的管理和加速。\n\n## 先决条件\n\n- Kubernetes version \u003e 1.16, 支持CSI\n- Golang 1.18+\n- Helm 3\n\n## 快速开始\n\n你可以通过 [快速开始](docs/zh/userguide/get_started.md) 在Kubernetes集群中测试Fluid.\n\n## 文档\n\n如果需要详细了解Fluid的使用，请参考文档 [docs](docs/README_zh.md)：\n\n- [English](docs/en/TOC.md)\n- [简体中文](docs/zh/TOC.md)\n\n你也可以访问[Fluid主页](https://fluid-cloudnative.github.io)来获取有关文档.\n\n## 快速演示\n\n\u003cdetails\u003e\n\u003csummary\u003e演示 1: 加速文件访问\u003c/summary\u003e\n\u003cpre\u003e\n\u003ca href=\"http://cloud.video.taobao.com/play/u/2987821887/p/1/e/6/t/1/277753111709.mp4\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/2ee9ef7de9eeb386f365a5d10f5defd12f08457f/687474703a2f2f6b756265666c6f772e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f5374617469632f72656d6f74655f66696c655f616363657373696e672e706e67\" alt=\"\" data-canonical-src=\"static/remote_file_accessing.png\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n\u003c/pre\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e演示 2: 加速机器学习\u003c/summary\u003e\n\u003cpre\u003e\n\u003ca href=\"http://cloud.video.taobao.com/play/u/2987821887/p/1/e/6/t/1/277528130570.mp4\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/5688ab788da9f8cd057e32f3764784ce616ff0fd/687474703a2f2f6b756265666c6f772e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f5374617469632f6d616368696e655f6c6561726e696e672e706e67\" alt=\"\" data-canonical-src=\"static/machine_learning.png\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n\u003c/pre\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e演示 3: 加速PVC\u003c/summary\u003e\n\u003cpre\u003e\n\u003ca href=\"http://cloud.video.taobao.com/play/u/2987821887/p/1/e/6/t/1/281779782703.mp4\" rel=\"nofollow\"\u003e\u003cimg src=\"https://camo.githubusercontent.com/7343be344cfebfd53619c1c8a70530ffd43d3d96/68747470733a2f2f696d672e616c6963646e2e636f6d2f696d6765787472612f69342f363030303030303030333331352f4f31434e303164386963425031614d4a614a576a5562725f2121363030303030303030333331352d302d7462766964656f2e6a7067\" alt=\"\" data-canonical-src=\"https://img.alicdn.com/imgextra/i4/6000000003315/O1CN01d8icBP1aMJaJWjUbr_!!6000000003315-0-tbvideo.jpg\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n\u003c/pre\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e演示 4: 数据预热\u003c/summary\u003e\n\u003cpre\u003e\n\u003ca href=\"http://cloud.video.taobao.com/play/u/2987821887/p/1/e/6/t/1/287213603893.mp4\" rel=\"nofollow\"\u003e\u003cimg src=\"https://img.alicdn.com/imgextra/i4/6000000005626/O1CN01JJ9Fb91rQktps7K3R_!!6000000005626-0-tbvideo.jpg\" alt=\"\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n\u003c/pre\u003e\n\u003c/details\u003e\n\n\u003cdetails open\u003e\n\u003csummary\u003e演示 5: 在线不停机数据集缓存扩缩容\u003c/summary\u003e\n\u003cpre\u003e\n\u003ca href=\"http://cloud.video.taobao.com/play/u/2987821887/p/1/e/6/t/1/302459823704.mp4\" rel=\"nofollow\"\u003e\u003cimg src=\"https://img.alicdn.com/imgextra/i4/6000000004852/O1CN013kKkea1liGNWo2DOE_!!6000000004852-0-tbvideo.jpg\" alt=\"\" style=\"max-width:100%;\"\u003e\u003c/a\u003e\n\u003c/pre\u003e\n\u003c/details\u003e\n\n## 如何贡献\n\n欢迎您的贡献，如何贡献请参考[CONTRIBUTING.md](CONTRIBUTING.md).\n\n## 欢迎加入与反馈\n\nFluid让Kubernetes真正具有分布式数据缓存的基础能力，开源只是一个起点，需要大家的共同参与。大家在使用过程发现Bug或需要的Feature，都可以直接在 [GitHub](https://github.com/fluid-cloudnative/fluid)上面提 issue 或 PR，一起参与讨论。另外我们有钉钉与微信交流群，欢迎您的参与和讨论。\n\n钉钉讨论群\n\u003cdiv\u003e\n  \u003cimg src=\"static/dingtalk.png\" width=\"280\" title=\"dingtalk\"\u003e\n\u003c/div\u003e\n\n微信讨论群:\n\n\u003cdiv\u003e\n  \u003cimg src=\"static/wechat.png\" width=\"280\" title=\"dingtalk\"\u003e\n\u003c/div\u003e\n\n微信官方公众号:\n\n\u003cdiv\u003e\n  \u003cimg src=\"https://fluid-imgs.oss-cn-shanghai.aliyuncs.com/public/imgs/wxgzh_code.png\" width=\"280\" title=\"dingtalk\"\u003e\n\u003c/div\u003e\n\nSlack 讨论群\n- 加入 [`CNCF Slack`](https://slack.cncf.io/) 通过搜索频道 ``#fluid`` 和我们进行讨论.\n\n## 开源协议\n\nFluid采用Apache 2.0 license开源协议，详情参见[LICENSE](./LICENSE)文件。\n\n## 漏洞报告\n\n安全性是Fluid项目高度关注的事务。如果您发现或遇到安全相关的问题，欢迎您给fluid.opensource.project@gmail.com邮箱发送邮件报告。具体细节请查看[SECURITY.md](SECURITY.md)。\n\n## 行为准则\n\nFluid 遵守 [CNCF 行为准则](https://github.com/cncf/foundation/blob/master/code-of-conduct.md)。\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluid-cloudnative%2Ffluid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluid-cloudnative%2Ffluid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluid-cloudnative%2Ffluid/lists"}