{"id":30151785,"url":"https://github.com/curvineio/curvine","last_synced_at":"2025-08-11T11:09:31.504Z","repository":{"id":302039412,"uuid":"991109362","full_name":"CurvineIO/curvine","owner":"CurvineIO","description":"High performance distributed cache system. Built by Rust.","archived":false,"fork":false,"pushed_at":"2025-08-07T10:15:58.000Z","size":1153,"stargazers_count":298,"open_issues_count":35,"forks_count":37,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-08-07T10:25:02.030Z","etag":null,"topics":["ai","ai-infra","bigdata","cache-storage","cloud-native","hdfs","high-performance-computing","io","rust","s3","shuffle","spark","train-acceleration"],"latest_commit_sha":null,"homepage":"https://curvineio.github.io","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CurvineIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-27T06:24:21.000Z","updated_at":"2025-08-07T07:33:16.000Z","dependencies_parsed_at":"2025-06-30T08:48:17.383Z","dependency_job_id":"6cc17ff6-e4db-4cdf-bc55-c468306c5613","html_url":"https://github.com/CurvineIO/curvine","commit_stats":null,"previous_names":["curvineio/curvine"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/CurvineIO/curvine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CurvineIO%2Fcurvine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CurvineIO%2Fcurvine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CurvineIO%2Fcurvine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CurvineIO%2Fcurvine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CurvineIO","download_url":"https://codeload.github.com/CurvineIO/curvine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CurvineIO%2Fcurvine/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269873158,"owners_count":24488993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-infra","bigdata","cache-storage","cloud-native","hdfs","high-performance-computing","io","rust","s3","shuffle","spark","train-acceleration"],"created_at":"2025-08-11T11:02:36.106Z","updated_at":"2025-08-11T11:09:31.478Z","avatar_url":"https://github.com/CurvineIO.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=center\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/static/img/curvine_logo.svg\",  width=\"180\" height=\"200\"\u003e\n\u003c/div\u003e\n\n![curvine-font-dark](https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/static/img/curvine_font_dark.svg#gh-light-mode-only)\n![curvine-font-light](https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/static/img/curvine_font_white.svg#gh-dark-mode-only)\n\n\u003cp align=\"center\"\u003e\n  English | \n  \u003ca href=\"https://github.com/CurvineIO/curvine/blob/main/README_zh.md\"\u003e简体中文\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=de\"\u003eDeutsch\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=es\"\u003eEspañol\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=fr\"\u003efrançais\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=ja\"\u003e日本語\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=ko\"\u003e한국어\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=pt\"\u003ePortuguês\u003c/a\u003e |\n  \u003ca href=\"https://readme-i18n.com/CurvineIO/curvine?lang=ru\"\u003eРусский\u003c/a\u003e\n\u003c/p\u003e\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Rust](https://img.shields.io/badge/Rust-1.86%2B-orange)](https://www.rust-lang.org)\n\n**Curvine** is a high-performance, concurrent distributed cache system written in Rust, designed for low-latency and high-throughput workloads.\n\n**[Roadmap 2025](https://github.com/CurvineIO/curvine/issues/29)**\n\n## 📚 Documentation Resources\n\nFor more detailed information, please refer to:\n\n- [Official Documentation](https://curvineio.github.io/docs/Overview/instroduction)\n- [Quick Start](https://curvineio.github.io/docs/Deploy/quick-start)\n- [User Manuals](https://curvineio.github.io/docs/category/user-manuals)\n- [Benchmark](https://curvineio.github.io/docs/category/benchmark)\n- [DeepWiki](https://deepwiki.com/CurvineIO/curvine)\n- [Commit convention](COMMIT_CONVENTION.md)\n- [Contribute guidelines](CONTRIBUTING.md)\n\n## Use Case\n![use_case](https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/docs/1-Overview/img/curvine-scene.png)\n\n- **Case1**: Training acceleration\n- **Case2**: Model distribution\n- **Case3**: Hot table data acceleration\n- **Case4**: Shuffle acceleration\n- **Case5**: Multi-cloud data caching\n\n\n## 🚀 Core Features\n\n- **High-performance RPC Framework**: An asynchronous communication framework based on Tokio, supporting high-concurrency request processing.\n- **Distributed Architecture**: A Master - Worker architecture design that supports horizontal scaling.\n- **Multi-level Cache**: Supports multi - level cache strategies for memory, SSD, and HDD.\n- **FUSE Interface**: Provides a FUSE file system interface for seamless integration into existing systems.\n- **Underlying Storage Integration**: Supports integration with multiple underlying storage systems.\n- **Raft Consensus**: Uses the Raft algorithm to ensure data consistency and high availability.\n- **Monitoring and Metrics**: Built - in monitoring and performance metric collection.\n- **Web Interface**: Provides a web management interface for convenient system monitoring and management.\n\n## 🧩 Modular Architecture\nCurvine adopts a modular design and is mainly composed of the following core components：\n\n- **orpc**: A high-performance network communication framework that supports asynchronous RPC calls.\n- **curvine-common**: A shared library containing protocol definitions, error handling, and common utilities.\n- **curvine-server**: A server component that includes Master and Worker implementations.\n- **curvine-client**: A client library that provides APIs for interacting with the server.\n- **curvine-fuse**: A FUSE file system interface that allows Curvine to be mounted as a local file system.\n- **curvine-libsdk**: An SDK library that supports multi - language access.\n- **curvine-web**: A web management interface and API.\n- **curvine-tests**: A testing framework and performance benchmarking tool.\n\n## 📦 System Requirements\n\n- Rust 1.86+\n- Linux or macOS (Limited support on Windows)\n- FUSE library (for file system functionality)\n\n## 🗂️ Cached File System Access\n### 🦀 Rust API (Recommended for Native Integration)\n```\nuse curvine_common::conf::ClusterConf;\nuse curvine_common::fs::Path;\nuse std::sync::Arc;\n\nlet conf = ClusterConf::from(conf_path);\nlet rt = Arc::new(conf.client_rpc_conf().create_runtime());\nlet fs = CurvineFileSystem::with_rt(conf, rt)?;\n\nlet path = Path::from_str(\"/dir\")?;\nfs.mkdir(\u0026path).await?;\n```\n\n### 📌 FUSE (Filesystem in Userspace)\n```\nls /curvine-fuse\n```\n\n**Officially Supported Linux Distributions**​\n\n| OS Distribution     | Kernel Requirement | Tested Version | Dependencies |\n|---------------------|--------------------|----------------|--------------|\n| ​**CentOS 7**​      | ≥3.10.0            | 7.6            | fuse2-2.9.2  |\n| ​**CentOS 8**​      | ≥4.18.0            | 8.5            | fuse3-3.9.1  |\n| ​**Rocky Linux 9**​ | ≥5.14.0            | 9.5            | fuse3-3.10.2 |\n| ​**RHEL 9**​        | ≥5.14.0            | 9.5            | fuse3-3.10.2 |\n| ​**Ubuntu 22**​      | ≥5.15.0            | 22.4           | fuse3-3.10.5 |\n\n### 🐘 Hadoop Compatible API\n```\nimport org.apache.hadoop.conf.Configuration;\nimport org.apache.hadoop.fs.*;\n\nConfiguration conf = new Configuration();\nconf.set(\"fs.cv.impl\", \"io.curvine.CurvineFileSystem\");\n\nFileSystem fs = FileSystem.get(URI.create(\"cv://master:8995\"), conf);\nFSDataInputStream in = fs.open(new Path(\"/user/test/file.txt\"));\n```\n\n## 🛠 Build Instructions\n\nThis project requires the following dependencies. Please ensure they are installed before proceeding:\n\n### 📋 Prerequisites\n\n- ​**GCC**: version 10 or later ([Installation Guide](https://gcc.gnu.org/install/))\n- ​**Rust**: version 1.86 or later ([Installation Guide](https://www.rust-lang.org/tools/install))\n- ​**Protobuf**: version 3.x\n- ​**Maven**: version 3.8 or later ([Install Guide](https://maven.apache.org/install.html))\n- ​**LLVM**: version 12 or later ([Installation Guide](https://llvm.org/docs/GettingStarted.html))\n- ​**FUSE**: libfuse2 or libfuse3 development packages\n- ​**JDK**: version 1.8 or later (OpenJDK or Oracle JDK)\n- ​**npm**: version 9 or later ([Node.js Installation](https://nodejs.org/))\n- ​**Python**: version 3.7 or later ([Installation Guide](https://www.python.org/downloads/))\n\nYou can either:\n1. Use the pre-configured `curvine-docker/compile/Dockerfile_rocky9` to build a compilation image\n2. Reference this Dockerfile to create a compilation image for other operating system versions\n3. We also supply `curvine/curvine-compile` image on dockerhub\n\n### 🚀 Build Steps (Linux - Ubuntu/Debian example)\n\n```bash\n# Compiled files are in build/dist\nmake all\n\n# or use curvine-compile:latest docker images to build\nmake docker-build\n# or use curvine-compile:build-cached docker images to build, this image already cached most dependency crates\nmake docker-build-cached\n```\n\nAfter successful compilation, target file will be generated in the build/dist directory. This file is the Curvine installation package that can be used for deployment or building images.\n\n### 🖥️  Start a single - node cluster\n```bash\ncd build/dist\n\n# Start the master node\nbin/curvine-master.sh start\n\n# Start the worker node\nbin/curvine-worker.sh start\n```\n\nMount the file system\n```bash\n# The default mount point is /curvine-fuse\nbin/curvine-fuse.sh start\n```\n\nView the cluster overview:\n```bash\nbin/curvine report\n```\n\nAccess the file system using compatible HDFS commands:\n```bash\nbin/curvine fs -mkdir /a\nbin/curvine fs -ls /\n```\n\nAccess Web UI：\n```\nhttp://your-hostname:9000\n```\n\nCurvine uses TOML - formatted configuration files. An example configuration is located at conf/curvine-cluster.toml. The main configuration items include:\n\n- Network settings (ports, addresses, etc.)\n- Storage policies (cache size, storage type)\n- Cluster configuration (number of nodes, replication factor)\n- Performance tuning parameters\n\n## 🏗️ Architecture Design\n\nCurvine adopts a master-slave architecture:\n\n- **Master Node**: Responsible for metadata management, worker node coordination, and load balancing.\n- **Worker Node**: Responsible for data storage and processing.\n- **Client**: Communicates with the Master and Worker nodes via RPC.\n\nThe system uses the Raft consensus algorithm to ensure metadata consistency and supports multiple storage strategies (memory, SSD, HDD) to optimize performance and cost.\n\n## 📈 Performance\n\nCurvine performs excellently in high-concurrency scenarios and supports:\n\n- High-throughput data read and write\n- Low-latency operations\n- Large-scale concurrent connections\n\n## Contributing\nPlease read Curvine [Contribute guidelines](CONTRIBUTING.md)\n\n## 📜 License\nCurvine is licensed under the ​**​[Apache License 2.0](LICENSE)​**.\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=CurvineIO/curvine\u0026type=Date)](https://www.star-history.com/#CurvineIO/curvine\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcurvineio%2Fcurvine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcurvineio%2Fcurvine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcurvineio%2Fcurvine/lists"}