https://github.com/curvineio/curvine

High performance distributed cache system. Built by Rust.
https://github.com/curvineio/curvine

ai ai-infra bigdata cache-storage cloud-native hdfs high-performance-computing io rust s3 shuffle spark train-acceleration

Last synced: 2 months ago
JSON representation

High performance distributed cache system. Built by Rust.

Host: GitHub
URL: https://github.com/curvineio/curvine
Owner: CurvineIO
License: apache-2.0
Created: 2025-05-27T06:24:21.000Z (5 months ago)
Default Branch: main
Last Pushed: 2025-08-07T10:15:58.000Z (2 months ago)
Last Synced: 2025-08-07T10:25:02.030Z (2 months ago)
Topics: ai, ai-infra, bigdata, cache-storage, cloud-native, hdfs, high-performance-computing, io, rust, s3, shuffle, spark, train-acceleration
Language: Rust
Homepage: https://curvineio.github.io
Size: 1.1 MB
Stars: 298
Watchers: 7
Forks: 37
Open Issues: 35
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Security: SECURITY.md

Awesome Lists containing this project

README

          






![curvine-font-dark](https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/static/img/curvine_font_dark.svg#gh-light-mode-only)

![curvine-font-light](https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/static/img/curvine_font_white.svg#gh-dark-mode-only)



  English | 

  简体中文 |

  Deutsch |

  Español |

  français |

  日本語 |

  한국어 |

  Português |

  Русский



[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

[![Rust](https://img.shields.io/badge/Rust-1.86%2B-orange)](https://www.rust-lang.org)

**Curvine** is a high-performance, concurrent distributed cache system written in Rust, designed for low-latency and high-throughput workloads.

**[Roadmap 2025](https://github.com/CurvineIO/curvine/issues/29)**

## 📚 Documentation Resources

For more detailed information, please refer to:

- [Official Documentation](https://curvineio.github.io/docs/Overview/instroduction)

- [Quick Start](https://curvineio.github.io/docs/Deploy/quick-start)

- [User Manuals](https://curvineio.github.io/docs/category/user-manuals)

- [Benchmark](https://curvineio.github.io/docs/category/benchmark)

- [DeepWiki](https://deepwiki.com/CurvineIO/curvine)

- [Commit convention](COMMIT_CONVENTION.md)

- [Contribute guidelines](CONTRIBUTING.md)

## Use Case

![use_case](https://raw.githubusercontent.com/CurvineIO/curvine-doc/refs/heads/main/docs/1-Overview/img/curvine-scene.png)

- **Case1**: Training acceleration

- **Case2**: Model distribution

- **Case3**: Hot table data acceleration

- **Case4**: Shuffle acceleration

- **Case5**: Multi-cloud data caching

## 🚀 Core Features

- **High-performance RPC Framework**: An asynchronous communication framework based on Tokio, supporting high-concurrency request processing.

- **Distributed Architecture**: A Master - Worker architecture design that supports horizontal scaling.

- **Multi-level Cache**: Supports multi - level cache strategies for memory, SSD, and HDD.

- **FUSE Interface**: Provides a FUSE file system interface for seamless integration into existing systems.

- **Underlying Storage Integration**: Supports integration with multiple underlying storage systems.

- **Raft Consensus**: Uses the Raft algorithm to ensure data consistency and high availability.

- **Monitoring and Metrics**: Built - in monitoring and performance metric collection.

- **Web Interface**: Provides a web management interface for convenient system monitoring and management.

## 🧩 Modular Architecture

Curvine adopts a modular design and is mainly composed of the following core components：

- **orpc**: A high-performance network communication framework that supports asynchronous RPC calls.

- **curvine-common**: A shared library containing protocol definitions, error handling, and common utilities.

- **curvine-server**: A server component that includes Master and Worker implementations.

- **curvine-client**: A client library that provides APIs for interacting with the server.

- **curvine-fuse**: A FUSE file system interface that allows Curvine to be mounted as a local file system.

- **curvine-libsdk**: An SDK library that supports multi - language access.

- **curvine-web**: A web management interface and API.

- **curvine-tests**: A testing framework and performance benchmarking tool.

## 📦 System Requirements

- Rust 1.86+

- Linux or macOS (Limited support on Windows)

- FUSE library (for file system functionality)

## 🗂️ Cached File System Access

### 🦀 Rust API (Recommended for Native Integration)

```

use curvine_common::conf::ClusterConf;

use curvine_common::fs::Path;

use std::sync::Arc;

let conf = ClusterConf::from(conf_path);

let rt = Arc::new(conf.client_rpc_conf().create_runtime());

let fs = CurvineFileSystem::with_rt(conf, rt)?;

let path = Path::from_str("/dir")?;

fs.mkdir(&path).await?;

```

### 📌 FUSE (Filesystem in Userspace)

```

ls /curvine-fuse

```

**Officially Supported Linux Distributions**

| OS Distribution     | Kernel Requirement | Tested Version | Dependencies |

|---------------------|--------------------|----------------|--------------|

| **CentOS 7**      | ≥3.10.0            | 7.6            | fuse2-2.9.2  |

| **CentOS 8**      | ≥4.18.0            | 8.5            | fuse3-3.9.1  |

| **Rocky Linux 9** | ≥5.14.0            | 9.5            | fuse3-3.10.2 |

| **RHEL 9**        | ≥5.14.0            | 9.5            | fuse3-3.10.2 |

| **Ubuntu 22**      | ≥5.15.0            | 22.4           | fuse3-3.10.5 |

### 🐘 Hadoop Compatible API

```

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.*;

Configuration conf = new Configuration();

conf.set("fs.cv.impl", "io.curvine.CurvineFileSystem");

FileSystem fs = FileSystem.get(URI.create("cv://master:8995"), conf);

FSDataInputStream in = fs.open(new Path("/user/test/file.txt"));

```

## 🛠 Build Instructions

This project requires the following dependencies. Please ensure they are installed before proceeding:

### 📋 Prerequisites

- **GCC**: version 10 or later ([Installation Guide](https://gcc.gnu.org/install/))

- **Rust**: version 1.86 or later ([Installation Guide](https://www.rust-lang.org/tools/install))

- **Protobuf**: version 3.x

- **Maven**: version 3.8 or later ([Install Guide](https://maven.apache.org/install.html))

- **LLVM**: version 12 or later ([Installation Guide](https://llvm.org/docs/GettingStarted.html))

- **FUSE**: libfuse2 or libfuse3 development packages

- **JDK**: version 1.8 or later (OpenJDK or Oracle JDK)

- **npm**: version 9 or later ([Node.js Installation](https://nodejs.org/))

- **Python**: version 3.7 or later ([Installation Guide](https://www.python.org/downloads/))

You can either:

1. Use the pre-configured `curvine-docker/compile/Dockerfile_rocky9` to build a compilation image

2. Reference this Dockerfile to create a compilation image for other operating system versions

3. We also supply `curvine/curvine-compile` image on dockerhub

### 🚀 Build Steps (Linux - Ubuntu/Debian example)

```bash

# Compiled files are in build/dist

make all

# or use curvine-compile:latest docker images to build

make docker-build

# or use curvine-compile:build-cached docker images to build, this image already cached most dependency crates

make docker-build-cached

```

After successful compilation, target file will be generated in the build/dist directory. This file is the Curvine installation package that can be used for deployment or building images.

### 🖥️  Start a single - node cluster

```bash

cd build/dist

# Start the master node

bin/curvine-master.sh start

# Start the worker node

bin/curvine-worker.sh start

```

Mount the file system

```bash

# The default mount point is /curvine-fuse

bin/curvine-fuse.sh start

```

View the cluster overview:

```bash

bin/curvine report

```

Access the file system using compatible HDFS commands:

```bash

bin/curvine fs -mkdir /a

bin/curvine fs -ls /

```

Access Web UI：

```

http://your-hostname:9000

```

Curvine uses TOML - formatted configuration files. An example configuration is located at conf/curvine-cluster.toml. The main configuration items include:

- Network settings (ports, addresses, etc.)

- Storage policies (cache size, storage type)

- Cluster configuration (number of nodes, replication factor)

- Performance tuning parameters

## 🏗️ Architecture Design

Curvine adopts a master-slave architecture:

- **Master Node**: Responsible for metadata management, worker node coordination, and load balancing.

- **Worker Node**: Responsible for data storage and processing.

- **Client**: Communicates with the Master and Worker nodes via RPC.

The system uses the Raft consensus algorithm to ensure metadata consistency and supports multiple storage strategies (memory, SSD, HDD) to optimize performance and cost.

## 📈 Performance

Curvine performs excellently in high-concurrency scenarios and supports:

- High-throughput data read and write

- Low-latency operations

- Large-scale concurrent connections

## Contributing

Please read Curvine [Contribute guidelines](CONTRIBUTING.md)

## 📜 License

Curvine is licensed under the **[Apache License 2.0](LICENSE)**.

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=CurvineIO/curvine&type=Date)](https://www.star-history.com/#CurvineIO/curvine&Date)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/curvineio/curvine

Awesome Lists containing this project

README