https://github.com/jonefeewang/stonemq
A high-performance and efficient message queue developed in Rust
https://github.com/jonefeewang/stonemq
message-queue-rust-kafka-pulsar-stonemq
Last synced: 10 months ago
JSON representation
A high-performance and efficient message queue developed in Rust
- Host: GitHub
- URL: https://github.com/jonefeewang/stonemq
- Owner: jonefeewang
- License: apache-2.0
- Created: 2023-12-28T13:13:38.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-19T01:14:27.000Z (about 1 year ago)
- Last Synced: 2025-02-19T01:31:49.236Z (about 1 year ago)
- Topics: message-queue-rust-kafka-pulsar-stonemq
- Language: Rust
- Homepage: https://wangjunfei.com/2025/02/10/Announcing-Stonemq-A-high-performance-and-efficient-message-queue-developed-in-Rust/
- Size: 13.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome - jonefeewang/stonemq - A high-performance and efficient message queue developed in Rust (<a name="Rust"></a>Rust)
- awesome-starred - jonefeewang/stonemq - A high-performance and efficient message queue developed in Rust (Rust)
README
# Stonemq
[](LICENSE)
[](https://github.com/username/stonemq/actions)
[](https://crates.io/crates/stonemq)
Stonemq aims to outperform Kafka in scenarios with massive-scale queue clusters, delivering enhanced performance and efficiency to achieve cost reduction and operational optimization.
In use cases involving large clusters with countless queues—particularly in enterprise business services and public cloud services—there can be tens of thousands of partitions and partition leaders. Regardless of whether the queues contain messages, the volume of messages, or the flow rate, the presence or absence of active consumers in each partition poses a significant burden for cluster operators. Partition growth negatively impacts cluster throughput, while node failures or restarts often result in partition leader and controller switching, creating critical operational challenges. Stonemq addresses these inefficiencies.
Cluster performance should remain consistent regardless of partition growth. Queues with varying traffic volumes need consolidation to enable more efficient message flow—akin to containerized shipping for multiple clients. This is precisely the vision of Stonemq. While solutions like Pulsar utilize journaling for centralized message handling, Stonemq seeks to retain Kafka’s replication , which simplifies and standardizes cluster operation and maintenance.We believe this protocol is both straightforward and highly efficient, forming the backbone of our solution.Additionally, by reusing Kafka's client-server communication protocol, StoneMQ ensures seamless migration without requiring any changes to the user's client applications. This approach significantly reduces adoption costs for users, enabling a smooth transition to StoneMQ while retaining the familiar and reliable interface they are accustomed to.
For a more detailed analysis and introduction to the project's inception, please refer to the blog: [Blog Link](https://wangjunfei.com/2025/02/10/Announcing-Stonemq-A-high-performance-and-efficient-message-queue-developed-in-Rust/).
---
## Current Status
- version 0.1.0 released
- Supports single-node message sending and receiving.
- Implements group consumption functionality.
---
## Installation and Usage
### Build From Source
1. Make sure you have Rust installed (recommended via [Rustup](https://rustup.rs/)):
```bash
rustc --version
```
2. Clone the repository:
```bash
git clone https://github.com/jonefeewang/stonemq.git
cd stonemq
```
3. Build the release version using Cargo:
```bash
cargo build --release
```
4. Run the application:
```bash
./target/release/stonemq
```
## **Quick Start**
### **1. Start the StoneMQ Server**
Run the following command to launch the StoneMQ server:
```bash
./stonemq
```
---
### **2. Connect to StoneMQ**
Use a **Kafka client library** (or REST API) with a minimum supported version of **Kafka v0.11** to connect to the StoneMQ server for publishing and subscribing to messages. StoneMQ uses the same default server port as Kafka: **9092**. This port can be customized in the configuration file.
---
### **3. Publish a Message**
StoneMQ comes with a built-in topic, **`topic_a`**, which is pre-configured with **2 partitions**:
- `topic_a-0`
- `topic_a-1`
By default, the system sets up with **2 journals**.
---
### **4. Configuration**
The configuration file, **`conf.toml`**, is located in the project’s root directory. Adjust settings as needed to customize the server's behavior.
### **5. Server Graceful Shutdown**
StoneMQ provides a graceful shutdown mechanism that allows the server to complete all pending tasks before exiting. This is particularly useful in scenarios where the server needs to ensure all messages are processed or stored before shutting down.
To trigger a graceful shutdown, you can send a termination signal to the server. In a Unix-like environment, you can use `Ctrl+C` to send the `SIGINT` signal or use the following command to send the `SIGINT` signal:
```bash
kill -2
```
This will allow the server to perform a graceful shutdown, ensuring all pending messages are processed and stored.
---
## Development Guide
### Local Development Setup
1. **Install Rust Toolchain**:
Ensure Rust is installed on your system. Use [Rustup](https://rustup.rs/) to get started:
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default stable
```
2. **Clone the Repository**:
```bash
git clone https://github.com/jonefeewang/stonemq.git
cd stonemq
```
3. **Run Tests**:
Run all unit tests to ensure everything is working correctly:
```bash
cargo test
```
4. **Check Code Formatting**:
Ensure the code adheres to the formatting rules:
```bash
cargo fmt -- --check
```
5. **Lint (Static Analysis)**:
Run clippy to check for potential issues:
```bash
cargo clippy
```
6. **Log Parsing**
An executable log_parser, source file is **`log_parser.rs`**, is available in the project's `bin` directory. This tool can be used to parse `log`,`index`,and `checkpoint` files.
7. **Benchmarking and Monitoring**
The entry file **`stonemq.rs`** in the `bin` directory allows you to specify the current running mode: **prod**, **dev**, or **perf**. The default mode is **prod**.
- **Dev Mode**: Suitable for development.
- **Perf Mode**: Enables the use of **tokio-console** to monitor the state of Tokio tasks within the process.
Additionally, you can specify whether to enable **trace**. When enabled, added spans can be observed in **Open Jaeger** for in-depth monitoring.
You can also specify the **log output level** through environment variables. The project uses **dotenv** crate to configure environment variables for ease of management and flexibility.
---
## Contribution Guide
Contributions are welcome and greatly appreciated! Whether it's reporting a bug, submitting code, improving documentation, or suggesting ideas, your help is valuable.
### Steps to Contribute
1. Fork the repository and create a new branch for your feature/bugfix.
2. Ensure all tests pass, and your code adheres to the project's standards.
3. Submit a pull request with a clear description of your changes.
---
## Community Support
If you encounter issues or have questions, feel free to reach out to the community:
- 🐛 Submit Issues: [GitHub Issues](https://github.com/jonefeewang/stonemq/issues)
- 📢 Join Discussions: [GitHub Discussions](https://github.com/jonefeewang/stonemq/discussions)
---
## License
Stonemq is distributed under the **Apache License 2.0**. See the [LICENSE](./LICENSE) file for more details.