https://github.com/richenlin/miniodb
An ultra-lightweight, high-performance, horizontally scalable distributed object storage and OLAP query analysis system.
https://github.com/richenlin/miniodb
duckdb minio object-storage olap
Last synced: 29 days ago
JSON representation
An ultra-lightweight, high-performance, horizontally scalable distributed object storage and OLAP query analysis system.
- Host: GitHub
- URL: https://github.com/richenlin/miniodb
- Owner: richenlin
- License: bsd-3-clause
- Created: 2025-06-25T06:51:36.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2025-09-01T05:57:11.000Z (10 months ago)
- Last Synced: 2025-09-01T07:44:05.789Z (10 months ago)
- Topics: duckdb, minio, object-storage, olap
- Language: Go
- Homepage:
- Size: 81 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# MinIODB - 基于MinIO、DuckDB和Redis的分布式OLAP系统
[](https://golang.org/)
[]()
[]()
## 项目简介
MinIODB是一个极致轻量化、高性能、可水平扩展的分布式对象存储与OLAP查询分析系统。采用存算分离架构，以MinIO作为分布式存储底座，DuckDB作为高性能OLAP查询引擎，Redis作为元数据中心，提供企业级的数据分析能力。
## 核心特性
- **🖥️ Dashboard控制台** - Web管理界面，支持实时监控、表管理、节点管理
- **🚀 灵活部署** - 支持单节点/分布式/All-in-One三种部署模式
- **⚡ 单节点模式** - 无需Redis依赖，仅需MinIO即可快速启动
- **📈 分布式扩展** - 支持水平扩展，通过增加节点线性提升处理能力
- **💾 轻量高效** - 资源占用少，适合资源受限环境
- **🛡️ 高可用** - 三层高可用体系（应用层/中间件层/存储层）
- **🔄 数据备份** - 热备+冷备完整备份体系，支持灾难恢复
- **🔐 权限体系** - 两级权限控制（功能权限+表级数据权限）
- **📊 表级管理** - 支持多表数据隔离和差异化配置
## 架构设计
```
+----------------+ +----------------+ +----------------+
| gRPC Client | | RESTful Client| | Browser |
+----------------+ +----------------+ +----------------+
| | |
v v v
+---------------------------------------------------------------+
| API Gateway / Query Node (Go) - :8081 |
| |
| - Request Parsing & Validation - Dashboard Server (新增) |
| - Query Coordination - SSE Real-time Events |
| - Result Aggregation - Static SPA (embedded) |
| - Metadata Manager |
+---------------------------------------------------------------+
^ | ^ | ^ |
| | | | | |
| v | v | v
+---------------------------------------------------------------+
| Connection Pool Manager |
| |
| Redis Pool | MinIO Pool | Dashboard Config|
| ├─ Standalone | ├─ Primary Pool | ├─ All-in-One |
| ├─ Sentinel | ├─ Backup Pool | └─ Standalone |
| ├─ Cluster | ├─ Health Check | |
| └─ Health Check | └─ Auto Failover | |
+---------------------------------------------------------------+
| ｜
v (Service Discovery) v (Query Planning)
+----------------+ +---------------------------------+
| Redis | | Worker Nodes (Go Service) |
| -------------- | |---------------------------------|
| - Service Reg. |<-----| - Heartbeat & Registration |
| - Data Index |----->| - DuckDB Instance (embedded) |
| - Hash Ring | | - Data Ingestion & Buffering |
| - Table Meta | | - Parquet File Generation |
| - Metadata Ver | | - Read/Write to MinIO |
| - Backup Index | | - Table-level Processing |
+----------------+ | - Connection Pool Client |
+---------------------------------+
^ |
| | (S3 API via Pool)
v v
+-------------------------+
| MinIO Cluster |
| (Distributed Object |
| Storage) |
| ├─ Primary Storage |
| ├─ Backup Storage |
| └─ TABLE/ID/YYYY-MM-DD/ |
+-------------------------+
```
**详细架构设计**：[docs/SOLUTION.md](docs/SOLUTION.md)
## 快速开始
### 前置要求
**单节点模式**（推荐开发测试）：
- Go 1.24+
- MinIO Server
- 4GB+ 内存
**分布式模式**（推荐生产环境）：
- Go 1.24+
- Redis 6.0+
- MinIO Cluster
- 8GB+ 内存
### 安装
```bash
git clone https://github.com/richenlin/minIODB.git
cd minIODB
go mod download
```
### 配置
```bash
cp config/config.yaml config/config.local.yaml
```
**单节点模式配置**：
```yaml
redis:
enabled: false # 关闭Redis，无需分布式协调
minio:
endpoint: "localhost:9000"
access_key: "minioadmin"
secret_key: "minioadmin"
bucket: "miniodb-data"
```
**分布式模式配置**：
```yaml
redis:
enabled: true
mode: "standalone" # 或 sentinel/cluster
addr: "localhost:6379"
minio:
endpoint: "minio-cluster:9000"
bucket: "miniodb-data"
```
### 启动服务
```bash
# 启动基础设施（Redis、MinIO）
./deploy/deploy.sh dev --install-deps
# 启动 MinIODB
go run cmd/main.go -c config/config.local.yaml
```
服务启动后：
- **gRPC端口**: :8080
- **REST API端口**: :8081
- **Swagger UI**: http://localhost:8081/api-docs/index.html
- **Prometheus指标**: http://localhost:8081/metrics
### 验证服务
```bash
# 健康检查
curl http://localhost:8081/v1/health
# 创建表
curl -X POST http://localhost:8081/v1/tables \
-H "Content-Type: application/json" \
-d '{"table_name": "users", "config": {"buffer_size": 1000}}'
# 写入数据
curl -X POST http://localhost:8081/v1/data \
-H "Content-Type: application/json" \
-d '{"table": "users", "id": "user-001", "timestamp": "2024-01-18T10:00:00Z", "payload": {"name": "张三"}}'
# 查询数据
curl -X POST http://localhost:8081/v1/query \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM users WHERE id = '\''user-001'\''"}'
```
## 文档
### 核心文档
- [架构设计文档](docs/SOLUTION.md) - 完整的系统架构和技术细节
- [部署指南](deploy/README.md) - Docker/Kubernetes/Ansible部署方式
### 模块文档
- [连接池管理](pkg/pool/README.md) - Redis和MinIO连接池设计
- [ID生成器](pkg/idgen/README.md) - UUID/Snowflake/自定义ID生成
- [日志系统](pkg/logger/README.md) - 结构化日志使用指南
## 许可证
本项目采用BSD-3-Clause许可证。详见[LICENSE](LICENSE)文件。
---
⭐ 如果这个项目对您有帮助，请给我们一个Star！