{"id":30687609,"url":"https://github.com/richenlin/miniodb","last_synced_at":"2026-05-15T22:39:20.014Z","repository":{"id":301124074,"uuid":"1008185178","full_name":"richenlin/minIODB","owner":"richenlin","description":"An ultra-lightweight, high-performance, horizontally scalable distributed object storage and OLAP query analysis system.","archived":false,"fork":false,"pushed_at":"2025-09-01T05:57:11.000Z","size":84957,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-01T07:44:05.789Z","etag":null,"topics":["duckdb","minio","object-storage","olap"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/richenlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-25T06:51:36.000Z","updated_at":"2025-09-01T05:57:15.000Z","dependencies_parsed_at":"2025-09-01T07:16:59.279Z","dependency_job_id":"9d3ee80a-1eaf-4918-a1e4-ff35f47a180f","html_url":"https://github.com/richenlin/minIODB","commit_stats":null,"previous_names":["richenlin/miniiodb"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/richenlin/minIODB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richenlin%2FminIODB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richenlin%2FminIODB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richenlin%2FminIODB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richenlin%2FminIODB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/richenlin","download_url":"https://codeload.github.com/richenlin/minIODB/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richenlin%2FminIODB/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273208777,"owners_count":25064204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duckdb","minio","object-storage","olap"],"created_at":"2025-09-02T00:03:30.255Z","updated_at":"2026-05-15T22:39:20.008Z","avatar_url":"https://github.com/richenlin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MinIODB - 基于MinIO、DuckDB和Redis的分布式OLAP系统\n\n[![Go Version](https://img.shields.io/badge/Go-1.24+-blue.svg)](https://golang.org/)\n[![Docker](https://img.shields.io/badge/Docker-Supported-blue.svg)]()\n[![Kubernetes](https://img.shields.io/badge/Kubernetes-Ready-blue.svg)]()\n\n## 项目简介\n\nMinIODB是一个极致轻量化、高性能、可水平扩展的分布式对象存储与OLAP查询分析系统。采用存算分离架构，以MinIO作为分布式存储底座，DuckDB作为高性能OLAP查询引擎，Redis作为元数据中心，提供企业级的数据分析能力。\n\n![Dashboard截图](docs/screenshot.png)\n\n## 核心特性\n\n- **🖥️ Dashboard控制台** - Web管理界面，支持实时监控、表管理、节点管理\n- **🚀 灵活部署** - 支持单节点/分布式/All-in-One三种部署模式\n- **⚡ 单节点模式** - 无需Redis依赖，仅需MinIO即可快速启动\n- **📈 分布式扩展** - 支持水平扩展，通过增加节点线性提升处理能力\n- **💾 轻量高效** - 资源占用少，适合资源受限环境\n- **🛡️ 高可用** - 三层高可用体系（应用层/中间件层/存储层）\n- **🔄 数据备份** - 热备+冷备完整备份体系，支持灾难恢复\n- **🔐 权限体系** - 两级权限控制（功能权限+表级数据权限）\n- **📊 表级管理** - 支持多表数据隔离和差异化配置\n\n## 架构设计\n\n```\n+----------------+      +----------------+      +----------------+\n|   gRPC Client  |      |  RESTful Client|      |  Browser       |\n+----------------+      +----------------+      +----------------+\n        |                      |                       |\n        v                      v                       v\n+---------------------------------------------------------------+\n|     API Gateway / Query Node (Go) - :8081                     |\n|                                                               |\n|  - Request Parsing \u0026 Validation    - Dashboard Server (新增)  |\n|  - Query Coordination              - SSE Real-time Events     |\n|  - Result Aggregation              - Static SPA (embedded)    |\n|  - Metadata Manager                                           |\n+---------------------------------------------------------------+\n      ^   |                    ^   |                    ^   |\n      |   |                    |   |                    |   | \n      |   v                    |   v                    |   v\n+---------------------------------------------------------------+\n|        Connection Pool Manager                                |\n|                                                               |\n|  Redis Pool      |    MinIO Pool         |   Dashboard Config|\n|  ├─ Standalone   |    ├─ Primary Pool    |   ├─ All-in-One   |\n|  ├─ Sentinel     |    ├─ Backup Pool     |   └─ Standalone   |\n|  ├─ Cluster      |    ├─ Health Check    |                    |\n|  └─ Health Check |    └─ Auto Failover   |                    |\n+---------------------------------------------------------------+\n      |                              ｜\n      v (Service Discovery)          v  (Query Planning)\n+----------------+      +---------------------------------+\n| Redis          |      | Worker Nodes (Go Service)       |\n| -------------- |      |---------------------------------|\n| - Service Reg. |\u003c-----| - Heartbeat \u0026 Registration      |\n| - Data Index   |-----\u003e| - DuckDB Instance (embedded)    |\n| - Hash Ring    |      | - Data Ingestion \u0026 Buffering    |\n| - Table Meta   |      | - Parquet File Generation       |\n| - Metadata Ver |      | - Read/Write to MinIO           |\n| - Backup Index |      | - Table-level Processing        |\n+----------------+      | - Connection Pool Client        |\n                        +---------------------------------+\n                                 ^         |\n                                 |         | (S3 API via Pool)\n                                 v         v\n                       +-------------------------+\n                       |   MinIO Cluster         |\n                       | (Distributed Object     |\n                       |      Storage)           |\n                       | ├─ Primary Storage      |\n                       | ├─ Backup Storage       |\n                       | └─ TABLE/ID/YYYY-MM-DD/ |\n                       +-------------------------+\n```\n\n**详细架构设计**：[docs/SOLUTION.md](docs/SOLUTION.md)\n\n## 快速开始\n\n### 前置要求\n\n**单节点模式**（推荐开发测试）：\n- Go 1.24+\n- MinIO Server\n- 4GB+ 内存\n\n**分布式模式**（推荐生产环境）：\n- Go 1.24+\n- Redis 6.0+\n- MinIO Cluster\n- 8GB+ 内存\n\n### 安装\n\n```bash\ngit clone https://github.com/richenlin/minIODB.git\ncd minIODB\ngo mod download\n```\n\n### 配置\n\n```bash\ncp config/config.yaml config/config.local.yaml\n```\n\n**单节点模式配置**：\n```yaml\nredis:\n  enabled: false  # 关闭Redis，无需分布式协调\n\nminio:\n  endpoint: \"localhost:9000\"\n  access_key: \"minioadmin\"\n  secret_key: \"minioadmin\"\n  bucket: \"miniodb-data\"\n```\n\n**分布式模式配置**：\n```yaml\nredis:\n  enabled: true\n  mode: \"standalone\"  # 或 sentinel/cluster\n  addr: \"localhost:6379\"\n\nminio:\n  endpoint: \"minio-cluster:9000\"\n  bucket: \"miniodb-data\"\n```\n\n### 启动服务\n\n```bash\n# 启动基础设施（Redis、MinIO）\n./deploy/deploy.sh dev --install-deps\n\n# 启动 MinIODB\ngo run cmd/main.go -c config/config.local.yaml\n```\n\n服务启动后：\n- **gRPC端口**: :8080\n- **REST API端口**: :8081\n- **Swagger UI**: http://localhost:8081/api-docs/index.html\n- **Prometheus指标**: http://localhost:8081/metrics\n\n### 验证服务\n\n```bash\n# 健康检查\ncurl http://localhost:8081/v1/health\n\n# 创建表\ncurl -X POST http://localhost:8081/v1/tables \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"table_name\": \"users\", \"config\": {\"buffer_size\": 1000}}'\n\n# 写入数据\ncurl -X POST http://localhost:8081/v1/data \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"table\": \"users\", \"id\": \"user-001\", \"timestamp\": \"2024-01-18T10:00:00Z\", \"payload\": {\"name\": \"张三\"}}'\n\n# 查询数据\ncurl -X POST http://localhost:8081/v1/query \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"sql\": \"SELECT * FROM users WHERE id = '\\''user-001'\\''\"}'\n```\n\n## 文档\n\n### 核心文档\n- [架构设计文档](docs/SOLUTION.md) - 完整的系统架构和技术细节\n- [部署指南](deploy/README.md) - Docker/Kubernetes/Ansible部署方式\n\n### 模块文档\n- [连接池管理](pkg/pool/README.md) - Redis和MinIO连接池设计\n- [ID生成器](pkg/idgen/README.md) - UUID/Snowflake/自定义ID生成\n- [日志系统](pkg/logger/README.md) - 结构化日志使用指南\n\n## 许可证\n\n本项目采用BSD-3-Clause许可证。详见[LICENSE](LICENSE)文件。\n\n---\n\n⭐ 如果这个项目对您有帮助，请给我们一个Star！","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichenlin%2Fminiodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frichenlin%2Fminiodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichenlin%2Fminiodb/lists"}