{"id":39359521,"url":"https://github.com/orcastor/orcas","last_synced_at":"2026-01-18T02:40:29.053Z","repository":{"id":57790476,"uuid":"178181047","full_name":"orcastor/orcas","owner":"orcastor","description":"🗄️【开放开箱即用内容寻址对象存储】支持主流操作系统和廉价低功耗设备 [OrcaS] Open Ready-to-use Content Addressable Storage - for popular OS \u0026 cheap and low power devices.","archived":false,"fork":false,"pushed_at":"2026-01-13T11:00:08.000Z","size":26533,"stargazers_count":27,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-01-13T14:06:32.474Z","etag":null,"topics":["ceph","content-addressable-storage","embedded","fastdfs","hacktoberfest","minio","nas","object-file-system","object-storage","object-store","osd","oss","s3","storage","tikv"],"latest_commit_sha":null,"homepage":"https://orcastor.github.io/doc/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orcastor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-03-28T10:31:22.000Z","updated_at":"2026-01-13T11:00:13.000Z","dependencies_parsed_at":"2024-03-12T17:02:40.962Z","dependency_job_id":"4d07967a-86a4-43ac-bbf5-6b3603a29a15","html_url":"https://github.com/orcastor/orcas","commit_stats":{"total_commits":323,"total_committers":1,"mean_commits":323.0,"dds":0.0,"last_synced_commit":"05c0eccca32112d5deab18efa0daa440b6f9443d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/orcastor/orcas","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orcastor%2Forcas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orcastor%2Forcas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orcastor%2Forcas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orcastor%2Forcas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orcastor","download_url":"https://codeload.github.com/orcastor/orcas/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orcastor%2Forcas/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28526633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceph","content-addressable-storage","embedded","fastdfs","hacktoberfest","minio","nas","object-file-system","object-storage","object-store","osd","oss","s3","storage","tikv"],"created_at":"2026-01-18T02:40:28.934Z","updated_at":"2026-01-18T02:40:29.032Z","avatar_url":"https://github.com/orcastor.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://orcastor.github.io/doc/\"\u003e\n    \u003cimg src=\"https://orcastor.github.io/doc/logo.svg\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003cstrong\u003eOrcaS: Open Ready-to-Use Content Addressable Storage\u003c/strong\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"/go.mod#L3\" alt=\"go version\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/go%20version-%3E=1.18-brightgreen?style=flat\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://goreportcard.com/badge/github.com/orcastor/orcas\" alt=\"goreport\"\u003e\n    \u003cimg src=\"https://goreportcard.com/badge/github.com/orcastor/orcas?v=20220901\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://app.fossa.com/projects/git%2Bgithub.com%2Forcastor%2Forcas?ref=badge_shield\" alt=\"FOSSA Status\"\u003e\n    \u003cimg src=\"https://app.fossa.com/api/projects/git%2Bgithub.com%2Forcastor%2Forcas.svg?type=shield\"/\u003e\n  \u003c/a\u003e\n  \u003ca href='https://orca-zhang.semaphoreci.com/branches/733643c4-54d7-4cc4-9b1e-c3c77c8ff7db' alt='Build Status'\u003e\n    \u003cimg src='https://orca-zhang.semaphoreci.com/badges/orcas/branches/master.svg?style=shields\u0026key=78c39699-2da0-4322-9372-0839f9dcde76'\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/orcastor/orcas/blob/master/LICENSE\" alt='MIT license'\u003e\n    \u003cimg src=\"https://img.shields.io/badge/license-MIT-blue.svg?style=flat\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://orcastor.github.io/doc/\" alt='docs'\u003e\n    \u003cimg src=\"https://img.shields.io/badge/docs-master-blue.svg?style=flat\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n- [English](README.md) | [中文](README.zh.md)\n\n## 🚀 What is OrcaS?\n\n**OrcaS** (Open Ready-to-Use Content Addressable Storage) is a **lightweight, high-performance object storage system** built with **Content Addressable Storage (CAS)** at its core. It provides enterprise-grade features like instant deduplication, multi-versioning, zero-knowledge encryption, and smart compression - all in a single binary that's ready to deploy.\n\n### Why OrcaS?\n\n- 🌐 **Open**: Open source (MIT license), transparent, community-driven development\n- ✅ **Ready-to-Use**: Content Addressable Storage ensures data integrity and automatic deduplication, production-ready out of the box\n- 🎯 **Content Addressable Storage**: Data is stored by content hash, enabling automatic deduplication and integrity verification\n- ⚡ **Instant Upload (Deduplication)**: Upload files in seconds, not minutes - identical files are detected instantly without uploading\n- 🔒 **Zero-Knowledge Encryption**: Your data, your keys - end-to-end encryption with industry-standard algorithms\n- 📦 **Production Ready**: S3-compatible API, VFS mount support, and comprehensive documentation\n- 🚀 **High Performance**: Optimized for both small and large files with intelligent packaging and chunking\n\n## ✨ Key Features\n\n### ⏱ Instant Upload (Object-level Deduplication)\n\n**What it does**: Upload identical files instantly without transferring data.\n\n**How it works**: \n- Calculates multiple checksums (XXH3, SHA-256) for each file\n- Before uploading, checks if identical content already exists\n- If found, creates a reference to existing data instead of uploading\n- **Result**: Upload time drops from minutes to milliseconds for duplicate files\n\n**Use cases**:\n- Backup systems (same files across multiple backups)\n- Version control systems (similar files across versions)\n- Multi-user environments (shared files)\n- CDN edge storage (cached content)\n\n**Benefits**:\n- 🚀 **99%+ faster** uploads for duplicate files\n- 💾 **Massive storage savings** - store 1 copy, reference it N times\n- ⚡ **Bandwidth savings** - no redundant data transfer\n- 🔍 **Automatic integrity verification** - content hash ensures data correctness\n\n![Deduplication Benefits](assets/deduplication-benefits.png)\n\n### 📦 Small Object Packaging\n\n**What it does**: Efficiently stores many small files together.\n\n**How it works**:\n- Groups small files (\u003c 64KB) into packages\n- Reduces metadata overhead and I/O operations\n- Maintains individual file access while optimizing storage\n\n**Benefits**:\n- 📈 **10x+ performance improvement** for small file operations\n- 💰 **Reduced storage costs** - less metadata overhead\n- ⚡ **Faster operations** - batch metadata writes\n\n### 🔪 Large Object Chunking\n\n**What it does**: Splits large files into manageable chunks.\n\n**How it works**:\n- Automatically chunks files larger than configured threshold (default 10MB)\n- Each chunk stored independently with its own checksum\n- Enables parallel upload/download and efficient updates\n\n**Benefits**:\n- 🔄 **Parallel processing** - upload/download chunks concurrently\n- 🛡️ **Resumable transfers** - retry failed chunks independently\n- ✏️ **Efficient updates** - only modified chunks need re-upload\n- 📊 **Better resource utilization** - process large files efficiently\n\n### 🗂 Object Multi-versioning\n\n**What it does**: Automatically maintains file version history.\n\n**How it works**:\n- Each file modification creates a new version\n- Old versions preserved automatically\n- Configurable retention policies\n- Space-efficient through content deduplication\n\n**Benefits**:\n- 🔙 **Point-in-time recovery** - restore any previous version\n- 🛡️ **Data protection** - accidental deletions are recoverable\n- 📚 **Audit trail** - track all changes over time\n- 💾 **Space efficient** - unchanged data shared across versions\n\n### 🔐 Zero-Knowledge Encryption\n\n**What it does**: End-to-end encryption where only you hold the keys.\n\n**How it works**:\n- AES-256 encryption (industry standard)\n- Encryption keys never leave your control\n- Optional per-bucket encryption keys\n- Transparent encryption/decryption\n\n**Benefits**:\n- 🔒 **Maximum security** - even storage admins can't read your data\n- ✅ **Compliance ready** - meets strict security requirements\n- 🛡️ **Data privacy** - your data, your control\n- 🌍 **International standards** - AES-256 encryption\n\n### 🗜 Smart Compression\n\n**What it does**: Automatically compresses data to save space.\n\n**How it works**:\n- Configurable compression algorithms (zstd, gzip, etc.)\n- Compression applied before encryption\n- Automatic detection of already-compressed data\n- Per-bucket compression settings\n\n**Benefits**:\n- 💾 **Storage savings** - typically 30-70% reduction\n- ⚡ **Bandwidth savings** - less data to transfer\n- 🎯 **Smart defaults** - works out of the box\n- ⚙️ **Configurable** - adjust per your needs\n\n## 🏗️ Architecture \u0026 Design\n\n### Content Addressable Storage (CAS) Core\n\nOrcaS is built on **Content Addressable Storage** principles, where data is stored and retrieved by its content hash rather than location.\n\n![Content Addressable Storage Architecture](assets/cas-architecture.png)\n\n**Key Benefits of CAS**:\n1. **Automatic Deduplication**: Identical content stored once, referenced many times\n2. **Integrity Verification**: Content hash ensures data hasn't been corrupted\n3. **Efficient Versioning**: New versions only store changed content\n4. **Simplified Backup**: Same content = same hash = no re-upload needed\n\n### System Architecture\n\n![System Architecture](assets/system-architecture.png)\n\n### Instant Upload Flow\n\n![Instant Upload Flow](assets/instant-upload-flow.png)\n\n### Data Storage Structure\n\n```\nStorage Layout:\n├── Metadata (SQLite)\n│   ├── Objects (files, directories)\n│   ├── DataInfo (content metadata)\n│   ├── Versions (version history)\n│   └── References (deduplication)\n│\n└── Data Blocks (File System)\n    └── \u003cbucket_id\u003e/\n        └── \u003chash_prefix\u003e/\n            └── \u003chash\u003e/\n                └── \u003cdataID\u003e_\u003cchunk_number\u003e\n```\n\n## 📊 Performance Highlights\n\n- **Instant Upload**: 99%+ faster for duplicate files (milliseconds vs minutes)\n- **Small Files**: 10x+ performance improvement with packaging\n- **Large Files**: Parallel chunk processing for optimal throughput\n- **Storage Efficiency**: 30-70% space savings with compression + deduplication\n- **Concurrent Operations**: Optimized for high concurrency\n\n**Performance Test Reports**:\n- [S3 API Performance Test Report](s3/docs/PERFORMANCE_TEST_REPORT.md)\n- [VFS Performance Optimization Report](vfs/PERFORMANCE_OPTIMIZATION_FINAL.md)\n\n## 🔧 Path Management\n\nOrcaS supports flexible path management, allowing you to use different storage paths within the same process. This is useful for multi-tenant scenarios or when managing multiple storage locations.\n\n### Creating Handlers with Paths\n\n#### LocalHandler\n\n`NewLocalHandler` requires both `basePath` and `dataPath` parameters:\n\n```go\nimport (\n    \"github.com/orcastor/orcas/core\"\n)\n\n// Create handler with custom paths\nhandler := core.NewLocalHandler(\"/custom/base/path\", \"/custom/data/path\")\ndefer handler.Close()\n\n// basePath: path for main database and bucket databases\n// dataPath: path for data file storage\n```\n\n#### NoAuthHandler\n\n`NewNoAuthHandler` only requires `dataPath` parameter. The `basePath` is automatically set to empty string (no main database):\n\n```go\n// Create NoAuthHandler (bypasses authentication)\nhandler := core.NewNoAuthHandler(\"/custom/data/path\")\ndefer handler.Close()\n\n// Only dataPath is needed, basePath is always empty for NoAuth mode\n```\n\n### Creating Admins with Paths\n\n#### LocalAdmin\n\n`NewLocalAdmin` requires both `basePath` and `dataPath` parameters:\n\n```go\n// Create admin with custom paths\nadmin := core.NewLocalAdmin(\"/custom/base/path\", \"/custom/data/path\")\n\n// basePath: path for main database and bucket databases\n// dataPath: path for data file storage\n```\n\n#### NoAuthAdmin\n\n`NewNoAuthAdmin` only requires `dataPath` parameter. The `basePath` is automatically set to empty string (no main database):\n\n```go\n// Create NoAuthAdmin (bypasses authentication and permission checks)\nadmin := core.NewNoAuthAdmin(\"/custom/data/path\")\n\n// Only dataPath is needed, basePath is always empty for NoAuth mode\n```\n\n### Path Usage Examples\n\n```go\n// Example: Using current directory for both paths\nhandler := core.NewLocalHandler(\".\", \".\")\nadmin := core.NewLocalAdmin(\".\", \".\")\n\n// Example: Separate paths for base and data\nhandler := core.NewLocalHandler(\"/var/orcas/base\", \"/var/orcas/data\")\nadmin := core.NewLocalAdmin(\"/var/orcas/base\", \"/var/orcas/data\")\n\n// Example: NoAuth mode (no main database, only data path)\nhandler := core.NewNoAuthHandler(\"/var/orcas/data\")\nadmin := core.NewNoAuthAdmin(\"/var/orcas/data\")\n```\n\n### Benefits\n\n- 🔄 **Multi-tenant Support**: Different contexts can use different storage paths\n- 🎯 **Flexible Configuration**: Specify paths directly when creating handlers/admins\n- ⚙️ **NoAuth Mode**: Simplified path management for NoAuth handlers/admins (only dataPath needed)\n- 🚀 **Process Isolation**: Multiple storage locations in the same process\n\n## 📚 Documentation\n\n- [Full Documentation](https://orcastor.github.io/doc/)\n- [VFS Mount Guide](vfs/MOUNT_GUIDE.md) - Complete guide for VFS filesystem mounting\n- [S3 API Documentation](s3/README.md)\n- [No Main Database Mode Guide](docs/NO_BASE_DB_GUIDE.md) - Run without main database (no user management)\n\n## 🤝 Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## 📄 License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## ⭐ Why Star This Project?\n\n- 🎯 **Production Ready**: Battle-tested, actively maintained\n- 🚀 **High Performance**: Optimized for real-world workloads\n- 🔒 **Security First**: Zero-knowledge encryption built-in\n- 💾 **Storage Efficient**: Automatic deduplication saves space and costs\n- 🛠️ **Easy to Use**: S3-compatible API, VFS mount, comprehensive docs\n- 🌟 **Innovative**: Content Addressable Storage with instant deduplication\n- 📈 **Actively Developed**: Regular updates and improvements\n- 🤝 **Open Source**: MIT licensed, community-driven\n\n**Star us if you find this project useful!** ⭐\n\n---\n\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Forcastor%2Forcas.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Forcastor%2Forcas?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forcastor%2Forcas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forcastor%2Forcas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forcastor%2Forcas/lists"}