{"id":41937986,"url":"https://github.com/microscaler/obsctl","last_synced_at":"2026-01-25T18:16:50.754Z","repository":{"id":302659358,"uuid":"1011816603","full_name":"microscaler/obsctl","owner":"microscaler","description":"A comprehensive, AWS CLI-compliant S3-compatible object storage management tool for any S3-compatible service.","archived":false,"fork":false,"pushed_at":"2025-07-03T14:59:33.000Z","size":785,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-03T15:40:49.082Z","etag":null,"topics":["s3","s3-bucket","s3-client","s3-storage"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microscaler.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-01T11:33:10.000Z","updated_at":"2025-07-03T14:29:38.000Z","dependencies_parsed_at":"2025-07-03T15:53:08.571Z","dependency_job_id":null,"html_url":"https://github.com/microscaler/obsctl","commit_stats":null,"previous_names":["microscaler/obsctl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/microscaler/obsctl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microscaler%2Fobsctl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microscaler%2Fobsctl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microscaler%2Fobsctl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microscaler%2Fobsctl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microscaler","download_url":"https://codeload.github.com/microscaler/obsctl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microscaler%2Fobsctl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28756433,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T16:32:25.380Z","status":"ssl_error","status_checked_at":"2026-01-25T16:32:09.189Z","response_time":113,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["s3","s3-bucket","s3-client","s3-storage"],"created_at":"2026-01-25T18:16:49.875Z","updated_at":"2026-01-25T18:16:50.746Z","avatar_url":"https://github.com/microscaler.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# obsctl - Object Storage Control Tool\n\nA comprehensive, AWS CLI-compliant S3-compatible object storage management tool for **any S3-compatible service**. Originally designed to solve specific challenges with Cloud.ru OBS, obsctl now supports **MinIO, AWS S3, Ceph, DigitalOcean Spaces, Wasabi, Backblaze B2, and any S3-compatible storage** with advanced features not found in traditional tools.\n\nBuilt with Rust for performance, safety, and reliability in production environments.\n\n---\n\n## 🔧 Why obsctl?\n\nTraditional S3 tools don't provide:\n* **Advanced wildcard patterns** - Bulk operations without complex scripting\n* **Universal S3 compatibility** - Works with any S3-compatible storage\n* **File safety checks** - Skip files being actively written\n* **Production observability** - OTEL traces and metrics\n* **Systemd integration** - Proper service lifecycle management\n* **AWS CLI compatibility** - Familiar command structure\n* **Robust error handling** - Comprehensive retry and recovery\n\n`obsctl` was **originally designed to solve specific challenges with Cloud.ru OBS** but has evolved into a universal S3-compatible tool suitable for mission-critical backup, archival, and data management scenarios across any S3-compatible storage provider in regulated and industrial environments.\n\n---\n\n\u003cimg src=\"docs/images/dashboard.png\" alt=\"obsctl dashboard\" width=\"800px\" align=\"centre s\" /\u003e\n\n---\n\n## 🚀 Features\n\n**AWS CLI Compatible Commands:**\n- `ls` - List objects in buckets with **wildcard pattern filtering** and **enterprise-grade advanced filtering** (equivalent to `aws s3 ls`)\n- `cp` - Copy files/objects (equivalent to `aws s3 cp`)\n- `sync` - Sync directories (equivalent to `aws s3 sync`)\n- `rm` - Remove objects (equivalent to `aws s3 rm`)\n- `mb` - Create buckets (equivalent to `aws s3 mb`)\n- `rb` - Remove buckets with **pattern-based bulk deletion** (equivalent to `aws s3 rb`)\n- `presign` - Generate presigned URLs (equivalent to `aws s3 presign`)\n- `head-object` - Show object metadata (equivalent to `aws s3api head-object`)\n- `du` - Storage usage statistics (custom extension)\n\n**🎯 Enterprise-Grade Advanced Filtering:**\n- **Date filtering** - Filter by creation/modification dates (YYYYMMDD + relative formats like 7d, 30d, 1y)\n- **Size filtering** - Filter by file size with multi-unit support (B, KB, MB, GB, TB, PB)\n- **Result management** - Head/tail operations with automatic sorting\n- **Multi-level sorting** - Sort by multiple criteria (e.g., `modified:desc,size:asc,name:asc`)\n- **Performance optimization** - Early termination for head operations, memory-efficient processing\n- **Comprehensive validation** - Mutual exclusion handling, logical validation\n\n**Production-Grade Safety:**\n- Detects files still being written (via file descriptor checks)\n- Skips recently modified files (2-second safety window)\n- **Safety confirmations for pattern-based bulk operations**\n- Systemd integration with health signaling\n- OpenTelemetry (OTEL) observability\n- Retry logic with exponential backoff\n- Configurable concurrency limits\n\n---\n## 🌐 **Universal S3 Compatibility**\n\nobsctl works seamlessly with **any S3-compatible object storage**:\n\n| Provider | Status | Endpoint Example |\n|----------|--------|------------------|\n| **AWS S3** | ✅ Fully Supported | `s3.amazonaws.com` |\n| **Cloud.ru OBS** | ✅ Fully Supported | `obs.ru-moscow-1.hc.sbercloud.ru` |\n| **MinIO** | ✅ Fully Supported | `localhost:9000` |\n| **Ceph RadosGW** | ✅ Fully Supported | `ceph.example.com` |\n| **DigitalOcean Spaces** | ✅ Fully Supported | `nyc3.digitaloceanspaces.com` |\n| **Wasabi** | ✅ Fully Supported | `s3.wasabisys.com` |\n| **Backblaze B2** | ✅ Fully Supported | `s3.us-west-000.backblazeb2.com` |\n| **Any S3 API** | ✅ Fully Supported | Custom endpoints |\n\n---\n\n## 🏗️ Architecture\n\n```mermaid\nsequenceDiagram\n    participant CLI as obsctl CLI\n    participant FS as Filesystem\n    participant PROC as /proc/\u003cpid\u003e/fd\n    participant S3 as S3-Compatible Storage\n    participant OTEL as Telemetry\n    participant SD as Systemd\n\n    CLI-\u003e\u003eSD: sd_notify(READY)\n    CLI-\u003e\u003eFS: Scan source files\n    loop For each file\n        CLI-\u003e\u003eFS: Check modification time\n        CLI-\u003e\u003ePROC: Check open file descriptors\n        alt File is safe\n            CLI-\u003e\u003eS3: Upload/Download/Copy\n            S3--\u003e\u003eCLI: Success/Failure\n            CLI-\u003e\u003eOTEL: Emit telemetry\n        else File in use\n            CLI-\u003e\u003eCLI: Skip with warning\n        end\n    end\n    CLI-\u003e\u003eOTEL: Emit summary\n    CLI-\u003e\u003eSD: sd_notify(STOPPING)\n```\n\n---\n\n## 🛡️ Safety Guarantees\n\n| Safety Feature | Description |\n|---------------|-------------|\n| **Open File Detection** | Skips files with active file descriptors via `/proc/\u003cpid\u003e/fd` |\n| **Modification Window** | Ignores files modified within 2 seconds |\n| **Atomic Operations** | Uses S3 multipart uploads for large files |\n| **Retry Logic** | Exponential backoff with configurable limits |\n| **Dry Run Mode** | Test operations without making changes |\n\n---\n\n## Processor Architecture Matrix\n\n### Supported Architectures\n| Architecture | Target Triple | Platform Support | Use Cases |\n|--------------|---------------|------------------|-----------|\n| **x86_64** | x86_64-unknown-linux-gnu | Linux, Windows | Servers, Desktops |\n| **ARM64** | aarch64-unknown-linux-gnu | Linux, macOS | Apple Silicon, ARM servers |\n| **ARMv7** | armv7-unknown-linux-gnueabihf | Linux | Raspberry Pi, IoT devices |\n| **macOS Intel** | x86_64-apple-darwin | macOS | Intel Macs |\n| **macOS ARM** | aarch64-apple-darwin | macOS | Apple Silicon Macs |\n| **Windows x64** | x86_64-pc-windows-msvc | Windows | Windows desktops/servers |\n\n---\n\n### **Multi-Provider Usage Examples**\n\n```bash\n# AWS S3\nobsctl cp ./data s3://bucket/data --region us-east-1\n\n# Cloud.ru OBS (original use case)\nobsctl cp ./data s3://bucket/data \\\n  --endpoint https://obs.ru-moscow-1.hc.sbercloud.ru \\\n  --region ru-moscow-1\n\n# MinIO (development/testing)\nobsctl cp ./data s3://bucket/data \\\n  --endpoint http://localhost:9000 \\\n  --region us-east-1\n\n# DigitalOcean Spaces\nobsctl cp ./data s3://bucket/data \\\n  --endpoint https://nyc3.digitaloceanspaces.com \\\n  --region nyc3\n\n# Wasabi\nobsctl cp ./data s3://bucket/data \\\n  --endpoint https://s3.wasabisys.com \\\n  --region us-east-1\n```\n\n---\n\n## 🎯 **Unique Differentiator: Advanced Pattern Matching with Auto-Detection**\n\nUnlike basic S3 tools, `obsctl` provides **intelligent pattern matching with automatic regex detection** for bucket operations - a feature that dramatically simplifies bulk operations and makes obsctl much more efficient than traditional tools **across any S3-compatible storage**.\n\n### **Intelligent Auto-Detection System**\n\nobsctl automatically detects whether your pattern is a simple wildcard or advanced regex, giving you the power of both worlds:\n\n- **Simple patterns** → Wildcard matching (user-friendly)\n- **Complex patterns** → Full regex power (advanced users)\n- **No flags needed** → Auto-detection handles everything\n\n### **Pattern-Based Bucket Management**\n\n```bash\n# Simple wildcard patterns (auto-detected)\nobsctl ls --pattern \"*-prod\"                   # Production buckets\nobsctl ls --pattern \"user-?-bucket\"           # Single-digit user buckets\nobsctl ls --pattern \"logs-202[0-9]\"           # Year-based logs\n\n# Advanced regex patterns (auto-detected)\nobsctl ls --pattern \"^backup-\\d{4}-\\d{2}$\"    # Date-formatted backups\nobsctl ls --pattern \"(dev|test|staging)-.*\"   # Multi-environment matching\nobsctl ls --pattern \"user-\\d+-data$\"          # Numeric user data buckets\n\n# Complex real-world scenarios\nobsctl rb --pattern \"^temp-session-[a-f0-9]{8}$\" --confirm  # Session cleanup\nobsctl ls --pattern \"logs-\\d{4}-(0[1-9]|1[0-2])\"          # Monthly logs\n```\n\n### **Comprehensive Pattern Reference**\n\n#### **Wildcard Patterns (Simple \u0026 Familiar)**\n| Pattern | Description | Example | Matches |\n|---------|-------------|---------|---------|\n| `*` | Any sequence of characters | `app-*` | `app-prod`, `app-staging`, `app-dev` |\n| `?` | Single character | `user-?` | `user-1`, `user-a`, `user-x` |\n| `[abc]` | Character set | `env-[dps]*` | `env-dev`, `env-prod`, `env-staging` |\n| `[a-z]` | Character range | `backup-[0-9]` | `backup-1`, `backup-5`, `backup-9` |\n| `[!abc]` | Negated set | `*-[!t]*` | Excludes test environments |\n\n#### **Regex Patterns (Advanced \u0026 Powerful)**\n| Pattern | Description | Example | Matches |\n|---------|-------------|---------|---------|\n| `^pattern$` | Exact match | `^app-prod$` | Only `app-prod` |\n| `\\d+` | One or more digits | `backup-\\d+` | `backup-123`, `backup-2024` |\n| `\\w{3,8}` | 3-8 word characters | `^\\w{3,8}$` | `app`, `bucket`, `data123` |\n| `(a\\|b\\|c)` | Alternation | `(dev\\|test\\|prod)-.*` | `dev-app`, `test-bucket`, `prod-data` |\n| `.*` | Any characters (regex) | `.*-backup-.*` | `app-backup-2024`, `user-backup-old` |\n| `\\d{4}-\\d{2}` | Date patterns | `logs-\\d{4}-\\d{2}` | `logs-2024-01`, `logs-2023-12` |\n| `[a-f0-9]{8}` | Hex patterns | `session-[a-f0-9]{8}` | `session-abc12345`, `session-def67890` |\n| `^(?!test).*` | Negative lookahead | `^(?!test).*-prod$` | `app-prod` but not `test-prod` |\n\n\u003e 💡 **Pro Tip**: Need help building complex regex patterns? Use [**Rubular.com**](https://rubular.com/) - an interactive regex editor where you can test your patterns against sample text in real-time. Perfect for validating bucket naming patterns before running obsctl commands!\n\n### **Real-World Examples by Use Case**\n\n#### **Production Environment Management**\n```bash\n# Production buckets only\nobsctl ls --pattern \".*-prod$\"\n\n# Non-production environments  \nobsctl ls --pattern \".*(dev|test|staging).*\"\n\n# Versioned releases\nobsctl ls --pattern \"^app-v\\d+\\.\\d+\\.\\d+-prod$\"\n```\n\n#### **Date-Based Operations**\n```bash\n# Monthly backups (2024)\nobsctl ls --pattern \"^backup-2024-(0[1-9]|1[0-2])$\"\n\n# Daily logs (January 2024)\nobsctl ls --pattern \"^logs-2024-01-([0-2][0-9]|3[01])$\"\n\n# Quarterly reports\nobsctl ls --pattern \"^reports-\\d{4}-Q[1-4]$\"\n```\n\n#### **User and Session Management**\n```bash\n# Numeric user IDs only\nobsctl ls --pattern \"^user-\\d+-.*\"\n\n# Temporary session buckets\nobsctl rb --pattern \"^temp-session-[a-f0-9]{8,16}$\" --confirm\n\n# User buckets with specific patterns\nobsctl ls --pattern \"^user-[a-z]{3,10}-workspace$\"\n```\n\n#### **Cleanup and Maintenance**\n```bash\n# All test and temporary buckets\nobsctl rb --pattern \"^(test|tmp|temp)-.*\" --confirm\n\n# Old backup buckets (before 2023)\nobsctl rb --pattern \"^backup-20(1[0-9]|2[0-2])-.*\" --confirm\n\n# Development branches\nobsctl rb --pattern \"^dev-feature-.*\" --confirm\n```\n\n#### **Complex Business Logic**\n```bash\n# Multi-region production buckets\nobsctl ls --pattern \"^(us|eu|ap)-(east|west|central)-\\d+-prod$\"\n\n# Compliance-specific patterns\nobsctl ls --pattern \"^(audit|compliance|security)-\\d{4}-(q[1-4]|annual)$\"\n\n# Service-specific buckets\nobsctl ls --pattern \"^(api|web|mobile|backend)-.*-(prod|staging)$\"\n```\n\n### **Auto-Detection Examples**\n\n```bash\n# These are automatically detected as WILDCARD patterns:\nobsctl ls --pattern \"*-prod\"           # Simple wildcard\nobsctl ls --pattern \"user-?\"           # Single character wildcard\nobsctl ls --pattern \"[abc]*\"           # Character class wildcard\n\n# These are automatically detected as REGEX patterns:\nobsctl ls --pattern \"^backup-\\d+$\"     # Contains ^ and \\d\nobsctl ls --pattern \"(dev|test|prod)\"  # Contains parentheses and |\nobsctl ls --pattern \"bucket{3,8}\"      # Contains curly braces\nobsctl ls --pattern \"app\\w+\"           # Contains backslash escape\n```\n\n\u003e 🔧 **Regex Testing**: For complex regex patterns, test them first at [**Rubular.com**](https://rubular.com/) with sample bucket names to ensure they match exactly what you expect before running production commands.\n\n**This intelligent pattern functionality eliminates the need for complex shell scripting with `grep`, `awk`, or multiple API calls** - operations that would require dozens of lines of bash can be done with a single obsctl command **regardless of your S3 provider**.\n\n\n---\n\n## 📖 Quick Start\n\n### Installation\n\n```bash\n# Build from source\ncargo build --release\nsudo cp target/release/obsctl /usr/local/bin/\n\n# Set credentials (works with any S3 provider)\nexport AWS_ACCESS_KEY_ID=\"your-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"your-secret-key\"\n```\n\n### Basic Usage\n\n```bash\n# List bucket contents (any S3 provider)\nobsctl ls s3://my-bucket/\n\n# List buckets with patterns\nobsctl ls --pattern \"*-prod\"                    # Production buckets\nobsctl ls --pattern \"user-[0-9]-*\"             # Numbered user buckets\nobsctl ls --pattern \"logs-202[3-4]\"            # Recent log buckets\n\n# Upload a file\nobsctl cp ./local-file.txt s3://my-bucket/remote-file.txt\n\n# Download a file\nobsctl cp s3://my-bucket/remote-file.txt ./local-file.txt\n\n# Sync directories\nobsctl sync ./local-dir s3://my-bucket/remote-dir/ --delete\n\n# Remove objects\nobsctl rm s3://my-bucket/old-file.txt\n\n# Create/remove buckets\nobsctl mb s3://new-bucket\nobsctl rb s3://empty-bucket --force\n\n# Pattern-based bucket deletion (with safety confirmation)\nobsctl rb --pattern \"test-*\" --confirm         # Delete all test buckets\nobsctl rb --pattern \"temp-[0-9]*\" --confirm    # Delete numbered temp buckets\n\n# Generate presigned URLs\nobsctl presign s3://my-bucket/file.txt --expires-in 3600\n\n# Check storage usage\nobsctl du s3://my-bucket/ --human-readable\n```\n\n### 🎯 **Enterprise-Grade Advanced Filtering**\n\nobsctl provides database-quality filtering capabilities for S3 object operations, enabling sophisticated data lifecycle management, operational monitoring, and cost optimization.\n\n#### **Date Filtering**\n```bash\n# Objects created after specific date (YYYYMMDD format)\nobsctl ls s3://logs/ --created-after 20240101 --recursive\n\n# Objects modified in the last 7 days (relative format)\nobsctl ls s3://data/ --modified-after 7d --recursive\n\n# Date range filtering\nobsctl ls s3://backups/ --created-after 20240101 --created-before 20240131 --recursive\n\n# Recent activity monitoring\nobsctl ls s3://user-data/ --modified-after 1d --sort-by modified:desc --head 50\n```\n\n#### **Size Filtering**\n```bash\n# Large files consuming storage (default unit: MB)\nobsctl ls s3://uploads/ --min-size 100 --recursive --sort-by size:desc\n\n# Small files for cleanup (explicit units)\nobsctl ls s3://cache/ --max-size 1KB --recursive --sort-by size:asc\n\n# Size range filtering\nobsctl ls s3://media/ --min-size 10MB --max-size 1GB --recursive\n\n# Storage optimization analysis\nobsctl ls s3://archive/ --max-size 1MB --created-before 20230101 --max-results 1000\n```\n\n#### **Multi-Level Sorting**\n```bash\n# Primary: modification date (desc), Secondary: size (asc)\nobsctl ls s3://logs/ --sort-by modified:desc,size:asc --recursive\n\n# Complex sorting: date, size, then name\nobsctl ls s3://data/ --sort-by created:desc,size:desc,name:asc --recursive\n\n# Simple reverse sorting\nobsctl ls s3://files/ --sort-by size --reverse --recursive\n```\n\n#### **Head/Tail Operations**\n```bash\n# Most recent 20 files (auto-sorted by modification date)\nobsctl ls s3://activity/ --tail 20 --recursive\n\n# First 100 results after filtering\nobsctl ls s3://bucket/ --min-size 10MB --head 100 --recursive\n\n# Largest 50 files\nobsctl ls s3://storage/ --sort-by size:desc --head 50 --recursive\n\n# Latest 10 log files\nobsctl ls s3://logs/ --pattern \"*.log\" --tail 10 --recursive\n```\n\n#### **Combined Advanced Filtering**\n```bash\n# Data lifecycle management: Find old large files for archival\nobsctl ls s3://production-data/ --recursive \\\n  --modified-before 20230101 \\\n  --min-size 100MB \\\n  --sort-by modified:asc,size:desc \\\n  --max-results 1000\n\n# Security audit: Recently modified sensitive files\nobsctl ls s3://sensitive-data/ --recursive \\\n  --modified-after 1d \\\n  --pattern \"*confidential*\" \\\n  --sort-by modified:desc \\\n  --head 100\n\n# Cost optimization: Small old files analysis\nobsctl ls s3://user-uploads/ --recursive \\\n  --created-before 20231201 \\\n  --max-size 1MB \\\n  --sort-by size:asc,created:asc \\\n  --max-results 5000\n\n# Operational monitoring: Recent large uploads\nobsctl ls s3://uploads/ --recursive \\\n  --created-after 7d \\\n  --min-size 50MB \\\n  --sort-by created:desc,size:desc \\\n  --head 20\n\n# Performance analysis: Files by modification pattern\nobsctl ls s3://app-data/ --recursive \\\n  --modified-after 30d \\\n  --pattern \"*.db\" \\\n  --sort-by modified:desc,size:desc \\\n  --tail 50\n```\n\n#### **Enterprise Use Cases**\n\n**Data Lifecycle Management:**\n```bash\n# Compliance: Files older than 7 years for deletion\nobsctl ls s3://compliance-data/ --recursive \\\n  --created-before 20170101 \\\n  --sort-by created:asc \\\n  --max-results 10000\n\n# Archive candidates: Large old files\nobsctl ls s3://active-storage/ --recursive \\\n  --modified-before 20231201 \\\n  --min-size 1GB \\\n  --sort-by size:desc,modified:asc\n```\n\n**Security \u0026 Auditing:**\n```bash\n# Incident response: Recently modified files\nobsctl ls s3://secure-vault/ --recursive \\\n  --modified-after 1d \\\n  --sort-by modified:desc \\\n  --max-results 500\n\n# Access pattern analysis: Large recent downloads\nobsctl ls s3://download-logs/ --recursive \\\n  --created-after 7d \\\n  --min-size 10MB \\\n  --sort-by created:desc,size:desc\n```\n\n**Cost Optimization:**\n```bash\n# Storage analysis: Small files consuming space\nobsctl ls s3://user-data/ --recursive \\\n  --max-size 100KB \\\n  --sort-by size:asc \\\n  --max-results 10000\n\n# Duplicate analysis: Files of same size\nobsctl ls s3://media-library/ --recursive \\\n  --sort-by size:desc,name:asc \\\n  --max-results 5000\n```\n\n**Performance Monitoring:**\n```bash\n# Hot data identification: Recently accessed large files\nobsctl ls s3://cache-storage/ --recursive \\\n  --modified-after 1d \\\n  --min-size 50MB \\\n  --sort-by modified:desc,size:desc \\\n  --head 100\n```\n\n#### **Advanced Filtering Options Reference**\n\n| Flag | Description | Format | Example |\n|------|-------------|--------|---------|\n| `--created-after` | Filter by creation date (after) | YYYYMMDD or relative | `20240101`, `7d`, `30d`, `1y` |\n| `--created-before` | Filter by creation date (before) | YYYYMMDD or relative | `20241231`, `30d` |\n| `--modified-after` | Filter by modification date (after) | YYYYMMDD or relative | `20240601`, `1d`, `7d` |\n| `--modified-before` | Filter by modification date (before) | YYYYMMDD or relative | `20240630`, `1d` |\n| `--min-size` | Minimum file size | Number + unit | `100MB`, `1GB`, `500KB` |\n| `--max-size` | Maximum file size | Number + unit | `1GB`, `100MB`, `10KB` |\n| `--max-results` | Limit total results | Number | `1000`, `5000`, `10000` |\n| `--head` | First N results | Number | `50`, `100`, `500` |\n| `--tail` | Last N results (auto-sorted) | Number | `20`, `50`, `100` |\n| `--sort-by` | Multi-level sorting | field:dir,field:dir | `modified:desc,size:asc` |\n| `--reverse` | Simple reverse sort | Flag | Use with single sort field |\n\n**Supported Units:** B, KB, MB, GB, TB, PB (decimal) and KiB, MiB, GiB, TiB, PiB (binary)  \n**Sort Fields:** name, size, created, modified  \n**Sort Directions:** asc (ascending), desc (descending)\n\n#### **Performance Features**\n- **Early termination** for head operations (stops processing when limit reached)\n- **Memory-efficient** streaming for large buckets (\u003e100K objects)\n- **Intelligent sorting** with automatic optimization\n- **Result limiting** to prevent memory exhaustion\n- **Auto-sorting** for tail operations (by modification date)\n\n### Provider-Specific Examples\n\n```bash\n# AWS S3 (default behavior)\nobsctl cp ./data/ s3://aws-bucket/data/ --recursive\n\n# Cloud.ru OBS (original use case)\nobsctl cp ./data/ s3://backups/daily/ \\\n  --endpoint https://obs.ru-moscow-1.hc.sbercloud.ru \\\n  --region ru-moscow-1 \\\n  --recursive \\\n  --max-concurrent 8\n\n# MinIO (development/testing)\nobsctl cp ./data/ s3://minio-bucket/data/ \\\n  --endpoint http://localhost:9000 \\\n  --region us-east-1 \\\n  --recursive\n\n# DigitalOcean Spaces\nobsctl cp ./data/ s3://do-space/data/ \\\n  --endpoint https://nyc3.digitaloceanspaces.com \\\n  --region nyc3 \\\n  --recursive\n\n# Wasabi\nobsctl cp ./data/ s3://wasabi-bucket/data/ \\\n  --endpoint https://s3.wasabisys.com \\\n  --region us-east-1 \\\n  --recursive\n```\n\n### Advanced Pattern Examples\n\n```bash\n# Simple wildcard patterns (auto-detected)\nobsctl ls --pattern \"app-*-[0-9][0-9]\"         # Versioned app buckets (app-prod-01, app-staging-15)\nobsctl ls --pattern \"*-[ds]*\"                  # Dev and staging environments  \nobsctl ls --pattern \"backup-202[0-9]-[01][0-9]\" # Monthly backups by year and month\n\n# Advanced regex patterns (auto-detected)\nobsctl ls --pattern \"^app-v\\d+\\.\\d+\\.\\d+$\"     # Semantic versioning (app-v1.2.3)\nobsctl ls --pattern \"^logs-\\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$\" # Daily logs\nobsctl ls --pattern \"^(frontend|backend|api)-.*-(prod|staging)$\" # Service-environment pattern\n\n# Complex business scenarios  \nobsctl ls --pattern \"^user-\\d{6,10}-workspace$\" # Employee ID workspaces\nobsctl ls --pattern \"^backup-\\d{4}-q[1-4]-final$\" # Quarterly backup finalization\nobsctl ls --pattern \"^temp-[a-f0-9]{8}-session$\" # Temporary session storage\n\n# Bulk cleanup operations (with safety confirmations)\nobsctl rb --pattern \"^tmp-.*\" --confirm        # Clean up all temporary buckets\nobsctl rb --pattern \"^test-.*-\\d{8}$\" --confirm # Remove dated test buckets  \nobsctl rb --pattern \"^dev-feature-.*\" --confirm # Remove feature branch buckets\nobsctl rb --pattern \"^(staging|dev)-.*-old$\" --confirm # Remove old non-prod buckets\n```\n\n### Advanced Options\n\n```bash\n# Sync with pattern filtering (any S3 provider)\nobsctl sync ./logs s3://log-bucket/app-logs/ \\\n  --include \"*.log\" \\\n  --exclude \"*.tmp\" \\\n  --delete \\\n  --dryrun\n```\n\n---\n\n## 📊 Observability\n\n### OpenTelemetry Integration\n\n```bash\nexport OTEL_EXPORTER_OTLP_ENDPOINT=\"https://otel-receiver.example.com/v1/traces\"\n\n# Operations automatically emit traces (any S3 provider)\nobsctl cp ./data s3://bucket/data --recursive\n```\n\n### Systemd Health Monitoring\n\n```bash\n# Check service status\nsystemctl status obsctl.service\n\n# View logs\njournalctl -u obsctl.service --since \"1 hour ago\"\n```\n\n---\n\n## 🔧 Configuration\n\n### Environment Variables\n\n```bash\n# AWS Credentials (works with any S3-compatible provider)\nexport AWS_ACCESS_KEY_ID=\"your-access-key\"\nexport AWS_SECRET_ACCESS_KEY=\"your-secret-key\"\n\n# Provider-specific endpoints\nexport AWS_ENDPOINT_URL=\"https://your-s3-provider.com\"\n\n# Observability\nexport OTEL_EXPORTER_OTLP_ENDPOINT=\"https://otel.example.com/otlp\"\n\n# Logging\nexport AWS_LOG_LEVEL=\"debug\"\nexport AWS_SMITHY_LOG=\"debug\"\n```\n\n### Global Options\n\n```bash\nobsctl [GLOBAL_OPTIONS] \u003cCOMMAND\u003e [COMMAND_OPTIONS]\n\nGlobal Options:\n  --debug \u003cLEVEL\u003e         Log level: trace, debug, info, warn, error [default: info]\n  -e, --endpoint \u003cURL\u003e    Custom S3 endpoint URL (for any S3-compatible provider)\n  -r, --region \u003cREGION\u003e   AWS region [default: us-east-1]\n  --timeout \u003cSECONDS\u003e     HTTP timeout [default: 10]\n```\n\n---\n\n## 🧪 Testing\n\n```bash\n# Run all tests\ncargo test\n\n# Run with coverage\ncargo test --coverage\n\n# Test specific functionality\ncargo test s3_uri\n```\n\nTest coverage includes:\n- S3 URI parsing and validation\n- Command-line argument parsing\n- File descriptor detection\n- Error handling scenarios\n- Multi-provider compatibility\n\n---\n\n## 📚 Documentation\n\n- [Operator Manual](docs/MANUAL.md) - Production deployment guide\n- [GitLab CI Integration](docs/GITLAB_CI.md) - CI/CD pipeline setup\n\n---\n\n## Roadmap\n\n### Why “obsctl mount” Matters\n\n#### …and why **our** take on FUSE is different\n\n1. **Zero-friction data access**\n   *Developers can `cd /mnt/obs` and use ordinary CLI tools, scripts, or desktop apps without ever touching S3 semantics.*\n   No more rewriting pipelines from `cp` to `aws s3`—everything “just looks like a folder.”\n\n2. **Legacy compatibility**\n   • Older software that **can’t** speak S3 (e.g., Oracle datapump, image viewers, Photoshop) suddenly works with Cloud.ru buckets.\n   • On-prem ↔ cloud migrations become lift-and-shift: point the path at `/mnt/obs` and you’re done.\n\n3. **Online editing \u0026 instant feedback**\n   Mounts enable IDEs, **VS Code remote workspaces**, or container bind-mounts to treat cloud objects as live files.\n   Paired with multipart write-through, “save” is seconds—not minutes.\n\n---\n\n### Our Unique Selling Point (USP)\n\n| Existing FUSE drivers                        | **obsctl mount**                                                             |\n| -------------------------------------------- | ---------------------------------------------------------------------------- |\n| Expose a bucket ✔️                           | Expose a bucket ✔️                                                           |\n| Basic metrics (if any)                       | **OpenTelemetry by default** — latency histograms, bytes, IOPS, error codes. |\n| Separate config steps                        | **One-liner**: credentials, endpoint, cache flags injected automatically.    |\n| Hard to debug (“it’s slow… why?”)            | **Span-level traces** show which op, object, and HTTP call caused the stall. |\n| Ops learn a new binary (`rclone`, `s3fs`, …) | Same familiar `obsctl` family.                                               |\n| No systemd health                            | Built-in **sd\\_notify**, watchdog, \u0026 auto-remount timer.                     |\n\n#### What “rich OTel” gives you\n\n* **Dashboards in minutes** – drop our pre-made Grafana JSON: see read/write throughput, 99p latency, cache hit ratio.\n* **Alertable KPIs** – error counters already tagged (`bucket`, `op`, `driver`).\n* **Cloud-agnostic** – the same trace schema for AWS S3, Cloud.ru OBS, MinIO, or Ceph.\n* **Cost insights** – bytes transferred per prefix, surfacing the real egress bill in Prometheus.\n\n\u003e **Bottom-line:**\n\u003e *obsctl* doesn’t try to reinvent FUSE; it **wraps** the best driver for the job and turns it into an **observable, self-healing mount** with zero manual plumbing.\n\n\n---\n\n## 🤝 Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature/new-command`\n3. Make changes following AWS CLI conventions\n4. Add tests for new functionality\n5. Submit a pull request\n\n---\n\n## 📄 License\n\nMIT or Apache 2.0\n\n---\n\n## 🏷️ Version History\n\n- **v0.4.0** - **🚀 MAJOR: Intelligent Pattern Matching with Auto-Detection** - Automatic wildcard/regex detection, comprehensive regex support with [Rubular](https://rubular.com/)-style patterns, real-world business logic examples, enhanced Universal S3 compatibility documentation\n- **v0.3.0** - **Advanced wildcard pattern support** for bucket operations, OpenTelemetry built-in by default, **Universal S3 compatibility**\n- **v0.2.0** - AWS CLI compliant command structure, comprehensive S3 operations\n- **v0.1.0** - Initial upload-only tool with safety features for Cloud.ru OBS\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicroscaler%2Fobsctl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicroscaler%2Fobsctl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicroscaler%2Fobsctl/lists"}