{"id":31770945,"url":"https://github.com/abitofhelp/adaptive_pipeline","last_synced_at":"2026-05-17T06:46:25.635Z","repository":{"id":317465629,"uuid":"1015167130","full_name":"abitofhelp/adaptive_pipeline","owner":"abitofhelp","description":"Adaptive Rust pipeline for high-throughput file processing—dynamic chunking, parallelism, AES/ChaCha encryption, backpressure, and Prometheus/tracing.","archived":false,"fork":false,"pushed_at":"2025-10-07T09:07:16.000Z","size":6059,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-07T11:20:26.652Z","etag":null,"topics":["adaptive-concurrency","backpressure","chunking","concurrency","data-pipeline","encryption","file-processing","metrics","observability","opentelemetry","parallelism","prometheus","rust","stream-processing","tracing"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abitofhelp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-07T05:29:07.000Z","updated_at":"2025-10-07T09:07:20.000Z","dependencies_parsed_at":"2025-10-07T11:34:03.987Z","dependency_job_id":null,"html_url":"https://github.com/abitofhelp/adaptive_pipeline","commit_stats":null,"previous_names":["abitofhelp/optimized_adaptive_pipeline_rs","abitofhelp/adaptive_pipeline"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/abitofhelp/adaptive_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abitofhelp%2Fadaptive_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abitofhelp%2Fadaptive_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abitofhelp%2Fadaptive_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abitofhelp%2Fadaptive_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abitofhelp","download_url":"https://codeload.github.com/abitofhelp/adaptive_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abitofhelp%2Fadaptive_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002104,"owners_count":26083307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adaptive-concurrency","backpressure","chunking","concurrency","data-pipeline","encryption","file-processing","metrics","observability","opentelemetry","parallelism","prometheus","rust","stream-processing","tracing"],"created_at":"2025-10-10T03:17:35.162Z","updated_at":"2025-10-10T03:17:37.689Z","avatar_url":"https://github.com/abitofhelp.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nAdaptive Pipeline\nCopyright (c) 2025 Michael Gardner, A Bit of Help, Inc.\nSPDX-License-Identifier: BSD-3-Clause\nSee LICENSE file in the project root.\n--\u003e\n\n# Adaptive Pipeline\n\nA **production-grade**, **high-performance** file processing system built with Rust, featuring advanced concurrency patterns, adaptive performance optimization, and enterprise-level reliability.\n\n[![License](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)\n[![Rust](https://img.shields.io/badge/rust-1.87%2B-orange.svg)](https://www.rust-lang.org/)\n[![crates.io](https://img.shields.io/crates/v/adaptive-pipeline.svg)](https://crates.io/crates/adaptive-pipeline)\n\n\u003e **⚡ 800+ MB/s sustained throughput** | **🎯 Zero-panic production code** | **🔐 Enterprise-grade security**\n\u003e\n\u003e Benchmarked on Mac Pro 2019 (Intel Xeon W-3235, 12-core, NVMe SSD): **811 MB/s** on 100MB files, **822 MB/s** on 1GB files.\n\u003e Run `adaptive_pipeline benchmark` to measure on your hardware.\n\n## 🚀 What Makes This Different\n\nThis isn't just another file processor - it's a **showcase of advanced Rust patterns** and **production engineering**:\n\n- **⚡ Blazing Fast**: 800+ MB/s sustained throughput with optimized chunk-based parallelism\n- **🔄 Smart Concurrency**: Reader → CPU Workers → Direct Writer pattern eliminates bottlenecks\n- **🎯 Adaptive Performance**: Self-tuning chunk sizes and worker scaling (or manual optimization for 20-60% gains)\n- **🛡️ Zero-Panic Production**: No unwrap/expect/panic in production code - enforced by CI\n- **🔐 Enterprise Security**: AES-256-GCM, ChaCha20-Poly1305, Argon2 KDF, secure key management\n- **📊 Observable**: Prometheus metrics endpoint, structured tracing, real-time dashboards\n\n## 📦 Workspace Structure\n\nThis project is organized as a multi-crate workspace following Clean Architecture principles:\n\n| Crate | Purpose | Documentation |\n|-------|---------|---------------|\n| **[adaptive-pipeline](adaptive_pipeline/)** | Application \u0026 infrastructure layers, CLI | [README](adaptive_pipeline/README.md) |\n| **[adaptive-pipeline-domain](adaptive_pipeline_domain/)** | Pure business logic, DDD entities \u0026 services | [README](adaptive_pipeline_domain/README.md) |\n| **[adaptive-pipeline-bootstrap](adaptive_pipeline_bootstrap/)** | Platform abstraction, signal handling, DI | [README](adaptive_pipeline_bootstrap/README.md) |\n\nEach crate is independently publishable on crates.io and has its own comprehensive documentation.\n\n## 📋 Table of Contents\n\n- [Workspace Structure](#-workspace-structure)\n- [Quick Start](#-quick-start)\n- [Command Line Reference](#-command-line-reference)\n- [Architecture](#-architecture)\n- [Performance](#-performance)\n- [Features](#-features)\n- [Development](#-development)\n- [Advanced Usage](#-advanced-usage)\n\n## 🏗️ Architecture\n\n### Clean Architecture Layers\n\n```\n┌─────────────────────────────────────────────┐\n│         Bootstrap Layer                     │\n│  (DI, Platform Detection, Signal Handling)  │\n│  📦 adaptive-pipeline-bootstrap             │\n└─────────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────────┐\n│         Presentation Layer                  │\n│  (CLI, API endpoints, DTOs)                 │\n│  📦 adaptive-pipeline (CLI)                 │\n└─────────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────────┐\n│         Application Layer                   │\n│  (Use cases, orchestration, async services) │\n│  📦 adaptive-pipeline (use cases)           │\n└─────────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────────┐\n│           Domain Layer                      │\n│  (Pure business logic - SYNC only)          │\n│  📦 adaptive-pipeline-domain                │\n└─────────────────────────────────────────────┘\n                    ↑\n┌─────────────────────────────────────────────┐\n│        Infrastructure Layer                 │\n│  (Database, File I/O, External Systems)     │\n│  📦 adaptive-pipeline (infrastructure)      │\n└─────────────────────────────────────────────┘\n```\n\n### Concurrency Model\n\n**Channel-Based Execution Pipeline**:\n\n```\n┌─────────────┐    Channel     ┌──────────────┐    Direct Write    ┌────────────┐\n│   Reader    │───────────────→│ CPU Workers  │───────────────────→│   Writer   │\n│   Task      │  Backpressure  │  (Parallel)  │  Random Access     │  (.adapipe)│\n└─────────────┘                └──────────────┘                    └────────────┘\n      ↓                              ↓ ↓ ↓                              ↓\n  File I/O                      Rayon Threads                    Concurrent Seeks\n  (Streaming)                   (CPU-bound)                      (No Mutex!)\n```\n\n**Key Design Decisions:**\n\n1. **Reader Task**: Streams file chunks with backpressure - prevents memory overload\n2. **CPU Workers**: Rayon thread pool for parallel stage execution (compress/encrypt)\n3. **Direct Writes**: Workers write directly to calculated positions - **no writer bottleneck**\n4. **Resource Manager**: Global semaphores prevent CPU/IO oversubscription\n\n**Architecture Principles:**\n\n- **Domain Layer** ([adaptive-pipeline-domain](adaptive_pipeline_domain/)): Pure Rust, no async, no I/O - just business logic\n- **Infrastructure Layer** ([adaptive-pipeline](adaptive_pipeline/)): All I/O, all async, all external dependencies\n- **Bootstrap Layer** ([adaptive-pipeline-bootstrap](adaptive_pipeline_bootstrap/)): Platform abstraction, DI, signal handling\n- **Dependency Inversion**: Domain defines interfaces, infrastructure implements\n- **Hexagonal Ports**: `FileIOService`, `CompressionService` are domain ports\n\nFor detailed architecture documentation, see each crate's README.\n\n## ⚡ Performance\n\n### Benchmarks (Mac Pro 2019, Intel Xeon W-3235 @ 3.3GHz, 12-core/24-thread, 48GB RAM, NVMe SSD)\n\nMeasured with `adaptive_pipeline benchmark` command:\n\n| File Size | Best Throughput | Optimal Config | Adaptive Config |\n|-----------|----------------|----------------|-----------------|\n| 100 MB    | **811 MB/s**   | 16MB chunks, 7 workers | 502 MB/s (16MB, 8 workers) |\n| 1 GB      | **822 MB/s**   | 64MB chunks, 5 workers | 660 MB/s (64MB, 10 workers) |\n\n**Key Findings:**\n- 🚀 **800+ MB/s sustained** throughput proves production-grade performance\n- 🎯 **Optimal != Maximum**: 5-7 workers often beat 12+ workers (less context switching overhead)\n- 📦 **Larger chunks win**: 16-64MB chunks maximize sequential I/O on modern SSDs\n- 📈 **Linear scaling**: Performance maintained from 100MB to multi-GB files\n- 🔧 **20-60% optimization headroom**: Adaptive baseline is solid; tuning unlocks more\n\n**Want to see YOUR numbers?**\n```bash\ncargo install adaptive-pipeline\nadaptive_pipeline benchmark --file \u003cyour-file\u003e\n```\n\nPerformance varies by CPU architecture, core count, and storage type. These numbers demonstrate what's possible with modern Rust on server-grade hardware.\n\n### Optimizations Implemented\n\n✅ **Memory Efficiency**\n- Streaming I/O (no full-file read)\n- Memory-mapped files for large data\n- Zero-copy operations where possible\n- Adaptive chunk sizing (1MB-64MB)\n\n✅ **CPU Optimization**\n- Rayon work-stealing for CPU-bound ops\n- SIMD acceleration (where available)\n- Lock-free metrics collection\n- Parallel chunk processing\n\n✅ **I/O Optimization**\n- Async I/O with Tokio\n- Direct concurrent writes (no mutex!)\n- Read-ahead buffering\n- Write coalescing\n\n✅ **Concurrency Patterns**\n- Channel backpressure prevents overload\n- Resource tokens prevent oversubscription\n- Graceful cancellation (CancellationToken)\n- Supervision tree for task management\n\n## 🚀 Quick Start\n\n### As a CLI Tool\n\n#### Prerequisites\n\n- **Rust**: 1.70 or later (install via [rustup](https://rustup.rs/))\n- **SQLite**: For pipeline persistence\n- **OS**: macOS, Linux, or Windows\n\n#### Installation\n\n**From crates.io:**\n```bash\ncargo install adaptive-pipeline\n```\n\n**From source:**\n```bash\n# Clone the repository\ngit clone https://github.com/abitofhelp/optimized_adaptive_pipeline_rs.git\ncd optimized_adaptive_pipeline_rs\n\n# Build optimized binary\nmake build-release\n\n# Binary location\n./target/release/adaptive-pipeline --help\n```\n\n### As a Library\n\nAdd to your `Cargo.toml`:\n\n```toml\n[dependencies]\nadaptive-pipeline = \"1.0\"\nadaptive-pipeline-domain = \"1.0\"  # For pure domain logic\nadaptive-pipeline-bootstrap = \"1.0\"  # For platform abstraction\n```\n\nSee the [adaptive-pipeline README](adaptive_pipeline/README.md) for library usage examples.\n\n### First Run\n\n```bash\n# Create a test file\ndd if=/dev/urandom of=test.dat bs=1M count=100\n\n# Process with compression + encryption\nadaptive-pipeline process \\\n  --input test.dat \\\n  --output test.adapipe \\\n  --compress \\\n  --encrypt\n\n# Check output\nls -lh test.adapipe\n\n# Restore original\nadaptive-pipeline restore \\\n  --input test.adapipe \\\n  --output restored.dat\n\n# Verify integrity\ndiff test.dat restored.dat\n```\n\n## 📟 Command Line Reference\n\n### Global Options\n\n```bash\nadaptive-pipeline [OPTIONS] \u003cCOMMAND\u003e\n\nOptions:\n  -v, --verbose              Enable verbose logging\n  -c, --config \u003cPATH\u003e        Configuration file path\n      --cpu-threads \u003cN\u003e      Override CPU worker thread count (default: num_cpus - 1)\n      --io-threads \u003cN\u003e       Override I/O worker thread count (default: auto-detect)\n      --storage-type \u003cTYPE\u003e  Storage device type: nvme, ssd, hdd (default: auto)\n      --channel-depth \u003cN\u003e    Channel depth for pipeline stages (default: 4)\n  -h, --help                 Print help\n  -V, --version              Print version\n```\n\n### Commands\n\n#### `process` - Process File Through Pipeline\n\nProcess a file through a configured pipeline with compression, encryption, or validation.\n\n```bash\nadaptive-pipeline process --input \u003cFILE\u003e --output \u003cFILE\u003e --pipeline \u003cNAME\u003e [OPTIONS]\n\nOptions:\n  -i, --input \u003cFILE\u003e         Input file path\n  -o, --output \u003cFILE\u003e        Output file path (.adapipe)\n  -p, --pipeline \u003cNAME\u003e      Pipeline name (e.g., \"compress-encrypt\")\n      --chunk-size-mb \u003cMB\u003e   Chunk size in MB (default: adaptive)\n      --workers \u003cN\u003e          Number of parallel workers (default: adaptive)\n\nExamples:\n  # Process with default pipeline\n  pipeline process -i large.dat -o large.adapipe -p compress-encrypt\n\n  # Process with custom settings\n  pipeline process -i data.bin -o data.adapipe -p secure \\\n    --chunk-size-mb 16 --workers 8\n\n  # Process on NVMe with optimized I/O\n  pipeline process -i huge.dat -o huge.adapipe -p fast \\\n    --storage-type nvme --io-threads 24\n```\n\n#### `create` - Create New Pipeline\n\nCreate a new processing pipeline with custom stages.\n\n```bash\nadaptive-pipeline create --name \u003cNAME\u003e --stages \u003cSTAGES\u003e [OPTIONS]\n\nOptions:\n  -n, --name \u003cNAME\u003e          Pipeline name (kebab-case)\n  -s, --stages \u003cSTAGES\u003e      Comma-separated stages: compression,encryption,integrity\n  -o, --output \u003cFILE\u003e        Save pipeline definition to file (optional)\n\nSupported Stages:\n  compression                Brotli compression (default)\n  compression:zstd           Zstandard compression\n  compression:lz4            LZ4 compression\n  encryption                 AES-256-GCM encryption (default)\n  encryption:chacha20        ChaCha20-Poly1305 encryption\n  integrity                  SHA-256 checksum\n  passthrough                No transformation (testing)\n\nExamples:\n  # Create compression-only pipeline\n  pipeline create -n compress-only -s compression\n\n  # Create secure pipeline with encryption\n  pipeline create -n secure-backup -s compression:zstd,encryption,integrity\n\n  # Create fast pipeline with LZ4\n  pipeline create -n fast-compress -s compression:lz4\n```\n\n#### `list` - List Available Pipelines\n\nList all configured pipelines in the database.\n\n```bash\nadaptive-pipeline list\n\nExample Output:\n  Found 3 pipeline(s):\n\n  Pipeline: compress-encrypt\n    ID: 550e8400-e29b-41d4-a716-446655440000\n    Status: active\n    Stages: 2\n    Created: 2025-01-15 10:30:45 UTC\n    Updated: 2025-01-15 10:30:45 UTC\n```\n\n#### `show` - Show Pipeline Details\n\nDisplay detailed information about a specific pipeline.\n\n```bash\nadaptive-pipeline show \u003cPIPELINE_NAME\u003e\n\nArguments:\n  \u003cPIPELINE_NAME\u003e  Name of the pipeline to show\n\nExample:\n  pipeline show compress-encrypt\n\nExample Output:\n  === Pipeline Details ===\n  ID: 550e8400-e29b-41d4-a716-446655440000\n  Name: compress-encrypt\n  Status: active\n  Created: 2025-01-15 10:30:45 UTC\n\n  Stages (2):\n    1. compression (Compression)\n       Algorithm: brotli\n       Enabled: true\n       Order: 0\n\n    2. encryption (Encryption)\n       Algorithm: aes256gcm\n       Enabled: true\n       Order: 1\n```\n\n#### `delete` - Delete Pipeline\n\nDelete a pipeline from the database.\n\n```bash\nadaptive-pipeline delete \u003cPIPELINE_NAME\u003e [OPTIONS]\n\nArguments:\n  \u003cPIPELINE_NAME\u003e  Name of the pipeline to delete\n\nOptions:\n      --force      Skip confirmation prompt\n\nExamples:\n  # Delete with confirmation\n  pipeline delete old-pipeline\n\n  # Force delete without confirmation\n  pipeline delete old-pipeline --force\n```\n\n#### `restore` - Restore Original File\n\nRestore an original file from a processed `.adapipe` file.\n\n```bash\nadaptive-pipeline restore --input \u003cFILE\u003e [OPTIONS]\n\nOptions:\n  -i, --input \u003cFILE\u003e         .adapipe file to restore from\n  -o, --output-dir \u003cDIR\u003e     Output directory (default: use original path)\n      --mkdir                Create directories without prompting\n      --overwrite            Overwrite existing files without prompting\n\nExamples:\n  # Restore to original location\n  pipeline restore -i backup.adapipe\n\n  # Restore to specific directory\n  pipeline restore -i data.adapipe -o /tmp/restored/ --mkdir\n\n  # Force overwrite existing file\n  pipeline restore -i file.adapipe --overwrite\n```\n\n#### `validate` - Validate Configuration\n\nValidate a pipeline configuration file (TOML/JSON/YAML).\n\n```bash\nadaptive-pipeline validate \u003cCONFIG_FILE\u003e\n\nArguments:\n  \u003cCONFIG_FILE\u003e  Path to configuration file\n\nSupported Formats:\n  - TOML (.toml)\n  - JSON (.json)\n  - YAML (.yaml, .yml)\n\nExample:\n  pipeline validate pipeline-config.toml\n```\n\n#### `validate-file` - Validate .adapipe File\n\nValidate the integrity of a processed `.adapipe` file.\n\n```bash\nadaptive-pipeline validate-file --file \u003cFILE\u003e [OPTIONS]\n\nOptions:\n  -f, --file \u003cFILE\u003e  .adapipe file to validate\n      --full         Perform full streaming validation (decrypt/decompress/verify)\n\nExamples:\n  # Quick format validation\n  pipeline validate-file -f output.adapipe\n\n  # Full integrity check (slower but thorough)\n  pipeline validate-file -f output.adapipe --full\n```\n\n#### `compare` - Compare Files\n\nCompare an original file against its `.adapipe` processed version.\n\n```bash\nadaptive-pipeline compare --original \u003cFILE\u003e --adapipe \u003cFILE\u003e [OPTIONS]\n\nOptions:\n  -o, --original \u003cFILE\u003e  Original file to compare\n  -a, --adapipe \u003cFILE\u003e   .adapipe file to compare against\n      --detailed         Show detailed differences\n\nExample:\n  pipeline compare -o original.dat -a processed.adapipe --detailed\n\nExample Output:\n  📊 File Comparison:\n     Original file: original.dat\n     .adapipe file: processed.adapipe\n\n  📏 Size Comparison:\n     Current file size: 104857600 bytes\n     Expected size: 104857600 bytes\n     ✅ Size matches\n\n  🔐 Checksum Comparison:\n     Expected: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n     Current:  e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n     ✅ Checksums match - files are identical\n```\n\n#### `benchmark` - System Benchmarking\n\nRun performance benchmarks to optimize configuration.\n\n```bash\nadaptive-pipeline benchmark [OPTIONS]\n\nOptions:\n  -f, --file \u003cFILE\u003e          Use existing file for benchmark\n      --size-mb \u003cMB\u003e         Test data size in MB (default: 100)\n      --iterations \u003cN\u003e       Number of iterations (default: 3)\n\nExamples:\n  # Quick benchmark with defaults\n  pipeline benchmark\n\n  # Comprehensive benchmark\n  pipeline benchmark --size-mb 1000 --iterations 5\n\n  # Benchmark with existing file\n  pipeline benchmark -f /path/to/large/file.dat\n\nOutput:\n  - Generates optimization report: pipeline_optimization_report.md\n  - Tests multiple chunk sizes and worker counts\n  - Recommends optimal configuration for your system\n```\n\n### Exit Codes\n\nThe CLI uses standard Unix exit codes (sysexits.h):\n\n| Code | Name         | Description                          |\n|------|--------------|--------------------------------------|\n| 0    | SUCCESS      | Command completed successfully       |\n| 1    | ERROR        | General error                        |\n| 65   | EX_DATAERR   | Invalid data or configuration        |\n| 66   | EX_NOINPUT   | Input file not found                 |\n| 70   | EX_SOFTWARE  | Internal software error              |\n| 74   | EX_IOERR     | I/O error (read/write failure)       |\n\n**Usage in scripts:**\n```bash\n#!/bin/bash\nadaptive-pipeline process -i input.dat -o output.adapipe -p compress-encrypt\nEXIT_CODE=$?\n\nif [ $EXIT_CODE -eq 0 ]; then\n    echo \"Success!\"\nelif [ $EXIT_CODE -eq 66 ]; then\n    echo \"Input file not found\"\nelif [ $EXIT_CODE -eq 74 ]; then\n    echo \"I/O error - check disk space and permissions\"\nelse\n    echo \"Error occurred (code: $EXIT_CODE)\"\nfi\n```\n\n### Environment Variables\n\n```bash\n# Database location\nexport ADAPIPE_SQLITE_PATH=\"./pipeline.db\"\n\n# Logging configuration\nexport RUST_LOG=\"pipeline=debug,tower_http=warn\"\n\n# Performance tuning\nexport RAYON_NUM_THREADS=8        # CPU thread pool size\nexport TOKIO_WORKER_THREADS=4     # Async I/O thread pool size\n```\n\n### Shell Completion\n\nGenerate shell completion scripts for your shell:\n\n```bash\n# Bash\nadaptive-pipeline --generate-completion bash \u003e /etc/bash_completion.d/pipeline\n\n# Zsh\nadaptive-pipeline --generate-completion zsh \u003e /usr/local/share/zsh/site-functions/_pipeline\n\n# Fish\nadaptive-pipeline --generate-completion fish \u003e ~/.config/fish/completions/pipeline.fish\n\n# PowerShell\nadaptive-pipeline --generate-completion powershell \u003e pipeline.ps1\n```\n\n## ✨ Features\n\n### Core Capabilities\n\n🔄 **Multi-Stage Processing Pipeline**\n- Configurable stages: compression, encryption, validation\n- Dynamic stage ordering and parallelization\n- Plugin architecture for custom stages\n\n🔐 **Enterprise Security**\n- **Encryption**: AES-256-GCM (hardware accelerated), ChaCha20-Poly1305\n- **Key Derivation**: Argon2id, scrypt, PBKDF2\n- **Memory Safety**: Automatic key zeroing, secure memory handling\n- **Integrity**: SHA-256 checksums, chunk-level verification\n\n⚡ **Adaptive Performance**\n- Auto-detects optimal chunk size (1MB-64MB)\n- Dynamic worker count (1-32 based on cores)\n- File size-based optimization profiles\n- Resource-aware backpressure\n\n📊 **Observability**\n- Prometheus metrics endpoint\n- Structured logging (tracing)\n- Performance dashboards (Grafana)\n- Queue depth monitoring\n\n🛡️ **Production Reliability**\n- Zero panic in production code\n- Comprehensive error handling\n- Graceful shutdown (SIGTERM/SIGINT/SIGHUP)\n- Supervision tree for task recovery\n\n### Supported Algorithms\n\n**Compression:**\n- Brotli (best ratio, good speed)\n- Zstd (balanced)\n- Gzip (widely compatible)\n- LZ4 (fastest)\n\n**Encryption:**\n- AES-256-GCM (default, hardware accelerated)\n- ChaCha20-Poly1305 (constant-time)\n- AES-128/192-GCM variants\n\n**Key Derivation:**\n- Argon2id (memory-hard, GPU-resistant)\n- scrypt (alternative memory-hard)\n- PBKDF2 (legacy compatibility)\n\n## 💻 Development\n\n### Makefile Targets\n\n```bash\n# Development\nmake check              # Fast syntax check\nmake build             # Debug build\nmake build-release     # Optimized build\n\n# Quality\nmake lint              # Development linting\nmake lint-strict       # Production linting (no unwrap/panic)\nmake format            # Auto-format code\nmake test              # Run all tests\n\n# CI/CD\nmake ci-local          # Full CI pipeline locally\nmake pre-commit        # Pre-commit checks\n\n# Performance\nmake bench             # Run benchmarks\nmake flamegraph        # Generate flamegraph\n\n# Documentation\nmake doc-open          # Generate \u0026 open docs\n```\n\n### Running Tests\n\n```bash\n# All tests\ncargo test --workspace\n\n# Unit tests only (fast)\ncargo test --lib\n\n# Integration tests\ncargo test --test '*'\n\n# With logging\nRUST_LOG=debug cargo test -- --nocapture\n\n# Specific test\ncargo test test_channel_pipeline\n```\n\n### Code Quality Standards\n\n**Zero Tolerance in Production:**\n```rust\n// ❌ Never in production code\n.unwrap()\n.expect(\"...\")\npanic!(\"...\")\ntodo!()\nunimplemented!()\n\n// ✅ Always use\n.map_err(|e| PipelineError::...)?\n.ok_or_else(|| PipelineError::...)?\nResult\u003cT, PipelineError\u003e\n```\n\n**Enforced by CI:**\n```bash\nmake lint-strict  # Must pass for merge\n\n# Denies:\n# - clippy::unwrap_used\n# - clippy::expect_used\n# - clippy::panic\n# - clippy::todo\n# - clippy::unimplemented\n```\n\n### Project Structure\n\n```\npipeline/src/\n├── application/\n│   ├── services/\n│   │   ├── pipeline_service.rs          # Channel-based orchestration\n│   │   ├── file_processor_service.rs    # File processing logic\n│   │   └── transactional_chunk_writer.rs # Concurrent writes\n│   └── use_cases/\n│       └── restore_file.rs               # File restoration\n│\n├── infrastructure/\n│   ├── adapters/\n│   │   ├── compression_service_adapter.rs\n│   │   ├── encryption_service_adapter.rs\n│   │   └── file_io_service_adapter.rs\n│   ├── config/\n│   │   └── rayon_config.rs               # CPU pool management\n│   ├── metrics/\n│   │   ├── metrics_service.rs            # Prometheus integration\n│   │   └── concurrency_metrics.rs        # Queue depth tracking\n│   ├── runtime/\n│   │   ├── resource_manager.rs           # CPU/IO tokens\n│   │   └── supervisor.rs                 # Task supervision\n│   └── repositories/\n│       └── sqlite_pipeline_repository.rs # Pipeline persistence\n│\n├── presentation/\n│   └── cli/\n│       └── commands.rs                    # CLI interface\n│\n└── main.rs                                # Entry point\n\nbootstrap/src/\n├── config.rs           # DI container\n├── signals.rs          # Signal handling\n└── platform/           # Platform abstractions\n\npipeline-domain/src/\n├── entities/           # Business entities\n├── value_objects/      # Domain values\n└── services/           # Domain services (sync)\n```\n\n## 🎯 Advanced Usage\n\n### Custom Pipeline Configuration\n\n```rust\nuse pipeline::PipelineBuilder;\n\n// Create custom pipeline\nlet pipeline = PipelineBuilder::new(\"secure-backup\")\n    .add_compression(\"zstd\", 9)\n    .add_encryption(\"aes256gcm\")\n    .add_integrity_check()\n    .chunk_size_mb(16)\n    .workers(8)\n    .build()?;\n\n// Execute\npipeline.process(\"input.dat\", \"output.adapipe\").await?;\n```\n\n### Benchmarking\n\n```bash\n# Auto-benchmark all file sizes\nadaptive-pipeline benchmark\n\n# Specific size with iterations\nadaptive-pipeline benchmark \\\n  --size-mb 1000 \\\n  --iterations 5 \\\n  --output-report bench_results.md\n\n# Compare configurations\nadaptive-pipeline compare \\\n  --configs baseline.toml,optimized.toml \\\n  --size-mb 500\n```\n\n### Monitoring\n\n```bash\n# Start with metrics\nadaptive-pipeline serve --metrics-port 9090\n\n# Query Prometheus\ncurl http://localhost:9090/metrics\n\n# Key metrics:\n# - pipeline_throughput_bytes_per_second\n# - pipeline_cpu_queue_depth\n# - pipeline_worker_utilization\n# - pipeline_chunk_processing_duration_ms\n```\n\n### Platform-Specific Builds\n\n```bash\n# Cross-compilation (requires cross)\nmake install-cross-targets\n\n# Linux x86_64\nmake build-linux-x86_64\n\n# macOS Apple Silicon\nmake build-macos-aarch64\n\n# Windows x64\nmake build-windows-x86_64\n\n# All platforms\nmake build-all-platforms\n```\n\n## 🔧 Configuration\n\n### Example: `pipeline.toml`\n\n```toml\n[pipeline]\nname = \"production-pipeline\"\nchunk_size_mb = 8        # Adaptive default\nparallel_workers = 0     # 0 = auto-detect\n\n[compression]\nalgorithm = \"zstd\"\nlevel = \"balanced\"       # fast | balanced | best\nparallel_processing = true\n\n[encryption]\nalgorithm = \"aes256gcm\"\nkey_derivation = \"argon2id\"\nmemory_cost = 65536     # 64 MB\niterations = 3\n\n[performance]\nmemory_limit_mb = 2048\nuse_memory_mapping = true\ncpu_throttle = false\nsimd_enabled = true\n\n[observability]\nmetrics_enabled = true\nmetrics_port = 9090\ntrace_level = \"info\"    # trace | debug | info | warn | error\n```\n\n### Environment Variables\n\n```bash\n# Database\nexport ADAPIPE_SQLITE_PATH=\"pipeline.db\"\n\n# Logging\nexport RUST_LOG=\"pipeline=debug,tower_http=warn\"\n\n# Performance\nexport RAYON_NUM_THREADS=8\nexport TOKIO_WORKER_THREADS=4\n```\n\n## 📚 Resources\n\n### Documentation\n\n**📖 Complete Guides:**\n- **[User Guide](https://abitofhelp.github.io/adaptive_pipeline/)** - Getting started, installation, and quick reference\n- **[Developer Guide](https://abitofhelp.github.io/adaptive_pipeline/developer/)** - Architecture deep-dive, patterns, and implementation details\n\n**📝 Additional Resources:**\n- [Channel-Based Architecture](docs/EXECUTION_VS_PROCESSING_PIPELINES.md)\n- [Database Setup](docs/DATABASE_SETUP.md)\n- [Performance Tuning](docs/adaptive-performance-optimization.md)\n- [API Documentation](https://docs.rs/adaptive-pipeline)\n\n## 📄 License\n\nThis project is licensed under the **BSD 3-Clause License** - see the [LICENSE](LICENSE) file for details.\n\n## 🙏 Acknowledgments\n\n- Architecture inspired by production systems at scale\n- Concurrency patterns from Tokio/Rayon ecosystems\n- Security design following NIST Cybersecurity Framework\n- Domain-Driven Design principles from Eric Evans\n- Built with ❤️ by the Rust community\n\n---\n\n**Built with Rust 🦀 | Production-Ready ✨ | Performance-First ⚡**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabitofhelp%2Fadaptive_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabitofhelp%2Fadaptive_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabitofhelp%2Fadaptive_pipeline/lists"}