{"id":37909538,"url":"https://github.com/guided-traffic/s3-encryption-proxy","last_synced_at":"2026-05-09T07:07:03.563Z","repository":{"id":327402322,"uuid":"1047333548","full_name":"guided-traffic/s3-encryption-proxy","owner":"guided-traffic","description":"A Go-based proxy that provides transparent encryption/decryption for S3 objects with envelope encryption.","archived":false,"fork":false,"pushed_at":"2026-02-13T07:30:38.000Z","size":4917,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-13T16:33:18.632Z","etag":null,"topics":["encryption","envelope-encryption","proxy","s3","security"],"latest_commit_sha":null,"homepage":"https://s3ep.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guided-traffic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-30T07:19:03.000Z","updated_at":"2026-02-13T07:27:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/guided-traffic/s3-encryption-proxy","commit_stats":null,"previous_names":["guided-traffic/s3-encryption-proxy"],"tags_count":145,"template":false,"template_full_name":null,"purl":"pkg:github/guided-traffic/s3-encryption-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guided-traffic%2Fs3-encryption-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guided-traffic%2Fs3-encryption-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guided-traffic%2Fs3-encryption-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guided-traffic%2Fs3-encryption-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guided-traffic","download_url":"https://codeload.github.com/guided-traffic/s3-encryption-proxy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guided-traffic%2Fs3-encryption-proxy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29612521,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T10:52:55.328Z","status":"ssl_error","status_checked_at":"2026-02-19T10:52:26.323Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["encryption","envelope-encryption","proxy","s3","security"],"created_at":"2026-01-16T17:18:10.635Z","updated_at":"2026-04-07T11:01:37.192Z","avatar_url":"https://github.com/guided-traffic.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 Encryption Proxy\n\nA Go-based proxy that provides transparent encryption/decryption for S3 objects with envelope encryption (RSA or AES), streaming multipart uploads, and HMAC integrity verification.\n\n\n## Overview\n\nThe S3 Encryption Proxy intercepts S3 API calls and automatically:\n- **Encrypts** objects before storing them in S3 using envelope encryption (unique DEK per object)\n- **Decrypts** objects when retrieving them from S3 with automatic provider detection\n- **Verifies** data integrity using HMAC-SHA256 with configurable modes\n- **Maintains** full S3 API compatibility with streaming support for large files\n\n**Key Features:**\n- 🔒 **Transparent Encryption**: No client-side changes required\n- 🔑 **Envelope Encryption**: RSA or AES KEK with unique AES DEK per object\n- 🚀 **S3 API Compatible**: Works with existing S3 clients and tools\n- � **Streaming Uploads**: Memory-efficient multipart uploads with configurable buffer sizes\n- 🛡️ **Integrity Verification**: HMAC-SHA256 with off/lax/strict/hybrid modes\n- 🔐 **Client Authentication**: AWS Signature V4 validation with rate limiting- 🌍 **Environment Variable Support**: Secrets via `${VAR}` references in config files- � **Production Ready**: Comprehensive testing, monitoring, and CI/CD\n\n## Quick Start\n\n### Local Demo (Fastest)\n\n```bash\n# Start MinIO + S3 Encryption Proxy + S3 Explorers\n./start-demo.sh\n\n# Access S3 Explorer (view encrypted data): http://localhost:9001\n# Access Direct S3 Explorer (view raw data): http://localhost:9002\n# Proxy endpoint: http://localhost:8080\n# Metrics endpoint: http://localhost:9090/metrics\n```\n\n### Docker (Recommended)\n\nChoose your encryption provider:\n\n```bash\n# RSA Envelope Encryption (Recommended for production)\ndocker run -p 8080:8080 -p 9090:9090 \\\n  -v $(pwd)/config:/config:ro \\\n  -e RSA_PRIVATE_KEY=\"$(cat private-key.pem)\" \\\n  -e RSA_PUBLIC_KEY=\"$(cat public-key.pem)\" \\\n  ghcr.io/guided-traffic/s3-encryption-proxy:latest \\\n  --config /config/rsa-example.yaml\n\n# AES Envelope Encryption (Simple development setup)\ndocker run -p 8080:8080 -p 9090:9090 \\\n  -v $(pwd)/config:/config:ro \\\n  -e AES_KEY=$(openssl rand -base64 32) \\\n  ghcr.io/guided-traffic/s3-encryption-proxy:latest \\\n  --config /config/aes-example.yaml\n```\n\n### From Source\n\n```bash\n# Clone and build\ngit clone https://github.com/guided-traffic/s3-encryption-proxy.git\ncd s3-encryption-proxy\nmake build\n\n# Generate keys (choose one)\nmake build-keygen \u0026\u0026 ./build/s3ep-keygen           # For AES (outputs base64 key)\ngo build ./cmd/rsa-keygen \u0026\u0026 ./rsa-keygen 2048     # For RSA (generates PEM files)\n\n# Update config file with generated keys\n# Edit config/aes-example.yaml or config/rsa-example.yaml\n\n# Run with configuration\n./build/s3-encryption-proxy --config config/aes-example.yaml\n```\n\n### Client Usage\n\nUse any S3 client with the proxy endpoint:\n\n```bash\n# AWS CLI\naws s3 --endpoint-url http://localhost:8080 cp file.txt s3://my-bucket/\n\n# Python boto3\nimport boto3\ns3 = boto3.client('s3', endpoint_url='http://localhost:8080')\ns3.put_object(Bucket='my-bucket', Key='file.txt', Body=b'data')\n```\n\n## Architecture\n\n```\n┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐\n│   S3 Client     │───►│ Encryption      │───►│   S3 Storage    │\n│   (boto3, aws   │    │ Proxy           │    │   (AWS/MinIO)   │\n│   cli, etc.)    │◄───│ (Go Service)    │◄───│                 │\n└─────────────────┘    └─────────────────┘    └─────────────────┘\n                              │\n                              ▼\n                      ┌─────────────┐\n                      │     KMS     │\n                      │ (Optional)  │\n                      └─────────────┘\n```\n\n## Encryption Providers\n\nThe S3 Encryption Proxy supports multiple encryption providers, each optimized for different use cases:\n\n### 🔐 Provider Comparison\n\n| Feature | **RSA Envelope** | **AES Envelope** | **None** |\n|---------|------------------|------------------|----------|\n| **Security Level** | 🟢 High | 🟢 High | ❌ None |\n| **Performance** | 🟡 Good | 🟢 Excellent | 🟢 Excellent |\n| **KMS Dependency** | ✅ None | ✅ None | ✅ None |\n| **Key Rotation** | � Manual | 🔄 Manual | ❌ N/A |\n| **Unique DEK per Object** | ✅ Yes | ✅ Yes | ❌ N/A |\n| **Setup Complexity** | 🟡 Medium | 🟢 Simple | 🟢 Simple |\n| **Production Ready** | ✅ Yes | ✅ Yes | ❌ Testing Only |\n\n### 1. **RSA Envelope Encryption (Recommended for Production)**\n\n**When to use:** Organizations wanting envelope security without KMS dependency\n```yaml\nproviders:\n  - alias: \"rsa-envelope\"\n    type: \"rsa\"\n    description: \"RSA envelope encryption (auto-selects AES-CTR for multipart, AES-GCM for whole files)\"\n    config:\n      public_key_pem: |\n        -----BEGIN PUBLIC KEY-----\n        ...\n        -----END PUBLIC KEY-----\n      private_key_pem: \"${RSA_PRIVATE_KEY}\"\n```\n\n**Advantages:**\n- 🔒 Strong envelope encryption (RSA + AES-GCM/AES-CTR)\n- 🏠 Self-contained, no external dependencies\n- 🔑 Unique DEK per object\n- 💰 No KMS costs\n- 🔄 Manual key rotation possible\n\n**Disadvantages:**\n- 🔧 Manual key pair management\n- 📁 Private key must be securely stored\n- 🔄 Key rotation requires manual process\n\n### 2. **AES Envelope Encryption (Recommended for Development)**\n\n**When to use:** Development, testing, or simple production setups\n```yaml\nproviders:\n  - alias: \"aes-envelope\"\n    type: \"aes\"\n    description: \"AES envelope encryption (auto-selects AES-CTR for multipart, AES-GCM for whole files)\"\n    config:\n      aes_key: \"base64-encoded-256-bit-key\"\n```\n\n**Advantages:**\n- ⚡ High performance with envelope security\n- 🟢 Simple setup and configuration\n- 🏠 No external dependencies\n- 🔑 Unique DEK per object\n- 🔧 Minimal operational complexity\n\n**Disadvantages:**\n- 🔑 Single master key for all DEK encryption\n- 🔄 Key compromise affects all data\n- 🛡️ Lower security than RSA (symmetric key distribution)\n\n### 3. **None Provider (Testing Only)**\n\n**When to use:** Development testing, performance benchmarking\n```yaml\nproviders:\n  - alias: \"default\"\n    type: \"none\"\n```\n\n**Advantages:**\n- ⚡ Maximum performance (no encryption)\n- 🔧 Zero configuration required\n\n**Disadvantages:**\n- ❌ No encryption or security\n- 🚫 Never use in production\n\n## Multi-Provider Support\n\nThe proxy supports multiple providers simultaneously for migration and compatibility:\n\n```yaml\nencryption:\n  # Active provider for new objects\n  encryption_method_alias: \"aes-current\"\n\n  # Integrity verification: off, lax, strict, hybrid\n  integrity_verification: \"strict\"\n\n  # All providers for reading existing objects\n  providers:\n    - alias: \"aes-current\"\n      type: \"aes\"\n      description: \"Current AES envelope encryption\"\n      config:\n        aes_key: \"XZmcGLpObUuGV8CFOmfLKs7rggrX2TwIk5/Lbt9Azl4=\"\n\n    - alias: \"rsa-backup\"\n      type: \"rsa\"\n      description: \"Backup RSA envelope encryption\"\n      config:\n        public_key_pem: |\n          -----BEGIN PUBLIC KEY-----\n          ...\n          -----END PUBLIC KEY-----\n        private_key_pem: |\n          -----BEGIN PRIVATE KEY-----\n          ...\n          -----END PRIVATE KEY-----\n```\n\n## Key Generation Tools\n\n### Generate AES Keys\n```bash\n# Build and run AES key generator\nmake build-keygen \u0026\u0026 ./build/s3ep-keygen\n```\n\n### Generate RSA Key Pairs\n```bash\n# Build and run RSA key generator\ngo build ./cmd/rsa-keygen \u0026\u0026 ./rsa-keygen 2048\n```\n\n## Configuration\n\n### Complete Configuration File Structure\n\n```yaml\n# Server Configuration\nbind_address: \"0.0.0.0:8080\"\nlog_level: \"debug\"  # debug, info, warn, error\nlog_format: \"text\"  # text or json\nlog_health_requests: false\n\n# S3 Backend Configuration\ns3_backend:\n  target_endpoint: \"https://s3.amazonaws.com\"\n  region: \"us-east-1\"\n  access_key_id: \"your-access-key\"\n  secret_key: \"your-secret-key\"\n  use_tls: true\n  insecure_skip_verify: false\n\n# S3 Client Authentication (Enterprise Security)\ns3_clients:\n  - type: \"static\"\n    access_key_id: \"client-user\"\n    secret_key: \"minimum-16-chars\"  # minimum 16 characters\n    description: \"Client authentication\"\n\n# S3 Security Configuration\ns3_security:\n  strict_signature_validation: true\n  max_clock_skew_seconds: 300  # 5 minutes\n  enable_rate_limiting: true\n  max_requests_per_minute: 60\n  enable_security_logging: true\n  max_failed_attempts: 5\n  unblock_ip_seconds: 60\n\n# Monitoring\nmonitoring:\n  enabled: true\n  bind_address: \":9090\"\n  metrics_path: \"/metrics\"\n\n# License\nlicense_file: \"config/license.jwt\"\n\n# Encryption Configuration\nencryption:\n  encryption_method_alias: \"current-provider\"\n  integrity_verification: \"strict\"  # off, lax, strict, hybrid\n  # metadata_key_prefix: \"s3ep-\"    # Optional custom prefix\n  providers:\n    - alias: \"current-provider\"\n      type: \"aes\"  # or \"rsa\", \"none\"\n      config: { ... }\n\n# Performance Optimizations\noptimizations:\n  streaming_buffer_size: 65536      # 64KB (4KB - 2MB)\n  streaming_segment_size: 12582912  # 12MB (5MB - 5GB)\n  enable_adaptive_buffering: false\n  streaming_threshold: 5242880      # 5MB\n  clean_aws_signature_v4_chunked: true\n  clean_http_transfer_chunked: false\n```\n\n### Environment Variable References\n\nConfiguration values can reference environment variables using the `${VAR_NAME}` syntax. This avoids storing secrets directly in config files.\n\n**Supported fields:**\n- `s3_backend.access_key_id`, `s3_backend.secret_key`\n- `s3_clients[].access_key_id`, `s3_clients[].secret_key`\n- All string values in `encryption.providers[].config` (e.g., `aes_key`, `public_key_pem`, `private_key_pem`)\n\n**Behavior:**\n- Only `${VAR}` syntax is expanded (bare `$VAR` is **not** expanded — safe for passwords containing `$`)\n- If a referenced variable is not set or empty, the proxy **refuses to start** with a clear error message\n- Partial expansion works: `\"prefix-${VAR}-suffix\"`\n- Values without `${...}` are used as-is (no change to existing configs)\n\n**Example configuration:**\n```yaml\ns3_backend:\n  access_key_id: \"${S3_ACCESS_KEY_ID}\"\n  secret_key: \"${S3_SECRET_KEY}\"\n\ns3_clients:\n  - type: \"static\"\n    access_key_id: \"${CLIENT_ACCESS_KEY}\"\n    secret_key: \"${CLIENT_SECRET_KEY}\"\n\nencryption:\n  providers:\n    - alias: \"aes-envelope\"\n      type: \"aes\"\n      config:\n        aes_key: \"${AES_ENCRYPTION_KEY}\"\n\n    # RSA keys via environment variables\n    - alias: \"rsa-envelope\"\n      type: \"rsa\"\n      config:\n        public_key_pem: \"${RSA_PUBLIC_KEY}\"\n        private_key_pem: \"${RSA_PRIVATE_KEY}\"\n```\n\n**Setting the variables:**\n```bash\n# S3 Backend credentials\nexport S3_ACCESS_KEY_ID=\"your-access-key\"\nexport S3_SECRET_KEY=\"your-secret-key\"\n\n# AES key\nexport AES_ENCRYPTION_KEY=\"$(./build/s3ep-keygen)\"\n\n# RSA keys (multiline values work)\nexport RSA_PUBLIC_KEY=\"$(cat public-key.pem)\"\nexport RSA_PRIVATE_KEY=\"$(cat private-key.pem)\"\n```\n\n### Configuration Examples\n\nSee complete examples in the `config/` directory:\n\n#### RSA Envelope Configuration (`config/rsa-example.yaml`)\n```yaml\nencryption:\n  encryption_method_alias: \"rsa-envelope\"\n  integrity_verification: \"strict\"\n  providers:\n    - alias: \"rsa-envelope\"\n      type: \"rsa\"\n      description: \"RSA envelope encryption (auto-selects AES-CTR for multipart, AES-GCM for whole files)\"\n      config:\n        public_key_pem: |\n          -----BEGIN PUBLIC KEY-----\n          MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...\n          -----END PUBLIC KEY-----\n        private_key_pem: \"${RSA_PRIVATE_KEY}\"\n```\n\n#### AES Envelope Configuration (`config/aes-example.yaml`)\n```yaml\nencryption:\n  encryption_method_alias: \"aes-envelope\"\n  integrity_verification: \"strict\"\n  providers:\n    - alias: \"aes-envelope\"\n      type: \"aes\"\n      description: \"AES envelope encryption (auto-selects AES-CTR for multipart, AES-GCM for whole files)\"\n      config:\n        aes_key: \"XZmcGLpObUuGV8CFOmfLKs7rggrX2TwIk5/Lbt9Azl4=\"\n```\n\n#### Multi-Provider Configuration (`config/multi-example.yaml`)\n```yaml\nencryption:\n  encryption_method_alias: \"aes-current\"\n  integrity_verification: \"strict\"\n  providers:\n    # Current encryption for new objects\n    - alias: \"aes-current\"\n      type: \"aes\"\n      description: \"Current AES envelope encryption\"\n      config:\n        aes_key: \"XZmcGLpObUuGV8CFOmfLKs7rggrX2TwIk5/Lbt9Azl4=\"\n\n    # Backup encryption for migration\n    - alias: \"rsa-backup\"\n      type: \"rsa\"\n      description: \"Backup RSA envelope encryption\"\n      config:\n        public_key_pem: \"${RSA_PUBLIC_KEY}\"\n        private_key_pem: \"${RSA_PRIVATE_KEY}\"\n```\n\n#### None Provider Configuration (`config/none-example.yaml`)\n```yaml\nencryption:\n  encryption_method_alias: \"default\"\n  integrity_verification: \"lax\"\n  providers:\n    - alias: \"default\"\n      type: \"none\"\n```\n\n## Documentation\n\nComprehensive documentation is available in the [`docs/`](./docs/) directory:\n\n### 📖 User Guides\n- **[Configuration Guide](./docs/configuration.md)** - Complete configuration reference with examples\n- **[Deployment Guide](./docs/deployment.md)** - Docker, Kubernetes, cloud deployment options\n- **[API Reference](./docs/api-reference.md)** - S3 API compatibility and client integration\n\n### 🏗️ Architecture \u0026 Development\n- **[Architecture Guide](./docs/architecture.md)** - System design and encryption flows\n- **[Development Guide](./docs/development.md)** - Developer setup and contribution guidelines\n- **[Security Guide](./docs/security.md)** - Security architecture and best practices\n\n### 📋 Reference\n- **[Project Summary](./PROJECT-SUMMARY.md)** - High-level project overview\n- **[Contributing Guidelines](./CONTRIBUTING.md)** - How to contribute\n- **[Security Policy](./SECURITY.md)** - Security reporting procedures\n\n## Deployment Options\n\n### Docker\n\n#### With Configuration File (Recommended)\n```bash\n# Build\ndocker build -t s3-encryption-proxy .\n\n# Run with config file\ndocker run -d \\\n  -p 8080:8080 \\\n  -v $(pwd)/config:/config:ro \\\n  s3-encryption-proxy --config /config/aes-example.yaml\n```\n\n#### With Environment Variables\n```bash\n# RSA Envelope\ndocker run -d \\\n  -p 8080:8080 \\\n  -e RSA_PUBLIC_KEY=\"$(cat keys/public-key.pem)\" \\\n  -e RSA_PRIVATE_KEY=\"$(cat keys/private-key.pem)\" \\\n  -v $(pwd)/config:/config:ro \\\n  s3-encryption-proxy --config /config/rsa-example.yaml\n\n# AES Envelope\ndocker run -d \\\n  -p 8080:8080 \\\n  -e AES_KEY=\"$(./build/s3ep-keygen)\" \\\n  -v $(pwd)/config:/config:ro \\\n  s3-encryption-proxy --config /config/aes-example.yaml\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  s3-encryption-proxy:\n    image: ghcr.io/guided-traffic/s3-encryption-proxy:latest\n    ports:\n      - \"8080:8080\"\n      - \"9090:9090\"  # Metrics\n    environment:\n      - RSA_PUBLIC_KEY=${RSA_PUBLIC_KEY}\n      - RSA_PRIVATE_KEY=${RSA_PRIVATE_KEY}\n    volumes:\n      - ./config:/config:ro\n    command: [\"--config\", \"/config/rsa-example.yaml\"]\n```\n\n### Kubernetes with Helm\n\n```bash\n# Use the provided Helm chart\ncd deploy/helm\n./install.sh production\n\n# Or manually with custom values\nhelm install s3-encryption-proxy ./s3-encryption-proxy \\\n  --values values-production.yaml \\\n  --set-file config.yaml=config/rsa-example.yaml \\\n  --set-file secrets.rsaPrivateKey=keys/private-key.pem\n```\n\nExample production values:\n\n```yaml\n# values-production.yaml\nreplicaCount: 3\n\nautoscaling:\n  enabled: true\n  minReplicas: 2\n  maxReplicas: 10\n  targetCPUUtilizationPercentage: 70\n\nresources:\n  limits:\n    cpu: 500m\n    memory: 512Mi\n  requests:\n    cpu: 100m\n    memory: 128Mi\n\nmonitoring:\n  enabled: true\n  serviceMonitor:\n    enabled: true\n```\n\nSee [Deployment Guide](./docs/deployment.md) for complete examples.\n\n## Security\n\n- **🔐 AES-GCM/AES-CTR Encryption**: Industry-standard authenticated encryption\n- **🔑 Envelope Encryption**: KEK/DEK separation for maximum security\n- **🛡️ Integrity Verification**: HMAC-SHA256 with configurable modes (off, lax, strict, hybrid)\n- **🔒 Client Authentication**: AWS Signature V4 validation with rate limiting\n- **📋 Compliance Ready**: Supports SOC 2, GDPR, HIPAA requirements\n\nSee [Security Guide](./docs/security.md) for detailed security information.\n\n## Development\n\n```bash\n# Setup development environment\nmake deps \u0026\u0026 make tools\n\n# Run tests\nmake test\n\n# Code quality checks\nmake quality\n\n# Local development server\nmake dev\n```\n\nSee [Development Guide](./docs/development.md) for complete developer information.\n\n## License\n\nSee [LICENSE](./LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguided-traffic%2Fs3-encryption-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguided-traffic%2Fs3-encryption-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguided-traffic%2Fs3-encryption-proxy/lists"}