{"id":43286600,"url":"https://github.com/moriyoshi/s3-router","last_synced_at":"2026-02-09T02:10:53.205Z","repository":{"id":334645476,"uuid":"1138060298","full_name":"moriyoshi/s3-router","owner":"moriyoshi","description":"S3 reverse proxy supporting complex routing","archived":false,"fork":false,"pushed_at":"2026-01-30T16:47:44.000Z","size":307,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-30T21:48:47.798Z","etag":null,"topics":["aws","aws-s3","proxy","routing","s3","secrets-manager"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moriyoshi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-01-20T07:35:31.000Z","updated_at":"2026-01-30T16:46:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/moriyoshi/s3-router","commit_stats":null,"previous_names":["moriyoshi/s3-router"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/moriyoshi/s3-router","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moriyoshi%2Fs3-router","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moriyoshi%2Fs3-router/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moriyoshi%2Fs3-router/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moriyoshi%2Fs3-router/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moriyoshi","download_url":"https://codeload.github.com/moriyoshi/s3-router/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moriyoshi%2Fs3-router/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28984813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T17:52:09.146Z","status":"ssl_error","status_checked_at":"2026-02-01T17:49:53.529Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","proxy","routing","s3","secrets-manager"],"created_at":"2026-02-01T18:01:38.007Z","updated_at":"2026-02-09T02:10:53.128Z","avatar_url":"https://github.com/moriyoshi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3-router\n\ns3-router is a S3 proxy server that intelligently routes requests to multiple S3-compatible backends based on flexible regex patterns and key transformations.\n\n## Features\n\n* **S3 API Compatibility** - S3 API compatibility for object operations (GET, PUT, DELETE, etc.), multipart uploads, and bucket operations\n* **Flexible Request Routing** - Route requests based on ordered regex patterns on object keys with deterministic rewrites\n* **Request Rewriting** - Multi-step rewrite rules with named and numbered capture group templates for transforming object keys\n* **Multiple Backend Support** - Proxy requests to multiple S3-compatible backends with independent configuration and credentials\n* **Virtual Host Support** - S3 virtual-hosted-style requests with explicit host mappings and wildcard/suffix matching\n* **Advanced Credential Management** - Support for multiple credential sources:\n  - AWS SDK default credential chain (IAM roles, environment variables, shared credentials)\n  - File-based credentials with JSON format\n  - AWS Secrets Manager integration for centralized secret management\n  - STS role assumption for cross-account access\n  - Inline static credentials (not recommended for production)\n* **SigV4 Authentication** - Full AWS SigV4 signature verification with support for header-based and query-based presigned URLs\n* **Circuit Breaker Pattern** - Automatic failure detection and backend isolation with automatic recovery\n* **Request/Response Streaming** - Optimized streaming for large file uploads and downloads with minimal memory usage\n* **Route Caching** - LRU cache for route matching decisions to improve performance\n* **Observability** - Prometheus metrics for monitoring request latency, throughput, and backend health\n* **Health Checks** - Built-in health check endpoints and per-backend health monitoring\n* **Configurable Timeouts** - Per-backend and server-level timeout configuration\n* **Multi-format Configuration** - Support for YAML, JSON, and TOML configuration files\n* **Comprehensive Logging** - Human-friendly and structured JSON logging with automatic format detection\n\n### Configuration language\n\ns3-router loads its routing table from a JSON, YAML, or TOML document with the following structure.\n\n```yaml\nbackends:        # map keyed by backend id\n  backend1:\n    endpoint: s3.ap-northeast-1.amazonaws.com  # optional override; defaults to AWS endpoint logic\n    bucket: bucket-1                           # required physical bucket name\n    prefix: prefix1/                           # optional prefix prepended to every rewritten key\n    timeout: 30s                               # optional per-backend HTTP client timeout (Go duration string)\n    retries: 3                                 # optional retry count for AWS SDK requests\n    credentials:                               # optional; falls back to AWS default chain when omitted\n      type: file                               # credential source type: default, file, aws-secrets-manager, inline\n      path: /etc/s3router/backend-creds.json  # required for type: file\n      assume_role:                             # optional STS role assumption\n        role_arn: arn:aws:iam::1234:role/example\n        session_name: s3-router                # optional; defaults to \"s3-router\"\n        duration: 3600                         # optional; duration in seconds (default: 3600)\n\nbuckets:         # supports both list and map formats (see examples below)\n  foo:                                      # virtual bucket exposed to clients (map key)\n    routes:                                 # first route that matches wins\n      - path: ^foo/(?P\u003crest\u003e.*)             # required RE2 regex applied to the object key\n        backend: backend1\n        rewrite:                            # optional sequential rewrite steps\n          - result: $rest                   # template applies captures from `path`\n          - pattern: ^special/(.*)          # optional secondary regex on current key\n            result: SPECIAL/$1              # supports numbered/named captures ($1, $rest)\n\n# Server configuration (optional; defaults shown)\nserver:\n  read_timeout: 15\n  write_timeout: 15\n  idle_timeout: 60\n  max_body_size: 4294967296\n  route_cache_size: 1000\n\n# Authentication configuration (optional; defaults shown)\nauth:\n  default_region: us-east-1\n  clock_skew_leeway: 15m\n\n# Virtual hosts configuration (optional; defaults shown)\nvirtual_hosts:\n  hosts: []\n\nfeatures:                                  # optional boolean map (currently unused)\n  enable_caching: true\n  enable_metrics: false\n\ncredentials_store: credentials.json         # optional; path to JSON file with static credentials\n```\n\n#### Backends\n\n| Field | Type | Required | Description |\n| --- | --- | --- | --- |\n| `endpoint` | string | no | Custom S3-compatible endpoint with optional template placeholders. Defaults to AWS regional/global endpoint resolution. See [Endpoint Configuration](#endpoint-configuration). |\n| `bucket` | string | yes | Physical bucket that stores the objects. |\n| `prefix` | string | no | Static prefix prepended before the rewritten key (e.g., `folder/`). |\n| `timeout` | string | no | Go duration parsed timeout applied to the backend HTTP client (e.g., `30s`, `2m`). |\n| `retries` | int | no | Retry attempts for SDK operations. Defaults to the SDK's standard behavior. |\n| `credentials` | object | no | Credential source specification with optional role assumption. See [Credential Sources](#credential-sources). |\n\n#### Endpoint Configuration\n\nThe `endpoint` field in backend configuration supports two modes:\n\n**Mode 1: Template-based Endpoint (Custom)**\n\nWhen the `endpoint` field contains placeholders, s3-router resolves them to construct a custom endpoint URL. This is useful for S3-compatible services (MinIO, Wasabi, DigitalOcean Spaces, etc.) or AWS regions with special requirements.\n\n**Supported Placeholders:**\n\n| Placeholder | Format | Description | Example |\n| --- | --- | --- | --- |\n| `bucket` | `${bucket}` or `$bucket` | The physical bucket name | `my-bucket` |\n| `region` | `${region}` or `$region` | AWS region from backend or client context | `us-west-2` |\n| `key` | `${key}` or `$key` | The object key (URL-encoded, preserving `/`) | `folder/myfile.txt` |\n| `fips` | `${fips}` or `$fips` | Set to `1` if FIPS is enabled, empty otherwise | `1` or `` |\n| `globalEndpoint` | `${globalEndpoint}` or `$globalEndpoint` | Set to `1` if using global endpoint, empty otherwise | `1` or `` |\n| `accelerate` | `${accelerate}` or `$accelerate` | Set to `1` if S3 Transfer Acceleration is enabled, empty otherwise | `1` or `` |\n| `dualStack` | `${dualStack}` or `$dualStack` | Set to `1` if dual-stack (IPv4/IPv6) is enabled, empty otherwise | `1` or `` |\n| `usePathStyle` | `${usePathStyle}` or `$usePathStyle` | Set to `1` if using path-style URLs, empty otherwise | `1` or `` |\n\n**Default Values:**\n\nPlaceholders support optional default values using the `${name:-default}` syntax:\n\n```yaml\nendpoint: \"https://${region:-us-east-1}.example.com/${bucket}/${key}\"\n```\n\nIf `${region}` is empty, `us-east-1` will be used.\n\n**Nested and Conditional Placeholders:**\n\nThe template system supports unlimited nesting depth and conditional expansion operators:\n\n1. **Nested defaults**: `${VAR:-${FALLBACK:-default}}`\n   - Falls back through multiple levels until a value is found\n   \n   ```yaml\n   endpoint: \"https://${custom_host:-${region}.example.com}/${bucket}/${key}\"\n   ```\n   If `custom_host` is unset, uses `${region}.example.com`; if both are unset, uses just the bucket and key path.\n\n2. **Conditional expansion (`:+` operator)**: `${VAR:+expansion}`\n   - If VAR is set (non-empty), substitutes the expansion; otherwise empty string\n   \n   ```yaml\n   endpoint: \"https://s3${fips:+-fips}.${region}.amazonaws.com/${bucket}/${key}\"\n   ```\n   If `fips` is set to `1`, the endpoint becomes `https://s3-fips.{region}.amazonaws.com/...`; otherwise `https://s3.{region}.amazonaws.com/...`\n\n3. **Complex nesting**: Both operators can be combined with arbitrary nesting depth\n   ```yaml\n   endpoint: \"https://${host:+${host}}${host:-minio.${region:-us-east-1}.local}/${bucket}/${key}\"\n   ```\n\n**Scheme Handling:**\n\nIf the resolved endpoint URL doesn't start with `http://` or `https://`, the router automatically prepends `https://`.\n\n**Template Examples:**\n\n```yaml\n# AWS S3 virtual-host-style (path-style not recommended for most cases)\nendpoint: \"https://${bucket}.s3.${region}.amazonaws.com/${key}\"\n\n# AWS S3 path-style\nendpoint: \"https://s3.${region}.amazonaws.com/${bucket}/${key}\"\n\n# MinIO on custom domain\nendpoint: \"https://minio.example.com/${bucket}/${key}\"\n\n# S3-compatible with conditional FIPS\nendpoint: \"https://s3${fips:--fips}.${region}.amazonaws.com/${bucket}/${key}\"\n\n# With default region\nendpoint: \"https://s3.${region:-us-east-1}.example.com/${bucket}/${key}\"\n```\n\n**Mode 2: AWS SDK Default Endpoint (When not using template)**\n\nWhen `endpoint` is omitted from the backend configuration, s3-router uses the AWS SDK's endpoint resolver. This automatically selects the correct regional or global endpoint based on:\n- The `region` field in backend config (if present)\n- Endpoint options: `use_path_style`, `use_fips`, `use_global_endpoint`, `use_dual_stack`, `accelerate`\n\nThe SDK endpoint resolver supports:\n- **Regional endpoints**: `https://s3.{region}.amazonaws.com` or virtual-host style\n- **FIPS endpoints**: `https://s3-fips.{region}.amazonaws.com`\n- **Dual-stack**: IPv4 and IPv6 support\n- **Transfer Acceleration**: `https://s3-accelerate.amazonaws.com` or region-specific accelerate endpoints\n\n```yaml\n# Uses SDK resolver with regional endpoint\nbackends:\n  backend1:\n    bucket: my-bucket\n    region: eu-west-1\n    # Resolved to: https://s3.eu-west-1.amazonaws.com or virtual-host style\n\n  # Uses SDK resolver with FIPS endpoint\n  backend2:\n    bucket: my-bucket\n    region: us-east-1\n    use_fips: true\n    # Resolved to: https://s3-fips.us-east-1.amazonaws.com\n\n  # Uses SDK resolver with multiple options\n  backend3:\n    bucket: my-bucket\n    region: us-west-2\n    use_fips: true\n    use_dual_stack: true\n    accelerate: true\n```\n\n#### Credential Sources\n\nBackend credentials are specified using the `credentials` field, which supports multiple source types. When credentials are omitted entirely, the AWS SDK default credential chain is used (environment variables, shared credentials file, IMDS, etc.).\n\n##### Credential Source Types\n\n**`type: default` - AWS SDK Default Credential Chain**\n\nUses the AWS SDK default credential chain, which includes:\n- Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`)\n- Shared credentials file (`~/.aws/credentials`)\n- Shared config file (`~/.aws/config`)\n- IMDSv2 (EC2 instance metadata)\n- ECS task credentials\n- WebIdentity token (for EKS, OIDC)\n\n```yaml\ncredentials:\n  type: default\n```\n\n**`type: inline` - Inline Static Credentials**\n\n\u003e ⚠️ **Not Recommended**: Embedding credentials in configuration files is a security risk. Use `default`, `file`, or `aws-secrets-manager` instead.\n\nEmbeds credentials directly in the configuration file.\n\n```yaml\ncredentials:\n  type: inline\n  access_key_id: AKIA...\n  secret_access_key: ...\n  session_token: optional_temporary_token  # optional\n```\n\n**`type: file` - Load from JSON File**\n\nLoads credentials from a JSON file on the filesystem. Useful for containerized deployments with mounted secrets or Kubernetes mounted ConfigMaps/Secrets.\n\n```yaml\ncredentials:\n  type: file\n  path: /etc/s3router/backend-creds.json\n```\n\nThe JSON file format:\n```json\n{\n  \"access_key_id\": \"AKIA...\",\n  \"secret_access_key\": \"...\",\n  \"session_token\": \"optional_temporary_token\"\n}\n```\n\n**`type: aws-secrets-manager` - Load from AWS Secrets Manager**\n\nLoads credentials from an AWS Secrets Manager secret. The secret must contain a JSON object with the same format as the file provider.\n\n```yaml\ncredentials:\n  type: aws-secrets-manager\n  secret_name: s3-router/my-backend\n  region: us-east-1  # optional; defaults to us-east-1\n```\n\nThe secret value in AWS Secrets Manager:\n```json\n{\n  \"access_key_id\": \"AKIA...\",\n  \"secret_access_key\": \"...\",\n  \"session_token\": \"optional_temporary_token\"\n}\n```\n\n##### STS Role Assumption\n\nAll credential sources support optional STS role assumption, enabling cross-account or cross-role access patterns:\n\n```yaml\ncredentials:\n  type: file\n  path: /etc/s3router/base-creds.json\n  assume_role:\n    role_arn: arn:aws:iam::123456789:role/S3RouterRole\n    session_name: s3-router-backend1  # optional; defaults to \"s3-router\"\n    duration: 3600  # optional; duration in seconds (default: 3600)\n```\n\nThe flow:\n1. Load base credentials from the specified source\n2. Use the base credentials to call STS `AssumeRole`\n3. Return the temporary credentials from the STS response\n4. Use temporary credentials for S3 operations\n\n##### Examples\n\n**Example 1: Default AWS SDK Chain**\n```yaml\nbackends:\n  backend1:\n    bucket: my-bucket\n    credentials:\n      type: default\n```\n\n**Example 2: File-based Credentials**\n```yaml\nbackends:\n  backend2:\n    bucket: my-bucket\n    credentials:\n      type: file\n      path: /run/secrets/s3-backend-creds.json\n```\n\n**Example 3: AWS Secrets Manager**\n```yaml\nbackends:\n  backend3:\n    bucket: my-bucket\n    credentials:\n      type: aws-secrets-manager\n      secret_name: prod/s3-router/backend3\n      region: us-west-2\n```\n\n**Example 4: File-based with Role Assumption**\n```yaml\nbackends:\n  backend4:\n    bucket: my-bucket\n    credentials:\n      type: file\n      path: /etc/s3router/cross-account-base-creds.json\n      assume_role:\n        role_arn: arn:aws:iam::987654321:role/CrossAccountS3Access\n        session_name: s3-router-cross-account\n        duration: 7200\n```\n\n**Example 5: Inline Credentials (Not Recommended)**\n```yaml\nbackends:\n  backend5:\n    bucket: my-bucket\n    credentials:\n      type: inline\n      access_key_id: AKIA...\n      secret_access_key: ...\n```\n\n\u003e ⚠️ **Not Recommended**: Embedding credentials in configuration files is a security risk. Use `type: default`, `type: file`, or `type: aws-secrets-manager` instead.\n\n#### Buckets and routes\n\n* Buckets are evaluated in the order listed. `name` defines the virtual bucket clients address.\n* Each route requires a `path` regular expression (RE2 syntax) that runs against the object key portion after the virtual bucket. Named (`(?P\u003cname\u003e...)`) and numbered captures can be referenced in rewrites.\n* `backend` points to a backend entry.\n* `rewrite` (optional) is a list of sequential rules:\n  * Every rule must include `result`, a template that can reference `$1`, `$2`, … and named captures like `$rest`.\n  * When `pattern` is absent, captures come from the route `path` match; when present, the pattern runs on the current key and exposes its own captures.\n  * Rules run in order, feeding the output of one step into the next.\n* Routing decisions are based on the path regex only; HTTP method and headers are not used for matching today.\n\nIf no rewrite rules are provided, the original key (plus any backend `prefix`) is forwarded unchanged.\n\n#### Path parsing and escaping\n\nThe router treats the request path as URL-escaped text and only decodes it right before backend calls:\n\n* The incoming object key is derived from `r.URL.EscapedPath()`, so percent-encoded bytes remain escaped during routing.\n* Virtual-hosted style:\n  * Bucket is taken from the host (including `virtual_hosts` mappings).\n  * Object key is the escaped path without the leading `/`.\n* Path-style:\n  * Bucket is the first path segment and is decoded with `url.PathUnescape`.\n  * Object key is the remainder of the escaped path (still percent-encoded).\n* Route matching and rewrite rules operate on the escaped key string, so patterns see `%2F`, `%20`, etc.\n* Backend keys are built as `backend.prefix + rewritten_key`.\n  * For most SDK-backed operations, the router applies a conservative unescape (valid `%XX` sequences are decoded; invalid escapes are preserved as literal `%`). This means `%2F` becomes `/` before the SDK call.\n  * Streaming PUT/UploadPart requests and copy-source headers use the rewritten key as-is (still URL-escaped) when building the upstream URL/header. Endpoint templates expect `${key}` to already be URL-encoded and to preserve `/`.\n\n#### Features\n\n`features` is an optional map reserved for future toggles. Flags are currently parsed but not used by runtime logic.\n\n#### Server Configuration\n\nThe `server` section (optional) allows tuning of HTTP server behavior:\n\n```yaml\nserver:\n  read_timeout: 30s        # HTTP read timeout (default: 15s)\n  write_timeout: 30s       # HTTP write timeout (default: 15s)\n  idle_timeout: 60s        # HTTP idle timeout (default: 60s)\n  max_body_size: 4GB         # Max request/response body size (default: 4GB)\n  route_cache_size: 1k             # LRU cache size for route matching (default: 1000)\n```\n\n**Value Formats:**\n\n- **Timeouts** (`read_timeout`, `write_timeout`, `idle_timeout`):\n  - Go duration format: `30s`, `2m`, `1h`, `15m` (recommended)\n  - Raw seconds (backward compatible): `30`, `60`, `3600`\n  - Default units: seconds when using plain numbers\n\n- **Size** (`max_body_size`):\n  - With units: `4GB`, `512MB`, `100KB`, `1024B`, `4GiB`, `512MiB`, `100KiB`\n  - Raw bytes: `4294967296` (default: 4GB)\n  - Supported units: B, KB, MB, GB, KiB, MiB, GiB (case-insensitive)\n  - Uses binary units: 1KB = 1024B, 1MB = 1024KB, etc.\n  - Binary units: 1KiB = 1024B, 1MiB = 1024KiB, 1GiB = 1024MiB\n\n- **Count** (`route_cache_size`):\n  - With units: `1k` (1000), `1m` (1000000)\n  - Raw number: `1000` (default: 1000)\n  - Supported units: k (thousands), m (millions)\n\n| Field | Type | Default | Description |\n| --- | --- | --- | --- |\n| `read_timeout` | string/int | 15s | HTTP server read timeout. Supports Go duration format (e.g., `30s`, `2m`) or seconds as a number. |\n| `write_timeout` | string/int | 15s | HTTP server write timeout. Supports Go duration format or seconds as a number. |\n| `idle_timeout` | string/int | 60s | HTTP server idle timeout. Supports Go duration format or seconds as a number. |\n| `max_body_size` | string/int | 4GB | Maximum size of request and response bodies. Supports units (GB, MB, KB, B, GiB, MiB, KiB) or raw bytes. Requests exceeding this limit are rejected. |\n| `route_cache_size` | string/int | 1000 | LRU cache size for route matching decisions. Supports units (k, m) or raw counts. Larger values improve performance but use more memory. Set to 0 to disable caching. |\n\n#### Authentication Configuration\n\nThe `auth` section (optional) controls signature verification behavior:\n\n```yaml\nauth:\n  default_region: us-east-1         # Default AWS region for SigV4 verification (default: us-east-1)\n  clock_skew_leeway: 15m             # Clock skew tolerance in Go duration format (default: 15m)\n```\n\n| Field | Type | Default | Description |\n| --- | --- | --- | --- |\n| `default_region` | string | us-east-1 | Default AWS region used when verifying SigV4 signatures. This region is used when the client doesn't explicitly specify a region in the Authorization header. Clients can override this by including a region in their credential scope. |\n| `clock_skew_leeway` | string | 15m | Clock skew tolerance as a Go duration string (e.g., `15m`, `900s`, `1h`). Requests with timestamps outside this range from server time are rejected to prevent replay attacks. AWS default is 15 minutes. |\n\n#### Virtual Hosts Configuration\n\nThe `virtual_hosts` section (optional) enables S3 virtual-hosted-style requests (bucket names in host headers). It is active when the `hosts` list is non-empty.\n\n```yaml\nvirtual_hosts:\n  hosts:                            # List of hosts - string (wildcard/suffix) or map for bucket mapping\n    - api.local: \"my-bucket\"        # Map: explicit bucket mapping (api.local -\u003e my-bucket)\n    - \"localhost:8080\": \"bucket-1\"  # Map with explicit port mapping\n    - \"*.s3.localhost\"              # Wildcard: bucket from subdomain (mybucket.s3.localhost -\u003e mybucket)\n    - \".example.com\"                # Suffix: bucket from subdomain (bucket.example.com -\u003e bucket)\n    - \"*.custom.local\": \"default\"   # Wildcard with explicit mapping (any.custom.local -\u003e default)\n```\n\n**Virtual Host Mode:**\n- When a request matches a configured virtual host, the bucket name is determined by the configuration\n- **Map entries**: The bucket name is explicitly mapped (e.g., `api.local: \"my-bucket\"` → bucket `my-bucket`)\n- **Wildcard/suffix strings**: The subdomain is extracted as the bucket name (e.g., `mybucket.s3.localhost` → bucket `mybucket`)\n- **Wildcards with mapping**: All matching hosts use the explicit bucket name\n- **Exact host strings without mapping are ignored** (fall through to path-style resolution)\n- This allows S3 clients to use virtual-hosted-style URLs instead of path-style URLs\n\n**Host Matching Rules:**\n- Exact matches: `localhost:8080` matches only requests to that exact host:port\n- Wildcard matches: `*.example.com` matches `bucket.example.com` but not `foo.bar.example.com`\n- Port support: Patterns can include explicit ports; omitting the port matches any port on that host\n\n| Field | Type | Default | Description |\n| --- | --- | --- | --- |\n| `hosts` | array | empty | List of host entries. Each entry can be a wildcard/suffix string or a map with one key-value pair (host: bucket). Supports wildcards and ports. |\n\n## Running s3-router\n\n### Command-line Flags\n\ns3-router supports the following command-line flags:\n\n| Flag | Type | Default | Description |\n| --- | --- | --- | --- |\n| `-config` | string | `config.yaml` | Path to configuration file |\n| `-credentials` | string | (none) | Path to credentials file (optional) |\n| `-listen` | string | `:8080` | Address to listen on for the main S3 API server |\n| `-admin` | string | `:9090` | Address to listen on for the admin server |\n| `-log-level` | string | `info` | Log level: `debug`, `info`, `warn`, `error` |\n| `-log-format` | string | `auto` | Log format: `auto`, `text`, `json`. When `auto`, uses human-friendly text format for terminal output and structured JSON for piped/redirected output |\n| `-timeout` | duration | `30s` | Request timeout (e.g., `30s`, `2m`) |\n| `-check-route` | string | (none) | Print routing decision for an `s3://bucket/key` and exit |\n\n### Logging\n\ns3-router provides human-friendly logging with automatic format detection:\n\n- **Terminal (interactive)**: Outputs human-readable text format with timestamps and structured key-value pairs\n- **Piped/Redirected**: Automatically switches to structured JSON format for easy parsing by log aggregation systems\n\nYou can override the automatic detection with the `-log-format` flag:\n\n```bash\n# Interactive terminal - auto-detects text format\n./s3router -config config.yaml\n\n# Force text format (human-readable)\n./s3router -config config.yaml -log-format text\n\n# Force JSON format (structured, for log aggregation)\n./s3router -config config.yaml -log-format json\n```\n\n### Authentication \u0026 Authorization\n\ns3-router implements **SigV4 signature verification** on all S3 operations. \n\n#### Authenticated Operations (Require SigV4 Signature)\n\nAll operations require authentication, including:\n\n- **Object Operations**: GetObject, PutObject, DeleteObject, HeadObject, CopyObject\n- **Multi-part Operations**: CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListParts\n- **Listing Operations**: ListObjects, ListObjectsV2  \n- **ACL/Tagging Operations**: GetObjectAcl, PutObjectAcl, GetObjectTagging, PutObjectTagging\n- **Other Operations**: DeleteObjects, GetObjectAttributes, etc.\n\n**Unauthenticated requests** to these operations are rejected with a 401 Unauthorized response.\n\n#### Unauthenticated Operations (Bypassed for Compatibility)\n\nThe following bucket-level operations return fixed responses without requiring authentication:\n\n- **ListBuckets**: Returns the configured virtual buckets (synthetic timestamps)\n- **CreateBucket**: Returns `409 BucketAlreadyExists` (no state mutation)\n- **DeleteBucket**: Returns `403 BucketNotEmpty` (no state mutation)\n\nThese operations are bypassed to support S3 clients that probe for bucket operations during initialization.\n\n#### Static Credential Store\n\nAuthentication uses a static credential store loaded from a JSON file mapping access key IDs to secret keys.\n\n**Credentials file format** (`credentials.json`):\n```json\n{\n  \"AKIAIOSFODNN7EXAMPLE\": {\n    \"secret_key\": \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\",\n    \"enabled\": true,\n    \"created_at\": \"2024-01-01T00:00:00Z\"\n  },\n  \"AKIATESTKEYEXAMPLE01\": {\n    \"secret_key\": \"test_secret_key_12345678901234567890\",\n    \"enabled\": true\n  }\n}\n```\n\n#### Configuration\n\nCredentials file can be specified in two ways:\n\n1. **CLI flag** (takes precedence):\n   ```bash\n   ./s3router -config config.yaml -credentials credentials.json\n   ```\n\n2. **Config file** (`config.yaml`):\n   ```yaml\n   credentials_store: /etc/s3router/credentials.json\n   \n   backends:\n     backend1:\n       bucket: my-bucket\n   \n   buckets:\n     - name: virtual-bucket\n       routes:\n         - path: ^(.*)$\n           backend: backend1\n   ```\n\n#### Signature Verification\n\ns3-router verifies **AWS SigV4 signatures** on incoming requests, supporting:\n\n- **Header-based signatures**: Authorization header with standard AWS SigV4 format\n  ```\n  Authorization: AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20240101/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date;x-amz-content-sha256, Signature=abc123...\n  X-Amz-Date: 20240101T120000Z\n  X-Amz-Content-Sha256: 5d41402abc4b2a76b9719d911017c592\n  ```\n\n- **Query-based signatures**: Presigned URLs with X-Amz-Signature parameter\n  ```\n  GET /?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F...\u0026X-Amz-Date=20240101T120000Z\u0026X-Amz-Signature=...\n  ```\n\n##### Multi-Region Support\n\ns3-router automatically extracts the region from the client's credential scope in the Authorization header. This allows clients from any region to use the router:\n\n```yaml\n# Client in ap-northeast-1\nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20240101/ap-northeast-1/s3/aws4_request, ...\n\n# Client in eu-west-1  \nAuthorization: AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20240101/eu-west-1/s3/aws4_request, ...\n```\n\nIf the client doesn't specify a region in the credential scope, the default region from the `auth` config section is used:\n\n```yaml\nauth:\n  default_region: ap-northeast-1  # Used when client doesn't specify a region\n```\n\n#### Security Features\n\n- **Timestamp validation**: Requests with timestamps outside the configured clock skew tolerance (default: ±15m) are rejected to prevent replay attacks. Configure via `auth.clock_skew_leeway` using Go duration format (e.g., `15m`, `900s`).\n- **Credential lookup**: Only requests signed with registered access keys are accepted\n- **Constant-time comparison**: Signature comparison uses constant-time HMAC comparison to prevent timing attacks\n- **Disabled credentials**: Credentials can be marked `\"enabled\": false` in the JSON file to revoke access\n\n#### Error Responses\n\nAuthentication errors return appropriate HTTP status codes:\n\n| Status | Error Code | Reason |\n|--------|-----------|--------|\n| 401 | MissingAuthenticationToken | No Authorization header or X-Amz-Signature |\n| 400 | InvalidAuthHeader | Malformed or missing authorization components |\n| 403 | InvalidAccessKeyId | Access key not found in credential store |\n| 403 | SignatureDoesNotMatch | Signature verification failed |\n| 403 | RequestTimeTooSkewed | Request timestamp too old or too far in future |\n\n#### Using with AWS SDK / CLI\n\nTo test s3-router with AWS CLI or SDK:\n\n```bash\n# AWS CLI - configure credentials and endpoint\naws configure --profile s3router\n# AWS Access Key ID: AKIAIOSFODNN7EXAMPLE\n# AWS Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n# Default region: us-east-1\n\n# Test with AWS CLI\naws --profile s3router --endpoint-url http://localhost:8080 s3 ls s3://virtual-bucket/\n\n# Python boto3 example\nimport boto3\nclient = boto3.client(\n    's3',\n    endpoint_url='http://localhost:8080',\n    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',\n    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',\n    region_name='us-east-1'\n)\nresponse = client.list_objects_v2(Bucket='virtual-bucket')\n```\n\n#### Credential Management Best Practices\n\n##### For Client Authentication Credentials (credentials_store)\n\n1. **File permissions**: Restrict credentials file to user/group that runs s3-router\n   ```bash\n   chmod 600 credentials.json\n   ```\n\n2. **Secrets management**: Use environment variable substitution or secret management tools\n   ```bash\n   export ROUTER_CREDENTIALS_PATH=/run/secrets/s3router_credentials.json\n   ./s3router -config config.yaml -credentials $ROUTER_CREDENTIALS_PATH\n   ```\n\n3. **Credential rotation**: Update credentials.json without restarting (restart currently required)\n   ```bash\n   kill \u003cpid\u003e\n   ./s3router -config config.yaml -credentials credentials.json\n   ```\n\n4. **Audit logging**: Enable debug logging to see authentication attempts\n   ```bash\n   ./s3router -config config.yaml -credentials credentials.json -log-level debug\n   ```\n\n##### For Backend Credentials\n\n**Recommended approach: Use externalized credential sources instead of inline credentials**\n\n1. **Use `credentials` field with externalized sources**\n   - File-based: Mount secrets from Kubernetes, Docker secrets, or configuration management\n   - AWS Secrets Manager: Automatically rotated, centrally managed, IAM-controlled\n   - Default chain: Leverage IAM roles on EC2/ECS, environment variables, or shared credentials\n\n2. **File-based credentials (`type: file`)**\n   ```bash\n   # Use restricted permissions\n   chmod 600 /etc/s3router/backend-creds.json\n   \n   # Mount via Docker/Kubernetes for easy rotation\n   docker run -v /secrets/backend-creds.json:/etc/s3router/backend-creds.json ...\n   ```\n\n3. **AWS Secrets Manager (`type: aws-secrets-manager`)**\n   ```yaml\n   credentials:\n     type: aws-secrets-manager\n     secret_name: prod/s3-router/backend1\n     region: us-east-1\n   ```\n   \n   Benefits:\n   - Automatic credential rotation via Lambda\n   - Centralized secret management\n   - IAM-based access control\n   - Audit trail via CloudTrail\n   - No plaintext credentials in config files\n\n4. **Default AWS Chain (`type: default`)**\n   ```yaml\n   credentials:\n     type: default\n   ```\n   \n   Best for:\n   - EC2 instances with IAM roles\n   - ECS tasks with IAM task roles\n   - Kubernetes pods with IRSA (IAM Roles for Service Accounts)\n   - Development with `~/.aws/credentials`\n\n5. **STS Role Assumption for Cross-Account Access**\n   ```yaml\n   credentials:\n     type: file\n     path: /etc/s3router/base-creds.json\n     assume_role:\n       role_arn: arn:aws:iam::ACCOUNT2:role/S3RouterRole\n       duration: 3600\n   ```\n   \n   Use for:\n   - Cross-account bucket access\n   - Least-privilege access (assume role with minimal permissions)\n   - Audit trails (STS calls are logged in CloudTrail)\n\n## Prefix Optimization\n\ns3-router includes an automatic prefix optimization feature that significantly improves performance for patterns with static prefixes. When a route pattern contains a static prefix (e.g., `^folder/(.*)$`), the router extracts this prefix and uses it in S3 ListObjects API calls instead of scanning all objects in the bucket.\n\n### How It Works\n\n1. **Pattern Analysis**: Routes are analyzed to detect static prefixes at the beginning of the pattern\n2. **Prefix Extraction**: Static prefixes are extracted and used directly in S3 ListObjects queries with the `prefix` parameter\n3. **Automatic Filtering**: Only objects matching the prefix are retrieved, reducing API calls and improving performance\n\n### Example\n\n```yaml\nbackends:\n  backend1:\n    bucket: my-bucket\n\nbuckets:\n  - name: data\n    routes:\n      - path: ^logs/(?P\u003crest\u003e.*)$    # Static prefix \"logs/\"\n        backend: backend1\n```\n\nWith this configuration:\n- Instead of listing all objects in `my-bucket`, the router queries for objects with prefix `logs/`\n- This reduces the result set, improves performance, and lowers S3 API costs\n- The router then applies the regex pattern to the filtered results\n\n### Requirements for Optimization\n\nPrefix optimization applies when:\n- The pattern starts with literal characters (e.g., `^logs/`)\n- The remainder uses only wildcard patterns (e.g., `.*`, `.+`)\n- No complex regex constraints follow the static prefix\n\nIf a pattern cannot be optimized (e.g., complex regex after the prefix), the router falls back to full bucket scans.\n\n## Streaming and Performance Considerations\n\n### PUT Object Streaming\n\ns3-router optimizes PutObject operations for streaming to minimize memory usage:\n\n#### Streaming Mode (No Checksum Trailers)\n\nWhen a client sends a PUT request **without** checksum trailers (the common case):\n- Request body is streamed directly to the S3 backend **without buffering**\n- Memory usage: Constant, regardless of object size\n- Best for: Large file uploads\n\n**Typical clients and scenarios:**\n- AWS CLI: `aws s3 cp file.bin s3://bucket/path/` ✅ Streamed\n- AWS SDK v3 (JavaScript/Python): Standard PUT requests ✅ Streamed\n- S3 multipart uploads with non-trailer checksums ✅ Streamed\n\n#### Buffering Mode (With Checksum Trailers)\n\nWhen a client sends a PUT request **with** checksum trailers (e.g., `x-amz-trailer: x-amz-checksum-sha256`):\n- Entire request body must be buffered to read trailers at the end\n- Memory usage: = Object size (pre-calculated via Content-Length, or full body if not provided)\n- Required because trailers arrive after the body in chunked encoding\n\n**Clients that use checksum trailers:**\n- AWS SDK v3 with flexible checksums enabled\n- S3Transfer clients with checksum_algorithm configured\n- Some S3-compatible clients that support trailer checksums\n\n### Memory Management\n\nFor environments with memory constraints:\n\n1. **Verify client doesn't require checksum trailers**\n   - Most clients don't send trailers by default\n   - Check if client explicitly requests trailer checksums\n\n2. **Configure max_body_size appropriately**\n   ```yaml\n   server:\n     max_body_size: 5GB  # Limit memory usage in checksum trailer mode\n   ```\n\n3. **Monitor memory usage**\n   - In checksum trailer mode, peak memory = largest object size\n   - Use `-log-level debug` to see which operations buffer trailers\n\n4. **Recommended approach: Upgrade clients to send checksums in headers**\n   - Avoids trailer buffering entirely\n   - Example (AWS SDK): Configure flexible checksums with headers-only mode\n\n### Performance Tips\n\n- **Disable route caching if not needed**\n  ```yaml\n  server:\n    route_cache_size: 0\n  ```\n\n- **Tune connection pooling for backend clients**\n  ```yaml\n  backends:\n    backend1:\n      timeout: 30s\n      retries: 3\n  ```\n\n- **Monitor circuit breaker activation**\n  - Circuit breaker automatically isolates failing backends\n  - Triggered when failure rate ≥ 60% over ≥3 requests\n  - Auto-heals after 10 seconds\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoriyoshi%2Fs3-router","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoriyoshi%2Fs3-router","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoriyoshi%2Fs3-router/lists"}