{"id":48728873,"url":"https://github.com/pbs-plus/pbs-s3gateway","last_synced_at":"2026-04-14T19:00:55.553Z","repository":{"id":350572640,"uuid":"1207431394","full_name":"pbs-plus/pbs-s3gateway","owner":"pbs-plus","description":"S3 gateway for backing up objects directly to PBS","archived":false,"fork":false,"pushed_at":"2026-04-11T22:33:47.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-12T00:34:29.015Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pbs-plus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-10T23:56:52.000Z","updated_at":"2026-04-11T22:33:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pbs-plus/pbs-s3gateway","commit_stats":null,"previous_names":["pbs-plus/pbs-s3gateway"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/pbs-plus/pbs-s3gateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pbs-plus%2Fpbs-s3gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pbs-plus%2Fpbs-s3gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pbs-plus%2Fpbs-s3gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pbs-plus%2Fpbs-s3gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pbs-plus","download_url":"https://codeload.github.com/pbs-plus/pbs-s3gateway/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pbs-plus%2Fpbs-s3gateway/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31810741,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"ssl_error","status_checked_at":"2026-04-14T18:05:01.765Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-12T00:26:21.868Z","updated_at":"2026-04-14T19:00:55.541Z","avatar_url":"https://github.com/pbs-plus.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pbs-s3gateway\n\nAn S3-compatible HTTP gateway that translates S3 API calls into [Proxmox Backup Server](https://pbs.proxmox.com/) operations. Designed to let applications that only speak S3 (like Kubernetes operators) store backups directly in PBS.\n\n## How It Works\n\n```\nS3 Client (mariadb-operator, etc.)\n  │\n  │  S3 HTTP (PUT/GET/DELETE/LIST)\n  ▼\n┌──────────────────────────┐\n│       pbs-s3gateway      │\n│                          │\n│  S3 bucket = PBS namespace       │\n│  S3 key    = encoded backup-id   │\n│  S3 object = PBS snapshot (1 blob)│\n│                          │\n│  Optional: AES-256-GCM   │\n│  client-side encryption  │\n└──────────┬───────────────┘\n           │\n           │  PBS HTTP API (REST + H2 backup protocol)\n           ▼\n┌──────────────────────────┐\n│   Proxmox Backup Server  │\n│   (datastore)            │\n└──────────────────────────┘\n```\n\n### Namespace Mapping\n\nPBS namespaces map directly to S3 buckets:\n\n| PBS Concept | S3 Equivalent |\n|---|---|\n| Top-level namespace (`team-a`) | Bucket (`team-a`) |\n| Sub-namespace (`team-a/proj-1`) | Folder prefix (`proj-1/`) |\n| Backup group (within a namespace) | Object key (base64url-encoded) |\n| Snapshot (within a group) | Object version (latest is served) |\n\n- `GET /` → lists all top-level PBS namespaces as S3 buckets\n- `GET /team-a` → lists objects in namespace `team-a`, including objects from sub-namespaces like `team-a/proj-1` appearing under prefix `proj-1/`\n- `PUT /team-a/backups/db.sql` → uploads to namespace `team-a`, key `backups/db.sql`\n\n### Key Encoding\n\nS3 keys are encoded into PBS-compatible backup-ids using base64url with a `k` prefix. This is reversible — the original S3 key is recovered on GET/LIST. For keys that exceed 128 characters (PBS's backup-id limit), a SHA-256 hash is used instead (not reversible, requires metadata lookup).\n\n### Upload Protocol\n\nUploads use PBS's HTTP/2 backup protocol:\n1. TLS connection to PBS\n2. HTTP/1.1 `Upgrade: proxmox-backup-protocol-v1` → 101 Switching Protocols\n3. HTTP/2 on the upgraded connection\n4. `POST /blob` with PBS-encoded blob data\n5. `POST /finish` to commit the snapshot\n\nDownloads, listings, and deletes use the standard PBS REST API over HTTPS.\n\n### Encryption\n\nOptional AES-256-GCM client-side encryption. When enabled:\n- Data is encrypted before upload, decrypted on download\n- PBS stores only ciphertext — the server never sees plaintext\n- Random 12-byte nonce per object, prepended to ciphertext\n- ETags are computed on ciphertext (consistent whether encrypted or not)\n\nWithout an encryption key, data passes through unchanged.\n\n## S3 API Support\n\n| Operation | S3 Method | Status |\n|---|---|---|\n| ListBuckets | `GET /` | Supported |\n| ListObjects | `GET /bucket` | Supported (prefix, marker) |\n| PutObject | `PUT /bucket/key` | Supported |\n| GetObject | `GET /bucket/key` | Supported |\n| HeadObject | `HEAD /bucket/key` | Supported |\n| DeleteObject | `DELETE /bucket/key` | Supported |\n\nNot supported: multipart upload, presigned URLs, bucket CRUD, ACLs, versioning (always serves latest snapshot).\n\n## Usage\n\n```\npbs-s3gateway [flags]\n```\n\n### Flags\n\n| Flag | Env Variable | Description |\n|---|---|---|\n| `--listen` | `LISTEN` | Listen address (default `:8080`) |\n| `--pbs-url` | `PBS_URL` | PBS API base URL (e.g. `https://pbs:8007`) |\n| `--pbs-datastore` | `PBS_DATASTORE` | PBS datastore name |\n| `--pbs-token` | `PBS_TOKEN` | PBS API token (`TOKENID:SECRET`). Optional when using `--credentials`. |\n| `--credentials` | `CREDENTIALS_FILE` | Path to S3 credentials file (JSON mapping accessKeyId → secretAccessKey) |\n| `--encryption-key` | `ENCRYPTION_KEY` | AES-256 key, hex-encoded (64 chars). Omit for passthrough. |\n| `--insecure-tls` | | Skip TLS certificate verification |\n\n### Authentication\n\nThe gateway supports two auth modes:\n\n**Static token** — a single PBS token used for all requests:\n```bash\npbs-s3gateway --pbs-token \"root@pam!s3gateway:secret\"\n```\n\n**Credential passthrough** — S3 Access Key ID and Secret are mapped to PBS credentials. Each S3 client authenticates with its own credentials, and the gateway translates them into PBS API tokens.\n\nSupported S3 auth methods: AWS Signature V4, AWS Signature V2, HTTP Basic Auth, query string auth.\n\n#### Creating PBS API Tokens\n\nThe gateway needs PBS API tokens with access to the target datastore. You can create tokens via the PBS web UI or API.\n\n**Option A: API token under a user (recommended)**\n\n1. In the PBS web UI, go to **Configuration → Access Control → API Tokens**\n2. Click **Add**\n3. Set **User** to the token owner (e.g. `root@pam`)\n4. Set **Token ID** (e.g. `s3gateway`) — the full token ID becomes `user@realm!tokenid` (e.g. `root@pam!s3gateway`)\n5. Uncheck **Privilege Separation** so the token inherits the user's permissions\n6. Save the generated **Secret** — it looks like `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`\n\nThen grant the user permissions on the datastore:\n\n1. Go to **Configuration → Access Control → Permissions**\n2. Add a permission:\n   - **Path**: `/datastore/\u003cdatastore-name\u003e`\n   - **User**: `root@pam` (or your chosen user)\n   - **Role**: `PVEDatastoreUser`\n\n`PVEDatastoreUser` includes `Datastore.Audit` and `Datastore.ReadDatastore` and `Datastore.Backup`, which covers listing, downloading, and uploading. To also allow deletes, use `PVEDatastoreAdmin` or assign `Datastore.Modify` and `Datastore.Prune` additionally.\n\n**Option B: Custom role with minimum permissions**\n\nFor finer-grained control, create a dedicated user and role with only the required privileges:\n\n1. Go to **Configuration → Access Control → Roles**, click **Add**:\n   - **Role ID**: `S3Gateway`\n   - **Privileges**: `Datastore.Audit`, `Datastore.Modify`, `Datastore.Backup`, `Datastore.Prune`\n\n2. Create a user under **Configuration → Access Control → User Management**:\n   - **User ID**: `s3gateway@pbs`\n   - Set a password\n\n3. Grant the role to the user on the datastore under **Permissions**:\n   - **Path**: `/datastore/\u003cdatastore-name\u003e`\n   - **User**: `s3gateway@pbs`\n   - **Role**: `S3Gateway`\n\n4. Create an API token for that user under **API Tokens**:\n   - **User**: `s3gateway@pbs`\n   - **Token ID**: `s3` (full ID becomes `s3gateway@pbs!s3`)\n   - **Privilege Separation**: unchecked\n\n#### Setting up the credentials file\n\nCreate `credentials.json` mapping each PBS token ID to its secret:\n\n```json\n{\n  \"root@pam!s3gateway\": \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\"\n}\n```\n\nThe mapping is: S3 Access Key ID = PBS token ID, S3 Secret Access Key = PBS token secret. The gateway forms `TOKENID:SECRET` as the PBS API token.\n\nFor multiple clients, add one entry per token:\n\n```json\n{\n  \"root@pam!team-a\": \"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee\",\n  \"root@pam!team-b\": \"11111111-2222-3333-4444-555555555555\"\n}\n```\n\n### Example\n\n```bash\n# Start the gateway with credentials passthrough\npbs-s3gateway \\\n  --listen :8080 \\\n  --pbs-url https://pbs.example.com:8007 \\\n  --pbs-datastore backup-store \\\n  --credentials credentials.json\n\n# Use with any S3 client\naws --endpoint-url http://localhost:8080 s3 cp backup.sql s3://team-a/backups/backup.sql\n```\n\n### With mariadb-operator\n\nConfigure the backup CRD with real PBS credentials as S3 access key and secret:\n\n```yaml\napiVersion: k8s.mariadb.com/v1alpha1\nkind: Backup\nmetadata:\n  name: mariadb-backup\nspec:\n  mariaDbRef:\n    name: mariadb\n  s3:\n    bucket: team-a\n    prefix: mariadb/\n    endpoint: http://pbs-s3gateway:8080\n    accessKeyId: \"root@pam!s3gateway\"\n    secretAccessKey: \"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\"\n```\n\n## Project Structure\n\n```\ncmd/pbs-s3gateway/        Entry point\ninternal/\n  auth/                   S3 credential extraction and PBS token mapping\n  crypto/                 AES-256-GCM encryption (sync.Pool for nonces, pre-computed cipher)\n  gateway/                S3 HTTP handler, routing, namespace mapping\n  keymapper/              S3 key \u003c-\u003e PBS backup-id encoding (cached via sync.Map)\n  pbs/                    PBS REST client + HTTP/2 upload protocol\n  s3/                     S3 XML response types and serialization\n```\n\n## Performance\n\nThe hot path is optimized for minimal allocations:\n- `sync.Pool` for body buffers, ETag formatting, encryption nonces\n- Pre-computed AES cipher/AEAD at construction (not per-request)\n- Cached key encoding via `sync.Map`\n- XML streamed directly to `ResponseWriter` (no intermediate `[]byte`)\n- Pre-allocated slices based on API response sizes\n- `strconv.FormatInt` and `encoding/hex` instead of `fmt.Sprintf`\n- Manual byte scanning in `splitPath` instead of `strings.TrimPrefix`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpbs-plus%2Fpbs-s3gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpbs-plus%2Fpbs-s3gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpbs-plus%2Fpbs-s3gateway/lists"}