{"id":49283302,"url":"https://github.com/impossibleforge/pfc-gateway","last_synced_at":"2026-04-25T20:01:58.532Z","repository":{"id":351781069,"uuid":"1212438617","full_name":"ImpossibleForge/pfc-gateway","owner":"ImpossibleForge","description":"HTTP REST query server for PFC cold archives, no DuckDB required","archived":false,"fork":false,"pushed_at":"2026-04-16T12:10:53.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-16T14:14:05.281Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ImpossibleForge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-16T11:32:16.000Z","updated_at":"2026-04-16T12:10:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ImpossibleForge/pfc-gateway","commit_stats":null,"previous_names":["impossibleforge/pfc-gateway"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ImpossibleForge/pfc-gateway","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ImpossibleForge","download_url":"https://codeload.github.com/ImpossibleForge/pfc-gateway/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-gateway/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32274982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T18:29:39.964Z","status":"ssl_error","status_checked_at":"2026-04-25T18:29:32.149Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-25T20:01:57.814Z","updated_at":"2026-04-25T20:01:58.520Z","avatar_url":"https://github.com/ImpossibleForge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pfc-gateway\n\n**Bidirectional HTTP gateway for PFC cold archives — no DuckDB required.**\n\npfc-gateway makes PFC archives on S3 (or local storage) queryable by **any tool** — Grafana, Python, curl, PowerBI — through a simple HTTP API. It also **receives** NDJSON from Fluent Bit, Vector, Telegraf, or any HTTP client and compresses it to `.pfc` archives automatically.\n\nPart of the [PFC Ecosystem](https://github.com/ImpossibleForge).\n\n---\n\n## What it does\n\n```\n[Fluent Bit / Vector / Telegraf / curl]\n          │\n          ▼  POST /ingest — push NDJSON rows\n     pfc-gateway  (this server)  ←─────────── also receives data\n          │\n          ├── .pfc_buffer.jsonl  (live buffer)\n          └── ingest_\u003cts\u003e.pfc    (auto-rotated on size or time)\n\n[Grafana / Python / curl / PowerBI / your own tools]\n          │\n          ▼  POST /query — HTTP REST, no client library needed\n     pfc-gateway  (this server)  ────────────► serves data\n          │\n          ▼  pfc_jsonl s3-fetch — HTTP Range requests\n     .pfc archives on S3 / local\n          │\n          ▼  only ~4% of the archive is read per query\n     NDJSON stream back to client\n```\n\n**One server. Ingest from any tool. Query from any tool.** No DuckDB, no custom plugins.\n\n---\n\n## Install\n\n```bash\n# 1. Install pfc_jsonl binary (required)\n# Linux x64:\ncurl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-linux-x64 \\\n     -o /usr/local/bin/pfc_jsonl \u0026\u0026 chmod +x /usr/local/bin/pfc_jsonl\n\n# macOS (Apple Silicon / M1–M4):\ncurl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-macos-arm64 \\\n     -o /usr/local/bin/pfc_jsonl \u0026\u0026 chmod +x /usr/local/bin/pfc_jsonl\n\n# 2. Install pfc-gateway\ngit clone https://github.com/ImpossibleForge/pfc-gateway\ncd pfc-gateway\npip install fastapi uvicorn boto3 python-dateutil\n\n# 3. Start the server\nPFC_API_KEY=your-secret-key uvicorn pfc_gateway:app --host 0.0.0.0 --port 8765\n```\n\n\u003e **License note:** This tool requires the `pfc_jsonl` binary. `pfc_jsonl` is free for personal and open-source use — commercial use requires a separate license. See [pfc-jsonl](https://github.com/ImpossibleForge/pfc-jsonl) for details.\n\nAWS credentials are read from the standard locations (`~/.aws/credentials`, environment variables, IAM role). No extra config needed.\n\n---\n\n## Ingest — receive data from any HTTP source\n\nEnable ingest by setting `PFC_INGEST_DIR`. The gateway appends rows to a buffer file\nand rotates it to a compressed `.pfc` file when a size or time threshold is reached.\n\n```bash\n# Start gateway with ingest enabled\nPFC_API_KEY=secret PFC_INGEST_DIR=/data/pfc \\\n  uvicorn pfc_gateway:app --host 0.0.0.0 --port 8765\n```\n\n### Send rows with curl\n\n```bash\n# JSON array\ncurl -s -X POST http://localhost:8765/ingest \\\n  -H \"X-API-Key: secret\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '[{\"ts\":\"2026-04-21T10:00:00Z\",\"level\":\"INFO\",\"msg\":\"server started\"}]'\n\n# NDJSON (Fluent Bit json_stream / Vector ndjson)\nprintf '{\"ts\":\"2026-04-21T10:00:01Z\",\"level\":\"WARN\",\"msg\":\"high cpu\"}\\n' | \\\n  curl -s -X POST http://localhost:8765/ingest \\\n  -H \"X-API-Key: secret\" \\\n  -H \"Content-Type: application/x-ndjson\" \\\n  --data-binary @-\n```\n\n### Fluent Bit HTTP output\n\n```ini\n[OUTPUT]\n    Name              http\n    Match             *\n    Host              your-server\n    Port              8765\n    URI               /ingest\n    Format            json          # sends JSON array — pfc-gateway auto-detects\n    Header            X-API-Key secret\n```\n\n### Vector HTTP sink\n\n```toml\n[sinks.pfc_gateway]\ntype     = \"http\"\ninputs   = [\"your_source\"]\nuri      = \"http://your-server:8765/ingest\"\nencoding.codec = \"ndjson\"\n\n[sinks.pfc_gateway.request.headers]\nX-API-Key = \"secret\"\n```\n\n### Force-flush the buffer\n\n```bash\ncurl -s -X POST http://localhost:8765/ingest/flush \\\n  -H \"X-API-Key: secret\"\n# → {\"flushed\": true, \"rows\": 4821, \"file\": \"/data/pfc/ingest_20260421T103045.pfc\"}\n```\n\n### Check buffer status\n\n```bash\ncurl -s http://localhost:8765/ingest/status -H \"X-API-Key: secret\"\n# → {\"enabled\": true, \"buffer_rows\": 142, \"buffer_mb\": 0.021,\n#    \"last_flush_age_sec\": 312.4, \"rotate_mb\": 64, \"rotate_sec\": 3600, ...}\n```\n\n---\n\n## Query a PFC archive\n\n### curl (S3 file)\n\n```bash\ncurl -s \\\n  -H \"X-API-Key: your-secret-key\" \\\n  -H \"Content-Type: application/json\" \\\n  -X POST http://localhost:8765/query \\\n  -d '{\n    \"file\":    \"s3://my-archive/pfc/logs_2026-03.pfc\",\n    \"from_ts\": \"2026-03-05T10:00:00Z\",\n    \"to_ts\":   \"2026-03-05T12:00:00Z\",\n    \"filter\":  {\"level\": \"ERROR\"}\n  }'\n```\n\n**Response: NDJSON stream**\n```json\n{\"ts\":\"2026-03-05T10:14:32Z\",\"level\":\"ERROR\",\"message\":\"connection refused\",\"host\":\"web-03\"}\n{\"ts\":\"2026-03-05T11:02:19Z\",\"level\":\"ERROR\",\"message\":\"disk full\",\"host\":\"db-01\"}\n```\n\n### curl (local file)\n\n```bash\ncurl -s \\\n  -H \"X-API-Key: your-secret-key\" \\\n  -X POST http://localhost:8765/query \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"file\":\"/data/archive/logs_march.pfc\",\"from_ts\":\"2026-03-01\",\"to_ts\":\"2026-04-01\"}'\n```\n\n### Python\n\n```python\nimport requests, json\n\nresp = requests.post(\n    \"http://localhost:8765/query\",\n    headers={\"X-API-Key\": \"your-secret-key\"},\n    json={\n        \"file\":    \"s3://my-archive/pfc/logs_2026-03.pfc\",\n        \"from_ts\": \"2026-03-05T10:00Z\",\n        \"to_ts\":   \"2026-03-05T12:00Z\",\n    },\n    stream=True,\n)\nfor line in resp.iter_lines():\n    row = json.loads(line)\n    print(row[\"ts\"], row.get(\"level\"), row.get(\"message\"))\n```\n\n---\n\n## Query multiple files (multi-month)\n\n```bash\ncurl -s \\\n  -H \"X-API-Key: your-secret-key\" \\\n  -X POST http://localhost:8765/query/batch \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"files\": [\n      \"s3://my-archive/pfc/logs_2026-01.pfc\",\n      \"s3://my-archive/pfc/logs_2026-02.pfc\",\n      \"s3://my-archive/pfc/logs_2026-03.pfc\"\n    ],\n    \"from_ts\": \"2026-01-15T00:00Z\",\n    \"to_ts\":   \"2026-03-15T00:00Z\"\n  }'\n```\n\nFiles are queried in order. Results stream back as a single combined NDJSON response.\n\n---\n\n\n## SQL Query via DuckDB (optional)\n\nIf DuckDB with the [pfc extension](https://github.com/ImpossibleForge/pfc-duckdb) is installed\non the gateway server, you can run full SQL queries against `.pfc` archives:\n\n```bash\ncurl -X POST http://localhost:8765/query/sql \\\n  -H \"x-api-key: secret\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"sql\": \"SELECT json_extract_string(line, '\"'\"'$.level'\"'\"') AS level, COUNT(*) AS cnt FROM pfc_scan('\"'\"'/var/lib/pfc/logs.pfc'\"'\"') GROUP BY level ORDER BY cnt DESC\"\n  }'\n```\n\nSupports any DuckDB SQL — `GROUP BY`, `AVG`, `JOIN` across multiple files, window functions:\n\n```sql\n-- Avg latency per service, last hour\nSELECT json_extract_string(line, '$.service') AS service,\n       ROUND(AVG(json_extract(line, '$.latency_ms')::FLOAT), 1) AS avg_ms\nFROM pfc_scan('/var/lib/pfc/logs.pfc')\nGROUP BY service ORDER BY avg_ms DESC;\n```\n\nCheck if SQL mode is available on your gateway instance:\n```bash\ncurl http://localhost:8765/ -H \"x-api-key: secret\"\n# {\"status\":\"ok\",\"version\":\"0.3.0\",\"binary\":\"...\",\"sql_mode\":true}\n```\n\n`sql_mode: false` means DuckDB is not installed — standard `/query` still works normally.\n\n**Setup:**\n```bash\n# Install DuckDB\ncurl -L https://github.com/duckdb/duckdb/releases/latest/download/duckdb_cli-linux-amd64.gz \\\n  | gunzip \u003e /usr/local/bin/duckdb \u0026\u0026 chmod +x /usr/local/bin/duckdb\n# Install pfc extension\nduckdb -c \"INSTALL pfc FROM community;\"\n```\n## Grafana Integration\n\npfc-gateway implements the [Grafana SimpleJSON data source](https://grafana.com/grafana/plugins/grafana-simple-json-datasource/) protocol.\n\n**Setup (takes 2 minutes):**\n\n1. In Grafana → Settings → Data Sources → Add data source\n2. Search for **SimpleJSON** (install plugin if needed)\n3. URL: `http://your-server:8765/grafana`\n4. Custom HTTP Header: `X-API-Key` = your secret key\n5. Save \u0026 Test → should show \"Data source is working\"\n\n**In a dashboard panel:**\n- Target: `s3://my-archive/pfc/logs_2026-03.pfc`\n- Optional filter: `s3://my-archive/logs.pfc|{\"level\":\"ERROR\"}`\n\nGrafana's time range picker controls `from_ts` and `to_ts` automatically.\n\n---\n\n## Live DB + cold archives in one dashboard\n\n**Without pfc-gateway:**  Only the last 30 days (hot live data) visible in Grafana.\n\n**With pfc-gateway:**\n- Panel 1: Live DB data source (last 30 days)\n- Panel 2: pfc-gateway data source (months/years of cold PFC archives)\n\nBoth panels in the same Grafana dashboard. No re-import. No DuckDB.\n\n---\n\n## API Reference\n\n### Query endpoints\n\n#### `POST /query`\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `file` | string | Local path or `s3://bucket/key.pfc` |\n| `from_ts` | string | ISO 8601 start time (inclusive) |\n| `to_ts` | string | ISO 8601 end time (exclusive) |\n| `filter` | object | Optional equality filter `{\"level\": \"ERROR\"}` |\n| `aws_profile` | string | Optional AWS profile name |\n\nReturns: `application/x-ndjson` stream.\n\n#### `POST /query/batch`\n\nSame as `/query` but with `files: [...]` array instead of single `file`.\n\n#### `GET /`\n\nHealth check. Returns `{\"status\": \"ok\", \"version\": \"0.2.0\"}`.\n\n### Ingest endpoints\n\n#### `POST /ingest`\n\nAccepts rows in three formats (auto-detected):\n- JSON array: `[{...}, {...}]`\n- Object with rows key: `{\"rows\": [{...}, ...]}`\n- Raw NDJSON: `{...}\\n{...}\\n`\n\nReturns: `{\"accepted\": N}`\n\nRequires `PFC_INGEST_DIR` to be set (returns 503 otherwise).\n\n#### `POST /ingest/flush`\n\nForce-compresses the current buffer to a `.pfc` file immediately.\n\nReturns: `{\"flushed\": true, \"rows\": N, \"file\": \"/path/to/ingest_\u003cts\u003e.pfc\"}` or `{\"flushed\": false, \"reason\": \"empty\"}`.\n\n#### `GET /ingest/status`\n\nReturns buffer statistics: row count, byte size, age since last flush, last output file, rotation thresholds.\n\n---\n\n## Environment Variables\n\n### Query / Auth\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PFC_API_KEY` | *(none — auth off)* | API key for `X-API-Key` header |\n| `PFC_JSONL_BINARY` | `/usr/local/bin/pfc_jsonl` | Path to pfc_jsonl binary |\n| `PFC_HOST` | `0.0.0.0` | Bind address |\n| `PFC_PORT` | `8765` | Port |\n| `PFC_PRESIGN_TTL` | `3600` | Pre-signed URL TTL in seconds |\n| `AWS_DEFAULT_REGION` | `eu-central-1` | AWS region for S3 |\n\n### Ingest\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PFC_INGEST_DIR` | *(none — ingest off)* | Directory for buffer + output `.pfc` files |\n| `PFC_INGEST_ROTATE_MB` | `64` | Rotate when buffer reaches this size (MB) |\n| `PFC_INGEST_ROTATE_SEC` | `3600` | Rotate when buffer is older than this (seconds) |\n| `PFC_INGEST_PREFIX` | `ingest` | Output filename prefix: `ingest_\u003cts\u003e.pfc` |\n\nStandard AWS variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_PROFILE`) are respected automatically.\n\n---\n\n## Run as systemd service\n\n```ini\n# /etc/systemd/system/pfc-gateway.service\n[Unit]\nDescription=pfc-gateway — PFC cold archive bidirectional gateway\nAfter=network.target\n\n[Service]\nType=simple\nUser=pfc\nWorkingDirectory=/opt/pfc-gateway\nExecStart=/usr/bin/uvicorn pfc_gateway:app --host 0.0.0.0 --port 8765\nRestart=on-failure\nEnvironment=PFC_API_KEY=your-secret-key\nEnvironment=AWS_DEFAULT_REGION=eu-central-1\nEnvironment=PFC_INGEST_DIR=/data/pfc          # omit to disable ingest\nEnvironment=PFC_INGEST_ROTATE_MB=64\nEnvironment=PFC_INGEST_ROTATE_SEC=3600\n\n[Install]\nWantedBy=multi-user.target\n```\n\n```bash\nsudo systemctl enable --now pfc-gateway\n```\n\n---\n\n## Run as Docker container\n\n```bash\ndocker run -d \\\n  -p 8765:8765 \\\n  -e PFC_API_KEY=your-secret-key \\\n  -e AWS_ACCESS_KEY_ID=... \\\n  -e AWS_SECRET_ACCESS_KEY=... \\\n  --name pfc-gateway \\\n  impossibleforge/pfc-gateway:latest\n```\n\n---\n\n## Architecture in the full PFC ecosystem\n\n```\nYour data sources\n    │\n    ├── pfc-migrate     (one-shot export)\n    ├── pfc-archiver-*  (autonomous daemon)\n    ├── pfc-fluentbit   (live pipeline)\n    └── pfc-gateway     (POST /ingest ← NEW)  ← this repo\n              │\n              ▼\n    .pfc archives (local / S3 / Azure / GCS)\n              │\n    ┌─────────┴──────────┐\n    │                    │\n    ▼                    ▼\npfc-duckdb          pfc-gateway  ← this repo\nSQL queries         HTTP REST\n(DuckDB needed)     (no DuckDB)\n    │                    │\n    ▼                    ▼\nPython / CLI        Grafana / PowerBI / curl / own tools\n                    Fluent Bit / Vector / Telegraf (ingest)\n```\n\n| Tool | What | DuckDB needed |\n|------|------|---------------|\n| `pfc-migrate` | One-shot export to `.pfc` | No |\n| `pfc-archiver-*` | Autonomous archive daemon | No |\n| `pfc-fluentbit` | Live pipeline → `.pfc` | No |\n| `pfc-duckdb` | SQL queries on PFC files | Yes |\n| **`pfc-gateway`** | **HTTP REST — any tool** | **No** |\n\n---\n\n## Related repos\n\n- [pfc-jsonl](https://github.com/ImpossibleForge/pfc-jsonl) — core binary (compress/decompress/query)\n- [pfc-migrate](https://github.com/ImpossibleForge/pfc-migrate) — one-shot export and archive conversion\n- [pfc-fluentbit](https://github.com/ImpossibleForge/pfc-fluentbit) — live Fluent Bit → PFC pipeline\n- [pfc-duckdb](https://github.com/ImpossibleForge/pfc-duckdb) — DuckDB extension for SQL queries on PFC files\n- [pfc-vector](https://github.com/ImpossibleForge/pfc-vector) — high-performance Rust ingest daemon for Vector.dev and Telegraf\n- [pfc-otel-collector](https://github.com/ImpossibleForge/pfc-otel-collector) — OpenTelemetry OTLP/HTTP log exporter\n- [pfc-kafka-consumer](https://github.com/ImpossibleForge/pfc-kafka-consumer) — Kafka / Redpanda consumer → PFC\n- [pfc-telegraf](https://github.com/ImpossibleForge/pfc-telegraf) — Telegraf HTTP output plugin → PFC\n- [pfc-grafana](https://github.com/ImpossibleForge/pfc-grafana) — Grafana data source plugin for PFC archives\n\n---\n\n*ImpossibleForge — [github.com/ImpossibleForge](https://github.com/ImpossibleForge)*\n*Contact: info@impossibleforge.com*\n\n---\n\n## License\n\npfc-gateway (this repository) is released under the MIT License — see [LICENSE](LICENSE).\n\nThe PFC-JSONL binary () is proprietary software — free for personal and open-source use. Commercial use requires a license: [info@impossibleforge.com](mailto:info@impossibleforge.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimpossibleforge%2Fpfc-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimpossibleforge%2Fpfc-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimpossibleforge%2Fpfc-gateway/lists"}