{"id":49283508,"url":"https://github.com/impossibleforge/pfc-vector","last_synced_at":"2026-04-25T20:03:18.121Z","repository":{"id":352900967,"uuid":"1216958831","full_name":"ImpossibleForge/pfc-vector","owner":"ImpossibleForge","description":"High-performance PFC ingest daemon for Vector.dev, Telegraf and HTTP sources — Rust + Tokio","archived":false,"fork":false,"pushed_at":"2026-04-21T15:31:07.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-21T17:36:13.195Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ImpossibleForge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-21T12:02:22.000Z","updated_at":"2026-04-21T15:31:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ImpossibleForge/pfc-vector","commit_stats":null,"previous_names":["impossibleforge/pfc-vector"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ImpossibleForge/pfc-vector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-vector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-vector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-vector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-vector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ImpossibleForge","download_url":"https://codeload.github.com/ImpossibleForge/pfc-vector/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-vector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32274987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T18:29:39.964Z","status":"ssl_error","status_checked_at":"2026-04-25T18:29:32.149Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-25T20:03:07.173Z","updated_at":"2026-04-25T20:03:18.107Z","avatar_url":"https://github.com/ImpossibleForge.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pfc-vector\n\n**High-performance PFC ingest daemon for Vector.dev, Telegraf, Fluent Bit, and any HTTP source.**\n\npfc-vector is a lightweight Rust HTTP server that receives log/metric events from any tool with HTTP output, buffers them in memory-safe Tokio async code, and compresses them to `.pfc` archives automatically.\n\nBuilt on **axum + tokio** — no GIL, no Python overhead, no lock held during compression.\n\nPart of the [PFC Ecosystem](https://github.com/ImpossibleForge).\n\n---\n\n## What it does\n\n```\n[Vector.dev / Telegraf / Fluent Bit / curl]\n          │\n          ▼  POST /ingest — push NDJSON rows\n     pfc-vector  (this Rust binary)\n          │\n          ├── .pfc_buffer.jsonl    (live ring buffer, append-only)\n          └── vector_\u003cts\u003e.pfc      (auto-rotated on size or time)\n                    │\n                    ▼\n             S3 / local storage\n```\n\n**Key design:**\n- Mutex held only for the **atomic rename** (microseconds), never during compression\n- Compression runs **outside the lock** → new rows accepted at full speed while old block compresses\n- On compress failure → buffer is **automatically restored**, zero data loss\n\n---\n\n## Install\n\n```bash\n# Prerequisites: pfc_jsonl binary\ncurl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-linux-x64 \\\n     -o /usr/local/bin/pfc_jsonl \u0026\u0026 chmod +x /usr/local/bin/pfc_jsonl\n\n# Build pfc-vector\ngit clone https://github.com/ImpossibleForge/pfc-vector\ncd pfc-vector\ncargo build --release\n\n# Start\nPFC_VECTOR_DIR=/data/pfc PFC_API_KEY=secret \\\n  ./target/release/pfc-vector\n```\n\n\u003e **License note:** This tool requires the `pfc_jsonl` binary. `pfc_jsonl` is free for personal and open-source use — commercial use requires a separate license. See [pfc-jsonl](https://github.com/ImpossibleForge/pfc-jsonl) for details.\n\n---\n\n## Vector.dev sink configuration\n\n```toml\n# vector.toml\n[sinks.pfc]\ntype   = \"http\"\ninputs = [\"your_source\"]\nuri    = \"http://your-server:8766/ingest\"\nencoding.codec = \"ndjson\"\n\n[sinks.pfc.request.headers]\nX-API-Key = \"secret\"\n\n# Optional: tune batch size to match PFC block size\n[sinks.pfc.batch]\nmax_bytes  = 10485760   # 10 MB\ntimeout_secs = 30\n```\n\n---\n\n## Telegraf HTTP output\n\n```toml\n[[outputs.http]]\nurl             = \"http://your-server:8766/ingest\"\nmethod          = \"POST\"\ndata_format     = \"json\"\n[outputs.http.headers]\n  X-API-Key   = \"secret\"\n  Content-Type = \"application/json\"\n```\n\n---\n\n## Fluent Bit HTTP output\n\n```ini\n[OUTPUT]\n    Name       http\n    Match      *\n    Host       your-server\n    Port       8766\n    URI        /ingest\n    Format     json\n    Header     X-API-Key secret\n```\n\n---\n\n## curl\n\n```bash\n# JSON array\ncurl -X POST http://localhost:8766/ingest \\\n  -H \"X-API-Key: secret\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '[{\"ts\":\"2026-04-21T10:00:00Z\",\"level\":\"INFO\",\"msg\":\"server started\"}]'\n\n# NDJSON (native Vector format)\nprintf '{\"ts\":\"2026-04-21T10:00:01Z\",\"level\":\"WARN\",\"msg\":\"high cpu\"}\\n' | \\\n  curl -X POST http://localhost:8766/ingest \\\n  -H \"X-API-Key: secret\" \\\n  -H \"Content-Type: application/x-ndjson\" \\\n  --data-binary @-\n```\n\n---\n\n## API Reference\n\n### `GET /` or `GET /health`\n\nHealth check.\n\n```json\n{\"status\": \"ok\", \"version\": \"0.1.0\", \"service\": \"pfc-vector\"}\n```\n\n### `POST /ingest`\n\nAccepts rows in three formats (auto-detected):\n- JSON array: `[{...}, {...}]`\n- Object with rows/events/logs key: `{\"rows\": [{...}]}`, `{\"events\": [{...}]}`\n- Raw NDJSON: `{...}\\n{...}\\n`\n\nReturns: `{\"accepted\": N}`\n\nRequires `PFC_VECTOR_DIR` to be set (returns 503 otherwise).\n\n### `POST /ingest/flush`\n\nForce-compress current buffer to a `.pfc` file immediately.\n\n```json\n{\"flushed\": true, \"rows\": 4821, \"file\": \"/data/pfc/vector_20260421T103045.pfc\"}\n```\n\n### `GET /ingest/status`\n\n```json\n{\n  \"enabled\": true,\n  \"buffer_rows\": 142,\n  \"buffer_mb\": 0.021,\n  \"last_flush_age_sec\": 312,\n  \"last_file\": \"/data/pfc/vector_20260421T100000.pfc\",\n  \"rotate_mb\": 64,\n  \"rotate_sec\": 3600,\n  \"version\": \"0.1.0\"\n}\n```\n\n---\n\n## Environment Variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PFC_VECTOR_DIR` | *(none — ingest off)* | Directory for buffer and output `.pfc` files |\n| `PFC_API_KEY` | *(none — auth off)* | API key for `X-API-Key` header |\n| `PFC_JSONL_BINARY` | `/usr/local/bin/pfc_jsonl` | Path to pfc_jsonl binary |\n| `PFC_VECTOR_ROTATE_MB` | `64` | Rotate when buffer reaches this size (MB) |\n| `PFC_VECTOR_ROTATE_SEC` | `3600` | Rotate when buffer is older than this (seconds) |\n| `PFC_VECTOR_PREFIX` | `vector` | Output filename prefix: `vector_\u003cts\u003e.pfc` |\n| `PFC_VECTOR_HOST` | `0.0.0.0` | Bind address |\n| `PFC_VECTOR_PORT` | `8766` | Port |\n\nStandard AWS variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_PROFILE`) are respected by `pfc_jsonl` automatically.\n\n---\n\n## Run as systemd service\n\n```ini\n# /etc/systemd/system/pfc-vector.service\n[Unit]\nDescription=pfc-vector — PFC ingest daemon for Vector/Telegraf/Fluent Bit\nAfter=network.target\n\n[Service]\nType=simple\nUser=pfc\nWorkingDirectory=/opt/pfc-vector\nExecStart=/opt/pfc-vector/pfc-vector\nRestart=on-failure\nEnvironment=PFC_API_KEY=your-secret-key\nEnvironment=PFC_VECTOR_DIR=/data/pfc\nEnvironment=PFC_VECTOR_ROTATE_MB=64\nEnvironment=PFC_VECTOR_ROTATE_SEC=3600\n\n[Install]\nWantedBy=multi-user.target\n```\n\n```bash\nsudo systemctl enable --now pfc-vector\n```\n\n---\n\n## Run as Docker container\n\n```bash\ndocker run -d \\\n  -p 8766:8766 \\\n  -e PFC_API_KEY=your-secret-key \\\n  -e PFC_VECTOR_DIR=/data/pfc \\\n  -v /data/pfc:/data/pfc \\\n  --name pfc-vector \\\n  impossibleforge/pfc-vector:latest\n```\n\n---\n\n## Architecture — Why Rust?\n\npfc-gateway (Python/FastAPI) is our bidirectional gateway — it's great for most workloads. pfc-vector exists for one reason: **extreme throughput with zero GIL pressure**.\n\n| | pfc-gateway (Python) | pfc-vector (Rust) |\n| [pfc-otel-collector](https://github.com/ImpossibleForge/pfc-otel-collector) | OpenTelemetry OTLP/HTTP log exporter |\n| [pfc-kafka-consumer](https://github.com/ImpossibleForge/pfc-kafka-consumer) | Kafka / Redpanda consumer |\n|---|---|---|\n| Language | Python + asyncio | Rust + Tokio |\n| GIL | Yes (no true parallelism) | None |\n| Lock scope | Was held during compression | Held only for rename (~µs) |\n| Compression | asyncio subprocess | Tokio subprocess (same pfc_jsonl) |\n| Best for | General use, query + ingest | High-frequency Vector/Telegraf pipelines |\n\nBoth use the same `pfc_jsonl` binary for compression. The PFC archive format is identical.\n\n---\n\n## Architecture in the full PFC ecosystem\n\n```\nYour data sources\n    │\n    ├── pfc-migrate     (one-shot export)\n    ├── pfc-archiver-*  (autonomous daemon)\n    ├── pfc-fluentbit   (live Fluent Bit → PFC pipeline)\n    ├── pfc-gateway     (POST /ingest — Python, bidirectional)\n    └── pfc-vector      (POST /ingest — Rust, high-frequency)  ← this repo\n              │\n              ▼\n    .pfc archives (local / S3)\n              │\n              ▼\n    pfc-gateway  ← query via HTTP REST (Grafana, Python, curl)\n```\n\n---\n\n## Related repos\n\n- [pfc-jsonl](https://github.com/ImpossibleForge/pfc-jsonl) — core binary (compress/decompress/query)\n- [pfc-gateway](https://github.com/ImpossibleForge/pfc-gateway) — HTTP REST gateway (query + ingest, Python)\n- [pfc-fluentbit](https://github.com/ImpossibleForge/pfc-fluentbit) — Fluent Bit → PFC pipeline\n- [pfc-migrate](https://github.com/ImpossibleForge/pfc-migrate) — one-shot export and archive conversion\n- [pfc-duckdb](https://github.com/ImpossibleForge/pfc-duckdb) — DuckDB extension for SQL queries on PFC files\n- [pfc-otel-collector](https://github.com/ImpossibleForge/pfc-otel-collector) — OpenTelemetry OTLP/HTTP log exporter\n- [pfc-kafka-consumer](https://github.com/ImpossibleForge/pfc-kafka-consumer) — Kafka / Redpanda consumer → PFC\n- [pfc-telegraf](https://github.com/ImpossibleForge/pfc-telegraf) — Telegraf HTTP output plugin → PFC\n- [pfc-grafana](https://github.com/ImpossibleForge/pfc-grafana) — Grafana data source plugin for PFC archives\n\n---\n\n*ImpossibleForge — [github.com/ImpossibleForge](https://github.com/ImpossibleForge)*  \n*Contact: info@impossibleforge.com*\n\n---\n\n\n\n---\n\n## License\n\npfc-vector (this repository) is released under the MIT License — see [LICENSE](LICENSE).\n\nThe PFC-JSONL binary () is proprietary software — free for personal and open-source use. Commercial use requires a license: [info@impossibleforge.com](mailto:info@impossibleforge.com)\n\n## Disclaimer\n\nPFC-Vector is an independent open-source project and is not affiliated with, endorsed by, or associated with Vector.dev or Datadog, Inc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimpossibleforge%2Fpfc-vector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimpossibleforge%2Fpfc-vector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimpossibleforge%2Fpfc-vector/lists"}