{"id":49283620,"url":"https://github.com/impossibleforge/pfc-otel-collector","last_synced_at":"2026-04-25T20:03:34.885Z","repository":{"id":353588588,"uuid":"1220034332","full_name":"ImpossibleForge/pfc-otel-collector","owner":"ImpossibleForge","description":"OpenTelemetry OTLP/HTTP log exporter for PFC-JSONL compression","archived":false,"fork":false,"pushed_at":"2026-04-24T14:17:54.000Z","size":31,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-24T16:28:40.100Z","etag":null,"topics":["compression","log-management","logs","observability","opentelemetry","otlp","pfc-jsonl","python","s3"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ImpossibleForge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-24T13:28:36.000Z","updated_at":"2026-04-24T14:18:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ImpossibleForge/pfc-otel-collector","commit_stats":null,"previous_names":["impossibleforge/pfc-otel-collector"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ImpossibleForge/pfc-otel-collector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-otel-collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-otel-collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-otel-collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-otel-collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ImpossibleForge","download_url":"https://codeload.github.com/ImpossibleForge/pfc-otel-collector/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImpossibleForge%2Fpfc-otel-collector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32274987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T18:29:39.964Z","status":"ssl_error","status_checked_at":"2026-04-25T18:29:32.149Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","log-management","logs","observability","opentelemetry","otlp","pfc-jsonl","python","s3"],"created_at":"2026-04-25T20:03:34.200Z","updated_at":"2026-04-25T20:03:34.868Z","avatar_url":"https://github.com/ImpossibleForge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pfc-otel-collector\n\n**OpenTelemetry log exporter for PFC-JSONL** — receive OTLP/HTTP log data and compress it directly to `.pfc` format.\n\nDrop `pfc-otel-collector` in front of any OpenTelemetry Collector pipeline to compress your log exports on arrival — no extra storage step, no conversion script.\n\n[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)\n[![Part of PFC-JSONL Ecosystem](https://img.shields.io/badge/ecosystem-PFC--JSONL-brightgreen)](https://github.com/ImpossibleForge/pfc-jsonl)\n\n---\n\n## How it fits in your pipeline\n\n```\nYour App / OTel SDK\n        │  OTLP/HTTP\n        ▼\nOpenTelemetry Collector\n        │  otlphttp exporter  →  http://pfc-otel-collector:4318\n        ▼\npfc-otel-collector          ← this service\n        │  pfc_jsonl compress\n        ▼\nlogs_20260115_100000.pfc    →  local disk or S3\n        │\n        ▼\nQuery with DuckDB / pfc-gateway\n```\n\n---\n\n## Quickstart\n\n### 1. Install\n\n```bash\npip install fastapi uvicorn toml\n# Optional S3 upload:\npip install boto3\n```\n\n### 2. Download pfc_jsonl binary\n\n```bash\n# Linux x86_64\ncurl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-linux-x86_64 \\\n     -o /usr/local/bin/pfc_jsonl \u0026\u0026 chmod +x /usr/local/bin/pfc_jsonl\n\n# macOS ARM64\ncurl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-macos-arm64 \\\n     -o /usr/local/bin/pfc_jsonl \u0026\u0026 chmod +x /usr/local/bin/pfc_jsonl\n```\n\n### 3. Create config\n\n```bash\ncp config/config.toml ./config.toml\n# Edit as needed\n```\n\n### 4. Start\n\n```bash\npython pfc_otel_collector.py --config config.toml\n# 2026-01-15T10:00:00 [pfc-otel] INFO pfc-otel-collector v0.1.0 started on port 4318\n```\n\n### 5. Point your OTel Collector at it\n\n```yaml\n# otel-collector-config.yaml\nexporters:\n  otlphttp/pfc:\n    endpoint: http://localhost:4318\n    logs_endpoint: http://localhost:4318/v1/logs\n\nservice:\n  pipelines:\n    logs:\n      exporters: [otlphttp/pfc]\n```\n\n---\n\n## Configuration\n\n```toml\n[server]\nhost    = \"0.0.0.0\"\nport    = 4318          # standard OTLP/HTTP port\napi_key = \"\"            # optional Bearer token auth\n\n[buffer]\nrotate_mb  = 64         # rotate when buffer reaches this size (MB)\nrotate_sec = 3600       # rotate after this many seconds even if not full\noutput_dir = \"/tmp/pfc-otel\"\nprefix     = \"otel\"     # output filename prefix\n\n[pfc]\nbinary = \"/usr/local/bin/pfc_jsonl\"\n\n[s3]\nenabled = false\nbucket  = \"my-log-archive\"\nprefix  = \"otel-logs/\"\nregion  = \"us-east-1\"\n```\n\n---\n\n## Output format\n\nEach OTLP log record becomes one flat JSONL line:\n\n```json\n{\n  \"timestamp\": \"2026-01-15T10:00:00.123Z\",\n  \"level\": \"ERROR\",\n  \"service\": \"payment-service\",\n  \"message\": \"charge failed: timeout\",\n  \"scope\": \"com.example.payments\",\n  \"http_method\": \"POST\",\n  \"http_status_code\": 500,\n  \"host_name\": \"prod-node-07\",\n  \"trace_id\": \"4bf92f3577b34da6a3ce929d0e0e4736\",\n  \"span_id\": \"00f067aa0ba902b7\"\n}\n```\n\nResource attributes (dots → underscores), log record attributes, scope name, and trace context are all included.\n\n---\n\n## Endpoints\n\n| Method | Path | Auth | Description |\n|--------|------|------|-------------|\n| `POST` | `/v1/logs` | optional | Ingest OTLP/HTTP log data |\n| `GET`  | `/health`  | none     | Health check + binary status |\n| `GET`  | `/stats`   | optional | Ingestion statistics |\n| `POST` | `/flush`   | optional | Force immediate rotation |\n\n### Health response\n\n```json\n{\n  \"status\": \"ok\",\n  \"version\": \"0.1.0\",\n  \"binary_found\": true,\n  \"buffered_lines\": 4271,\n  \"buffered_bytes\": 892440,\n  \"total_ingested\": 128500,\n  \"total_files_compressed\": 3\n}\n```\n\n---\n\n## Authentication\n\nSet `api_key` in config. All endpoints except `/health` will require:\n\n```\nAuthorization: Bearer \u003cyour-api-key\u003e\n```\n\n---\n\n## S3 upload\n\nSet `[s3] enabled = true` and provide your bucket. Credentials via environment:\n\n```bash\nexport AWS_ACCESS_KEY_ID=...\nexport AWS_SECRET_ACCESS_KEY=...\n# or use an IAM instance role (recommended)\n```\n\nAfter successful upload the local `.pfc` file is removed.\n\n---\n\n## CLI flags\n\n```\npfc-otel-collector --config config.toml   # use config file\npfc-otel-collector --port 4318            # override port\npfc-otel-collector --host 0.0.0.0         # override host\npfc-otel-collector --version              # print version\n```\n\n---\n\n## Querying compressed logs\n\n```sql\n-- DuckDB\nINSTALL pfc FROM community;\nLOAD pfc;\n\nSELECT level, count(*) \nFROM read_pfc_jsonl('otel_20260115_100000.pfc',\n                    ts_from=1768471200::BIGINT,\n                    ts_to=1768471500::BIGINT)\nWHERE line LIKE '%payment-service%'\nGROUP BY level;\n```\n\nOr decompress and query directly:\n\n```bash\npfc_jsonl query otel_20260115_100000.pfc \\\n  --from '2026-01-15T10:00' --to '2026-01-15T10:05'\n```\n\n---\n\n## Running tests\n\n```bash\npip install pytest pytest-asyncio httpx\npytest tests/ -v\n# 87 passed\n```\n\n---\n\n## Part of the PFC-JSONL Ecosystem\n\n| Repo | What it does |\n|------|-------------|\n| [pfc-jsonl](https://github.com/ImpossibleForge/pfc-jsonl) | Core compressor (BWT + rANS) |\n| [pfc-duckdb](https://github.com/ImpossibleForge/pfc-duckdb) | DuckDB community extension |\n| [pfc-fluentbit](https://github.com/ImpossibleForge/pfc-fluentbit) | Native Fluent Bit output plugin |\n| [pfc-vector](https://github.com/ImpossibleForge/pfc-vector) | High-performance HTTP ingest daemon |\n| [pfc-gateway](https://github.com/ImpossibleForge/pfc-gateway) | HTTP query gateway |\n| [pfc-migrate](https://github.com/ImpossibleForge/pfc-migrate) | Migrate from gzip/zstd/S3/Azure/GCS |\n| [pfc-kafka-consumer](https://github.com/ImpossibleForge/pfc-kafka-consumer) | Kafka / Redpanda consumer |\n| **pfc-otel-collector** | **OpenTelemetry OTLP/HTTP exporter** |\n| [pfc-grafana](https://github.com/ImpossibleForge/pfc-grafana) | Grafana data source plugin for PFC archives |\n\n---\n\n\n---\n\n## Disclaimer\n\nPFC-OTel-Collector is an independent open-source project and is not affiliated with, endorsed by, or associated with the Cloud Native Computing Foundation (CNCF) or the OpenTelemetry project.\n## License\n\npfc-otel-collector (this repository) is released under the MIT License — see [LICENSE](LICENSE).\n\nThe PFC-JSONL binary (`pfc_jsonl`) is proprietary software — free for personal and open-source use. Commercial use requires a license: [info@impossibleforge.com](mailto:info@impossibleforge.com)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimpossibleforge%2Fpfc-otel-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimpossibleforge%2Fpfc-otel-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimpossibleforge%2Fpfc-otel-collector/lists"}