{"id":38279815,"url":"https://github.com/y-scope/fluent-bit-clp","last_synced_at":"2026-01-17T01:56:43.872Z","repository":{"id":242914364,"uuid":"807228987","full_name":"y-scope/fluent-bit-clp","owner":"y-scope","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-16T18:44:34.000Z","size":16806,"stargazers_count":3,"open_issues_count":4,"forks_count":3,"subscribers_count":3,"default_branch":"irv2-beta","last_synced_at":"2026-01-16T21:43:10.245Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/y-scope.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-28T17:45:06.000Z","updated_at":"2026-01-16T18:44:38.000Z","dependencies_parsed_at":"2024-07-30T08:11:23.645Z","dependency_job_id":"c4ceaa97-b5de-469c-b886-0d810eb21455","html_url":"https://github.com/y-scope/fluent-bit-clp","commit_stats":null,"previous_names":["y-scope/fluent-bit-clp"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/y-scope/fluent-bit-clp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Ffluent-bit-clp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Ffluent-bit-clp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Ffluent-bit-clp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Ffluent-bit-clp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/y-scope","download_url":"https://codeload.github.com/y-scope/fluent-bit-clp/tar.gz/refs/heads/irv2-beta","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Ffluent-bit-clp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28491911,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T00:50:05.742Z","status":"ssl_error","status_checked_at":"2026-01-17T00:43:11.982Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T01:56:43.263Z","updated_at":"2026-01-17T01:56:43.860Z","avatar_url":"https://github.com/y-scope.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fluent Bit CLP Plugins\n\n[Fluent Bit][fluent-bit] output plugins that compress logs using [CLP][clp-blog] for efficient\nstorage and search on S3.\n\n## Overview\n\n```mermaid\nflowchart LR\n    A[Fluent Bit] --\u003e B[CLP Plugin]\n    B --\u003e C[CLP IR + Zstd]\n    C --\u003e D[(S3)]\n    D --\u003e E[YScope Log Viewer]\n```\n\n**[CLP][clp-blog]** (Compressed Log Processor) achieves 10-100x better compression than gzip while\nenabling fast search. These plugins compress logs into CLP's Intermediate Representation (IR)\nformat with [Zstd][zstd] compression, then upload to S3.\n\nView compressed logs directly in the browser with [YScope Log Viewer][log-viewer], or ingest into\nCLP for search at scale.\n\n## Plugins\n\nTwo plugins with different log delivery strategies:\n\n| Plugin | Strategy | Description |\n|--------|----------|-------------|\n| **[out_clp_s3_v2](plugins/out_clp_s3_v2/README.md)** | Continuous sync | Compressed logs continuously synced to S3; sync frequency tunable to balance freshness vs cost |\n| **[out_clp_s3](plugins/out_clp_s3/README.md)** | Batch upload | Logs buffered locally until size threshold reached, then uploaded as batch |\n\n### out_clp_s3_v2 — Continuous Sync\n\nLog rotation is delegated to the log appender. The plugin continuously syncs compressed log files\nto S3.\n\n- Higher severity logs (e.g., ERROR) trigger faster sync of the entire log file\n- Dual-timer strategy (hard + soft deadlines) prevents upload storms\n- Designed for Kubernetes (sidecar and DaemonSet patterns)\n\n### out_clp_s3 — Batch Upload\n\nLogs are buffered locally and uploaded when a size threshold is reached.\n\n- Upload when buffer reaches size threshold (default 16 MB)\n- Disk buffering with crash recovery\n- IAM role assumption for cross-account access\n\n## Quick Start\n\nThe fastest way to try locally with Docker Compose:\n\n```shell\ngit clone --recursive https://github.com/y-scope/fluent-bit-clp.git\ncd fluent-bit-clp\n\n# Choose one:\ncd plugins/out_clp_s3_v2/examples/docker-compose  # Continuous sync\ncd plugins/out_clp_s3/examples/docker-compose     # Batch upload\n\ndocker compose up\n```\n\nThis starts MinIO (S3-compatible storage), Fluent Bit with the plugin, and a log generator.\nView logs at http://localhost:9001 (minioadmin/minioadmin).\n\n## Installation\n\n### Docker Images\n\nPre-built Fluent Bit images with CLP plugins:\n\n```shell\ndocker pull ghcr.io/y-scope/fluent-bit-clp-s3-v2:latest  # Continuous sync\ndocker pull ghcr.io/y-scope/fluent-bit-clp-s3:latest     # Batch upload\n```\n\nImages are tagged with `latest` (main branch), branch names, and commit SHAs.\n\n### Pre-built Binaries\n\nStandalone `.so` plugin files available from [GitHub Releases](../../releases):\n\n| File | Architecture |\n|------|--------------|\n| `out_clp_s3_v2_linux_amd64.so` | x86_64 |\n| `out_clp_s3_v2_linux_arm64.so` | ARM64 |\n| `out_clp_s3_linux_amd64.so` | x86_64 |\n| `out_clp_s3_linux_arm64.so` | ARM64 |\n\nAlso available as zip from [Actions → build](../../actions/workflows/build.yaml).\n\n## Kubernetes\n\nSee [Kubernetes Examples](plugins/out_clp_s3_v2/examples/kubernetes/README.md) for:\n- Local development with k3d\n- Production deployment with init containers\n- Sidecar and DaemonSet patterns\n\n## Documentation\n\n| Document | Description |\n|----------|-------------|\n| [out_clp_s3_v2 README](plugins/out_clp_s3_v2/README.md) | Configuration and flush strategy |\n| [out_clp_s3 README](plugins/out_clp_s3/README.md) | Configuration and disk buffering |\n\n## Development\n\n### Prerequisites\n\n- Go 1.24+\n- [Task](https://taskfile.dev/installation/)\n\n### Building\n\n```shell\n# Clone with submodules\ngit clone --recursive https://github.com/y-scope/fluent-bit-clp.git\ncd fluent-bit-clp\n\n# Download clp-ffi-go native libraries\nbash third-party/clp-ffi-go/scripts/download-libs.sh\n\n# Build plugins\ntask build\n```\n\n### Linting\n\n```shell\ntask lint:check  # Run checks\ntask lint:fix    # Auto-fix\n```\n\n\u003e **Note:** Linting for `out_clp_s3` is temporarily disabled.\n\n[clp-blog]: https://www.uber.com/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp\n[fluent-bit]: https://fluentbit.io/\n[log-viewer]: https://github.com/y-scope/yscope-log-viewer\n[zstd]: https://facebook.github.io/zstd/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fy-scope%2Ffluent-bit-clp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fy-scope%2Ffluent-bit-clp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fy-scope%2Ffluent-bit-clp/lists"}