{"id":13466256,"url":"https://github.com/y-scope/clp","last_synced_at":"2026-01-21T07:55:17.746Z","repository":{"id":42126172,"uuid":"372437728","full_name":"y-scope/clp","owner":"y-scope","description":"Compressed Log Processor (CLP) is a free log management tool capable of compressing logs and searching the compressed logs without decompression.","archived":false,"fork":false,"pushed_at":"2026-01-20T09:39:46.000Z","size":9795,"stargazers_count":1029,"open_issues_count":362,"forks_count":88,"subscribers_count":12,"default_branch":"main","last_synced_at":"2026-01-20T10:43:56.083Z","etag":null,"topics":["analytics","compression","log-management","log-parser","logging","search"],"latest_commit_sha":null,"homepage":"https://yscope.com","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/y-scope.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-05-31T08:35:38.000Z","updated_at":"2026-01-20T04:52:21.000Z","dependencies_parsed_at":"2023-10-16T11:35:37.815Z","dependency_job_id":"502ddf14-70b7-4ac8-adc9-6909b0466608","html_url":"https://github.com/y-scope/clp","commit_stats":null,"previous_names":["y-scope/clp-core"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/y-scope/clp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Fclp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Fclp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Fclp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Fclp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/y-scope","download_url":"https://codeload.github.com/y-scope/clp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/y-scope%2Fclp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28629917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-21T04:47:28.174Z","status":"ssl_error","status_checked_at":"2026-01-21T04:47:22.943Z","response_time":86,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","compression","log-management","log-parser","logging","search"],"created_at":"2024-07-31T15:00:41.592Z","updated_at":"2026-01-21T07:55:17.741Z","avatar_url":"https://github.com/y-scope.png","language":"C++","readme":"\u003cimg alt=\"CLP\" src=\"docs/src/clp-logo.png\" width=\"300\"/\u003e\n\n[![Open bug reports](https://img.shields.io/github/issues/y-scope/clp/bug?label=bugs)](https://github.com/y-scope/clp/issues?q=is%3Aissue+is%3Aopen+label%3Abug)\n[![Open feature requests](https://img.shields.io/github/issues/y-scope/clp/enhancement?label=feature-requests)](https://github.com/y-scope/clp/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement)\n[![Discord][badge-discord]][yscope-community-discord]\n[![Slack][badge-slack]][yscope-community-slack]\n[![CLP on Zulip][badge-zulip]][yscope-community-zulip]\n\nYScope's Compressed Log Processor (CLP) compresses your logs, and allows you to search the\ncompressed logs without decompression. CLP supports both JSON logs and unstructured (i.e., free\ntext) logs. It also supports real-time log compression within several logging libraries. CLP also\nincludes purpose-built web interfaces for searching and viewing the compressed logs. To learn more\nabout it, read our [2021 paper][clp-paper-21] about handling unstructured logs and our \n[2024 paper][clp-paper-24] on extending it to JSON logs.\n\n# Benchmarks\n\n![CLP Benchmark on JSON Logs](docs/src/clp-json-benchmark.png)\n![CLP Benchmark on Unstructured Logs](docs/src/clp-unstructured-benchmark.png)\n\nThe figures above show CLP's compression and search performance compared to other tools. We separate\nthe experiments between JSON and unstructured logs because (1) some tools can only handle one type\nof logs, and (2) tools that can handle both types often have different designs for each type (such\nas CLP).\n\nCompression ratio is measured as the average across a variety of log datasets. Some of these\ndatasets can be found [here][datasets]. Search performance is measured using queries on the\nMongoDB logs (for JSON) and the Hadoop logs (for unstructured logs). Note that CLP uses an\nindex-less design, so for a fair comparison, we disabled MongoDB and PostgreSQL's indexes; If we\nleft them enabled, MongoDB and PostgreSQL's compression ratio would be worse. We didn't disable\nindexing for Elasticsearch or Splunk since these tools are fundamentally index-based (i.e., logs\ncannot be searched without indexes). More details about our experimental methodology can be found in\nthe [2021 paper][clp-paper-21] and the [2024 paper][clp-paper-24].\n\n# System Overview\n\n![CLP systems overview](docs/src/clp-complete-solution.png)\n\nCLP provides an end-to-end log management pipeline consisting of compression, search, analytics, and\nviewing. The figure above shows the CLP ecosystem architecture. It consists of the following\nfeatures:\n\n- **Compression and Search**: CLP compresses logs into archives, which can be searched and analyzed\n  in a [web UI](components/webui). The input can either be raw logs or CLP's compressed IR\n  (intermediate representation) produced by CLP's logging libraries.\n\n- **Real-time Compression with CLP Logging Libraries**: CLP provides logging libraries for\n  [Python][clp-loglib-py] and Java ([Log4j1][log4j1-appenders], [Log4j2][log4j2-appenders] and \n  [Logback][logback-appenders]). The logging libraries compress logs in real-time, so only\n  compressed logs are written to disk or transmitted over the network. The compressed logs use CLP's\n  intermediate representation (IR) format which achieves a higher compression ratio than general\n  purpose compressors like Zstandard. Compressing IR into archives can further double the\n  compression ratio and enable global search, but this requires more memory usage as it needs to\n  buffer enough logs. More details on IR versus archives can be found in this\n  [Uber Engineering Blog][uber-blog].\n\n- **[Log Viewer][log-viewer]**: the compressed IR can be viewed in a web-based log viewer. Compared\n  to viewing the logs in an editor, CLP's log viewer supports advanced features like filtering logs\n  based on log level verbosity (e.g., only displaying logs with log level equal or higher than\n  ERROR). These features are possible because CLP's logging libraries parse the logs before\n  compressing them into IR.\n\n- **IR Analytics Libraries**: we also provide a [Python library][clp-ffi-py] and a\n  [Go library][clp-ffi-go] that can analyze compressed IR.\n\n- **[Log parser][log-surgeon]**: CLP also includes a custom pushdown-automata-based log parser that\n  is 3x faster than state-of-the-art regular expression engines like [RE2][re2]. The log parser is\n  available as a library that can be used by other applications.\n\n# Getting Started\n\nYou can download a [release package](https://github.com/y-scope/clp/releases) which includes support\nfor distributed compression and search. Or, to quickly try CLP's *core* compression and search, you\ncan use a [prebuilt container][core-container].\n\nWe also have guides for building the [package][build-package] and [CLP core][core] from source.\n\nFor some logs you can use to test CLP, check out our open-source [datasets][datasets].\n\n# Docs\n\nYou can find our docs [online][clp-docs] or view the source in `docs/src`. \n\n# Providing Feedback\n\nYou can use GitHub issues to [report a bug][bug-report] or [request a feature][feature-req].\n\n# Community\n\nNeed help? Join us on one of our community servers:\n\n* [![Discord][badge-discord]][yscope-community-discord]\n* [![Slack][badge-slack]][yscope-community-slack]\n* [![CLP on Zulip][badge-zulip]][yscope-community-zulip]\n\n# Next Steps\n\nThis is our open-source release which we will be constantly updating with bug fixes, features, etc.\nIf you would like a feature or want to report a bug, please file an issue and we'll be happy to engage.\n\n[badge-discord]: https://img.shields.io/discord/1377353873068392580?style=flat\u0026logo=discord\u0026logoColor=white\u0026label=Discord\u0026labelColor=5561f5\n[badge-slack]: https://img.shields.io/badge/Slack-yscope--community-1e724f?style=flat\u0026logo=slack\u0026logoColor=white\u0026labelColor=4A154B\n[badge-zulip]: https://img.shields.io/badge/Zulip-yscope--clp-1888FA?logo=zulip\n[bug-report]: https://github.com/y-scope/clp/issues/new?assignees=\u0026labels=bug\u0026template=bug-report.yaml\n[build-package]: http://docs.yscope.com/clp/main/dev-docs/building-package\n[clp-docs]: https://docs.yscope.com/clp/main\n[clp-ffi-go]: https://github.com/y-scope/clp-ffi-go\n[clp-ffi-py]: https://github.com/y-scope/clp-ffi-py\n[clp-loglib-py]: https://github.com/y-scope/clp-loglib-py\n[clp-paper-21]: https://www.usenix.org/system/files/osdi21-rodrigues.pdf\n[clp-paper-24]: https://www.usenix.org/system/files/osdi24-wang-rui.pdf\n[core]: http://docs.yscope.com/clp/main/dev-docs/components-core\n[core-container]: http://docs.yscope.com/clp/main/user-docs/core-container\n[datasets]: https://docs.yscope.com/clp/main/user-docs/resources-datasets\n[feature-req]: https://github.com/y-scope/clp/issues/new?assignees=\u0026labels=enhancement\u0026template=feature-request.yaml\n[log-surgeon]: https://github.com/y-scope/log-surgeon\n[log-viewer]: https://github.com/y-scope/yscope-log-viewer\n[log4j1-appenders]: https://github.com/y-scope/log4j1-appenders\n[log4j2-appenders]: https://github.com/y-scope/log4j2-appenders\n[logback-appenders]: https://github.com/y-scope/logback-appenders\n[re2]: https://github.com/google/re2\n[uber-blog]: https://www.uber.com/en-US/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp\n[yscope-community-discord]: https://discord.gg/7kZA2m5G87\n[yscope-community-slack]: https://communityinviter.com/apps/yscopecommunity/yscope-community\n[yscope-community-zulip]: https://yscope-clp.zulipchat.com\n","funding_links":[],"categories":["C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fy-scope%2Fclp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fy-scope%2Fclp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fy-scope%2Fclp/lists"}