{"id":13629978,"url":"https://github.com/ucbrise/confluo","last_synced_at":"2025-05-15T23:06:39.959Z","repository":{"id":33269134,"uuid":"50422569","full_name":"ucbrise/confluo","owner":"ucbrise","description":"Real-time Monitoring and Analysis of Data Streams","archived":false,"fork":false,"pushed_at":"2022-05-20T20:55:41.000Z","size":11828,"stargazers_count":1442,"open_issues_count":25,"forks_count":200,"subscribers_count":104,"default_branch":"single-machine","last_synced_at":"2025-04-08T10:22:47.799Z","etag":null,"topics":["atomic","lock-free","log"],"latest_commit_sha":null,"homepage":"https://ucbrise.github.io/confluo/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ucbrise.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-26T10:49:47.000Z","updated_at":"2025-02-25T15:10:57.000Z","dependencies_parsed_at":"2022-09-11T19:51:13.834Z","dependency_job_id":null,"html_url":"https://github.com/ucbrise/confluo","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fconfluo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fconfluo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fconfluo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbrise%2Fconfluo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ucbrise","download_url":"https://codeload.github.com/ucbrise/confluo/tar.gz/refs/heads/single-machine","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atomic","lock-free","log"],"created_at":"2024-08-01T22:01:26.303Z","updated_at":"2025-05-15T23:06:34.934Z","avatar_url":"https://github.com/ucbrise.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"# Confluo\n\n[![Build Status](https://amplab.cs.berkeley.edu/jenkins/job/confluo/badge/icon)](https://amplab.cs.berkeley.edu/jenkins/job/confluo/)\n[![License](http://img.shields.io/:license-Apache%202-red.svg)](LICENSE)\n\nConfluo is a system for real-time monitoring and analysis of data, that supports:\n* high-throughput concurrent writes of millions of data points from multiple data streams;\n* online queries at millisecond timescale; and \n* ad-hoc queries using minimal CPU resources.\n\nPlease find detailed documentation [here](https://ucbrise.github.io/confluo/).\n\n## Installation\n\nRequired dependencies:\n\n* MacOS X or Unix-based OS; Windows is not yet supported.\n* C++ compiler that supports C++11 standard (e.g., GCC 5.3 or later)\n* CMake 3.2 or later\n* Boost 1.58 or later\n\nFor python client, you will additionally require:\n\n* Python 2.7 or later\n* Python Packages: setuptools, six 1.7.2 or later\n\nFor java client, you will additionally require:\n\n* Java JDK 1.7 or later\n* ant 1.6.2 or later\n\n### Source Build\n\nTo download and install Confluo, use the following commands:\n\n```bash\ngit clone https://github.com/ucbrise/confluo.git\ncd confluo\nmkdir build\ncd build\ncmake ..\nmake -j \u0026\u0026 make test \u0026\u0026 make install\n```\n\n## Using Confluo\n\nWhile Confluo supports multiple execution modes, the simplest way to get \nstarted is to start Confluo as a server daemon and query it using one of\nits client APIs.\n\nTo start the server daemon, run:\n\n```bash\nconfluod --address=127.0.0.1 --port=9090\n```\n\nHere's some sample usage of the Python API:\n\n```python\nimport sys\nfrom confluo.rpc.client import RpcClient\nfrom confluo.rpc.storage import StorageMode\n\n# Connect to the server\nclient = RpcClient(\"127.0.0.1\", 9090)\n\n# Create an Atomic MultiLog with given schema for a performance log\nschema = \"\"\"{\n  timestamp: ULONG,\n  op_latency_ms: DOUBLE,\n  cpu_util: DOUBLE,\n  mem_avail: DOUBLE,\n  log_msg: STRING(100)\n}\"\"\"\nstorage_mode = StorageMode.IN_MEMORY\nclient.create_atomic_multilog(\"perf_log\", schema, storage_mode)\n\n# Add an index\nclient.add_index(\"op_latency_ms\")\n\n# Add a filter\nclient.add_filter(\"low_resources\", \"cpu_util\u003e0.8 || mem_avail\u003c0.1\")\n\n# Add an aggregate\nclient.add_aggregate(\"max_latency_ms\", \"low_resources\", \"MAX(op_latency_ms)\")\n\n# Install a trigger\nclient.install_trigger(\"high_latency_trigger\", \"max_latency_ms \u003e 1000\")\n\n# Load some data\noff1 = client.append([100.0, 0.5, 0.9,  \"INFO: Launched 1 tasks\"])\noff2 = client.append([500.0, 0.9, 0.05, \"WARN: Server {2} down\"])\noff3 = client.append([1001.0, 0.9, 0.03, \"WARN: Server {2, 4, 5} down\"])\n\n# Read the written data\nrecord1 = client.read(off1)\nrecord2 = client.read(off2)\nrecord3 = client.read(off3)\n\n# Query using indexes\nrecord_stream = client.execute_filter(\"cpu_util\u003e0.5 || mem_avail\u003c0.5\")\nfor r in record_stream:\n  print r\n\n# Query using filters\nrecord_stream = client.query_filter(\"low_resources\", 0, sys.maxsize)\nfor r in record_stream:\n  print r\n\n# Query an aggregate\nprint client.get_aggregate(\"max_latency_ms\", 0, sys.maxsize)\n\n# Query alerts generated by a trigger\nalert_stream = client.get_alerts(0, sys.maxsize, \"high_latency_trigger\")\nfor a in alert_stream:\n  print a\n```\n\n## Contributing\n\nPlease create a GitHub issue to file a bug or request a feature. We welcome pull-requests, but request that you review the [pull-request process](CONTRIBUTING.md) before submitting one.\n\nPlease subscribe to the mailing list (confluo-dev@googlegroups.com) for project announcements, and discussion regarding use-cases and development.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucbrise%2Fconfluo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fucbrise%2Fconfluo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucbrise%2Fconfluo/lists"}