{"id":17258494,"url":"https://github.com/thibauts/styx","last_synced_at":"2025-08-28T19:24:39.719Z","repository":{"id":49376184,"uuid":"341168467","full_name":"thibauts/styx","owner":"thibauts","description":"Simple, high-performance event streaming broker","archived":false,"fork":false,"pushed_at":"2021-04-01T11:28:31.000Z","size":432,"stargazers_count":64,"open_issues_count":0,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-18T14:03:02.105Z","etag":null,"topics":["broker","event-streaming","messaging","pubsub","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thibauts.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-22T10:54:24.000Z","updated_at":"2024-10-27T20:37:09.000Z","dependencies_parsed_at":"2022-08-25T11:40:12.288Z","dependency_job_id":null,"html_url":"https://github.com/thibauts/styx","commit_stats":null,"previous_names":["dataptive/styx"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thibauts%2Fstyx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thibauts%2Fstyx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thibauts%2Fstyx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thibauts%2Fstyx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thibauts","download_url":"https://codeload.github.com/thibauts/styx/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245003631,"owners_count":20545639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broker","event-streaming","messaging","pubsub","stream-processing"],"created_at":"2024-10-15T07:20:50.602Z","updated_at":"2025-03-22T18:34:27.861Z","avatar_url":"https://github.com/thibauts.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Styx\n\nStyx is a simple and high-performance event streaming broker. It aims to provide teams of all sizes with a simple to operate, disk-persisted publish-subscribe system for event streams. Styx is deployed as a single binary with no dependencies, it exposes a high-performance binary protocol as well as HTTP and WebSockets APIs for both event production and consumption.\n\nDesigned around the concept of a Commit Log as popularized by projects like Apache Kafka or Apache Pulsar, Styx provides durable storage and atomicity on tail-able event logs.\n\nMain features:\n\n- Deployed as a **single binary** with **no dependency**\n- Designed to run and provide durability as a **single-node** system\n- Out-of-the-box **Prometheus** and **Statsd** monitoring\n- **Full-featured CLI** for event log management, **backup/restore** and flexible **bulk load/export**\n- Event records are **immutable**, **atomic**, **durable**, and fsynced to permanent storage before being acked\n- Configurable **retention policies**\n- **Small footprint**. Very low memory and CPU consumption\n- Native **REST API** for event log **management**\n- Native **REST API** for event **production and consumption**, including batched with long-polling support\n- Native **WebSockets API** both for event production and consumption\n- **Millions of events/second**, **GB/s throughput**, low latency\n- Scales to **thousands of producers, consumers, and event logs**\n- **Decoupled storage engine** that can be used standalone\n\nStyx is designed to be a no-brainer for teams that don't want to invest resources in complex (and sometimes fragile) clustered solutions, while providing data safety, plenty of room to scale and instant integration in any language.\n\n## How does it work ?\n\nStyx is a Publish-Subscribe system. `Producers` publish events to event `logs`, and `consumers` subscribe to these logs for further processing.\n\n```\n                                           +--------------+\n                                           |              |\n                                     +----\u003e+   Consumer   |\n                                     |     |              |\n                                     |     +--------------+\n+------------+     +------------+    |     +--------------+\n|            |     |            +----+     |              |\n|  Producer  +----\u003e+  Styx log  +---------\u003e+   Consumer   |\n|            |     |            +----+     |              |\n+------------+     +------------+    |     +--------------+\n                                     |     +--------------+\n                                     |     |              |\n                                     +----\u003e+   Consumer   |\n                                           |              |\n                                           +--------------+\n```\n\n\nUsage of the word `log` in Styx should not be confused with textual application logs, and designate an ordered sequence of event `records` that are persisted to disk.\n\n```\n+------------+------------+------------+-----------+------------+\n|  record 0  |  record 1  |  record 2  |  .......  |  record N  |\n+------------+------------+------------+-----------+------------+\n```\n\nEvent `logs` are append-only and immutable. As they are persisted to disk, event `logs` can be consumed in real-time by susbcribers, and can also be replayed from the beggining or from any arbitrary `position`. This unlocks keeping a complete history of published events, regenerating downstream data when updating your app, or replaying complete streams through new processing logic as unexpected needs surface.\n\n## Using Styx\n\nStyx usage can differ from traditional Message Queues and Streaming Platforms. In particular we encourage usage patterns that remove the need for consumer acks while providing stronger guarantees. You should consider reading the [Reliable Processing Howto](./docs/howto/reliable-processing.md) as an introduction.\n\nReliable event production from external sources to Styx logs can also be achieved with data sources exhibiting particular properties. We plan on developing this subject and other usage patterns in Howto's and tutorial as soon as possible.\n\n## Getting Started\n\n### Using docker\n\nYou can launch a Styx container to try it out with\n\n```bash\n$ docker pull dataptive/styx\n$ docker run --name styx -p 7123:7123 dataptive/styx\n2021-03-31T08:06:06.987817093Z INFO server: starting Styx server version 0.1.4\n2021-03-31T08:06:06.987971158Z INFO logman: starting log manager (data_directory=./data)\n2021-03-31T08:06:06.988911129Z INFO server: listening for client connections on 0.0.0.0:7123\n```\n\nStyx will now be reachable at http://localhost:7123/.\n\nThe Styx CLI is available from the docker container\n\n```bash\n$ docker exec -it styx styx\nUsage: styx COMMAND\n\nA command line interface (CLI) for the Styx API.\n\nCommands:\n        logs  Manage logs\n\nGlobal Options:\n        -f, --format string     Output format [text|json] (default \"text\")\n        -H, --host string       Server to connect to (default \"http://localhost:7123\")\n        -h, --help              Display help\n```\n\n### Precompiled binaries\n\nPrecompiled binaries and packages are available from the [Releases](https://github.com/dataptive/styx/releases) section for various operating systems and architectures.\n\nInstalling on Debian-based systems\n\n```bash\nwget https://github.com/dataptive/styx/releases/download/v0.1.4/styx-0.1.4-amd64.deb\ndpkg -i styx-0.1.4-amd64.deb\n```\n\nInstalling on Redhat-based systems\n\n```bash\nwget https://github.com/dataptive/styx/releases/download/v0.1.4/styx-0.1.4-amd64.rpm\nrpm -i styx-0.1.4-amd64.rpm\n```\n\n### Building from source\n\nTo build or install Styx from source, you will need to have a working Go environment.\n\n```bash\n$ go get github.com/dataptive/styx/cmd/styx-server\n$ go get github.com/dataptive/styx/cmd/styx\n```\n\nCreate a directory to host event log data, and update your `config.toml` to make `data_directory` point to your newly created directory (a sample config file is provided at the root of this repository).\n\nAssuming you're hosting both `data_directory` and `config.toml` in your current working directory, run\n\n```bash\n$ mkdir $PWD/data\n```\n\nThen run the Styx server with this command\n\n```bash\n$ styx-server --config=$PWD/config.toml\n```\n\nThe Styx CLI should be available as the `styx` command\n\n```bash\n$ styx\nUsage: styx COMMAND\n\nA command line interface (CLI) for the Styx API.\n\nCommands:\n        logs  Manage logs\n\nGlobal Options:\n        -f, --format string     Output format [text|json] (default \"text\")\n        -H, --host string       Server to connect to (default \"http://localhost:7123\")\n        -h, --help              Display help\n```\n\nTo build the binaries without installing them, run\n\n```bash\n$ git clone https://github.com/dataptive/styx.git\n$ cd styx\n$ go build -o styx-server cmd/styx-server/main.go\n$ go build -o styx cmd/styx/main.go\n```\n\nThe `styx-server` and `styx` binaries should be available in your working directory.\n\n### Producing and consuming records\n\nWhile the server is running, create a `test` event log and tail its contents using the CLI\n\n```bash\n$ styx logs create test\n$ styx logs consume test --follow\n```\n\nThe CLI will hang, waiting for incoming events (you can quit with Ctrl+D)\n\nIn another terminal, try pushing events in the log with curl\n\n```bash\n$ curl localhost:7123/logs/test/records -X POST -d 'Hello, world !' \n```\n\nAll CLI commands support the `--help` flag, play with them to get a quick tour of Styx's features. For example\n\n```bash\n$ styx logs create --help\nUsage: styx logs create NAME [OPTIONS]\n\nCreate a new log\n\nOptions:\n        --max-record-size bytes         Maximum record size\n        --index-after-size bytes        Write a segment index entry after every size\n        --segment-max-count records     Create a new segment when current segment exceeds this number of records\n        --segment-max-size bytes        Create a new segment when current segment exceeds this size\n        --segment-max-age seconds       Create a new segment when current segment exceeds this age\n        --log-max-count records         Expire oldest segment when log exceeds this number of records\n        --log-max-size bytes            Expire oldest segment when log exceeds this size\n        --log-max-age seconds           Expire oldest segment when log exceeds this age\n\nGlobal Options:\n        -f, --format string             Output format [text|json] (default \"text\")\n        -H, --host string               Server to connect to (default \"http://localhost:7123\")\n        -h, --help                      Display help\n```\n\n### Benchmarking\n\nUse the `benchmark` command from the CLI to run a benchmark on your own server.\n\nThe command will launch a batch of produce and consume tasks with various payload sizes and produce a report. You should have about 10GB of free storage for everything to run smoothly. Used disk space will automatically be freed when the benchmark ends.\n\nAlthough you'll get better numbers when running outside of docker, the following commands will give you a quick estimate of what you can expect from Styx in your own context.\n\n```bash\n$ docker pull dataptive/styx\n$ docker run --name styx -d -p 7123:7123 dataptive/styx\n$ docker exec styx styx benchmark\n```\n\nIn case you've built Styx from source, simply run\n\n```bash\n$ styx benchmark\n```\n\n## Documentation\n\nThe documentation is a work in progress, please open an issue if you find something unclear or missing.\n\n* [Getting started](./docs/administration)\n\t* [Installing](./docs/administration/installation.md)\n\t* [Configuration](./docs/administration/configuration.md)\n\t* [Monitoring with Prometheus and Statsd](./docs/administration/monitoring.md)\n\t* [CLI reference](./docs/administration/CLI.md)\n* [API reference](./docs/api)\n\t* [Managing event logs](./docs/api/manage.md)\n\t* [Producing events through the REST API](./docs/api/produce_HTTP.md)\n\t* [Consuming events through the REST API](./docs/api/consume_HTTP.md)\n\t* [Producing events with WebSockets](./docs/api/produce_websocket.md)\n\t* [Consuming events with WebSockets](./docs/api/consume_websocket.md)\n\t* [Binary protocol specs](./docs/api/styx_protocol.md)\n\t\t* [Opening a connection for producing events](./docs/api/produce_styx.md)\n\t\t* [Opening a connection for consuming events](./docs/api/consume_styx.md)\n\t* [API Media Types](./docs/api/media_types.md)\n\t* [API errors](./docs/api/errors.md)\n* [Howto](./docs/howto)\n\t* [Consuming events with HTTP long-polling](./docs/howto/longpolling.md)\n* [Code examples](./docs/examples)\n\t* [Using the Golang client Producer](./docs/examples/golang-fast-producer)\n\t* [Using WebSockets in python to consume a real-time event feed](./docs/examples/python-gdax-connector)\n\t* [Using python+requests to produce concurrently](./docs/examples/python-multi-producers)\n\t* [Building a reliable processing pipeline using python and WebSockets](./docs/examples/python-reliable-processing)\n\t* [Building a HTTP-\u003eStyx event gateway in NodeJS](./docs/examples/nodejs-event-gateway)\n\t* [Using NodeJS to build a simple event processing pipeline](./docs/examples/nodejs-simple-processing)\n\n## Design\n\nTODO !\n\n## Contributing\n\nYou're more than welcome to open issues should you encounter any bug or instability in the code. Feature suggestions are welcome. We may restrict PRs until we have setup a CLA.\n\n## Community\n\nYou're welcome to join our [Slack](https://join.slack.com/t/dataptive/shared_invite/zt-mzl99jf9-zaFiIVYbZRN6m2nqkcgZmg) to discuss the project or ask for help !\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthibauts%2Fstyx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthibauts%2Fstyx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthibauts%2Fstyx/lists"}