{"id":13499495,"url":"https://github.com/tremor-rs/tremor-runtime","last_synced_at":"2025-05-13T16:10:49.367Z","repository":{"id":36974847,"uuid":"211798430","full_name":"tremor-rs/tremor-runtime","owner":"tremor-rs","description":"Main Tremor Project Rust Codebase","archived":false,"fork":false,"pushed_at":"2025-02-20T05:12:54.000Z","size":24236,"stargazers_count":893,"open_issues_count":150,"forks_count":127,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-04-23T20:55:49.455Z","etag":null,"topics":["hacktoberfest","hacktoberfest2021","rust","tremor","tremor-languages","tremor-query","tremor-runtime","tremor-script"],"latest_commit_sha":null,"homepage":"https://www.tremor.rs","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tremor-rs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-30T07:13:12.000Z","updated_at":"2025-04-16T02:58:31.000Z","dependencies_parsed_at":"2024-04-17T06:28:03.801Z","dependency_job_id":"db5cbf1c-b4e1-4c63-a209-cf55e92388c5","html_url":"https://github.com/tremor-rs/tremor-runtime","commit_stats":{"total_commits":4129,"total_committers":53,"mean_commits":77.90566037735849,"dds":0.6597239040930007,"last_synced_commit":"81897ea4585ec00df56df49c5b1cb7562a5c406a"},"previous_names":[],"tags_count":107,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tremor-rs%2Ftremor-runtime","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tremor-rs%2Ftremor-runtime/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tremor-rs%2Ftremor-runtime/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tremor-rs%2Ftremor-runtime/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tremor-rs","download_url":"https://codeload.github.com/tremor-rs/tremor-runtime/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250514767,"owners_count":21443208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","hacktoberfest2021","rust","tremor","tremor-languages","tremor-query","tremor-runtime","tremor-script"],"created_at":"2024-07-31T22:00:33.698Z","updated_at":"2025-04-23T20:56:37.520Z","avatar_url":"https://github.com/tremor-rs.png","language":"Rust","funding_links":[],"categories":["Rust","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"\u003cp align=center\u003e\u003cimg src=\"https://raw.githubusercontent.com/cncf/artwork/master/projects/tremor/horizontal/color/tremor-horizontal-color.png\" width='35%'\u003e\u003c/p\u003e\n\u003cp align=center\u003e\u003ca href=\"https://landscape.cncf.io/selected=tremor\"\u003eCNCF Early Stage Sandbox Project\u003c/p\u003e\n\u003cp align=center\u003e\u003ca href=\"https://landscape.cncf.io/category=streaming-messaging\u0026format=card-mode\u0026grouping=category\"\u003eCNCF Streaming \u0026amp; Messaging\u003c/p\u003e\n\n\u003chr\u003e\n\n[![Gitpod]][gitpod-hook]\n\n\u003chr\u003e\n\n[![Build Status]][actions-tests]\n[![Quality Checks]][actions-checks]\n[![License Checks]][actions-license-audit]\n[![Security Checks]][actions-security-audit]\n[![codecov]][codecov report]\n[![Dependabot Status]][dependabot status]\n[![CII Best Practices]][bestpractices]\n[![GitHub]](LICENSE)\n[![Discord]][discord-invite]\n\n[build status]: https://github.com/tremor-rs/tremor-runtime/workflows/Tests/badge.svg\n[actions-tests]: https://github.com/tremor-rs/tremor-runtime/actions?query=workflow%3ATests\n[Quality Checks]: https://github.com/tremor-rs/tremor-runtime/workflows/Checks/badge.svg\n[actions-checks]: https://github.com/tremor-rs/tremor-runtime/actions?query=workflow%3AChecks\n[actions-license-audit]: https://github.com/tremor-rs/tremor-runtime/actions?query=workflow%3A%22License+audit%22\n[License Checks]: https://github.com/tremor-rs/tremor-runtime/workflows/License%20audit/badge.svg\n[actions-security-audit]: https://github.com/tremor-rs/tremor-runtime/actions?query=workflow%3A%22Security+audit%22\n[Security Checks]: https://github.com/tremor-rs/tremor-runtime/workflows/Security%20audit/badge.svg\n[dependabot status]: https://flat.badgen.net/github/dependabot/tremor-rs/tremor-runtime\n[cii best practices]: https://bestpractices.coreinfrastructure.org/projects/4356/badge\n[bestpractices]: https://bestpractices.coreinfrastructure.org/projects/4356\n[github]: https://img.shields.io/github/license/tremor-rs/tremor-runtime\n[discord]: https://img.shields.io/discord/752801695066488843.svg?label=\u0026logo=discord\u0026logoColor=ffffff\u0026color=7389D8\u0026labelColor=6A7EC2\n[discord-invite]: https://bit.ly/tremor-discord\n[gitpod]: https://gitpod.io/button/open-in-gitpod.svg\n[gitpod-hook]: https://gitpod.io/#https://github.com/tremor-rs/tremor-runtime\n[codecov]: https://codecov.io/gh/tremor-rs/tremor-runtime/branch/main/graph/badge.svg?token=d1bhuZGcOK\n[codecov report]: https://codecov.io/gh/tremor-rs/tremor-runtime\n\n---\n\nIn short, Tremor is an event- or stream-processing system. It is designed to perform well for high-volumetric data both in terms of consumption of memory and CPU resources and in terms of latency. The goal of Tremor is to be a convenient tool for the operator at the time of configuring Tremor and at runtime in a production setup. We provide our own LSP for Tremor configurations and take great care of providing insightful metrics and helpful error messages at runtime. All this while keeping the hot data-path as performant as possible.\n\nTremor is well suited for ETL workloads on structured and binary data (both schemaless and strictly schematic), aggregations, traffic shaping and routing purposes. \n\nTremor speaks various protocols (TCP, UDP, HTTP, Websockets, DNS) and can connect to various external systems such as Kafka, Influx compatible stores, syslog, Open telemetry, Google Pubsub, Google BigQuery, S3 and many more.\n\n* [Documentation v0.12](https://tremor.rs/docs/0.12/index)\n* [Install instructions](https://tremor.rs/docs/0.12/getting-started/install)\n* [Guides](https://tremor.rs/docs/0.12/guides/overview)\n* [Reference](https://tremor.rs/docs/0.12/reference/index)\n* [Architecture overview](https://tremor.rs/docs/0.12/about/architecture)\n\n## Audience\n\nTremor is a real-time event processing engine built for users that have a high message volume to deal with and want to build pipelines to process, route, or limit this event stream. Tremor supports vast number of connectors to interact with: TCP, UDP, HTTP, Websockets, Kafka, Elasticsearch, S3 and many more.\n\n### When to use Tremor\n\n- You want to apply traffic-shaping to a high volume of incoming events\n- You want to distribute events based on their contents\n- You want to protect a downstream system from overload\n- You wish to perform ETL like tasks on data.\n\n### When not to use Tremor\n\nNote: Some of those restrictions are subject to change as tremor is a growing project.\n\nWe currently do not recommend tremor where:\n\n- Your event structure is not mappable to a JSON-like data structure.\n  - If in doubt, please reach out and create a ticket so we can assist and advice\n  - In many cases ( textual formats ) a [preprocessor](https://www.tremor.rs/docs/0.12/reference/preprocessors/), [postprocessor](https://www.tremor.rs/docs/0.12/reference/postprocessors/)\n- You need connectivity to a system, protocol or technology that is not currently supported directly or indirectly by the existing set of [connectors](https://www.tremor.rs/docs/0.12/reference/connectors/).\n  - If in doubt, please reach out and create a ticket so we can assist and advise.\n- You require complex and expensive operations on your event streams like joins of huge streams. Tremor is not built for huge analytical datasets, rather for tapping into infinite datastreams at their source (e.g. k8s events, syslog, kafka).\n\nWe accept and encourage contributions no matter how small so if tremor is compelling for your use case or project, then please get in touch, reach out, raise a ticket and we're happy to collaborate and guide contributions and contributors.\n\n## Examples\n\nSee our [Demo](demo/README.md) for a complex Tremor setup that can easily be run locally by using docker compose.\n\nCheckout the [Guides](https://www.tremor.rs/docs/0.12/guides/) on our website for more examples on how to use tremor.\n\n## Packages\n\nWe do provide RPM, DEB and pre-compiled binaries (`x86_64` only) for each release.\n\nCheck out our [Releases Page](https://github.com/tremor-rs/tremor-runtime/releases).\n\n## Docker\n\nDocker images are published to both [Docker Hub](https://hub.docker.com/r/tremorproject/tremor) and [Github Packages Container Registry](https://ghcr.io).\n\n| Container registry | Image name                        |\n| ------------------ | --------------------------------- |\n| docker.io          | `tremorproject/tremor`            |\n| ghcr.io            | `tremor-rs/tremor-runtime/tremor` |\n\nWe publish our images with a set of different tags as explained below\n\n| Image tags | Explanation                         | Example  |\n| ---------- | ----------------------------------- | -------- |\n| `edge`     | Tracking the `main` branch          |          |\n| `latest`   | The latest release                  |          |\n| `0.X.Y`    | The exact release                   | `0.12.1` |\n| `0.X`      | The latest bugfix release for `0.X` | `0.12`   |\n| `0`        | The latest minor release for `0`    | `0`      |\n\n### Building the Docker Image\n\nTremor runs in a docker image. If you wish to build a local image, clone this repository, and either run `make image` or run `docker compose build`. Both will create an image called `tremorproject/tremor:latest`.\n\nNote that since the image is building tremor in release mode it requires some serious resources. We recommend allowing docker to use at least **12 but better 16 gigabytes of memory** and as many cores as there are to spare. Depending on the system building, the image can take up to an hour.\n\nProviding too little resources to the docker machine can destabilize the docker build process. If you're encountering logs/errors like:\n\n```\n(signal: 9, SIGKILL: kill)\n# OR\nERROR: Service 'tremor' failed to build : The command '/bin/sh -c cargo build --release --all --verbose' returned a non-zero code: 101\n```\n\nIt is likely that your docker resources are starved. Consider increasing your resources ([Windows](https://docs.docker.com/docker-for-windows/#resources)/[Mac](https://docs.docker.com/docker-for-mac/#resources)) before trying again, posting in Discord, or raising an issue.\n\n### Running\n\nTo run `tremor` locally and introspect its docker environment, do the following:\n\n```bash\nmake image\ndocker run tremorproject/tremor:latest\n```\n\nA local shell can be acquired by finding the container id of the running docker container and using that to attach a shell to the image.\n\n```bash\ndocker ps\n```\n\nThis returns:\n\n```text\nCONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS               NAMES\nfa7e3b4cec86        tremorproject/tremor:latest      \"/tremor-runtime.sh\"   43 seconds ago      Up 42 seconds                           gracious_shannon\n```\n\nExecuting a shell on that container will then give you local access:\n\n```bash\ndocker exec -it fa7e3b4cec86 sh\n```\n\n## Building From Source\n\n\u003e :warning: Local builds are not supported and purely at your own risk. For contributing to Tremor please checkout our [Development Quick Start Guide](https://www.tremor.rs/community/development/quick-start)\n\nIf you are not comfortable with managing library packages on your system or don't have experience with, please use the Docker image provided above. \n\nFor local builds, tremor requires rust 2021 (version `1.76` or later), along with all the tools needed to build rust programs. Eg: for CentOS, the packages `gcc`, `make`, `cmake`, `clang`, `openssl`, and `libstdc++` are required. For different distributions or operating systems, please install the packages accordingly.\n**NOTE** AVX2, SSE4.2 or NEON are needed to build [simd-json](https://github.com/simd-lite/simd-json#cpu-target) used by tremor. So if you are building in vm, check which processor instruction are passed to it. Like `lscpu | grep Flags`\nFor a more detailed guide on local builds, please refer to the [tremor development docs](https://www.tremor.rs/community/development/quick-start).\n\n### ARM/aarch64/NEON\n\nTo run and compile with neon use:\n\n```bash\nRUSTCFLAGS=\"-C cpu-target=native\" cargo +nightly build --features neon --all\n```\n\n\n## Configuration\n\nTremor is configured using `.troy` files written in our own [Troy](https://www.tremor.rs/docs/0.12/language/) language.\n\nCustom [Troy](https://www.tremor.rs/docs/0.12/language/) modules can be loaded from any directory pointed to by the environment variable `TREMOR_PATH`.\nDirectory entries need to be separated by a colon `:`.\n\n### Docker\n\nFor use in docker [Troy](https://www.tremor.rs/docs/0.12/language/) files should be mounted to `/etc/tremor/config`.\n\nCustom [Troy](https://www.tremor.rs/docs/0.12/language/) modules and libraries should be mounted to `/usr/local/share/tremor`.\n### Example\n\nThis very simple example will consume lines from stdin and send them to stdout.\n\n```tremor\ndefine flow example\nflow\n  # import some common pre-defined pipeline and connector definitions\n  # to use here and save some typing\n  use tremor::pipelines;\n  use tremor::connectors;\n\n  # create instances of the connectors and pipelines we need\n  create connector console from connectors::console;\n  create pipeline pass from pipelines::passthrough;\n\n  # connect everything to form an event flow\n  connect /connector/console to /pipeline/pass;\n  connect /pipeline/pass to /connector/console;\nend;\n\ndeploy flow example;\n```\n\nRun this example in file `example.troy` with docker:\n\n```console\n$ docker run -i -v\"$PWD:/etc/tremor/config\" tremorproject/tremor:latest\n```\n\nPlease also look at the [demo](demo/configs/tremor/config) for a fully documented example.\n\nFor more involved examples check out our [Recipes](https://www.tremor.rs/docs/0.12/language/common_patterns).\n\n\n## Local Demo\n\n**Note**: Docker should run with at least 4GB of memory!\n\nTo demo run `make demo`, this requires the `tremorproject/tremor` image to exist on your machine.\n\n### Design\n\nThe demo mode logically follows the flow outlined below. It reads the data from data.json.xz, sends it at a fixed rate to the `demo` bucket on Kafka and from there reads it into the tremor container to apply classification and bucketing. Finally, it off-ramps statistics of the data based on those steps.\n\n```text\n╔════════════════════╗   ╔════════════════════╗   ╔════════════════════╗\n║      loadgen       ║   ║       Kafka        ║   ║       tremor       ║\n║ ╔════════════════╗ ║   ║ ┌────────────────┐ ║   ║ ┌────────────────┐ ║\n║ ║ tremor-runtime ║─╬───╬▶│  bucket: demo  │─╬───╬▶│ tremor-runtime │ ║\n║ ╚════════════════╝ ║   ║ └────────────────┘ ║   ║ └────────────────┘ ║\n║          ▲         ║   ╚════════════════════╝   ║          │         ║\n║          │         ║                            ║          │         ║\n║          │         ║                            ║          ▼         ║\n║ ┌────────────────┐ ║                            ║ ┌────────────────┐ ║\n║ │  data.json.xz  │ ║                            ║ │     tremor     │ ║\n║ └────────────────┘ ║                            ║ └────────────────┘ ║\n╚════════════════════╝                            ║          │         ║\n                                                  ║          │         ║\n                                                  ║          ▼         ║\n                                                  ║ ┌────────────────┐ ║\n                                                  ║ │    grouping    │ ║\n                                                  ║ └────────────────┘ ║\n                                                  ║          │         ║\n                                                  ║          │         ║\n                                                  ║          ▼         ║\n                                                  ║ ┌────────────────┐ ║\n                                                  ║ │  stats output  │ ║\n                                                  ║ └────────────────┘ ║\n                                                  ╚════════════════════╝\n```\n\n### Configuration\n\n#### Config file\n\nThe demo configuration can be inspected and changed in the `demo/configs/tremor/config/main.troy` file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftremor-rs%2Ftremor-runtime","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftremor-rs%2Ftremor-runtime","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftremor-rs%2Ftremor-runtime/lists"}