{"id":13406027,"url":"https://github.com/streamdal/streamdal","last_synced_at":"2025-05-14T11:08:41.625Z","repository":{"id":186203578,"uuid":"674687849","full_name":"streamdal/streamdal","owner":"streamdal","description":"Code-Native Data Privacy","archived":false,"fork":false,"pushed_at":"2024-11-15T15:30:27.000Z","size":306486,"stargazers_count":602,"open_issues_count":10,"forks_count":15,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-11T23:17:26.400Z","etag":null,"topics":["astrojs","data-contracts","deno","docker","event-driven","go","javascript","message-queues","nodejs","observability","python","reactjs","rust","streaming-data","tail-f","wasi","wasm"],"latest_commit_sha":null,"homepage":"https://www.streamdal.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamdal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-04T14:26:21.000Z","updated_at":"2025-05-06T14:47:24.000Z","dependencies_parsed_at":"2023-09-23T11:18:08.544Z","dependency_job_id":"f0cd412b-7d8e-435a-9f7a-49a49ddc21df","html_url":"https://github.com/streamdal/streamdal","commit_stats":{"total_commits":3608,"total_committers":24,"mean_commits":"150.33333333333334","dds":0.6643569844789357,"last_synced_commit":"b290f47f969e888134ca8d11c0d094a4ef6e1120"},"previous_names":["streamdal/snitch","streamdal/streamdal"],"tags_count":411,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fstreamdal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fstreamdal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fstreamdal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fstreamdal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamdal","download_url":"https://codeload.github.com/streamdal/streamdal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254129481,"owners_count":22019628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["astrojs","data-contracts","deno","docker","event-driven","go","javascript","message-queues","nodejs","observability","python","reactjs","rust","streaming-data","tail-f","wasi","wasm"],"created_at":"2024-07-30T19:02:18.941Z","updated_at":"2025-05-14T11:08:41.573Z","avatar_url":"https://github.com/streamdal.png","language":"TypeScript","readme":"\u003cdiv align=\"center\"\u003e\n   \n\u003cimg src=\"./assets/img/streamdal-logo-dark.png#gh-dark-mode-only\"\u003e\u003cimg src=\"./assets/img/streamdal-logo-light.png#gh-light-mode-only\"\u003e  \n\n[![GitHub](https://img.shields.io/github/license/streamdal/streamdal)](https://github.com/streamdal/streamdal)\n[![Discord](https://img.shields.io/badge/Community-Discord-4c57e8.svg)](https://discord.gg/streamdal)\n\n\u003c/div\u003e  \n\nStreamdal is an open-source _**'Code-Native Data Privacy'**_ solution for \nrunning data tasks _directly_ in your application code.\n\n\u003csub\u003eThink of it as a \"workflow engine\" or a \"pre/post data processor\" that\nis executed client-side via WebAssembly in your application code.\u003c/sub\u003e\n\nIt is at least _10x faster_, _10x cheaper_ and _10x easier_ to operate than\ntraditional data pipelines.\n\n----\n\n\u003csub\u003e_This is what it looks like:_\u003c/sub\u003e\n\u003cimg src=\"assets/img/console.png\"\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[Manifesto](#manifesto) •\n[Benefits](#benefits) •\n[Demo](#live-demo) •\n[Getting Started](#getting-started) •\n[How Does It Work?](#how-does-it-work) •\n[Community](#community) •\n[Resources](#resources)\n\n\u003c/div\u003e\n\n# Benefits\n\nThere are major benefits to running pipelines directly within your app:\n\n* **Eliminates** the need for a separate data pipeline infrastructure\n   * Pipelines execute from within your app, using existing compute that your\n     app is already using\n* **Eliminates** the need for a separate data pipeline team\n   * No more waiting for the data pipeline team to make pipeline changes\n* Is **ridiculously** fast\n   * Streamdal uses Wasm to execute pipelines at near-native speeds\n* Is **actually** real-time\n   * Not \"near real-time\" or \"max-30-seconds-real-time\" - but _actually_\n     real-time - data is processed as soon as your app reads or writes data\n* [And **many** other reasons](https://docs.streamdal.com/en/getting-started/use-cases/)\n\n# Live Demo\n\nYou don't have to install the [server](https://github.com/streamdal/streamdal/tree/main/apps/server), the [console](https://github.com/streamdal/streamdal/tree/main/apps/console) or [instrument](https://docs.streamdal.com/en/guides/instrumentation/)\nany of your apps to see Streamdal in action. We've got a live demo :)\n\n### [DEMO.STREAMDAL.COM](https://demo.streamdal.com)\n\nWhile the demo is read-only, it will give you a good idea about the capabilities\nof Streamdal.\n\nThe demo showcases **real-time**:\n\n1. PII detection \u0026 masking\n1. Schema inference/detection \u0026 validation\n1. Data extraction \u0026 truncation\n1. Stream debugging \u0026 monitoring\n1. Data source and destination visualization \u0026 exploration\n\n\u003csub\u003e_You can read more about how this is achieved in the [\"how does it work?\"](https://docs.streamdal.com/en/getting-started/how-streamdal-works/) docs._\u003c/sub\u003e\n\n# Getting Started\n\nGetting started consists of two steps:\n\n1. **Installing** the server, console and their dependencies\n2. **Instrumenting** your code with one of our [SDKs](https://docs.streamdal.com/en/core-components/sdk/)\n\n### Install\n\nThe _easiest_ way to get Streamdal running is via `curl | bash`:\n\n```\ncurl -sSL https://sh.streamdal.com | bash\n```\n\n1. The install script will verify that you have `git`, `docker` and `docker-compose` installed\n2. The install script will clone this repo to `~/streamdal`\n3. The install script will bring up all components via `docker-compose`\n\nOnce done:\n\n🎉 **Open`http://localhost:8080` in your browser!** 🎉\n\nYou should be presented with a _beautiful_ (but empty) UI! To populate it,\nwe will need to instrument some code. _Onto the next section!_\n\n\u003csub\u003e_For alternative installation methods, check the [docs](./docs) dir._\u003c/sub\u003e\n\n### Instrument\n\nOnce you have installed the server and console, you can begin instrumenting your\ncode.\n\n_Instrumentation_ involves calling our SDKs `.Process()` method either **after**\nyour application reads data from somewhere _OR_ **before** your app is about to\nwrite data somewhere.\n\n#### SDKs\n\n- [Go](https://github.com/streamdal/streamdal/tree/main/sdks/go) ━ [Example](https://github.com/streamdal/streamdal/tree/main/sdks/go#example-usage)\n- [Python](https://github.com/streamdal/streamdal/tree/main/sdks/python) ━ [Example](https://github.com/streamdal/streamdal/tree/main/sdks/python#example-usage)\n- [Node](https://github.com/streamdal/streamdal/tree/main/sdks/node) ━ [Example](https://github.com/streamdal/streamdal/tree/main/sdks/node#getting-started)\n\n\u003e To see an example of a _complete_ instrumentation, take a look at one of our\n\u003e [shim examples](https://docs.streamdal.com/en/core-components/libraries-shims/) \n\u003e or the [Go demo client](./apps/server/test-utils/demo-client/) that is bundled\n\u003e with the [./apps/server](./apps/server/test-utils/demo-client/).\n\n# How Does It Work?\n\nStreamdal consists of **three** main components:\n\n- **[Server](https://github.com/streamdal/streamdal/tree/main/apps/server)**\n- **[Console](https://github.com/streamdal/streamdal/tree/main/apps/console)**\n- **[SDKs](https://docs.streamdal.com/en/core-components/sdk/)**\n\nThe basic flow is that you [install](#getting-started) the server and console and wrap any \nreads or writes in your app with one of our SDKs. Once that's done, you will be \nable to see the app and the data your app is reading or writing in the \n[console](https://github.com/streamdal/console) (or use the [CLI](https://github.com/streamdal/cli)).\n\nYou will also be able to enforce rules on your data (such as _\"this should be \nvalid JSON\"_, _\"message should contain a field called `foo`\"_, _\"strip all email\naddresses\"_ and so on).\n\n\u003e [!IMPORTANT]\n\u003e For a more in-depth explanation of the flow and the various components, visit \n\u003e our [docs](https://docs.streamdal.com/en/getting-started/how-streamdal-works/).\n\n# Repo Layout\n\nThis repo is a [monorepo](https://en.wikipedia.org/wiki/Monorepo) that has the following layout and usage:\n\n```plaintext\n# ┌── assets                 \u003c--- Static assets \n# │   ├── img\n# │   └── ...\n# ├── apps\n# │   ├── cli                \u003c--- CLI UI \n# │   ├── console            \u003c--- Web-based UI\n# │   ├── docs               \u003c--- https://docs.streamdal.com \n# │   ├── server             \u003c--- Server component\n# │   └── ...\n# ├── docs\n# │   ├── install\n# │\t│    ├── bare-metal\n# │\t│    ├── docker\n# │\t│    └── ...\n# |   ├── instrument\n# |   └── ...\n# ├── libs\n# │   ├── protos             \u003c--- Common protobuf schemas\n# │   ├── wasm               \u003c--- Wasm funcs used in pipeline steps\n# │   ├── wasm-detective     \u003c--- Wasm lib used for data parsing and validation \n# │   ├── wasm-transformer   \u003c--- Wasm lib used for data transformation\n# │   └── ...\n# ├── scripts                   \n# │   ├── install\n# │   │\t  └── install.sh     \u003c--- Install script for installing Streamdal\n# │   └── ...\n# ├── LICENSE\n# ├── Makefile               \u003c--- Makefile with common tasks; run `make help` for more info\n# └── README.md\n```\n\n### Tested Versions\n\nThe Streamdal platform is made up of several components that must all be at specific versions to function properly together.\n\nWhen running in production, verify versions of the components against this list:\n\n| Date       | Server | Console | Protos | Wasm   | Helm   |\n|------------|--------|---------|--------|--------|--------|\n| 05.08.2024 | 0.0.38 | 0.0.69  | 0.1.56 | 0.1.38 | 0.1.61 |\n\n# Community\n\nWe're building Streamdal in the open and we'd love for you to join us!\n\nJoin our [Discord](https://discord.gg/streamdal)!\n\n### Getting Help\n\nStuck? Something not working right? Have questions?\n\n* First and _easiest_ way to get help is to join our [Discord](https://discord.gg/streamdal)\n* If you're not in the mood to chat - there's [docs](https://docs.streamdal.com)\n* If all else fails, [open an issue](https://github.com/streamdal/streamdal/issues/new)!\n\n### Roadmap\n\nYou have control over what we're building - [our roadmap is 100% public](https://github.com/orgs/streamdal/projects/1)!\n\nFeel free to stop by to discuss features, suggest new ones or just keep an eye\non what's in the pipeline.\n\n### Contributing\n\nWe :heart: contributions! But... before you craft a beautiful PR, please read\nthrough our [contributing docs](https://docs.streamdal.com/en/resources-support/contributing/).\n\n### License\n\nThis project is licensed under the `Apache-2.0` license. \n\nSee the [LICENSE](LICENSE) file for more info.\n","funding_links":[],"categories":["TypeScript","Rust","Real-time Data"],"sub_categories":["Android Things"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Fstreamdal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamdal%2Fstreamdal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Fstreamdal/lists"}