{"id":13499465,"url":"https://github.com/infinyon/fluvio","last_synced_at":"2025-05-13T17:02:38.681Z","repository":{"id":37029658,"uuid":"205473061","full_name":"infinyon/fluvio","owner":"infinyon","description":"🦀 event stream processing for developers to stream and process data in motion to power responsive data intensive applications.","archived":false,"fork":false,"pushed_at":"2025-05-06T15:37:54.000Z","size":35707,"stargazers_count":4916,"open_issues_count":160,"forks_count":514,"subscribers_count":44,"default_branch":"master","last_synced_at":"2025-05-06T16:08:02.430Z","etag":null,"topics":["cloud-native","data-analytics","data-flow","data-integration","data-pipelines","distributed-systems","event-driven-architecture","real-time","rust","serverless","stateful","stream-processing","stream-processing-engine","streaming","streaming-analytics","streaming-data","streaming-data-pipelines","streaming-data-processing","webassembly"],"latest_commit_sha":null,"homepage":"https://www.fluvio.io/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/infinyon.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-08-31T00:11:58.000Z","updated_at":"2025-05-06T14:48:17.000Z","dependencies_parsed_at":"2023-09-21T19:42:59.952Z","dependency_job_id":"f6ffee55-b88d-4866-b0f3-c6e5e718342d","html_url":"https://github.com/infinyon/fluvio","commit_stats":{"total_commits":2124,"total_committers":62,"mean_commits":34.25806451612903,"dds":0.6699623352165724,"last_synced_commit":"3bc85be53046eb15d279f24e870c160a2a8c7892"},"previous_names":[],"tags_count":183,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Ffluvio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Ffluvio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Ffluvio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Ffluvio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/infinyon","download_url":"https://codeload.github.com/infinyon/fluvio/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253990449,"owners_count":21995773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-native","data-analytics","data-flow","data-integration","data-pipelines","distributed-systems","event-driven-architecture","real-time","rust","serverless","stateful","stream-processing","stream-processing-engine","streaming","streaming-analytics","streaming-data","streaming-data-pipelines","streaming-data-processing","webassembly"],"created_at":"2024-07-31T22:00:33.395Z","updated_at":"2025-05-13T17:02:38.624Z","avatar_url":"https://github.com/infinyon.png","language":"Rust","readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eFluvio\u003c/h1\u003e\n  \u003ca href=\"https://fluvio.io\" target=\"_blank\"\u003e\n    \u003cstrong\u003eComposable, Declarative, Stateful Data Streaming System\u003c/strong\u003e\n  \u003c/a\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n\n[![CI Status](https://github.com/infinyon/fluvio/actions/workflows/hourly.yml/badge.svg)](https://github.com/infinyon/fluvio/actions/workflows/hourly.yml)\n  [![CD Status](https://github.com/infinyon/fluvio/workflows/CD_Dev/badge.svg)](https://github.com/infinyon/fluvio/actions/workflows/cd_dev.yaml)\n  [![fluvio Crates.io version](https://img.shields.io/crates/v/fluvio?style=flat)](https://crates.io/crates/fluvio)\n  [![Fluvio Rust documentation](https://docs.rs/fluvio/badge.svg)](https://docs.rs/fluvio)\n  [![Fluvio dependency status](https://deps.rs/repo/github/infinyon/fluvio/status.svg)](https://deps.rs/repo/github/infinyon/fluvio)\n  [![Fluvio Discord](https://img.shields.io/discord/695712741381636168.svg?logo=discord\u0026style=flat)](https://discordapp.com/invite/bBG2dTz)\n\n  \u003cbr\u003e\n\n  [![An animated visual of fluvio distributed streaming runtime](.github/assets/infinyon-fluvio-sdf.gif)](https://fluvio.io)\n\n  \u003cbr\u003e\n\u003c/div\u003e\n\n**Fluvio** is a lean and mean distributed data streaming engine written in Rust. Combined with **Stateful DataFlow** distributed stream processing framework, Fluvio provides a *unified* *composable* *distributed streaming* and *stream processin*g paradigm for developers. It is the foundation of [InfinyOn Cloud](https://infinyon.cloud/).\n\n## Quick Start - Get started with Fluvio and Stateful DataFlow in 5 minutes or less on your system!\n\n### Step 1. Download Fluvio Version Manager:\n\nFluvio is installed via the **Fluvio Version Manager**, shortened to `fvm`.\n\nTo install `fvm`, run the following command:\n\n```bash\ncurl -fsS https://hub.infinyon.cloud/install/install.sh | bash\n```\nAs part of the initial setup, `fvm` will also install the Fluvio CLI available in the stable channel as of the moment of installation.\n\nFluvio is stored in `$HOME/.fluvio`, with the executable binaries stored in `$HOME/.fluvio/bin`.\n\n\u003e For the best compatibliity on Windows, InfinyOn recommends WSL2\n\n### Step 2. Start a cluster:\n\nStart cluster on you local machine with the following command:\n\n```bash\nfluvio cluster start\n```\n\n### Step 3. Install SDF CLI\n\nStateful dataflows are managed via `sdf cli` that we install it using `fvm`.\n\n```bash\nfvm install sdf-beta9\n```\n\n### Step 4. Create the Dataflow file\n\nCreate a dataflow file in the directory `split-sentence` directory:\n\n```bash\nmkdir -p split-sentence-inline\ncd split-sentence-inline\n```\n\nCreate the `dataflow.yaml` and add the following content:\n\n```yaml\napiVersion: 0.5.0\n\nmeta:\n  name: split-sentence-inline\n  version: 0.1.0\n  namespace: example\n\nconfig:\n  converter: raw\n\ntopics:\n  sentence:\n    schema:\n      value:\n        type: string\n        converter: raw\n  words:\n    schema:\n      value:\n        type: string\n        converter: raw\n\nservices:\n  sentence-words:\n    sources:\n      - type: topic\n        id: sentence\n\n    transforms:\n      - operator: flat-map\n        run: |\n          fn sentence_to_words(sentence: String) -\u003e Result\u003cVec\u003cString\u003e\u003e {\n            Ok(sentence.split_whitespace().map(String::from).collect())\n          }\n      - operator: map\n        run: |\n          pub fn augment_count(word: String) -\u003e Result\u003cString\u003e {\n            Ok(format!(\"{}({})\", word, word.chars().count()))\n          }\n\n    sinks:\n      - type: topic\n        id: words\n```\n\n### Step 5. Run the DataFlow\n\nUse sdf command line tool to run the dataflow:\n\n```bash\nsdf run --ui\n```\n\u003e The --ui flag serves the graphical representation of the dataflow on SDF Studio.\n\n### Step 6. Test the DataFlow\n\nProduce sentences to in `sentence` topic:\n```bash\nfluvio produce sentence\n```\nInput some text, for example:\n```bash\nHello world\nHi there\n```\nConsume from `words` to retrieve the result:\n```bash\nfluvio consume words -Bd\n```\nSee the results, for example:\n```bash\nHello(1)\nworld(1)\nHi(1)\nthere(1)\n```\n\n### Step 6. Inspect State\n\nThe dataflow collects runtime metrics that you can inspect in the runtime terminal.\n\nCheck the sentence-to-words counters:\n```bash\nshow state sentence-words/sentence-to-words/metrics\n```\nSee results, for example:\n```bash\n Key    Window  succeeded  failed\n stats  *       2          0\n ```\n Check the augment-count counters:\n ```bash\n show state sentence-words/augment-count/metrics\n ```\n See results, for example:\n ```bash\n Key    Window  succeeded  failed\n stats  *       4          0\n ```\nCongratulations! You've successfully built and run a composable dataflow!\n\nMore examples of Stateful DataFlow are on GitHub - https://github.com/infinyon/stateful-dataflows-examples/.\n\n#### Check Fluvio Core Documentation\nFluvio documentation will provide additional context on how to use the Fluvio clusters, CLI, clients, a development kits.\n- [Fluvio overview](https://www.fluvio.io/docs/fluvio/overview)\n- [Fluvio CLI docs home](https://www.fluvio.io/docs/fluvio/cli/overview)\n- [Fluvio Architecture](https://www.fluvio.io/docs/fluvio/concepts/architecture/overview)\n\n#### Check Stateful DataFlow Documentation\nStateful DataFlow designed to handle complex data processing workflows, allowing for customization and scalability through various programming languages and system primitives.\n\n- [SDF overview](https://www.fluvio.io/sdf/)\n- [SDF Architecture](https://www.fluvio.io/sdf/concepts/architecture)\n\n#### Learn how to build custom connectors\nFluvio can connect to practically any system that you can think of.\n- For first party systems, fluvio clients can integrate with the edge system or application to source data.\n- For third party systems fluvio connectors connect at the protocol level and collects data into fluvio topics.\n\nOut of the box Fluvio has native http, webhook, mqtt, kafka inbound connectors. In terms of outbound connectors out of the box Fluvio supports http, SQL, kafka, and experimental builds of DuckDB, Redis, S3, Graphite etc.\n\nUsing Connector Development Kit, its intuitive to build connectors to any system fast.\n\nCheck out the docs and let us know if you need help building any connector.\n- [Connector docs](https://www.fluvio.io/docs/connectors/overview)\n- [Connector Development Kit (cdk) docs](https://www.fluvio.io/docs/connectors/cdk)\n\n#### Learn how to build custom smart modules\nFluvio applies wasm based stream processing and data transformations. We call these reusable transformation functions smart modules. Reusable Smart modules are built using Smart Module Development Kit and can be distributed using InfinyOn Cloud hub.\n\n- [Smart Modules docs](https://www.fluvio.io/docs/smartmodules/overview)\n- [Smart Modules Development Kit (smdk) docs](https://www.fluvio.io/docs/smartmodules/smdk)\n\n#### Try workflows on InfinyOn Cloud\nInfinyOn Cloud is Fluvio on the cloud as a managed service.\n- [Check InfinyOn Cloud Guides](https://infinyon.com/docs/guides/)\n- [Check out experimental data flows on InfinyOn Labs Repo](https://github.com/infinyon/labs-projects)\n\n### Clients\n- [Fluvio Client API docs home](https://www.fluvio.io/docs/fluvio/apis/overview)\n\n**Language Specifc API docs:**\n- [Rust API docs](https://docs.rs/fluvio/latest/fluvio/)\n- [Python API docs](https://infinyon.github.io/fluvio-client-python/fluvio.html)\n- [Javascript API docs](https://infinyon.github.io/fluvio-client-node/)\n\n\n**Community Maintained:**\n- [Go API docs](https://github.com/avinassh/fluvio-go)\n- [Java API docs](https://github.com/infinyon/fluvio-client-java)\n- [Elixir API docs](https://github.com/viniarck/fluvio-ex)\n\n## Contributing\n\nIf you'd like to contribute to the project, please read our\n[Contributing guide](CONTRIBUTING.md).\n\n## Community\n\nMany fluvio users and developers have made projects to share with the community.\nHere a a few listed below:\n\n### Projects Using Fluvio\n- [Swiftide Project](https://github.com/bosun-ai/swiftide): a Rust native library for building LLM applications\n- [Real Time Stock Charts](https://github.com/KeptCodes/stock-charts): See how Fluvio is used to update real time stock charts\n\n### Community Connectors\n- [Qdrant Connector](https://qdrant.tech/documentation/data-management/fluvio/)\n- [Google Sheets Connector](https://github.com/fluvio-connectors/sheets-connector): Send data from Fluvio to Google Sheets\n- [Elastic Connector](https://github.com/fluvio-connectors/elastic-connector): Send data from Fluvio to Elastic Search\n\n## Community Development Resources\n\nMore projects and utilities are available in the  [Fluvio Community Github Org](https://github.com/fluvio-community/)\n\n- [Hello World Fluvio Connector](https://github.com/fluvio-community/connector-hello-source): Sample Fluvio connector template to build your own connector\n- [Gurubase](https://gurubase.io/g/fluvio): Third-party AI/LLM Docs query\n\n### Contributors are awesome\n\u003ca href=\"https://github.com/infinyon/fluvio/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=infinyon/fluvio\" /\u003e\n\u003c/a\u003e\n\nMade with [contrib.rocks](https://contrib.rocks).\n\n## License\n\nThis project is licensed under the [Apache license](LICENSE).\n","funding_links":[],"categories":["Rust","Libraries","库 Libraries","Table of Contents","Uncategorized","webassembly"],"sub_categories":["Data streaming","数据流 Data streaming","Data Pipeline","Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfinyon%2Ffluvio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfinyon%2Ffluvio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfinyon%2Ffluvio/lists"}