Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/infinyon/fluvio

Lean and mean distributed stream processing system written in rust and web assembly.
https://github.com/infinyon/fluvio

cloud-native data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-data streaming-data-pipelines streaming-data-processing webassembly

Last synced: 8 days ago
JSON representation

Lean and mean distributed stream processing system written in rust and web assembly.

Host: GitHub
URL: https://github.com/infinyon/fluvio
Owner: infinyon
License: apache-2.0
Created: 2019-08-31T00:11:58.000Z (about 5 years ago)
Default Branch: master
Last Pushed: 2023-12-19T07:20:20.000Z (11 months ago)
Last Synced: 2023-12-19T11:48:37.456Z (11 months ago)
Topics: cloud-native, data-flow, data-integration, data-pipelines, distributed-systems, event-driven-architecture, real-time, rust, serverless, stateful, stream-processing, stream-processing-engine, streaming, streaming-data, streaming-data-pipelines, streaming-data-processing, webassembly
Language: Rust
Homepage: https://www.fluvio.io/
Size: 22.9 MB
Stars: 2,284
Watchers: 34
Forks: 166
Open Issues: 142
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE-OF-CONDUCT.md

Awesome Lists containing this project

awesome-rust-cloud-native - infinyon/fluvio - native real-time data streaming platform with in-line computation capabilities (Uncategorized / Uncategorized)
awesome-rust - infinyon/fluvio - Programmable data streaming platform [![CI](https://github.com/infinyon/fluvio/workflows/CI/badge.svg?branch=stable)](https://github.com/infinyon/fluvio/actions) (Libraries / Data streaming)
awesome-repositories - infinyon/fluvio - Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one. (Rust)
awesome-rust-cn - infinyon/fluvio - (库 Libraries / 数据流 Data streaming)
awesome-streaming - fluvio - Real-time programmable data streaming platform with in-line computation capabilities. (Table of Contents / Data Pipeline)
awesome-rust - infinyon/fluvio - Programmable data streaming platform [![CI](https://github.com/infinyon/fluvio/workflows/CI/badge.svg?branch=stable)](https://github.com/infinyon/fluvio/actions) (Libraries / Data streaming)
fucking-awesome-rust - infinyon/fluvio - Programmable data streaming platform [![CI](https://github.com/infinyon/fluvio/workflows/CI/badge.svg?branch=stable)](https://github.com/infinyon/fluvio/actions) (Libraries / Data streaming)
fucking-awesome-rust - infinyon/fluvio - Programmable data streaming platform [![CI](https://github.com/infinyon/fluvio/workflows/CI/badge.svg?branch=stable)](https://github.com/infinyon/fluvio/actions) (Libraries / Data streaming)

README

Fluvio

Composable, Declarative, Stateful Data Streaming System

[![CI Status](https://github.com/infinyon/fluvio/workflows/CI/badge.svg?branch=staging)](https://github.com/infinyon/fluvio/actions/workflows/ci.yml)
[![CD Status](https://github.com/infinyon/fluvio/workflows/CD_Dev/badge.svg)](https://github.com/infinyon/fluvio/actions/workflows/cd_dev.yaml)
[![fluvio Crates.io version](https://img.shields.io/crates/v/fluvio?style=flat)](https://crates.io/crates/fluvio)
[![Fluvio Rust documentation](https://docs.rs/fluvio/badge.svg)](https://docs.rs/fluvio)
[![Fluvio dependency status](https://deps.rs/repo/github/infinyon/fluvio/status.svg)](https://deps.rs/repo/github/infinyon/fluvio)
[![Fluvio Discord](https://img.shields.io/discord/695712741381636168.svg?logo=discord&style=flat)](https://discordapp.com/invite/bBG2dTz)

[![An animated visual of fluvio distributed streaming runtime](.github/assets/fluvio-event-streaming.gif)](https://fluvio.io)

Fluvio is a lightweight high-performance distributed data streaming system written in Rust and Web Assembly.

## Quick Start - Get started with Fluvio in 2 minutes or less!

### Step 1. Download Fluvio Version Manager:

On your terminal run

```bash
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
```

Follow the instructions and copy/paste the path to the `bin` directory to your startup script file.

Fluvio version manager will give you the ability to download different versions of Fluvio:
- Including our read-only edge cluster with built-in compression, caching, and mirroring to never lose data even with extended downtimes.
- Or our Developer Preview of Stateful Streaming which we are building using the web assembly component model to support all web assembly compatible languages.

### Step 2. Start local cluster:

The following command will start a local cluster by default:

```bash
fluvio cluster start
```

### Step 3. Create Topic:

The following command will create a topic called hello-fluvio:

```bash
fluvio topic create hello-fluvio
```

### Step 4. Produce to Topic, Consume From Topic:

Produce data to your topic. Run the command first and then type some messages:

```bash
fluvio produce hello-fluvio
> hello fluvio
Ok!
> test message
Ok!
```

Consume data from the topic, Run the following command in a different terminal:

```bash
fluvio consume hello-fluvio -B -d
```

Just like that! You have a local cluster running.

## Using Pre-Build Fluvio Versions

You may want to prefer other Fluvio versions than the latest stable release. You can do so by specifying the version in the `VERSION` environment variable.

**Install Latest Release (as of `master` branch)**

```bash
$ curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION=latest bash
```

**Install Specific Version**

```bash
$ curl -fsS https://hub.infinyon.cloud/install/install.sh | VERSION=x.y.z bash
```

## Next Steps
Now that you have a cluster running you can try building data flows in different paradigms.

#### Check Fluvio Core Documentation
Fluvio documentation will provide additional context on how to use the Fluvio clusters, CLI, clients, a development kits.
- [Fluvio docs home](https://www.fluvio.io)
- [Fluvio CLI docs home](https://www.fluvio.io/docs/fluvio/cli/overview)
- [Fluvio Architecture](https://www.fluvio.io/docs/fluvio/concepts/architecture/overview)

#### Learn how to build custom connectors
Fluvio can connect to practically any system that you can think of.
- For first party systems, fluvio clients can integrate with the edge system or application to source data.
- For third party systems fluvio connectors connect at the protocol level and collects data into fluvio topics.

Out of the box Fluvio has native http, webhook, mqtt, kafka inbound connectors. In terms of outbound connectors out of the box Fluvio supports SQL, DuckDB, Graphite, experimental builds of Redis, S3 etc.

Using Connector Development Kit, we built our existing connectors in a matter of few days. Check out the docs and let us know if you need help building any connector.
- [Connector docs](https://www.fluvio.io/docs/connectors/overview)
- [Connector Development Kit (cdk) docs](https://www.fluvio.io/docs/connectors/cdk)

#### Learn how to build custom smart modules
Fluvio applies wasm based stream processing and data transformations. We call these reusable transformation functions smart modules. Reusable Smart modules are built using Smart Module Development Kit and can be distributed using InfinyOn Cloud hub.

- [Smart Modules docs](https://www.fluvio.io/docs/smartmodules/overview)
- [Smart Modules Development Kit (smdk) docs](https://www.fluvio.io/docs/smartmodules/smdk)

There are some limitations on the amount of polyglot development interface support. While bindings can be generated for wasm compatible languages, there are quirks in that approach. We have a better solution with Stateful Service Development Kit, which we are implementing using the web assembly component model. In the upcoming releases we will be able to natively support all wasm compatible programming languages.

- Stateful Service Development Kit docs- *Coming Soon* [Request Developer Preview Invite](https://infinyon.com/request/ss-early-access/)

#### Try workflows on InfinyOn Cloud
InfinyOn Cloud is Fluvio on the cloud as a managed service. All new users get $3000 worth of credits to build data flows on InfinyOn Cloud.
- [Check InfinyOn Cloud Guides](https://infinyon.com/docs/guides/)
- [Check out experimental data flows on InfinyOn Labs Repo](https://github.com/infinyon/labs-projects)

### Clients
- [Fluvio Client API docs home](https://www.fluvio.io/docs/fluvio/apis/overview)

**Language Specifc API docs:**
- [Rust API docs](https://docs.rs/fluvio/latest/fluvio/)
- [Python API docs](https://infinyon.github.io/fluvio-client-python/fluvio.html)
- [Javascript API docs](https://infinyon.github.io/fluvio-client-node/)
- [Java API docs](https://infinyon.github.io/fluvio-client-java/com/infinyon/fluvio/package-summary.html)

**Community Maintained:**
- [Go API docs](https://github.com/avinassh/fluvio-go)
- [Java API docs](https://github.com/infinyon/fluvio-client-java)
- [Elixir API docs](https://github.com/viniarck/fluvio-ex)

## Contributing

If you'd like to contribute to the project, please read our
[Contributing guide](CONTRIBUTING.md).

### Contributors are awesome

Made with [contrib.rocks](https://contrib.rocks).

## License

This project is licensed under the [Apache license](LICENSE).