{"id":13671758,"url":"https://github.com/JasonThon/lightflus","last_synced_at":"2025-04-27T18:31:40.822Z","repository":{"id":63307027,"uuid":"471603926","full_name":"JasonThon/lightflus","owner":"JasonThon","description":"A Lightweight, Cloud-Native Stateful Distributed Dataflow Engine","archived":false,"fork":false,"pushed_at":"2023-04-09T05:57:12.000Z","size":1948,"stargazers_count":98,"open_issues_count":3,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-11T09:44:06.827Z","etag":null,"topics":["dataflow","deno","distributed-computing","rust","streaming","typescript"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JasonThon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-19T05:47:15.000Z","updated_at":"2024-10-21T16:09:43.000Z","dependencies_parsed_at":"2024-11-11T13:15:29.350Z","dependency_job_id":null,"html_url":"https://github.com/JasonThon/lightflus","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasonThon%2Flightflus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasonThon%2Flightflus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasonThon%2Flightflus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasonThon%2Flightflus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JasonThon","download_url":"https://codeload.github.com/JasonThon/lightflus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251187271,"owners_count":21549613,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataflow","deno","distributed-computing","rust","streaming","typescript"],"created_at":"2024-08-02T09:01:17.908Z","updated_at":"2025-04-27T18:31:38.748Z","avatar_url":"https://github.com/JasonThon.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eLightflus\u003c/h1\u003e\n  \u003cp\u003e\n    \u003cstrong\u003eLightflus is a cloud-native distributed stateful dataflow engine. \u003c/strong\u003e\n  \u003c/p\u003e\n  \u003cp\u003e\n\n[![CI](https://github.com/Lady-Summer/lightflus-runtime/actions/workflows/workflow.yml/badge.svg)](https://github.com/Lady-Summer/lightflus-runtime/actions/workflows/workflow.yml) [![Join the chat at https://gitter.im/lightflus/community](https://badges.gitter.im/lightflus/community.svg)](https://gitter.im/lightflus/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n[![codecov](https://codecov.io/gh/JasonThon/lightflus/branch/master/graph/badge.svg?token=7Y1MMDWNG5)](https://codecov.io/gh/JasonThon/lightflus)\n\u003c/p\u003e\n\u003c/div\u003e\n\n**Important Notes**\n\n1. Lightflus has released the **demo version**. Welcome to try it! \n\n2. Your feedback is very important and we will take it very seriously!\n\n\n## Design Philosophy\n\n### Target Users\n\nLightflus is **designed for most developer teams even no one is familiar with streaming computation**. Any of your team member can write a dataflow task and deploy it on production. Lightflus can connect with any event source (Kafka, MQTT, etc) in your cloud infra and emit the output result into the storage sink (mysql, redis, etc) which is processed by user-defined Dataflow. \n\n\n### Typescript API + Rust Runtime\n\nLightflus is powered by [Deno](https://github.com/denoland/deno) and implemented in Rust which can ensure memory safe and real-time performance. We embed `v8` engine into Lightflus engine with minimum dependencies makes it light and fast; With the help of `Deno`, you can **run `Typescript` user-defined functions or `WebAssembly` encoded bytes code (for better performance) in Lightflus with stream-like API**; \n\n### References\n\nLightflus mainly refers to Google's Paper [The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing](https://research.google/pubs/pub43864/) and [Streaming System](https://www.oreilly.com/library/view/streaming-systems/9781491983867/). Some other papers in the field of streaming system are also our important source of references. \n\n\n## Document\nYou can read the [document](https://humorous-bream-e48.notion.site/Lightflus-Document-217eedc73610413ba2a4f0c374d66c77) for more details about Lightflus;\n\n\n## Community\n\nYou can join [Gitter](https://gitter.im/lightflus/community) community!\n\n\n## Set Up Lightflus\n\n### Start from Source\n\n```bash\n$ cargo run --manifest-path src/taskmanager/Cargo.toml\n$ cargo run --manifest-path src/coordinator/Cargo.toml\n$ cargo run --manifest-path src/apiserver/Cargo.toml\n```\n\n### Start by Docker (**Recommended**)\n\n```bash\n$ docker-compose up\n```\n\n## Try Lightflus\n\n### Preparation\n\n1. Install Node.JS environment\n2. Use WebStorm / VS Code to create a new Node.JS project\n3. intialize typescript project\n   1. install typescript dependency: `npm install typescript`\n   2. initialize `tsconfig.json`: \n    ```bash \n    yarn tsc -p .\n    ```\n4. install `lightflus-api` dependency: \n   ```bash \n   npm i lightflus-api\n   ```\n\n### Deploy Your first Lightflus Task\nWe use `word count` as the example to show you how to deploy a Lightflus dataflow task\n\n1. Modify `tsconfig.json`\n\nWe recommand you to set up properties in `tsconfig.json` file like below:\n\n```json\n{\n  \"compilerOptions\": {\n    \"module\": \"commonjs\",\n    \"target\": \"es2016\",\n    \"sourceMap\": true,\n    \"baseUrl\": \"./\",\n    \"incremental\": true,\n    \"skipLibCheck\": true,\n    \"strictNullChecks\": false,\n    \"forceConsistentCasingInFileNames\": false,\n    \"strictPropertyInitialization\": false,\n    \"esModuleInterop\": true,\n    \"moduleResolution\": \"node\"\n  }\n}\n\n```\n\n2. Implement Word Count\n\n```typescript\n// wordCount example\n\nimport {context} from \"lightflus-api/src/stream/context\";\nimport {kafka, redis} from \"lightflus-api/src/connectors/connectors\";\nimport ExecutionContext = context.ExecutionContext;\nimport Kafka = kafka.Kafka;\nimport Redis = redis.Redis;\n\nasync function wordCount(ctx: ExecutionContext) {\n    // fetch string stream from kafka\n    let source = Kafka\n        .builder()\n        .brokers([\"kafka:9092\"])\n        // topic\n        .topic(\"topic_2\")\n        // groupId\n        .group(\"word_count\")\n        // deserialization type\n        .build\u003cstring\u003e(undefined, typeof \"\");\n\n    // It will persist the counting values in Redis\n    let sink = Redis.new\u003c{ t0: number, t1: string }\u003e()\n        .host(\"redis\")\n        .keyExtractor((v) =\u003e v.t1)\n        .valueExtractor((v) =\u003e v.t0.toString());\n\n    // create a Dataflow\n    let stream = source.createFlow(ctx);\n\n    // We design the Dataflow\n    await stream.flatMap(value =\u003e value.split(\" \").map(v =\u003e {\n        return {t0: 1, t1: v};\n    }))\n        .keyBy(v =\u003e v.t1)\n        .reduce((v1, v2) =\u003e {\n            return {t1: v1.t1, t0: v1.t0 + v2.t0};\n        })\n        // write the results into Redis sink\n        .sink(sink)\n        // Then execute\n        .execute();\n}\n\nwordCount(ExecutionContext.new(\"wordCount\", \"default\")).then();\n```\n\n3. Compile typescript codes\n\n```bash\n$ yarn tsc -p .\n```\n\n4. Set environment variables\n\n```bash\n$ export LIGHTFLUS_ENDPOINT=localhost:8080\n```\n\n5. Run Javascript code after compilation\n\n```bash\n$ node wordCount.js\n```\n\n### Make the Dataflow Run!\n\nYou can send message to Kafka\n\n```text\nhello hello hello world world world\n```\n\nAnd you can get values in Redis\n\n```bash\nredis\u003e GET hello\n\"3\"\n\nredis\u003e GET world\n\"3\"\n```\n\n# Contribution\nPlease read [CONTRIBUTING.md](CONTRIBUTING.md) document\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJasonThon%2Flightflus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJasonThon%2Flightflus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJasonThon%2Flightflus/lists"}