{"id":19028583,"url":"https://github.com/project-flogo/stream","last_synced_at":"2025-04-15T21:17:20.019Z","repository":{"id":33494904,"uuid":"143154806","full_name":"project-flogo/stream","owner":"project-flogo","description":"Elegant stream processing pipeline written entirely in Golang","archived":false,"fork":false,"pushed_at":"2023-06-15T16:34:07.000Z","size":199,"stargazers_count":65,"open_issues_count":7,"forks_count":33,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-15T21:17:08.587Z","etag":null,"topics":["edge","flogo","stream-processing","streaming"],"latest_commit_sha":null,"homepage":"http://flogo.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/project-flogo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-08-01T12:45:12.000Z","updated_at":"2025-01-30T21:59:21.000Z","dependencies_parsed_at":"2024-04-08T02:03:41.293Z","dependency_job_id":"67014762-c56e-45bc-9201-db079c618d7a","html_url":"https://github.com/project-flogo/stream","commit_stats":null,"previous_names":["flogo-oss/stream"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-flogo%2Fstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-flogo%2Fstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-flogo%2Fstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project-flogo%2Fstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/project-flogo","download_url":"https://codeload.github.com/project-flogo/stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249153953,"owners_count":21221330,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["edge","flogo","stream-processing","streaming"],"created_at":"2024-11-08T21:11:38.293Z","updated_at":"2025-04-15T21:17:19.997Z","avatar_url":"https://github.com/project-flogo.png","language":"Go","funding_links":[],"categories":["Go","_Table of Contents_"],"sub_categories":["On-Prem"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src =\"https://raw.githubusercontent.com/TIBCOSoftware/flogo/master/images/flogo-ecosystem_streams.png\" /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\" \u003e\n  \u003cb\u003eStreams is a pipeline based, stream processing action for the Project Flogo Ecosystem\u003c/b\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://travis-ci.org/project-flogo/stream.svg?branch=master\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/dependencies-up%20to%20date-green.svg\"/\u003e\n  \u003cimg src=\"https://img.shields.io/badge/license-BSD%20style-blue.svg\"/\u003e\n  \u003ca href=\"https://gitter.im/project-flogo/Lobby?utm_source=share-link\u0026utm_medium=link\u0026utm_campaign=share-link\"\u003e\u003cimg src=\"https://badges.gitter.im/Join%20Chat.svg\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n# Flogo Stream\n\nEdge devices have the potential for producing millions or even billions of events at rapid intervals, often times the events on their own are meaningless, hence the need to provide basic streaming operations against the slew of events.\n\nA native streaming action as part of the Project Flogo Ecosystem accomplishes the following primary objectives:\n\n- Enables apps to implement basic streaming constructs in a simple pipeline fashion\n- Provides non-persistent state for streaming operations\n  - Streams are persisted in memory until the end of the pipeline\n- Serves as a pre-process pipeline for raw data to perform basic mathematical and logical operations. Ideal for feeding ML models\n\nSome of the key highlights include:\n\n😀 **Simple pipeline** construct enables a clean, easy way of dealing with streams of data\u003cbr/\u003e\n⏳ **Stream aggregation** across streams using time or event tumbling \u0026 sliding windows\u003cbr/\u003e\n🙌 **Join streams** from multiple event sources\u003cbr/\u003e\n🌪 **Filter** out the noise with stream filtering capabilities\u003cbr/\u003e\n\n## Getting Started\n\nWe’ve made building powerful streaming pipelines as easy as possible. Develop your pipelines using:\n\n- A simple, clean JSON-based DSL\n- Golang API\n\nSee the sample below of an aggregation pipeline (for brevity, the triggers and metadata of the resource has been omitted). Also don’t forget to check out the [examples](https://github.com/project-flogo/stream/tree/master/examples) in the repo.\n\n```json\n  \"stages\": [\n    {\n      \"ref\": \"github.com/project-flogo/stream/activity/aggregate\",\n      \"settings\": {\n        \"function\": \"sum\",\n        \"windowType\": \"timeTumbling\",\n        \"windowSize\": \"5000\"\n      },\n      \"input\": {\n        \"value\": \"=$.input\"\n      }\n    },\n    {\n      \"ref\": \"github.com/project-flogo/contrib/activity/log\",\n      \"input\": {\n        \"message\": \"=$.result\"\n      }\n    }\n  ]\n```\n\n## Try out the example\n\nFirstly you should install the install the [Flogo CLI](https://github.com/project-flogo/cli).\n \nNext you should download our aggregation example [agg-flogo.json](https://github.com/project-flogo/stream/blob/master/examples/agg-flogo.json).\n\nWe'll create a our application using the example file, we'll call it myApp\n\n```bash\n$ flogo create -f agg-flogo.json myApp\n```\n\nNow, build it...\n\n```bash\n$ cd myApp/\n$ flogo build\n```\n\n## Activities\n\nFlogo Stream also provides some activities to assist in stream processing.\n\n* [Aggregate](activity/aggregate/README.md) : This activity allows you to aggregate data and calculate an average or sliding average.\n* [Filter](activity/filter/README.md) : This activity allows you to filter out data in a streaming pipeline.\n\n## License \nFlogo source code in [this](https://github.com/project-flogo/stream) repository is under a BSD-style license, refer to [LICENSE](https://github.com/project-flogo/strem/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject-flogo%2Fstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproject-flogo%2Fstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject-flogo%2Fstream/lists"}