{"id":13482256,"url":"https://github.com/lmco/streamflow","last_synced_at":"2025-03-27T12:32:45.388Z","repository":{"id":21535565,"uuid":"24855056","full_name":"lmco/streamflow","owner":"lmco","description":"StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows.","archived":false,"fork":false,"pushed_at":"2023-12-17T03:25:09.000Z","size":4904,"stargazers_count":250,"open_issues_count":36,"forks_count":69,"subscribers_count":44,"default_branch":"develop","last_synced_at":"2024-08-01T17:32:04.421Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/lmco/streamflow/wiki","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2014-10-06T16:57:01.000Z","updated_at":"2024-04-26T21:11:46.000Z","dependencies_parsed_at":"2022-08-21T17:01:04.972Z","dependency_job_id":"5f291cca-e455-435b-99cb-29e1429bb902","html_url":"https://github.com/lmco/streamflow","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmco%2Fstreamflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmco%2Fstreamflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmco%2Fstreamflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmco%2Fstreamflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmco","download_url":"https://codeload.github.com/lmco/streamflow/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222252119,"owners_count":16955969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:00.343Z","updated_at":"2024-10-30T15:31:56.022Z","avatar_url":"https://github.com/lmco.png","language":"Java","funding_links":[],"categories":["Table of Contents","Building","大数据"],"sub_categories":["Toolkit","Workflows"],"readme":"# StreamFlow\u0026trade; \n\n[![Build Status](https://travis-ci.org/lmco/streamflow.svg?branch=develop)](https://travis-ci.org/lmco/streamflow)\n\n[![Join the chat at https://gitter.im/lmco/streamflow](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/lmco/streamflow?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n## Overview\n\nStreamFlow\u0026trade; is a stream processing tool designed to help build and monitor processing \nworkflows.  The ultimate goal of StreamFlow is to make working with stream processing frameworks \nsuch as Apache Storm easier, faster, and with \"enterprise\" like management functionality.  \nStreamFlow also provides a mechanism for non-developers such as data scientists, analysts, or \noperational users to rapidly build scalable data flows and analytics.\n\n![Sample topology](https://raw.githubusercontent.com/wiki/lmco/streamflow/sample-topology.png)\n\nStreamFlow provides the following capabilities: \n\n1. A **responsive web interface** for building and monitoring Storm topologies.\n2. An interactive **drag and drop topology builder** for authoring new topologies\n3. A **dashboard** for monitoring the status and performance of topologies as well as viewing aggregated topology logs.\n4. A specialized **topology engine** which solves some Storm complexities such as ClassLoader isolation and serialization and provides a mechanism for dependency injection.\n5. A **modular framework** for publishing and organizing new capabilities in the form of Spouts and Bolts.\n\n\n## How it works\n\nThe following is a simple depiction of the StreamFlow stack. The web interface is built using open \nsource web frameworks and is backed by a series of reusable web services. StreamFlow is capable of \nauthoring and managing topologies dynamically using a series of reusable Frameworks.  These \nFrameworks are simply JAR files comprised of standard Storm Spouts and Bolts with a metadata \nconfiguration file which exposes the frameworks.  StreamFlow utilizes a custom topology driver \nwhich is used to bootstrap and execute a topology along with StreamFlow specific configuration logic.\n\n\n## Concepts\n\nThe following is a description of some core StreamFlow concepts and terminology.\n\n#### Component\nComponents represent business logic modules which are draggable in the StreamFlow UI.  Examples of Components include Storm Spouts and Storm Bolts.\n\n#### Framework\nA grouping of related *Components* and their associated metadata. Ideally elements of a framework \nshould all be compatible when wired together on a topology as they share the same protocol. \nFrameworks might be organized around a set of technologies or domains. An analogy would be a \nJava Library or Objective C Framework. Topologies have frameworks as dependencies.\n\n#### Resource\nA resource is an object used by spouts/bolts in order to externalize common state. For example, an \nobject which represents a technical asset in the environment/cluster such as a database or Kafka \nqueue. Alternatively, a resource might provide an uploaded file or container of global state. \nResources should be used to encapsulate functionality outside of a bolt/spout if that information \nis used in several places in a topology or within multiple topologies. Resources also provide a \nuseful mechanism for injecting parameters, connections, or state into a bolt/spout making the spout \nor bolt simpler, easier to write, and more testable.\n\n#### Serialization\nSeriaizations allow for the definition of custorm serializers/deserializers.  Specifically these serializations should be specified in the Kryo format to properly integrate with Storm. \n\n#### Topology\nTopologies in Storm define the processing logic and link between nodes to describe the data flow.  StreamFlow utilizes registered components to allow users to dynamically build topologies in a drag and drop interface.  This allows topologies to be built using existing components without requiring additional code.\n\n\n## Find out more\n\nThe StreamFlow [Wiki](https://github.com/lmco/streamflow/wiki) is the best place to go to learn more \nabout the StreamFlow architecture and how to install and configure a StreamFlow server in your \nenvironment.\n\n[https://github.com/lmco/streamflow/wiki](https://github.com/lmco/streamflow/wiki);\n\nHere are some quick links to help get you started with StreamFlow:\n\n* [Getting Started with StreamFlow](https://github.com/lmco/streamflow/wiki/Getting-Started)\n* [Installing and Running StreamFlow](https://github.com/lmco/streamflow/wiki/Getting-Started#startup)\n* [Configuring StreamFlow](https://github.com/lmco/streamflow/wiki/Configuration)\n* [How Does StreamFlow Work?](https://github.com/lmco/streamflow/wiki/How-It-Works)\n* [Developing StreamFlow Frameworks](https://github.com/lmco/streamflow/wiki/Developing-Frameworks)\n\n\n## Questions or need help?\n\nIf you have any questions or issues please feel free to contact the development team using one of the following methods.\n\n* [Google Group](https://groups.google.com/forum/#!forum/streamflow-user)\n* [GitHub Issues](https://github.com/lmco/streamflow/issues)\n\n\n## License\n\nStreamFlow is copyright 2014 Lockheed Martin Corporation. \n\nLicensed under the **[Apache License, Version 2.0] [license]** (the \"License\");\nyou may not use this software except in compliance with the License.\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\nThis product incorporates open source software components covered by the terms \nof third party license agreements contained in the /Licenses \nfolder of this project.\n\n## Documentation Version\n\n*Last Updated: 1/7/2015*\n\n\n[license]: http://www.apache.org/licenses/LICENSE-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmco%2Fstreamflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmco%2Fstreamflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmco%2Fstreamflow/lists"}