{"id":13481960,"url":"https://github.com/apache/streampipes","last_synced_at":"2025-12-11T20:46:23.415Z","repository":{"id":38810891,"uuid":"130603282","full_name":"apache/streampipes","owner":"apache","description":"Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.","archived":false,"fork":false,"pushed_at":"2025-05-09T12:51:40.000Z","size":326131,"stargazers_count":642,"open_issues_count":158,"forks_count":196,"subscribers_count":25,"default_branch":"dev","last_synced_at":"2025-05-10T17:16:46.387Z","etag":null,"topics":["analytics","dashboard","edge","iiot","iot","iot-analytics","iot-platform","microservices","opcua","s7","self-service","stream-processing"],"latest_commit_sha":null,"homepage":"https://streampipes.apache.org","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-04-22T20:06:55.000Z","updated_at":"2025-05-09T12:51:42.000Z","dependencies_parsed_at":"2024-02-19T13:04:04.056Z","dependency_job_id":"8ab2a0aa-1568-4886-b3e1-6cbbb9986dd2","html_url":"https://github.com/apache/streampipes","commit_stats":{"total_commits":9843,"total_committers":113,"mean_commits":87.10619469026548,"dds":0.6229807985370314,"last_synced_commit":"878e68d697dc0dd6f5c3f8478370372a81e6b074"},"previous_names":["apache/incubator-streampipes","streampipes/streampipes"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fstreampipes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fstreampipes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fstreampipes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fstreampipes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/streampipes/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253878831,"owners_count":21977882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","dashboard","edge","iiot","iot","iot-analytics","iot-platform","microservices","opcua","s7","self-service","stream-processing"],"created_at":"2024-07-31T17:00:57.774Z","updated_at":"2025-12-11T20:46:23.356Z","avatar_url":"https://github.com/apache.png","language":"Java","readme":"\u003c!--\n  ~ Licensed to the Apache Software Foundation (ASF) under one or more\n  ~ contributor license agreements.  See the NOTICE file distributed with\n  ~ this work for additional information regarding copyright ownership.\n  ~ The ASF licenses this file to You under the Apache License, Version 2.0\n  ~ (the \"License\"); you may not use this file except in compliance with\n  ~ the License.  You may obtain a copy of the License at\n  ~\n  ~    http://www.apache.org/licenses/LICENSE-2.0\n  ~\n  ~ Unless required by applicable law or agreed to in writing, software\n  ~ distributed under the License is distributed on an \"AS IS\" BASIS,\n  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n  ~ See the License for the specific language governing permissions and\n  ~ limitations under the License.\n  ~\n  --\u003e\n\n[![Github Actions](https://img.shields.io/github/actions/workflow/status/apache/streampipes/build.yml)](https://github.com/apache/streampipes/actions/)\n[![Docker pulls](https://img.shields.io/docker/pulls/apachestreampipes/backend.svg)](https://hub.docker.com/r/apachestreampipes/backend/)\n![](https://img.shields.io/badge/java--version-17-blue.svg)\n[![Maven central](https://img.shields.io/maven-central/v/org.apache.streampipes/streampipes-service-core.svg)](https://img.shields.io/maven-central/v/org.apache.streampipes/streampipes-service-core.svg)\n[![License](https://img.shields.io/github/license/apache/streampipes.svg)](http://www.apache.org/licenses/LICENSE-2.0)\n[![Last commit](https://img.shields.io/github/last-commit/apache/streampipes.svg)]()\n[![Apache StreamPipes](https://img.shields.io/endpoint?url=https://dashboard.cypress.io/badge/detailed/q1jdu2\u0026style=flat\u0026logo=cypress)](https://dashboard.cypress.io/projects/q1jdu2/runs)\n[![Contributors](https://img.shields.io/github/contributors/apache/streampipes)](https://github.com/apache/streampipes/graphs/contributors)\n![GitHub commit activity](https://img.shields.io/github/commit-activity/y/apache/streampipes)\n[![GitHub issues by-label](https://img.shields.io/github/issues/apache/streampipes/good%20first%20issue)](https://github.com/apache/streampipes/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)\n\u003cbr\u003e\n[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white)](https://linkedin.com/company/apache-streampipes)\n[![Twitter](https://img.shields.io/badge/Twitter-1DA1F2?style=for-the-badge\u0026logo=twitter\u0026logoColor=white)](https://twitter.com/StreamPipes)\n\n\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n   \u003cimg src=\"https://streampipes.apache.org/img/sp-logo-color.png\"\n   alt=\"StreamPipes Logo\" title=\"Apache StreamPipes Logo\" width=\"50%\"/\u003e\n  \u003cbr\u003e\n\u003c/h1\u003e\n\u003ch3 align=\"center\"\u003eSelf-Service Data Analytics for the (Industrial) IoT\u003c/h3\u003e\n\u003ch4 align=\"center\"\u003eStreamPipes is a self-service (Industrial) IoT toolbox to enable non-technical users to connect\n, analyze and explore IoT data streams. \u003c/h4\u003e\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/apache/streampipes/dev/images/streampipes-overview.png\" alt=\"StreamPipes Overview\"/\u003e\n\u003c/p\u003e\n\n\n***\n\n## Table of contents\n\n  * [About Apache StreamPipes](#about-apache-streampipes)\n  * [User interface](#userinterface)\n  * [Installation](#installation)\n  * [Documentation](#documentation)\n  * [Building StreamPipes](#building-streampipes)\n  * [Pipeline Elements](#pipeline-elements)\n  * [Extending StreamPipes](#extending-streampipes)\n  * [Bugs and Feature Requests](#bugs-and-feature-requests)\n  * [Get help](#get-help)\n  * [Contribute](#contribute)\n  * [Feedback](#feedback)\n  * [License](#license)\n\n***\n\n## About Apache StreamPipes\n\nApache StreamPipes makes industrial data analytics easy!\n\nStreamPipes is an end-to-end toolbox for the industrial IoT.\nIt comes with a rich graphical user interface targeted at non-technical users and provides the following features:\n\n\n* Quickly connect \u003e20 industrial protocols such as OPC-UA, PLCs, MQTT, REST, Pulsar, Kafka and others.\n* Create data harmonization and analytics pipelines using \u003e 100 algorithms and data sinks to forward data to third-party systems.\n* Use the data explorer to visually explore historical data with many widgets tailored for time-series data.\n* A live dashboard to display real-time data from data sources and pipelines, e.g., for shopfloor monitoring.\n\n\nStreamPipes is highly extensible and includes a Java SDK to create new\npipeline elements and adapters. Python support is available in an early development stage - stay tuned!\nPipeline elements are standalone microservices that can run anywhere -\ncentrally on your server or close at the edge.\nYou want to employ your own machine learning model on live data?\nJust write your own data processor and make it reusable as a pipeline element.\n\nBesides that, StreamPipes includes features for production deployments:\n\n* Assign resources such as pipelines, data streams and dashboards to assets for better organization\n* Monitoring \u0026 metrics of pipelines and adapters\n* Built-in user and access rights management\n* Export and import resources\n\n## User interface\n\n* Connect data from an OPC-UA server following a three-step configuration process:\n\n![StreamPipes Connect](https://raw.githubusercontent.com/apache/streampipes/dev/images/streampipes-connect.gif)\n\n* Create a pipeline to detect a continuous decrease using a trend detection data processor and a ``Notification``sink:\n\n![StreamPipes Pipeline Editor](https://raw.githubusercontent.com/apache/streampipes/dev/images/streampipes-pipelines.gif)\n\n* Visually analyze data using the data explorer:\n\n![StreamPipes Data Explorer](https://raw.githubusercontent.com/apache/streampipes/dev/images/streampipes-data-explorer.gif)\n\n\n## Installation\n\nThe quickest way to run StreamPipes including the latest extensions (adapters, pipeline elements) is by using our Docker-based [installation \u0026 operation options](installer), namely:\n\n* **[StreamPipes Compose](installer/compose)** - The User's Choice\n* **[StreamPipes CLI](installer/cli)** - The Developer's Favorite\n* **[StreamPipes k8s](installer/k8s)** - The Operator's Dream\n\n\u003e [!IMPORTANT]\n\u003e StreamPipes CLI \u0026 k8s are highly recommended for developers or operators. Standard users should stick to StreamPipes Compose.\n\nPlease follow the instructions provided in the corresponding `README.md` to get started.\n\nFor a more in-depth manual, read the [installation guide](https://streampipes.apache.org/docs/try-installation.html).\n\n\u003e [!NOTE]\n\u003e TL;DR: Download the latest release, switch to the ``installer/compose`` directory and run ``docker-compose up -d``.\n\n## Documentation\n\nThe full documentation is available [here](https://streampipes.apache.org/docs/user-guide-introduction).\n\nQuick Links:\n\n* [Installation](https://streampipes.apache.org/docs/try-installation.html)\n* [Create adapters](https://streampipes.apache.org/docs/use-connect.html)\n* [Create pipelines](https://streampipes.apache.org/docs/use-pipeline-editor.html)\n* [Write your own pipeline elements](https://streampipes.apache.org/docs/extend-archetypes.html)\n\n## Building StreamPipes\n\nTo properly build the StreamPipes core, the following tools should be installed:\n\n### Prerequisites\n* Java 17 JDK (We officially only support Java 17, JDKs above 17 might work as well, but we don't provide any guarantee)\n* Maven (tested with 3.8)\n* NodeJS + NPM (tested with v12+/ v6+)\n* Docker + Docker-Compose\n\n### Building\n\nTo build the core project, do the following:\n\n```\n    mvn clean package\n```\n\nTo build the ui, switch to the ``ui`` folder and perform the following steps:\n\n```\n    npm install\n    npm run build\n```\n\n### Starting\n\nTo start StreamPipes, run ``docker-compose up --build -d`` from the root directory.\n\nYou can also use the installer or CLI as described in the ``Installation`` section.\n\n## Pipeline Elements\nStreamPipes includes a repository of extensions for adapters and pipeline elements:\n* **Connect adapters** for a variety of IoT data sources as well as\n* **Data Processors** and **Data Sinks** as ready-to-use pipeline elements.\n\nThe source code of all included pipeline elements and adapters can be found [here](https://github.com/apache/streampipes/tree/dev/streampipes-extensions).\n\n## Extending StreamPipes\n\nYou can easily add your own data streams, processors or sinks. A [Java-based SDK](https://streampipes.apache.org/docs/extend-tutorial-data-processors.html) can be used to integrate your existing processing logic into StreamPipes.\nPipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.\n\n👉 Check our [developer guide](https://streampipes.apache.org/docs/extend-setup.html).\n\n## Bugs and Feature Requests\n\nIf you've found a bug or have a feature that you'd love to see in StreamPipes, feel free to create an issue on GitHub:\n\n👉 [Bugs](https://github.com/apache/streampipes/issues)\n👉 [Feature requests](https://github.com/apache/streampipes/discussions/categories/ideas)\n\n## Get help\n\nIf you have any problems during the installation or questions around StreamPipes, you'll get help through one of our\ncommunity channels:\n\n👉 [Mailing Lists](https://streampipes.apache.org/community/mailing-lists/)\n\nOr directly subscribe to [users-subscribe@streampipes.apache.org](mailto:users-subscribe@streampipes.apache.org)!\n\n👉 And don't forget to follow us on [Twitter](https://twitter.com/streampipes)!\n\n## Contribute\n\nWe welcome all kinds of contributions to StreamPipes. If you are interested in contributing, let us know! You'll\n get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.\n\nHere are some first steps in case you want to contribute:\n* Subscribe to our dev mailing list [dev-subscribe@streampipes.apache.org](mailto:dev-subscribe@streampipes.apache.org)\n* Send an email, tell us about your interests and which parts of StreamPipes you'd like to contribute (e.g., core or UI)!\n* Ask for a mentor who helps you to understand the code base and guides you through the first setup steps\n* Find an issue on [GitHub](https://github.com/apache/streampipes/issues) which is tagged with a _good first issue_ label\n* Have a look at our developer [wiki](https://cwiki.apache.org/confluence/display/STREAMPIPES) to learn more about StreamPipes development.\n\nHave fun!\n\n## Feedback\n\nWe'd love to hear your feedback! Subscribe to [users@streampipes.apache.org](mailto:users@streampipes.apache.org)\n\n## License\n\n[Apache License 2.0](LICENSE)\n","funding_links":[],"categories":["Table of Contents"],"sub_categories":["IoT"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fstreampipes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fstreampipes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fstreampipes/lists"}