Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
https://github.com/numaproj/numaflow
data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing
Last synced: 24 days ago
JSON representation
Kubernetes-native platform to run massively parallel data/streaming jobs
- Host: GitHub
- URL: https://github.com/numaproj/numaflow
- Owner: numaproj
- License: apache-2.0
- Created: 2022-05-20T22:05:07.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-17T14:20:27.000Z (7 months ago)
- Last Synced: 2024-04-17T15:22:32.153Z (7 months ago)
- Topics: data-processing, hacktoberfest, k8s, kubernetes, map-reduce, pipeline, stream-processing
- Language: Go
- Homepage: https://numaflow.numaproj.io
- Size: 27.5 MB
- Stars: 913
- Watchers: 16
- Forks: 88
- Open Issues: 143
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-streaming - Numaflow - Kubernetes native stream processing platform with language agnostic framework. Scalable and cost-efficient (Table of Contents / Streaming Engine)
README
# Numaflow
[](https://goreportcard.com/report/github.com/numaproj/numaflow)
[](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg)
[](https://godoc.org/github.com/numaproj/numaflow/pkg/apis)
[](LICENSE)
[](https://github.com/numaproj/numaflow/releases/latest)
[](https://bestpractices.coreinfrastructure.org/projects/6078)
Welcome to Numaflow! A Kubernetes-native, serverless platform for running scalable and reliable event-driven applications. Numaflow decouples event sources and sinks from the processing logic, allowing each component to independently auto-scale based on demand. With out-of-the-box sources and sinks, and built-in observability, developers can focus on their processing logic without worrying about event consumption, writing boilerplate code, or operational complexities. Each step of the pipeline can be written in any programming language, offering unparalleled flexibility in using the best programming language for each step and ease of using the languages you are most familiar with.
Numaflow, created by the Intuit Argo team to address community needs for continuous event processing, leverages their expertise to deliver a scalable and robust, serverless platform for event-driven applications.
## Use Cases
- Event driven applications: Process events as they happen, e.g., updating inventory and sending customer notifications in e-commerce.
- Real time analytics: Analyze data instantly, e.g., social media analytics, observability data processing.
- Inference on streaming data: Perform real-time predictions, e.g., anomaly detection.
- Workflows running in a streaming manner.
## Key Features
- Kubernetes-native: If you know Kubernetes, you already know how to use Numaflow.
- Serverless: Focus on your code and let the system scale up and down based on demand.
- Language agnostic: Use your favorite programming language.
- Exactly-Once semantics: No input element is duplicated or lost even as pods are rescheduled or restarted.
- Auto-scaling with back-pressure: Each vertex automatically scales from zero to whatever is needed.
## Data Integrity Guarantees
- Minimally provide at-least-once semantics
- Provide exactly-once semantics for unbounded and near real-time data sources
- Preserving order is not required
## Roadmap
- Mono Vertex to bypass ISB for simple use cases (1.4)
## Demo
[](https://youtu.be/TOqKOYX0nrE)
## Resources
- [QUICK_START](docs/quick-start.md)
- [EXAMPLES](examples)
- [DEVELOPMENT](docs/development/development.md)
- [CONTRIBUTING](https://github.com/numaproj/numaproj/blob/main/CONTRIBUTING.md)