https://github.com/mbarcia/pipelineframework
The Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps with automatic gRPC and REST adapter generation.
https://github.com/mbarcia/pipelineframework
functional-programming grpc microservices quarkus-extension reactive-programming rest
Last synced: 2 months ago
JSON representation
The Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps with automatic gRPC and REST adapter generation.
- Host: GitHub
- URL: https://github.com/mbarcia/pipelineframework
- Owner: mbarcia
- License: other
- Created: 2022-10-06T17:37:30.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2026-01-01T23:12:09.000Z (3 months ago)
- Last Synced: 2026-01-04T18:37:30.754Z (3 months ago)
- Topics: functional-programming, grpc, microservices, quarkus-extension, reactive-programming, rest
- Language: Java
- Homepage: https://pipelineframework.org
- Size: 11.2 MB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# The Pipeline Framework
[](https://search.maven.org/search?q=g:%22org.pipelineframework%22%20AND%20a:%22pipelineframework%22)
[](https://opensource.org/licenses/Apache-2.0)
[](https://adoptium.net/)
[](https://quarkus.io)
[](https://coderabbit.ai)
The Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps with automatic gRPC and REST adapter generation.
## 🚀 Quick Start
### Visual Designer
The fastest way to get started is using our visual canvas designer:
- Visit [https://app.pipelineframework.org](https://app.pipelineframework.org)
- Design your pipeline visually
- Download the complete source code for your pipeline application
## 📋 Featuring capabilities
### Product Owners
- **Microservices Architecture**: Each pipeline step can be deployed and scaled independently
- **Data Integrity**: Immutability ensures that all transformations are preserved, providing robust data integrity
- **Audit Trail**: Every input transformation is permanently recorded, enabling complete auditability
- **Reliability**: Built-in dead letter queue for error handling ensures resilient processing
### Architects
- **Immutability by Design**: No database updates during pipeline execution - only appends/preserves
- **Kubernetes & Serverless Ready**: Native builds enable quick start-up times for Kubernetes and serverless platforms (AWS Lambda, Google Cloud Run)
- **Cost Efficiency**: Reduced resource consumption with native builds and reactive processing
- **Scalability**: Each pipeline step can be scaled independently based on demand
- **gRPC & REST Flexibility**: High-performance gRPC for throughput-intensive tasks, REST for ease of integration
- **Multiple Persistence Options**: Choose from reactive (Panache) or virtual thread-based persistence models
- **Rich Step Types**: Support for OneToOne, OneToMany, ManyToOne, ManyToMany, and SideEffect processing patterns
### Developers
- **Monorepo Support**: Complete pipeline can be debugged step-by-step in a single repository
- **Compile-Time Safety**: All entities in common package provide type safety across steps
- **Reactive Processing**: Mutiny's event/worker thread model for efficient concurrency
- **Build-Time Generation**: Automatic gRPC and REST adapter generation at build time
- **gRPC & REST Flexibility**: Fast gRPC for high-throughput scenarios, REST for ease of development and integration
- **TLS Ready**: Reference implementation includes TLS configuration out of the box
- **Multiple Processing Patterns**: OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect and blocking variants
- **Health Monitoring**: Built-in health check capabilities for infrastructure monitoring
### QA Engineers
- **Unit Testing Excellence**: Each step deliberately kept small, making unit tests easy to write and fast to run
- **Deterministic Behavior**: Immutability ensures consistent test results across runs
- **AI-Assisted Development**: Modular structure is ideal for AI coding assistants
- **Focused Testing**: Each step's limited scope enables comprehensive testing
- **Multiple Execution Modes**: Test with different processing patterns (OneToOne, OneToMany, etc.) to validate behavior
- **Health Check Testing**: Built-in health monitoring provides additional testable endpoints
### CTOs
- **Cost Reduction**: Native builds and reactive processing reduce infrastructure costs
- **Developer Productivity**: Rapid development with visual designer and auto-generation
- **Operational Excellence**: Built-in observability, health checks, and error handling
- **Technology Modernization**: Reactive, cloud-native architecture with industry standards
- **Risk Mitigation**: Dead letter queue and multiple processing patterns provide resilience
## ✨ Capabilities summary
- Reactive, immutable architecture with strong compile-time safety
- Automatic adapter generation for REST & gRPC
- Modular monorepo with clear separation of runtime vs. deployment
- Native-ready builds for ultra‑fast startup and cloud efficiency
- Built-in health monitoring, observability, metrics, and tracing
- Visual pipeline designer for rapid development
- Multiple execution patterns (OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect, blocking)
- Comprehensive error handling with dead-letter queue support
- Multiple persistence models (reactive or virtual‑thread‑based)
- Strong testing support: deterministic, focused, and easy to automate
## 📚 Documentation
- **[Full Documentation](https://pipelineframework.org)** - Complete documentation site
- **[Getting Started Guide](https://pipelineframework.org/guide/getting-started)** - Step-by-step tutorial
- **[Canvas Designer Guide](https://pipelineframework.org/CANVAS_GUIDE)** - Visual design tool
- **[API Reference](https://pipelineframework.org/annotations/pipeline-step)** - Annotation documentation
## 🏗️ Architecture
The framework follows a modular architecture:
```text
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Orchestrator │───▶│ Backend Service │───▶│ Backend Service │
│ (Coordinates) │ │ (Step 1) │ │ (Step 2) │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Input Data │ │ Processing │ │ Output Data │
│ │ │ Pipeline │ │ │
└─────────────────┘ └──────────────────┘ └─────────────────┘
```
Each pipeline step is implemented as an independent service that can be scaled and deployed separately.
## 🛠️ Example Implementation
Here's a simple pipeline step implementation:
```java
@PipelineStep(
order = 1,
inputType = PaymentRecord.class,
outputType = PaymentStatus.class,
stepType = StepOneToOne.class,
backendType = GenericGrpcReactiveServiceAdapter.class,
autoPersist = true,
)
@ApplicationScoped
public class ProcessPaymentService implements ReactiveStreamingClientService {
@Override
public Uni process(PaymentRecord input) {
// Business logic here
PaymentStatus status = new PaymentStatus();
// Process the payment record
return Uni.createFrom().item(status);
}
}
```
## Continuous Integration
See our [CI documentation](CI.md)
## 🧪 Testing
See our [Testing documentation](TESTING.md)
## 🛡️ Security
If you discover a security vulnerability, please see our [security policy](SECURITY.md) for details on how to report it responsibly.
## 🤝 Contributing
Contributions of all kinds are welcome — code, documentation, ideas, translations and questions.
Please read our [contribution guidelines](CONTRIBUTING.md) for details on how to get started.