{"id":37019053,"url":"https://github.com/mbarcia/pipelineframework","last_synced_at":"2026-01-14T02:02:49.997Z","repository":{"id":311443415,"uuid":"546815788","full_name":"mbarcia/pipelineframework","owner":"mbarcia","description":"The Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps with automatic gRPC and REST adapter generation.","archived":false,"fork":false,"pushed_at":"2026-01-01T23:12:09.000Z","size":11698,"stargazers_count":3,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-04T18:37:30.754Z","etag":null,"topics":["functional-programming","grpc","microservices","quarkus-extension","reactive-programming","rest"],"latest_commit_sha":null,"homepage":"https://pipelineframework.org","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mbarcia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-10-06T17:37:30.000Z","updated_at":"2025-12-25T01:25:47.000Z","dependencies_parsed_at":"2025-12-31T22:02:37.552Z","dependency_job_id":null,"html_url":"https://github.com/mbarcia/pipelineframework","commit_stats":null,"previous_names":["mbarcia/csv-payments-poc","mbarcia/pipelineframework"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/mbarcia/pipelineframework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbarcia%2Fpipelineframework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbarcia%2Fpipelineframework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbarcia%2Fpipelineframework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbarcia%2Fpipelineframework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mbarcia","download_url":"https://codeload.github.com/mbarcia/pipelineframework/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mbarcia%2Fpipelineframework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functional-programming","grpc","microservices","quarkus-extension","reactive-programming","rest"],"created_at":"2026-01-14T02:02:47.493Z","updated_at":"2026-01-14T02:02:49.983Z","avatar_url":"https://github.com/mbarcia.png","language":"Java","readme":"# The Pipeline Framework\n\n[![Maven Central](https://img.shields.io/maven-central/v/org.pipelineframework/pipelineframework.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22org.pipelineframework%22%20AND%20a:%22pipelineframework%22)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Java 21+](https://img.shields.io/badge/Java-21+-brightgreen.svg)](https://adoptium.net/)\n[![Quarkus](https://img.shields.io/badge/Quarkus-3.29.4-orange)](https://quarkus.io)\n[![CodeRabbit](https://img.shields.io/coderabbit/prs/github/mbarcia/pipelineframework?label=CodeRabbit\u0026color=purple\n)](https://coderabbit.ai)\n\nThe Pipeline Framework is a powerful framework for building reactive pipeline processing systems. It simplifies the development of distributed systems by providing a consistent way to create, configure, and deploy pipeline steps with automatic gRPC and REST adapter generation.\n\n## 🚀 Quick Start\n\n### Visual Designer\nThe fastest way to get started is using our visual canvas designer:\n- Visit [https://app.pipelineframework.org](https://app.pipelineframework.org)\n- Design your pipeline visually\n- Download the complete source code for your pipeline application\n\n## 📋 Featuring capabilities\n\n### Product Owners\n- **Microservices Architecture**: Each pipeline step can be deployed and scaled independently\n- **Data Integrity**: Immutability ensures that all transformations are preserved, providing robust data integrity\n- **Audit Trail**: Every input transformation is permanently recorded, enabling complete auditability\n- **Reliability**: Built-in dead letter queue for error handling ensures resilient processing\n\n### Architects\n- **Immutability by Design**: No database updates during pipeline execution - only appends/preserves\n- **Kubernetes \u0026 Serverless Ready**: Native builds enable quick start-up times for Kubernetes and serverless platforms (AWS Lambda, Google Cloud Run)\n- **Cost Efficiency**: Reduced resource consumption with native builds and reactive processing\n- **Scalability**: Each pipeline step can be scaled independently based on demand\n- **gRPC \u0026 REST Flexibility**: High-performance gRPC for throughput-intensive tasks, REST for ease of integration\n- **Multiple Persistence Options**: Choose from reactive (Panache) or virtual thread-based persistence models\n- **Rich Step Types**: Support for OneToOne, OneToMany, ManyToOne, ManyToMany, and SideEffect processing patterns\n\n### Developers\n- **Monorepo Support**: Complete pipeline can be debugged step-by-step in a single repository\n- **Compile-Time Safety**: All entities in common package provide type safety across steps\n- **Reactive Processing**: Mutiny's event/worker thread model for efficient concurrency\n- **Build-Time Generation**: Automatic gRPC and REST adapter generation at build time\n- **gRPC \u0026 REST Flexibility**: Fast gRPC for high-throughput scenarios, REST for ease of development and integration\n- **TLS Ready**: Reference implementation includes TLS configuration out of the box\n- **Multiple Processing Patterns**: OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect and blocking variants\n- **Health Monitoring**: Built-in health check capabilities for infrastructure monitoring\n\n### QA Engineers\n- **Unit Testing Excellence**: Each step deliberately kept small, making unit tests easy to write and fast to run\n- **Deterministic Behavior**: Immutability ensures consistent test results across runs\n- **AI-Assisted Development**: Modular structure is ideal for AI coding assistants\n- **Focused Testing**: Each step's limited scope enables comprehensive testing\n- **Multiple Execution Modes**: Test with different processing patterns (OneToOne, OneToMany, etc.) to validate behavior\n- **Health Check Testing**: Built-in health monitoring provides additional testable endpoints\n\n### CTOs\n- **Cost Reduction**: Native builds and reactive processing reduce infrastructure costs\n- **Developer Productivity**: Rapid development with visual designer and auto-generation\n- **Operational Excellence**: Built-in observability, health checks, and error handling\n- **Technology Modernization**: Reactive, cloud-native architecture with industry standards\n- **Risk Mitigation**: Dead letter queue and multiple processing patterns provide resilience\n\n## ✨ Capabilities summary\n\n- Reactive, immutable architecture with strong compile-time safety  \n- Automatic adapter generation for REST \u0026 gRPC  \n- Modular monorepo with clear separation of runtime vs. deployment  \n- Native-ready builds for ultra‑fast startup and cloud efficiency  \n- Built-in health monitoring, observability, metrics, and tracing  \n- Visual pipeline designer for rapid development  \n- Multiple execution patterns (OneToOne, OneToMany, ManyToOne, ManyToMany, SideEffect, blocking)  \n- Comprehensive error handling with dead-letter queue support  \n- Multiple persistence models (reactive or virtual‑thread‑based)  \n- Strong testing support: deterministic, focused, and easy to automate  \n\n## 📚 Documentation\n\n- **[Full Documentation](https://pipelineframework.org)** - Complete documentation site\n- **[Getting Started Guide](https://pipelineframework.org/guide/getting-started)** - Step-by-step tutorial\n- **[Canvas Designer Guide](https://pipelineframework.org/CANVAS_GUIDE)** - Visual design tool\n- **[API Reference](https://pipelineframework.org/annotations/pipeline-step)** - Annotation documentation\n\n## 🏗️ Architecture\n\nThe framework follows a modular architecture:\n\n```text\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│   Orchestrator  │───▶│ Backend Service  │───▶│ Backend Service │\n│   (Coordinates) │    │     (Step 1)     │    │     (Step 2)    │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n         │                        │                       │\n         ▼                        ▼                       ▼\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│    Input Data   │    │   Processing     │    │   Output Data   │\n│                 │    │   Pipeline       │    │                 │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n```\n\nEach pipeline step is implemented as an independent service that can be scaled and deployed separately.\n\n## 🛠️ Example Implementation\n\nHere's a simple pipeline step implementation:\n\n```java\n@PipelineStep(\n    order = 1,\n    inputType = PaymentRecord.class,\n    outputType = PaymentStatus.class,\n    stepType = StepOneToOne.class,\n    backendType = GenericGrpcReactiveServiceAdapter.class,\n    autoPersist = true,\n)\n@ApplicationScoped\npublic class ProcessPaymentService implements ReactiveStreamingClientService\u003cPaymentRecord, PaymentStatus\u003e {\n    \n    @Override\n    public Uni\u003cPaymentStatus\u003e process(PaymentRecord input) {\n        // Business logic here\n        PaymentStatus status = new PaymentStatus();\n        // Process the payment record\n        return Uni.createFrom().item(status);\n    }\n}\n```\n\n## Continuous Integration\nSee our [CI documentation](CI.md)\n\n## 🧪 Testing\nSee our [Testing documentation](TESTING.md)\n\n## 🛡️ Security\n\nIf you discover a security vulnerability, please see our [security policy](SECURITY.md) for details on how to report it responsibly.\n\n## 🤝 Contributing\n\nContributions of all kinds are welcome — code, documentation, ideas, translations and questions.\n\nPlease read our [contribution guidelines](CONTRIBUTING.md) for details on how to get started.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbarcia%2Fpipelineframework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmbarcia%2Fpipelineframework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmbarcia%2Fpipelineframework/lists"}