An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with pipelines

A curated list of projects in awesome lists tagged with pipelines .

https://github.com/concourse/concourse

Concourse is a container-based automation system written in Go.

ci ci-cd concourse containerd containers continuous-delivery continuous-integration elm go pipelines runc

Last synced: 09 Sep 2025

https://github.com/ploomber/ploomber

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

data-engineering data-science jupyter jupyter-notebooks machine-learning mlops notebooks papermill pipelines pycharm vscode workflow

Last synced: 29 Apr 2025

https://github.com/salesforce/transmogrifai

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

ai automated-machine-learning automl dsl einstein estimators feature-engineering features machine-learning ml pipelines salesforce scala spark sparkml structured-data transformations transformers transmogrification transmogrify

Last synced: 14 May 2025

https://github.com/salesforce/TransmogrifAI

TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

ai automated-machine-learning automl dsl einstein estimators feature-engineering features machine-learning ml pipelines salesforce scala spark sparkml structured-data transformations transformers transmogrification transmogrify

Last synced: 26 Mar 2025

https://github.com/meltano/meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

connectors data data-engineering data-pipelines dataops dataops-platform elt extract-data integration loaders meltano meltano-sdk open-source opensource pipelines singer tap taps target targets

Last synced: 12 May 2025

https://github.com/substantic/rain

Framework for large distributed pipelines

distributed-computing pipelines python-api rust workflows

Last synced: 27 Apr 2025

https://github.com/onepanelio/onepanel

The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

ai aiops annotation computer-vision deeplearning etl hyperparameter-tuning inference jupyterlab labeling machinelearning mlops pipelines pytorch tensorboard tensorflow training workflows

Last synced: 07 Oct 2025

https://github.com/juliankemmerer/pipelinec

A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.

c fpga fpga-acceleration fpga-accelerators fpga-programming hardware hardware-description hardware-description-language high-level-synthesis hls open-source-hardware pipelines python vhdl

Last synced: 15 May 2025

https://github.com/JulianKemmerer/PipelineC

A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.

c fpga fpga-acceleration fpga-accelerators fpga-programming hardware hardware-description hardware-description-language high-level-synthesis hls open-source-hardware pipelines python vhdl

Last synced: 22 Apr 2025

https://github.com/pypyr/pypyr

pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

automation cd ci ci-cd continuous-deployment continuous-integration devops pipeline pipeline-processor pipeline-runner pipelines pipelines-yaml python script script-loader task-manager task-runner taskrunner tool

Last synced: 21 Oct 2025

https://github.com/mdubourg001/glci

🦊 Test your Gitlab CI Pipelines changes locally using Docker.

ci docker gitlab gitlab-ci jobs pipelines

Last synced: 13 Apr 2025

https://github.com/live-keys/livekeys

Livekeys - Automation Platform with support for live scripting.

image-processing live-coding pipelines video-processing

Last synced: 17 Apr 2025

https://github.com/tektoncd/triggers

Event triggering with Tekton!

hacktoberfest pipelines tekton triggers

Last synced: 04 Oct 2025

https://github.com/acln0/zerocopy

Zero-copy I/O primitives and pipelines for Go. Linux-specific.

go linux pipelines splice tee zero-copy

Last synced: 04 Apr 2025

https://github.com/timoschwarzer/gitlab-monitor

A web-based monitor dashboard for GitLab CI

dashboard gitlab gitlabci hacktoberfest monitor pipelines

Last synced: 04 Apr 2025

https://github.com/harness/drone-cli

Command Line Tools for Drone CI

continuous-delivery continuous-integration pipelines

Last synced: 15 May 2025

https://github.com/smooks/smooks

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

analytics chunking enterprise-integration etl event-driven java pipelines sax smooks stream-processing xml

Last synced: 13 May 2025

https://github.com/calebwin/pipelines

An experimental programming language for data flow

compiler language nim parallel pipeline pipelines python

Last synced: 09 Apr 2025

https://github.com/zorbash/opus

A framework for pluggable business logic components

dsl elixir pipelines railway-oriented-programming

Last synced: 08 Apr 2025

https://github.com/pipefunc/pipefunc

Lightweight fast function pipeline (DAG) creation in pure Python for scientific workflows 🕸️🧪

dag hpc parallel-computing pipeline-framework pipelines reproducible-research slurm workflow-engine

Last synced: 16 Dec 2025

https://github.com/pytorch/torchx

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

airflow aws-batch components deep-learning distributed-training kubernetes machine-learning pipelines python pytorch ray slurm

Last synced: 14 May 2025

https://github.com/sharpliner/sharpliner

Use C# instead of YAML to define your Azure DevOps pipelines

azure-devops azure-devops-pipelines csharp dotnet pipelines yaml

Last synced: 06 Oct 2025

https://github.com/nf-core/modules

Repository to host tool-specific module files for the Nextflow DSL2 community!

dsl2 modules nextflow nf-core nf-test pipelines workflows

Last synced: 27 Mar 2025

https://github.com/intel/dffml

The easiest way to use Machine Learning. Mix and match underlying ML libraries and data set sources. Generate new datasets or modify existing ones with ease.

ai-inference ai-machine-learning ai-training analytics asyncio dag data-flow dataflows datasets dffml event-based flow-based-programming frameworks hyperautomation libraries machine-learning models pipelines python swrepo

Last synced: 08 May 2025

https://github.com/project-codeflare/codeflare

Simplifying the definition and execution, scaling and deployment of pipelines on the cloud.

automl data-science hyperparameter-optimization machine-learning pipelines ray sklearn workflows

Last synced: 11 Oct 2025

https://github.com/reubenbond/hagar

Fast, flexible, and version-tolerant serializer for .NET

dotnet dotnet-core pipelines serialization-library serializer span

Last synced: 04 Apr 2025

https://github.com/dataplane-app/dataplane

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

airflow data data-analysis data-engineering data-integration data-pipelines data-science dataplane datawarehouse etl finance golang kubernetes pipelines robotics-process-automation rpa scheduler workflow workflow-automation workflows

Last synced: 27 Dec 2025

https://github.com/tseemann/shovill

⚡♠️ Assemble bacterial isolate genomes from Illumina paired-end reads

genome-assembler illumina-sequencing pipelines

Last synced: 12 Apr 2025

https://github.com/DamZiobro/gstreamerCheatsheet

GStreamer pipelines and CLI commands for different GStreamer based features (process MPEG2-TS files, get video from DVB, deinterlace video, capture RTSP stream etc.)

capture-rtsp-stream capture-screen cheatsheet dvb gst gstreamer gstreamer-pipeline hls mpeg mpegts pipelines rtsp video youtube

Last synced: 16 Apr 2025

https://github.com/logrange/logrange

High performance data aggregating storage

database go golang log-aggregation logging pipelines stream-processing streams

Last synced: 23 Mar 2025

https://github.com/ivanpaulovich/fluentmediator

:twisted_rightwards_arrows: FluentMediator is an unobtrusive library that allows developers to build custom pipelines for Commands, Queries and Events.

chain-methods chain-of-responsibility craftmanship csharp ddd-cqrs dotnet-core event-driven event-handlers event-sourcing fluent-design fluent-interface mediator-pattern message-bus pipeline-framework pipelines tdd

Last synced: 09 Apr 2025

https://github.com/ivanpaulovich/FluentMediator

:twisted_rightwards_arrows: FluentMediator is an unobtrusive library that allows developers to build custom pipelines for Commands, Queries and Events.

chain-methods chain-of-responsibility craftmanship csharp ddd-cqrs dotnet-core event-driven event-handlers event-sourcing fluent-design fluent-interface mediator-pattern message-bus pipeline-framework pipelines tdd

Last synced: 14 Mar 2025

https://github.com/mrpaulandrew/procfwk

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.

adf adfprocfwk azure azure-functions azure-sql-database data-engineering data-factory framework metadata pipelines processing procfwk

Last synced: 08 Apr 2025

https://github.com/microsoftlearning/eshoponweb

Repository maintained by AZ-400 course and Learn content community. Project used for AZ-400 Labs. Forked from: https://github.com/dotnet-architecture/eShopOnWeb Sample - ASP.NET Core 8.0 reference application, powered by Microsoft, demonstrating a layered application architecture with monolithic deployment model.

az-400 bicep devops github github-actions pipelines

Last synced: 13 Apr 2025

https://github.com/azure/azops-accelerator

Integrated CI/CD Solution for Microsoft Azure

actions azure cicd deployments devops gitops pipelines

Last synced: 07 Apr 2025

https://github.com/siboehm/shallowspeed

Small scale distributed training of sequential deep learning models, built on Numpy and MPI.

deep-learning distributed-computing pipelines

Last synced: 26 Jun 2025

https://github.com/gobeam/mongo-go-pagination

Golang Mongodb Pagination for official mongodb/mongo-go-driver package which supports both normal queries and Aggregation pipelines with all information like Total records, Page, Per Page, Previous, Next, Total Page and query results.

aggregate aggregation database driver go go-mongo golang match mongo mongo-go-driver mongo-go-pagination mongo-pagination mongodb mongodb-driver pagination pagination-library paginator perpage pipelines totalpage

Last synced: 06 Apr 2025

https://github.com/anicetngrt/jiro-nn

A Deep Learning and preprocessing framework in Rust with support for CPU and GPU.

adam classification cuda data-analysis deep-learning dropout gpu gpu-computing machine-learning ml nalgebra neural-networks nn opencl pipelines regression rust sgd

Last synced: 09 Apr 2025

https://github.com/AnicetNgrt/jiro-nn

A Deep Learning and preprocessing framework in Rust with support for CPU and GPU.

adam classification cuda data-analysis deep-learning dropout gpu gpu-computing machine-learning ml nalgebra neural-networks nn opencl pipelines regression rust sgd

Last synced: 25 Sep 2025

https://github.com/ktomk/pipelines

Pipelines - Run Bitbucket Pipelines Wherever They Dock

bitbucket-pipelines docker local-build pipeline-runner pipelines

Last synced: 12 Apr 2025

https://github.com/juliaml/tabletransforms.jl

Transforms and pipelines with tabular data in Julia

data-science machine-learning pipelines statistics table transforms

Last synced: 14 Mar 2025

https://gitlab.com/dbsystel/gitlab-ci-python-library

GitLab CI Python Library (GCIP) used to create a GitLab CI Pipeline with Python 3.

automate cicd gcip gitlab pipelines python python3

Last synced: 29 Jun 2025

https://github.com/nf-core/configs

Config files used to define parameters specific to compute environments at different Institutions

bioinformatics configuration nextflow nf-core pipelines workflows

Last synced: 11 Sep 2025

https://github.com/ndw/xmlcalabash1

XML Calabash, an XProc processor

pipelines xml xproc

Last synced: 22 Jun 2025

https://github.com/redhat-scholars/tekton-tutorial

Get started with Tekton Pipelines :smiley_cat: for https://dn.dev/master. Learn from basics to advanced

cd ci cli demos kubernetes pipelines redhat-developers tekton-pipelines tekton-triggers tektoncd triggers tutorial

Last synced: 04 Apr 2025

https://github.com/eora-ai/inferoxy

Service for quick deploying and using dockerized Computer Vision models

computer-vision machine-learning mlops pipelines python

Last synced: 19 Jul 2025

https://github.com/ctuning/ck-tensorflow

Collective Knowledge components for TensorFlow (code, data sets, models, packages, workflows):

apis automation collective-knowledge knowledge-management metadata package-management pipelines workflow-automation

Last synced: 19 Jul 2025

https://github.com/neptune-ai/examples

📝 Examples of how to use Neptune for different use cases and with various MLOps tools

compare dashboard experiment fastai jupyter keras metadata ml mlops optuna pandas pipelines pytorch sklearn store tensorflow tracker tracking visualization xgboost

Last synced: 05 Apr 2025

https://github.com/hahwul/gee

🏵 Gee is tool of stdin to each files and stdout. It is similar to the tee command, but there are more functions for convenience. In addition, it was written as go

bugbounty-tool command-line-tool golang pipelines security-tools tee utility

Last synced: 06 Apr 2025

https://github.com/connectomicslab/connectomemapper3

Connectome Mapper 3 is a BIDS App that implements full anatomical, diffusion, resting/state functional MRI, and recently EEG processing pipelines, from raw T1 / DWI / BOLD , and preprocessed EEG data to multi-resolution brain parcellation with corresponding connection matrices.

bids bids-apps bidsapp brain chuv connectome dmri docker eeg fmri parcellation pipelines python singularity

Last synced: 29 Apr 2025

https://github.com/4dn-dcic/tibanna

Tibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.

aws aws-lambda aws-step-function bioinformatics conda cwl cwl-workflow docker pipelines snakemake snakemake-workflow wdl wdl-workflow

Last synced: 16 Apr 2025

https://github.com/shini4i/argo-watcher

A service that bridges traditional pipelines and GitOps, improving deployment visibility. Includes a built-in GitOps repo updater.

argocd automation ci-cd devops gitops kubernetes pipelines

Last synced: 09 May 2025

https://github.com/typestreamio/typestream

Open Source streaming platform. Write and run typed data pipelines with a minimal, familiar syntax.

kafka kafka-streams pipelines

Last synced: 11 May 2025

https://github.com/nicolay-r/arekit

Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML

batching bulk-operation frames language-model nlp pipelines pipelines-library relation-extraction relationship-extraction sentiment-analysis

Last synced: 13 Oct 2025

https://github.com/nicolay-r/AREkit

Document level Attitude and Relation Extraction toolkit (AREkit) for sampling and processing large text collections with ML and for ML

batching bulk-operation frames language-model nlp pipelines pipelines-library relation-extraction relationship-extraction sentiment-analysis

Last synced: 31 Mar 2025

https://github.com/cloud-pipelines/pipeline-editor

Cloud Pipelines Editor is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.

google-cloud google-cloud-platform google-cloud-vertex google-cloud-vertex-ai kubeflow-pipelines machine-learning pipeline pipelines vertex-ai

Last synced: 20 Oct 2025

https://github.com/robertoprevato/azuredevops-agents

Images for self-hosted DevOps agents

azure devops devops-agents pipelines

Last synced: 30 Apr 2025

https://github.com/groovy-sky/devops-101

Intro to DevOps from scratch.

ansible azure ci-cd devops docker iac linux pipelines

Last synced: 25 Dec 2025

https://github.com/Cloud-Pipelines/pipeline-editor

Cloud Pipelines Editor is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.

google-cloud google-cloud-platform google-cloud-vertex google-cloud-vertex-ai kubeflow-pipelines machine-learning pipeline pipelines vertex-ai

Last synced: 06 May 2025

https://github.com/anki-code/xontrib-pipeliner

Let your pipe lines flow thru the Python code in xonsh.

data-engineering data-engineering-pipeline pipe pipeline pipelines python shell xonsh xontrib

Last synced: 22 Aug 2025

https://github.com/izniburak/pipeline-go

Pipeline pattern for your jobs, tasks, etc.. (Golang)

design-patterns go golang pipeline pipelines

Last synced: 02 May 2025