Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with pipeline

A curated list of projects in awesome lists tagged with pipeline .

https://github.com/vectordotdev/vector

A high-performance observability data pipeline.

events forwarder logs metrics observability parser pipeline router rust stream-processing vector

Last synced: 12 May 2025

https://github.com/prefecthq/prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

automation data data-engineering data-ops data-science infrastructure ml-ops observability orchestration pipeline prefect python workflow workflow-engine

Last synced: 12 May 2025

https://github.com/airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake

Last synced: 12 May 2025

https://github.com/kestra-io/kestra

:zap: Workflow Automation Platform. Orchestrate & Schedule code in any language, run anywhere, 600+ plugins. Alternative to Airflow, n8n, Rundeck, VMware vRA, Zapier ...

automation data-orchestration devops high-availability infrastructure-as-code java low-code lowcode orchestration pipeline pipeline-as-code workflow

Last synced: 13 May 2025

https://github.com/PrefectHQ/prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

automation data data-engineering data-ops data-science infrastructure ml-ops observability orchestration pipeline prefect python workflow workflow-engine

Last synced: 24 Mar 2025

https://github.com/marimo-team/marimo

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. All in a modern, AI-native editor.

artificial-intelligence dag data-science data-visualization dataflow developer-tools machine-learning notebooks pipeline python reactive sql web-app

Last synced: 13 May 2025

https://github.com/kedro-org/kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

experiment-tracking hacktoberfest kedro machine-learning machine-learning-engineering mlops pipeline python

Last synced: 13 May 2025

https://github.com/prql/prql

PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

data pipeline sql

Last synced: 13 May 2025

https://github.com/PRQL/prql

PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

data pipeline sql

Last synced: 24 Mar 2025

https://github.com/tektoncd/pipeline

A cloud-native Pipeline resource.

cdf hacktoberfest kubernetes pipeline tekton

Last synced: 12 May 2025

https://github.com/projectdiscovery/httpx

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.

bugbounty cli cybersecurity hacktoberfest http lib osint pentest-tool pipeline ssl-certificate

Last synced: 12 May 2025

https://github.com/iam-veeramalla/jenkins-zero-to-hero

Install Jenkins, configure Docker as slave, set up cicd, deploy applications to k8s using Argo CD in GitOps way.

argocd cicd docker gitops jenkins kubernetes pipeline

Last synced: 14 May 2025

https://github.com/tc39/proposal-pipeline-operator

A proposal for adding a useful pipe operator to JavaScript.

javascript operator pipeline proposal syntax tc39

Last synced: 07 Apr 2025

https://github.com/brunch/brunch

:fork_and_knife: Web applications made easy. Since 2011.

brunch build-automation javascript pipeline workflow

Last synced: 24 Jan 2025

https://github.com/iam-veeramalla/Jenkins-Zero-To-Hero

Install Jenkins, configure Docker as slave, set up cicd, deploy applications to k8s using Argo CD in GitOps way.

argocd cicd docker gitops jenkins kubernetes pipeline

Last synced: 30 Mar 2025

https://github.com/nteract/papermill

📚 Parameterize, execute, and analyze notebooks

julia jupyter notebook notebook-generator notebooks nteract pipeline publishing python r scala

Last synced: 13 May 2025

https://github.com/gonglei007/gamedevmind

最全面的游戏开发技术图谱。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。

3d cpp devops framework game game-development game-framework game-server gamedev management mmorpg pipeline programming roadmap scrum shader unity unity3d unreal-engine unrealengine

Last synced: 13 May 2025

https://github.com/gonglei007/GameDevMind

最全面的游戏开发技术图谱。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。

3d cpp devops framework game game-development game-framework game-server gamedev management mmorpg pipeline programming roadmap scrum shader unity unity3d unreal-engine unrealengine

Last synced: 18 Apr 2025

https://github.com/jenkins-x/jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton

accelerator cicd continuous-delivery continuous-integration devops environments gitops golang hacktoberfest kubernetes openshift pipeline tekton

Last synced: 14 May 2025

https://github.com/alirezadir/production-level-deep-learning

A guideline for building practical production-level deep learning systems to be deployed in real world applications.

ai artificial-intelligence deep-learning deployment kubeflow machine-learning pipeline practical-machine-learning production-system scalable-applications system-design tfx

Last synced: 14 May 2025

https://github.com/alirezadir/Production-Level-Deep-Learning

A guideline for building practical production-level deep learning systems to be deployed in real world applications.

ai artificial-intelligence deep-learning deployment kubeflow machine-learning pipeline practical-machine-learning production-system scalable-applications system-design tfx

Last synced: 16 Apr 2025

https://github.com/marker-inc-korea/autorag

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

analysis automl benchmarking document-parser embeddings evaluation llm llm-evaluation llm-ops open-source ops optimization pipeline python qa rag rag-evaluation retrieval-augmented-generation

Last synced: 12 May 2025

https://github.com/tencentmusic/cube-studio

cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，支持sso登录，多租户，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，serverless，标注平台，自动化标注，数据集管理，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式

ai aihub argo automl gpt inference kubeflow kubernetes llmops mlops notebook pipeline pytorch spark vgpu workflow

Last synced: 14 May 2025

https://github.com/firecow/gitlab-ci-local

Tired of pushing to test your .gitlab-ci.yml?

cd ci git gitlab gitlab-ci local pipeline push uncomitted untracked

Last synced: 12 May 2025

https://github.com/epicgamesext/blendertools

Blender addons that improve the game development workflow between Blender and Unreal.

blender blender-addon pipeline python unreal

Last synced: 14 May 2025

https://github.com/EpicGamesExt/BlenderTools

Blender addons that improve the game development workflow between Blender and Unreal.

blender blender-addon pipeline python unreal

Last synced: 29 Nov 2024

https://github.com/alibaba/pipcook

Machine learning platform for Web developers

js machine-learning pipeline tensorflow

Last synced: 13 May 2025

https://alibaba.github.io/pipcook/

Machine learning platform for Web developers

js machine-learning pipeline tensorflow

Last synced: 04 May 2025

https://github.com/EntilZha/PyFunctional

Python library for creating data pipelines with chain functional programming

data datascience functional-programming pipeline python

Last synced: 26 Mar 2025

https://github.com/entilzha/pyfunctional

Python library for creating data pipelines with chain functional programming

data datascience functional-programming pipeline python

Last synced: 14 May 2025

https://github.com/instill-ai/instill-core

🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications

ai api cli developer-tools etl generative-ai golang gpt hacktoberfest llm low-code no-code open-source pipeline python stable-diffusion typescript unstructured-data

Last synced: 14 May 2025

https://github.com/apptension/saas-boilerplate

SaaS Boilerplate - Open Source and free SaaS stack that lets you build SaaS products faster in React, Django and AWS. Focus on essential business logic instead of coding repeatable features!

aws aws-esc backend boilerplate business cdk django docker frontend hacktoberfest pipeline product python react saas serverless stack typescript vitejs workers

Last synced: 14 May 2025

https://github.com/mara/mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

data data-integration etl pipeline postgresql python

Last synced: 14 May 2025

https://github.com/chunelfeng/cgraph

【A common used C++ & Python DAG framework】 一个通用的、无三方依赖的、跨平台的、收录于awesome-cpp的、基于流图的并行计算框架。欢迎star & fork & 交流

cpp dag graph pipeline pybind11 python taskflow threadpool workflow

Last synced: 14 May 2025

https://github.com/ChunelFeng/CGraph

【A common used C++ DAG framework】 一个通用的、无三方依赖的、跨平台的、收录于awesome-cpp的、基于流图的并行计算框架。欢迎star & fork & 交流

ai cpp dag graph hpc pipeline pybind11 python taskflow threadpool workflow

Last synced: 18 Mar 2025

https://github.com/numaproj/numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing

Last synced: 13 May 2025

https://github.com/hu17889/go_spider

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

crawler go pipeline schedule spider

Last synced: 25 Mar 2025

https://github.com/destel/rill

Go toolkit for clean, composable, channel-based concurrency

channels concurrency functional-programming generics go golang goroutines pipeline streaming

Last synced: 14 Apr 2025

https://github.com/ropensci/drake

An R-focused pipeline toolkit for reproducibility and high-performance computing

data-science drake high-performance-computing makefile peer-reviewed pipeline r r-package reproducibility reproducible-research ropensci rstats workflow

Last synced: 13 May 2025

https://github.com/tenox7/ttyplot

a realtime plotting utility for terminal/console with data input from stdin

chart cli cli-app command-line-tool commandline console console-tool cpu graph ping pipe pipeline plot realtime sar snmp snmp-network-throughput snmpget stdin

Last synced: 20 Mar 2025

https://github.com/alibaba/dawn

:sunrise: Dawn is a lightweight task management and build tool for front-end and nodejs.

build build-tool construction dawn dawn-cli front-end javascript middleware nodejs pack pipeline task

Last synced: 15 May 2025

https://alibaba.github.io/dawn/

:sunrise: Dawn is a lightweight task management and build tool for front-end and nodejs.

build build-tool construction dawn dawn-cli front-end javascript middleware nodejs pack pipeline task

Last synced: 01 Apr 2025

https://github.com/moby/datakit

Connect processes into powerful data pipelines with a simple git-like filesystem interface

data-flow database datakit docker filesystem-api pipeline

Last synced: 16 May 2025

https://github.com/scipipe/scipipe

Robust, flexible and resource-efficient pipelines using Go and the commandline

bioinformatics bioinformatics-pipeline cheminformatics dataflow fbp go golang pipeline scientific-workflows scipipe workflow workflow-engine

Last synced: 15 May 2025

https://github.com/cocoindex-io/cocoindex

ETL framework to turn your data AI-ready - with realtime incremental updates and support custom logic like lego.

ai change-data-capture data data-engineering data-indexing data-infrastructure data-processing dataflow etl help-wanted indexing knowledge-graph llm pipeline python rag real-time rust semantic-search streaming

Last synced: 14 May 2025

https://github.com/sematic-ai/sematic

An open-source ML pipeline development platform

ai data-science machine-learning ml ml-ops ml-pipeline ml-pipelines mlops pipeline python python3

Last synced: 14 May 2025

https://github.com/trivago/gollum

An n:m message multiplexer written in Go

golang gollum log logger logging logs message-bus multiplexer pipeline stream

Last synced: 16 May 2025

https://github.com/trivaGo/Gollum

An n:m message multiplexer written in Go

golang gollum log logger logging logs message-bus multiplexer pipeline stream

Last synced: 12 Mar 2025

https://github.com/tektoncd/dashboard

A dashboard for Tekton!

dashboard hacktoberfest pipeline tekton ui

Last synced: 13 May 2025

https://github.com/databiosphere/toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language

Last synced: 29 Apr 2025

https://github.com/paddlepaddle/serving

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

dag deep-learning docker gpu micro-service microservice-toolkit online-service paddle paddle-serving pipeline prediction predictor python rpc-service serving

Last synced: 15 May 2025

https://github.com/PaddlePaddle/Serving

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

dag deep-learning docker gpu micro-service microservice-toolkit online-service paddle paddle-serving pipeline prediction predictor python rpc-service serving

Last synced: 20 Mar 2025

https://github.com/DataBiosphere/toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language

Last synced: 07 Apr 2025

https://github.com/neumtry/neumai

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

ai chatgpt data data-engineering database embeddings etl llm llmops mlops ops pipeline python rag retrieval vector-database vectors

Last synced: 21 Apr 2025

https://github.com/shenwei356/rush

A cross-platform command-line tool for executing jobs in parallel

bioinformatics command cross-platform execute golang parallel pipeline shell windows

Last synced: 06 Apr 2025

https://github.com/NeumTry/NeumAI

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

ai chatgpt data data-engineering database embeddings etl llm llmops mlops ops pipeline python rag retrieval vector-database vectors

Last synced: 11 Apr 2025

https://github.com/internlm/huixiangdou

HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

application assistance chatbot dsl lark llm multimodal ocr pipeline rag robot wechat

Last synced: 08 Apr 2025

https://github.com/InternLM/HuixiangDou

HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

application assistance chatbot dsl lark llm multimodal ocr pipeline rag robot wechat

Last synced: 24 Mar 2025

https://github.com/henomis/lingoose

🪿 LinGoose is a Go framework for building awesome AI/LLM applications.

ai chatgpt embeddings go golang index llm openai pinecone pipeline prompt vector

Last synced: 15 May 2025

https://github.com/bregman-arie/howtheydevops

A curated collection of publicly available resources on how companies around the world practice DevOps

cd ci devops pipeline release

Last synced: 05 Apr 2025

https://github.com/czheo/syntax_sugar_python

A library adding some anti-Pythonic syntatic sugar to Python

functional parallelism pipeline python syntax-sugar

Last synced: 12 Apr 2025

https://github.com/pdpipe/pdpipe

Easy pipelines for pandas DataFrames.

data data-science dataframe dataframes pandas pandas-dataframe pipeline

Last synced: 17 Apr 2025

https://github.com/pipelight/pipelight

Tiny automation pipelines. Bring CI/CD to the smallest projects. Self-hosted, Lightweight, CLI only.

automation bash cicd docker git pipeline rust toml typescript yaml

Last synced: 15 May 2025

https://github.com/tektoncd/catalog

Catalog of shared Tasks and Pipelines.

catalog hacktoberfest k8s pipeline re-useable task tekton

Last synced: 24 Mar 2025

https://github.com/pypyr/pypyr

pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

automation cd ci ci-cd continuous-deployment continuous-integration devops pipeline pipeline-processor pipeline-runner pipelines pipelines-yaml python script script-loader task-manager task-runner taskrunner tool

Last synced: 26 Mar 2025

https://github.com/neuraxio/neuraxle

The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.

deep-learning framework hyperparameter-optimization hyperparameter-search hyperparameter-tuning hyperparameters machine-learning neuraxle parallel pipeline pipeline-framework python-library scikit-learn

Last synced: 16 May 2025

https://github.com/Neuraxio/Neuraxle

The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.

deep-learning framework hyperparameter-optimization hyperparameter-search hyperparameter-tuning hyperparameters machine-learning neuraxle parallel pipeline pipeline-framework python-library scikit-learn

Last synced: 05 Apr 2025

https://github.com/crowdhailer/ok

Elegant error/exception handling in Elixir, with result monads.

elixir elixir-pipelines macros monad pipeline

Last synced: 15 May 2025

https://github.com/CrowdHailer/OK

Elegant error/exception handling in Elixir, with result monads.

elixir elixir-pipelines macros monad pipeline

Last synced: 30 Mar 2025