https://github.com/zacernst/metalpipe

Modules for ETL Pipelines
https://github.com/zacernst/metalpipe

data-ingestion etl stream-processing

Last synced: 5 months ago
JSON representation

Modules for ETL Pipelines

• Host: GitHub
• URL: https://github.com/zacernst/metalpipe
• Owner: zacernst
• License: mit
• Created: 2018-07-20T20:39:31.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T02:29:27.000Z (over 3 years ago)
• Last Synced: 2025-03-14T22:17:05.526Z (over 1 year ago)
• Topics: data-ingestion, etl, stream-processing
• Language: TeX
• Homepage:
• Size: 17.3 MB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 19
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

          
[![Build Status](https://travis-ci.org/zacernst/metalpipe.svg?branch=master)](https://travis-ci.org/zacernst/metalpipe)

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)

[![Readthedocs](https://readthedocs.org/projects/metalpipe/badge/)](https://metalpipe.readthedocs.io/en/latest/#)

# MetalPipe: Modules for ETL Pipelines

MetalPipe is a lightweight, multithreaded framework for building ETL pipelines. It utilizes a design pattern similar to stream-processing frameworks such as Spark or Storm. But unlike those heavyweight systems, MetalPipe is designed for ETL, not data analytics.

The goals of MetalPipe are:

1. To speed up ETL pipeline development by replacing as much code as possible with simple configurations.

2. To make ETL pipelines faster by eliminating IO bottlenecks.

3. To enable robust monitoring and error-handling into all ETL pipelines by default.

4. To eliminate the need for specialized, heavyweight infrastructure for ETL jobs.

Documentation lives here:

https://metalpipe.readthedocs.io/en/latest/

MetalPipe is a work in progress. Although we use it in production, it should not generally be considered stable.