https://github.com/pachyderm/pachyderm
Data-Centric Pipelines and Data Versioning
https://github.com/pachyderm/pachyderm
analytics big-data containers data-analysis data-science distributed-systems docker go kubernetes pachyderm
Last synced: 27 days ago
JSON representation
Data-Centric Pipelines and Data Versioning
- Host: GitHub
- URL: https://github.com/pachyderm/pachyderm
- Owner: pachyderm
- License: apache-2.0
- Created: 2014-09-04T07:50:02.000Z (about 12 years ago)
- Default Branch: master
- Last Pushed: 2025-02-03T22:27:18.000Z (almost 2 years ago)
- Last Synced: 2026-10-12T17:27:23.035Z (about 1 month ago)
- Topics: analytics, big-data, containers, data-analysis, data-science, distributed-systems, docker, go, kubernetes, pachyderm
- Language: Go
- Homepage: https://www.pachyderm.com/
- Size: 626 MB
- Stars: 6,311
- Watchers: 152
- Forks: 576
- Open Issues: 940
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG-1.x.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-ccamel - pachyderm/pachyderm - Data-Centric Pipelines and Data Versioning (Go)
- awesome-llmops - Pachyderm - square) | (Data / Data Management)
- awesome-production-machine-learning - Pachyderm - Open source distributed processing framework build on Kubernetes focused mainly on dynamic building of production machine learning pipelines - [(Video)](https://www.youtube.com/watch?v=LamKVhe2RSM) (Model and Data Versioning)
- awesome-production-machine-learning - Pachyderm - Open source distributed processing framework build on Kubernetes focused mainly on dynamic building of production machine learning pipelines - [(Video)](https://www.youtube.com/watch?v=LamKVhe2RSM). (Data Pipeline)
- awesome-ai-ml-testing - Pachyderm - Data lineage and version control for data science. (🔄 CI/CD for ML)
- awesome-go - pachyderm/pachyderm - Data-Centric Pipelines and Data Versioning (Machine Learning and AI / Data Science and Analytics)
- awesome-starred - pachyderm/pachyderm - Data-Centric Pipelines and Data Versioning (go)
- awesome-cloud-native - pachyderm - Reproducible Data Science at Scale! (Data Processing & Analytics)
- awesome-open-data-centric-ai - Pachyderm
- awesome-list-docker - pachyderm
- awesome-etl - Pachyderm - "provides parallelized processing of multi-stage, language-agnostic pipelines with data versioning and data lineage tracking." (Go / Libraries)
- awesome-data-science-refs - Pachyderm: Data Versioning, Data Pipelines, and Data Lineage
- jimsghstars - pachyderm/pachyderm - Data-Centric Pipelines and Data Versioning (Go)
- awesome-golang-repositories - pachyderm
- llmops - Pachyderm - square) | (Data Management / Resources)
- awesome-fuse-fs - pachyderm - Containerized Data (Built On)
- Awesome-RAG-Production - Pachyderm - versioned pipeline orchestration | End-to-end provenance tracking across ingestion → embedding → index pipelines | (Data & Index Versioning / Frameworks)
README
[](https://github.com/pachyderm/pachyderm/releases)
[](https://github.com/pachyderm/pachyderm/blob/master/LICENSE)
[](https://pkg.go.dev/github.com/pachyderm/pachyderm/v2/src/client)
[](https://goreportcard.com/report/github.com/pachyderm/pachyderm)
[](https://www.pachyderm.com/slack)
[](https://cla-assistant.io/pachyderm/pachyderm)
# Pachyderm – Automate data transformations with data versioning and lineage
Pachyderm is cost-effective at scale, enabling data engineering teams to automate complex pipelines with sophisticated data transformations across any type of data. Our unique approach provides parallelized processing of multi-stage, language-agnostic pipelines with data versioning and data lineage tracking. Pachyderm delivers the ultimate CI/CD engine for data.
## Features
- Data-driven pipelines automatically trigger based on detecting data changes.
- Immutable data lineage with data versioning of any data type.
- Autoscaling and parallel processing built on Kubernetes for resource orchestration.
- Uses standard object stores for data storage with automatic deduplication.
- Runs across all major cloud providers and on-premises installations.
## Getting Started
To start deploying your end-to-end version-controlled data pipelines, run Pachyderm [locally](https://docs.pachyderm.com/latest/set-up/local-deploy/) or you can also [deploy on AWS/GCE/Azure](https://docs.pachyderm.com/latest/set-up/cloud-deploy) in about 5 minutes.
You can also refer to our complete [documentation](https://docs.pachyderm.com) to see tutorials, check out example projects, and learn about advanced features of Pachyderm.
If you'd like to see some examples and learn about core use cases for Pachyderm:
- [Examples](https://github.com/pachyderm/examples)
- [Use Cases](https://www.pachyderm.com/use-cases/)
- [Case Studies](https://www.pachyderm.com/case-studies/)
## Documentation
[Official Documentation](https://docs.pachyderm.com/)
## Community
Keep up to date and get Pachyderm support via:
- [](https://twitter.com/pachyderminc) Follow us on Twitter.
- [](https://slack.pachyderm.io) Join our community [Slack Channel](https://www.pachyderm.com/slack) to get help from the Pachyderm team and other users.
## Contributing
To get started, sign the [Contributor License Agreement](https://cla-assistant.io/pachyderm/pachyderm).
You should also check out our [contributing guide](https://docs.pachyderm.com/latest/contributing/setup/).
Send us PRs, we would love to see what you do! You can also check our GH issues for things labeled "help-wanted" as a good place to start. We're sometimes bad about keeping that label up-to-date, so if you don't see any, just let us know.
## Usage Metrics
Pachyderm automatically reports anonymized usage metrics. These metrics help us
understand how people are using Pachyderm and make it better. They can be
disabled by setting the env variable `METRICS` to `false` in the pachd
container.