{"id":50111388,"url":"https://github.com/npow/awesome-metaflow","last_synced_at":"2026-05-23T12:32:22.397Z","repository":{"id":340917996,"uuid":"1168170125","full_name":"npow/awesome-metaflow","owner":"npow","description":"Every Metaflow extension worth knowing, curated and organized","archived":false,"fork":false,"pushed_at":"2026-03-09T06:27:24.000Z","size":27,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-09T11:34:00.377Z","etag":null,"topics":["awesome-list","data-science","extensions","machine-learning","metaflow","mlops","python","workflow-orchestration"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/npow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-27T04:48:49.000Z","updated_at":"2026-03-09T06:27:27.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/npow/awesome-metaflow","commit_stats":null,"previous_names":["npow/awesome-metaflow"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/npow/awesome-metaflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npow%2Fawesome-metaflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npow%2Fawesome-metaflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npow%2Fawesome-metaflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npow%2Fawesome-metaflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/npow","download_url":"https://codeload.github.com/npow/awesome-metaflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/npow%2Fawesome-metaflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33396574,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-23T04:15:53.637Z","status":"ssl_error","status_checked_at":"2026-05-23T04:15:53.242Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome-list","data-science","extensions","machine-learning","metaflow","mlops","python","workflow-orchestration"],"created_at":"2026-05-23T12:32:20.456Z","updated_at":"2026-05-23T12:32:22.382Z","avatar_url":"https://github.com/npow.png","language":null,"funding_links":[],"categories":["Other Lists"],"sub_categories":["Vue Lists"],"readme":"# Awesome Metaflow [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n\u003e A curated list of Metaflow extensions, plugins, integrations, and resources.\n\n[Metaflow](https://metaflow.org) is a human-friendly Python/R framework for real-life ML, AI, and data science. Originally built at Netflix and open-sourced in 2019.\n\n---\n\n## Contents\n\n- [Core Infrastructure](#core-infrastructure)\n- [Infrastructure \u0026 IaC](#infrastructure--iac)\n- [Dependency Management](#dependency-management)\n- [Extension Mechanism \u0026 Templates](#extension-mechanism--templates)\n- [Orchestration \u0026 Scheduling](#orchestration--scheduling)\n- [Distributed Compute \u0026 Training](#distributed-compute--training)\n- [Model \u0026 Artifact Management](#model--artifact-management)\n- [Observability \u0026 Monitoring](#observability--monitoring)\n- [Cards \u0026 Visualization](#cards--visualization)\n- [Third-Party Integrations](#third-party-integrations)\n- [Developer Tooling](#developer-tooling)\n- [Examples \u0026 Tutorials](#examples--tutorials)\n\n---\n\n## Core Infrastructure\n\n- [metaflow](https://github.com/Netflix/metaflow) - Core framework with built-in `@kubernetes`, `@batch`, Argo, Airflow, and Step Functions support.\n- [metaflow-local-service](https://github.com/npow/metaflow-local-service) - Track Metaflow runs anywhere without a database — starts on demand, stops when idle.\n- [metaflow-serverless](https://github.com/npow/metaflow-serverless) - Serverless Metaflow metadata service — free-tier Postgres, zero setup.\n- [metaflow-service](https://github.com/Netflix/metaflow-service) - Metadata tracking REST API and UI backend.\n- [metaflow-ui](https://github.com/Netflix/metaflow-ui) - React web UI for real-time run monitoring with a plugin system.\n\n---\n\n## Infrastructure \u0026 IaC\n\n- [local_metaflow_deployment](https://github.com/outerbounds/local_metaflow_deployment) - Docker-based local stack: metadata service, PostgreSQL, and UI.\n- [metaflow-docker-deployment](https://github.com/gabrieltardochi/metaflow-docker-deployment) - Community Docker Compose: metadata service + UI + MinIO + PostgreSQL.\n- [metaflow-tools](https://github.com/outerbounds/metaflow-tools) - CloudFormation, Terraform (AWS/Azure/GCP/Nebius), and Helm charts for Kubernetes.\n- [metaflow-with-airflow-minio](https://github.com/outerbounds/metaflow-with-airflow-minio) - Metaflow + Airflow + MinIO on Minikube.\n- [terraform-aws-metaflow](https://github.com/outerbounds/terraform-aws-metaflow) - Official Terraform module for AWS (Batch, S3, RDS, Step Functions, ECS, optional EKS+Argo).\n\n---\n\n## Dependency Management\n\n- [metaflow-nflx-extensions](https://github.com/Netflix/metaflow-nflx-extensions) - Netflix's enhanced `@conda`/`@pypi`: named environments, mixed conda+pip, and faster resolving via micromamba. Requires Metaflow ≥ 2.8.3.\n- [metaflow_extensions](https://github.com/nestauk/metaflow_extensions) - Adds `@pip` and a `preinstall` shell-hook for system-level deps on remote nodes. ⚠️ Built against Metaflow 2.7.x; verify compatibility.\n\n---\n\n## Extension Mechanism \u0026 Templates\n\n- [config-examples](https://github.com/outerbounds/config-examples) - Examples using Metaflow's `Config` object for flow-level configuration.\n- [custom-decorator-examples](https://github.com/outerbounds/custom-decorator-examples) - Examples of user-defined decorators and mutators, including `@memoize` and `BaseFlow` patterns.\n- [metaflow-extensions-template](https://github.com/Netflix/metaflow-extensions-template) - Official template for building `metaflow_extensions` namespace packages.\n- [metaflow-orchestrator-kit](https://github.com/npow/metaflow-orchestrator-kit) - Compliance test suite and scaffold for building correct Metaflow orchestrators.\n\n---\n\n## Orchestration \u0026 Scheduling\n\n- [metaflow-dagster](https://github.com/npow/metaflow-dagster) - Dagster scheduling, observability, and UI for Metaflow pipelines.\n- [metaflow-flyte](https://github.com/npow/metaflow-flyte) - Schedule and monitor Metaflow pipelines through Flyte without rewriting them.\n- [metaflow-kestra](https://github.com/npow/metaflow-kestra) - Kestra scheduling, triggers, and UI for Metaflow pipelines.\n- [metaflow-kubeflow](https://github.com/outerbounds/metaflow-kubeflow) - Deploy Metaflow flows to Kubeflow Pipelines with no infra changes required.\n- [metaflow-mage](https://github.com/npow/metaflow-mage) - Mage pipeline orchestration and UI for Metaflow flows.\n- [metaflow-prefect](https://github.com/npow/metaflow-prefect) - Prefect scheduling, deployments, and UI for Metaflow pipelines.\n- [metaflow-slurm](https://github.com/outerbounds/metaflow-slurm) - Run steps on HPC Slurm clusters via SSH + `sbatch`. Beta.\n- [metaflow-temporal](https://github.com/npow/metaflow-temporal) - Temporal scheduling and durable workflows for Metaflow pipelines.\n- [metaflow-windmill](https://github.com/npow/metaflow-windmill) - Windmill workflow automation and UI for Metaflow pipelines.\n\n---\n\n## Distributed Compute \u0026 Training\n\n- [metaflow-deepspeed](https://github.com/outerbounds/metaflow-deepspeed) - `@deepspeed` for multi-node DeepSpeed training with S3/Azure checkpoint uploads. Experimental.\n- [metaflow-mpi](https://github.com/outerbounds/metaflow-mpi) - `@mpi` for multi-node MPI programs (C/Fortran/mpi4py). Experimental.\n- [metaflow-pyspark](https://github.com/outerbounds/metaflow-pyspark) - PySpark decorator for Metaflow steps. Experimental, low adoption.\n- [metaflow-ray](https://github.com/outerbounds/metaflow-ray) - Ephemeral Ray clusters on AWS Batch or Kubernetes. Supports Ray Core, Train, Tune, and Data.\n- [metaflow-sandbox](https://github.com/npow/metaflow-sandbox) - Metaflow steps in millisecond-start sandboxes with cloud-scale fanout and consistent deps.\n- [metaflow-tensorflow](https://github.com/outerbounds/metaflow-tensorflow) - `@tensorflow` that auto-configures `TF_CONFIG` for `tf.distribute.Strategy`. Experimental.\n- [metaflow-torchrun](https://github.com/outerbounds/metaflow-torchrun) - Run tasks as nodes in a `torchrun` DDP job on Batch or Kubernetes.\n\n---\n\n## Model \u0026 Artifact Management\n\n- [metaflow-checkpoint](https://github.com/outerbounds/metaflow-checkpoint) - `@checkpoint`, `@model`, and `@huggingface_hub` for fault-tolerant training and model lineage. APIs may change.\n- [metaflow-checkpoint-examples](https://github.com/outerbounds/metaflow-checkpoint-examples) - Examples covering PyTorch, Keras, Lightning, and distributed DDP.\n\n---\n\n## Observability \u0026 Monitoring\n\n- [metaflow-gpu-profile](https://github.com/outerbounds/metaflow-gpu-profile) - `@gpu_profile` decorator that renders GPU utilization as a Metaflow card.\n- [metaflow-measure](https://github.com/outerbounds/metaflow-measure) - Emit step metrics to Datadog and other backends via a `measure` API.\n- [metaflow-profiler](https://github.com/npow/metaflow-profiler) - Flamegraph profiling card for Metaflow steps.\n- [metaflow-sentry-logger](https://github.com/rsmith013/metaflow-sentry-logger) - Sentry logging via `@sentry`. ⚠️ Relies on an unsupported extension API; broken as of Metaflow 2.7.20.\n- [metaflowbot](https://github.com/outerbounds/metaflowbot) - Slack bot for real-time flow monitoring with CloudFormation deploy. ⚠️ Older; verify against current Metaflow.\n- [resource-tracker](https://github.com/SpareCores/resource-tracker) - Zero-dependency CPU/memory/GPU tracker with Metaflow card output and cloud cost recommendations.\n\n---\n\n## Cards \u0026 Visualization\n\n- [dynamic-card-examples](https://github.com/outerbounds/dynamic-card-examples) - Live cards with progress bars, Altair/Vega charts, maps, and custom JS.\n- [metaflow-card-altair](https://github.com/outerbounds/metaflow-card-altair) - Render Altair charts in Metaflow cards.\n- [metaflow-card-html](https://github.com/outerbounds/metaflow-card-html) - Render raw HTML as a card. Reference implementation for custom card authors. ⚠️ Community reports of breakage.\n- [metaflow-card-notebook](https://github.com/outerbounds/metaflow-card-notebook) - Render Jupyter Notebooks as Metaflow cards. ⚠️ Last commit 2022.\n- [metaflow-card-scatter3d](https://github.com/outerbounds/metaflow-card-scatter3d) - Live 2D and 3D scatter plot card.\n- [metaflow-card-template](https://github.com/outerbounds/metaflow-card-template) - Starter template for building new card types.\n- [metaflow-dataprofiler](https://github.com/npow/metaflow-dataprofiler) - Instant EDA reports on every DataFrame in your Metaflow steps — zero code changes.\n- [metaflow-traincard](https://github.com/npow/metaflow-traincard) - Live loss curves, GPU telemetry, and checkpoints in your Metaflow run card.\n\n---\n\n## Third-Party Integrations\n\n- [airflow-metaflow-demo](https://github.com/astronomer/airflow-metaflow-demo) - Metaflow + Airflow in Docker Compose with `KubernetesPodOperator` steps.\n- [comet_ml](https://www.comet.com/docs/v2/integrations/third-party-tools/metaflow/) - `@comet_flow` and `@comet_step` ship in the `comet_ml` package, including automatic Card export.\n- [hamilton-metaflow](https://github.com/outerbounds/hamilton-metaflow) - Hamilton as a feature engineering layer inside Metaflow steps.\n- [sap-ai-core-metaflow](https://pypi.org/project/sap-ai-core-metaflow/) - Generates Argo Workflow Templates from Metaflow flows for SAP AI Core.\n- [wandb](https://docs.wandb.ai/models/integrations/metaflow) - `@wandb_log` ships in the `wandb` package for experiment tracking.\n- [zdatasets](https://github.com/zillow/zdatasets) - Zillow's Dataset SDK with a `DatasetParameter` integration for Metaflow flows.\n\n---\n\n## Developer Tooling\n\n- [gha-metaflow](https://github.com/outerbounds/gha-metaflow) - GitHub Actions workflows that trigger Metaflow runs on push/PR.\n- [metaflow-contracts](https://github.com/npow/metaflow-contracts) - Catch bad data between Metaflow steps before it corrupts your pipeline.\n- [metaflow-dev-vscode](https://github.com/outerbounds/metaflow-dev-vscode) - VS Code extension with shortcuts for running flows and steps from the editor.\n- [metaflow-diff](https://github.com/outerbounds/metaflow-diff) - Diff your working directory against a past Metaflow run.\n- [metaflow-mcp-server](https://github.com/npow/metaflow-mcp-server) - MCP server for inspecting runs, logs, and artifacts from any AI agent.\n- [metaflow-optuna](https://github.com/npow/metaflow-optuna) - Parallel hyperparameter tuning with true adaptive TPE — no sequential bottleneck.\n- `metaflow-stubs` - Type stubs for IDE autocompletion. `pip install metaflow-stubs`\n\n---\n\n## Examples \u0026 Tutorials\n\n- [diffusion-metaflow](https://github.com/outerbounds/diffusion-metaflow) - Stable Diffusion text-to-image and text-to-video pipelines.\n- [dsbook](https://github.com/outerbounds/dsbook) - Code for *Effective Data Science Infrastructure* by Ville Tuulos.\n- [full-stack-ML-metaflow-tutorial](https://github.com/outerbounds/full-stack-ML-metaflow-tutorial) - Workshop: prototype to production ML with dbt, Great Expectations, W\u0026B, and SageMaker.\n- [hacker-news-sentiment](https://github.com/outerbounds/hacker-news-sentiment) - LLM-powered topic and sentiment analysis on Hacker News data.\n- [metaflow-instruction-tuning](https://github.com/outerbounds/metaflow-instruction-tuning) - LLM instruction tuning (Alpaca-LoRA, LLaMA 7B) with `@torchrun`.\n- [metaflow-trainium](https://github.com/outerbounds/metaflow-trainium) - AWS Trainium examples: LLaMA-2 pretraining, BERT fine-tuning.\n- [metaflow-transformers-tutorials](https://github.com/chiphuyen/metaflow-transformers-tutorials) - HuggingFace DistilBERT fine-tuning tutorials by Chip Huyen.\n- [post-modern-stack](https://github.com/jacopotagliabue/post-modern-stack) - Modern data stack + ML stack using dbt, SageMaker, and Metaflow.\n- [rag-demo](https://github.com/outerbounds/rag-demo) - End-to-end RAG pipeline with LlamaIndex and LanceDB/Pinecone.\n- [recs-at-reasonable-scale](https://github.com/jacopotagliabue/recs-at-reasonable-scale) - Recommendations at scale with dbt, NVIDIA Merlin, and Metaflow.\n- [triton-metaflow-starter-pack](https://github.com/outerbounds/triton-metaflow-starter-pack) - NVIDIA Triton Inference Server starter pack.\n- [tutorials](https://github.com/outerbounds/tutorials) - Official tutorials on workflow graphs, versioning, data access, and scheduling.\n- [user2020-metaflow-tutorial](https://github.com/Netflix/user2020-metaflow-tutorial) - R tutorial from useR! 2020.\n- [you-dont-need-a-bigger-boat](https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat) - End-to-end RecSys with Metaflow, Snowflake, SageMaker, dbt, and W\u0026B.\n\n---\n\n## Contributing\n\n1. Project must be Metaflow-related with a clear install path\n2. One-line description only\n3. Flag experimental/broken items with ⚠️\n4. No item without a working link\n\n---\n\n## License\n\n[![CC0](https://mirrors.creativecommons.org/presskit/buttons/88x31/svg/cc-zero.svg)](https://creativecommons.org/publicdomain/zero/1.0/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnpow%2Fawesome-metaflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnpow%2Fawesome-metaflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnpow%2Fawesome-metaflow/lists"}