{"id":19329720,"url":"https://github.com/outerbounds/metaflow-deepspeed","last_synced_at":"2025-07-31T09:11:19.881Z","repository":{"id":204227703,"uuid":"702761070","full_name":"outerbounds/metaflow-deepspeed","owner":"outerbounds","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-17T01:00:04.000Z","size":157,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-22T21:49:15.665Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-10T00:55:30.000Z","updated_at":"2024-10-17T00:59:34.000Z","dependencies_parsed_at":"2024-02-29T03:22:16.247Z","dependency_job_id":"8e7b419c-60d5-4d82-bf2c-fefac152cb07","html_url":"https://github.com/outerbounds/metaflow-deepspeed","commit_stats":null,"previous_names":["outerbounds/metaflow-deepspeed"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/outerbounds/metaflow-deepspeed","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-deepspeed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-deepspeed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-deepspeed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-deepspeed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/metaflow-deepspeed/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-deepspeed/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268016992,"owners_count":24181657,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-31T02:00:08.723Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T02:29:33.299Z","updated_at":"2025-07-31T09:11:19.832Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":["Distributed Compute \u0026 Training"],"sub_categories":[],"readme":"### Introduction\n[Deepspeed](https://www.deepspeed.ai/) is a highly scalable framework from Microsoft for distributed training and model serving. The Metaflow `@deepspeed` decorator helps you run these workflows inside of Metaflow tasks. \n\n### Features\n- **Automatic SSH configuration**: [Multi-node Deepspeed jobs are built around OpenMPI or Horovod](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node). Like Metaflow's [`@mpi` decorator](https://github.com/outerbounds/metaflow-mpi), the `@deepspeed` decorator automatically configures SSH requirements between nodes, so you can focus on research code.\n- **Seamless Python interface**: Metaflow's `@deepspeed` exposes a method `current.deepspeed.run` to make it easy to run Deepspeed commands on your transient MPI cluster, in the same way you'd launch Deepspeed from the terminal independent of Metaflow. A major design goal is to get the orchestration and other benefits of Metaflow, without requiring modification to research code.\n\n### Installation\nInstall this experimental module:\n```\npip install metaflow-deepspeed\n```\n\n### Getting Started\nAfter installing the module, you can import the `deepspeed` decorator and use it in your Metaflow steps.\nThis exposes the `current.deepspeed.run` method, which you can map your terminal commands for running Deepspeed. \n\n```python\nfrom metaflow import FlowSpec, step, deepspeed, current, batch, environment\n\nclass HelloDeepspeed(FlowSpec):\n\n    @step\n    def start(self):\n        self.next(self.train, num_parallel=2)\n\n    @environment(vars={\"NCCL_SOCKET_IFNAME\": \"eth0\"})\n    @batch(gpu=8, cpu=64, memory=256000)\n    @deepspeed\n    @step\n    def train(self):\n        current.deepspeed.run(\n            entrypoint=\"my_torch_dist_script.py\"\n        )\n        self.next(self.join)\n\n    @step\n    def join(self, inputs):\n        self.next(self.end)\n\n    @step\n    def end(self):\n        pass\n        \nif __name__ == \"__main__\":\n    HelloDeepspeed()\n```\n\n### Examples\n| Directory | MPI program description |\n| :--- | ---: |\n| [CPU Check](examples/cpu-only-check/README.md) | The easiest way to check your Deepspeed infrastructure on CPUs. |\n| [Hello Deepspeed](examples/hello/README.md) | The easiest way to check your Deepspeed infrastructure on GPUs. |  \n| [BERT](examples/bert/README.md) | Train your BERT model using Deepspeed! |  \n| [Dolly](examples/dolly/README.md) | A multi-node implementation of [Databricks' Dolly](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm). |  \n\n#### Cloud-specific use cases\n| Directory | MPI program description |\n| :--- | ---: |\n| [Automatically upload a directory on AWS](examples/upload-output-dir-aws/README.md) | Push a checkpoint of any directory to S3 after the Deepspeed process completes. |\n| [Automatically upload a directory on Azure](examples/upload-output-dir-azure/README.md) | Push a checkpoint of any directory to Azure Blob storage after the Deepspeed process completes. |\n| [Use Metaflow S3 client from the Deepspeed process](examples/metaflow-s3-from-script/README.md) | Upload arbitrary bytes to S3 storage from the Deepspeed process. |  \n| [Use Metaflow Azure Blob client from the Deepspeed process](examples/metaflow-azureblob-from-script/README.md) | Upload arbitrary bytes to Azure Blob storage from the Deepspeed process. |  \n| [Use a Metaflow Huggingface checkpoint on S3](examples/metaflow-s3-hf-trainer-callback/README.md) | Push a checkpoint to S3 at the end of each epoch using a customizable Huggingface callback. See the implementation [here](./metaflow_extensions/deepspeed/plugins/hf_callbacks.py) to build your own. |  \n| [Use a Metaflow Huggingface checkpoint on Azure](examples/metaflow-azure-hf-trainer-callback/README.md) | Push a checkpoint to Azure Blob storage at the end of each epoch using a customizable Huggingface callback. See the implementation [here](./metaflow_extensions/deepspeed/plugins/hf_callbacks.py) to build your own. |  \n\n\n### License\n`metaflow-deepspeed` is distributed under the \u003cu\u003eApache License\u003c/u\u003e.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-deepspeed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Fmetaflow-deepspeed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-deepspeed/lists"}