{"id":19329748,"url":"https://github.com/outerbounds/metaflow-directory-sync-examples","last_synced_at":"2026-06-08T21:33:22.799Z","repository":{"id":221619040,"uuid":"754812144","full_name":"outerbounds/metaflow-directory-sync-examples","owner":"outerbounds","description":null,"archived":false,"fork":false,"pushed_at":"2024-02-09T04:17:47.000Z","size":8,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-24T06:46:39.222Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-08T20:10:42.000Z","updated_at":"2024-03-13T00:34:06.000Z","dependencies_parsed_at":"2024-12-09T18:16:52.863Z","dependency_job_id":null,"html_url":"https://github.com/outerbounds/metaflow-directory-sync-examples","commit_stats":null,"previous_names":["outerbounds/metaflow-directory-sync-examples"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/outerbounds/metaflow-directory-sync-examples","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-directory-sync-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-directory-sync-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-directory-sync-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-directory-sync-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/metaflow-directory-sync-examples/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fmetaflow-directory-sync-examples/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34082130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T02:29:45.068Z","updated_at":"2026-06-08T21:33:22.784Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repository contains a Metaflow tool you can use inside `@step` functions, that will sync a local directory with S3 periodically, in a background thread.\n\nThis is particularly useful when with long-running training jobs that save checkpoints at long intervals, and you want to establish a two-way sync to ensure that those checkpoints are being pushed to S3. \n\nThe repository contains three examples. Each example folder contains the file `dir_sync.py`. To use it, place this file in the same directory as your Metaflow flow and import the `DirectorySyncManager` in \n1. [Example without Metaflow](./basic_example/): How to use the `DirectorySyncManager` in Python code.\n2. [Example with Metaflow](./basic_metaflow_example/): How to use the `DirectorySyncManager` in a Metaflow workflow.\n3. [Example with HuggingFace Trainer Workflow](./hf_trainer_example/): How to use the `DirectorySyncManager` in a Metaflow workflow that does multinode HuggingFace Training.\n\nThe steps to use the `DirectorySyncManager` are:\n1. Create the object `manager = DirectorySyncManager(...)` using these arguments.\n    - `root`: You must set this to the directory on the `@step` you want to sync to S3.\n    - `s3_root`: Specify this or `run`. This can be any S3 path your task can write to.\n    - `run`: Specify this or `s3_root`. This is the Metaflow Run object. You can use `self` when set inside a `FlowSpec`.\n    - `interval`: How often to check for changes in the `root`, and if detected, push the new contents to S3.\n    - `node_index`: Use this if you are running a `num_parallel` job in the Metaflow task. Set it with `current.parallel.node_index`.\n2. Start the manager `manager.start()`.\n3. Run the code you want to sync during.\n4. Stop the manager `manager.stop()`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-directory-sync-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Fmetaflow-directory-sync-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fmetaflow-directory-sync-examples/lists"}