{"id":15650713,"url":"https://github.com/klieret/wandb-offline-sync-hook","last_synced_at":"2025-05-07T20:09:23.856Z","repository":{"id":65347334,"uuid":"561498902","full_name":"klieret/wandb-offline-sync-hook","owner":"klieret","description":"A convenient way to trigger synchronizations to wandb / Weights \u0026 Biases if your compute nodes don't have internet! ","archived":false,"fork":false,"pushed_at":"2025-05-05T21:09:49.000Z","size":248,"stargazers_count":78,"open_issues_count":12,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-05T22:25:57.511Z","etag":null,"topics":["hyperparameter-optimization","hyperparameter-tuning","machine-learning","ray","ray-tune","wandb","weights-and-biases"],"latest_commit_sha":null,"homepage":"https://wandb-offline-sync-hook.rtfd.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/klieret.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-03T20:32:28.000Z","updated_at":"2025-04-29T15:43:11.000Z","dependencies_parsed_at":"2023-11-10T11:27:49.486Z","dependency_job_id":"216da843-624a-411c-82b1-fa786080885f","html_url":"https://github.com/klieret/wandb-offline-sync-hook","commit_stats":{"total_commits":165,"total_committers":5,"mean_commits":33.0,"dds":0.296969696969697,"last_synced_commit":"9f18e6bda1a21f7329a523d87a7acf6d6fbcd3ed"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klieret%2Fwandb-offline-sync-hook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klieret%2Fwandb-offline-sync-hook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klieret%2Fwandb-offline-sync-hook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/klieret%2Fwandb-offline-sync-hook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/klieret","download_url":"https://codeload.github.com/klieret/wandb-offline-sync-hook/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252586861,"owners_count":21772363,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hyperparameter-optimization","hyperparameter-tuning","machine-learning","ray","ray-tune","wandb","weights-and-biases"],"created_at":"2024-10-03T12:35:35.558Z","updated_at":"2025-05-07T20:09:23.807Z","avatar_url":"https://github.com/klieret.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eWandb Offline Sync Hook\u003c/h1\u003e\n\u003cem\u003eA convenient way to trigger synchronizations to wandb if your compute nodes don't have internet!\u003c/em\u003e\n\u003cp\u003e\u003c/p\u003e\n\n[![Documentation Status](https://readthedocs.org/projects/wandb-offline-sync-hook/badge/?version=latest)](https://wandb-offline-sync-hook.readthedocs.io/en/latest/?badge=latest)\n[![PyPI version](https://badge.fury.io/py/wandb-osh.svg)](https://pypi.org/project/wandb-osh)\n[![Python 3.8‒3.11](https://img.shields.io/badge/python-3.8%E2%80%923.11-blue)](https://www.python.org)\n[![PR welcome](https://img.shields.io/badge/PR-Welcome-%23FF8300.svg)](https://git-scm.com/book/en/v2/GitHub-Contributing-to-a-Project)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/klieret/wandb-offline-sync-hook/main.svg)](https://results.pre-commit.ci/latest/github/klieret/wandb-offline-sync-hook/main)\n[![.github/workflows/test.yaml](https://github.com/klieret/wandb-offline-sync-hook/actions/workflows/test.yaml/badge.svg)](https://github.com/klieret/wandb-offline-sync-hook/actions/workflows/test.yaml)\n[![link checker](https://github.com/klieret/wandb-offline-sync-hook/actions/workflows/check-links.yaml/badge.svg)](https://github.com/klieret/wandb-offline-sync-hook/actions)\n[![codecov](https://codecov.io/github/klieret/wandb-offline-sync-hook/branch/main/graph/badge.svg?token=6MQZ4LODE5)](https://app.codecov.io/github/klieret/wandb-offline-sync-hook)\n[![gitmoji](https://img.shields.io/badge/gitmoji-%20😜%20😍-FFDD67.svg)](https://gitmoji.dev)\n[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black)\n\n\u003c/div\u003e\n\n## 🤔 What is this?\n\n- ✅ You use [`wandb`/Weights \u0026 Biases](https://wandb.ai/) to record your machine learning trials?\n- ✅ Your ML experiments run on compute nodes without internet access (for example, using a batch system)?\n- ✅ Your compute nodes and your head/login node (with internet) have access to a shared file system?\n\nThen this package might be useful. For alternatives, see [below](https://github.com/klieret/wandb-offline-sync-hook#what-alternatives-are-there).\n\n\u003cdiv align=\"center\"\u003e\n\n![](https://user-images.githubusercontent.com/13602468/200086359-507b8653-e999-4cb3-ac93-ba1d175d2016.png)\n\n\u003c/div\u003e\n\n### What you might have been doing so far\n\nYou probably have been using `export WANDB_MODE=\"offline\"` on the compute nodes and then ran something like\n\n```bash\ncd /.../result_dir/\nfor d in $(ls -t -d */); do cd $d; wandb sync --sync-all; cd ..; done\n```\n\nfrom your head node (with internet access) every now and then.\nHowever, obviously this is not very satisfying as it doesn't update live.\nSure, you could throw this in a `while True` loop, but if you have a lot of trials in your directory, this will take forever, [cause unnecessary network traffic](https://github.com/wandb/wandb/issues/2887) and it's just not very elegant.\n\n### How does `wandb-osh` solve the problem?\n\n1. You add a hook that is called every time an epoch concludes (that is, when we want to trigger a sync).\n2. You start the `wandb-osh` script in your head node with internet access. This script will now trigger `wandb sync` upon request from one of the compute nodes.\n\n### How is this implemented?\n\nVery simple: Every time an epoch concludes, the hook gets called and creates a file in the _communication directory_ (`~/.wandb_osh_communication` by default).\nThe `wandb-osh` script that is running on the head node (with internet) reads these files and performs the synchronization.\n\n### What alternatives are there?\n\nWith [ray tune][ray-tune], you can use your ray head node as the place to synchronize from (rather than deploying it via the batch system as well, as the [current docs][ray-tune-slurm-docs] suggest). See the note below or my [demo repository][ray-tune-slurm-test].\nSimilar strategies might be possible for `wandb` as well (let me know!).\n\n## 📦 Installation\n\n```bash\npip3 install wandb-osh\n```\n\nFor completeness, the extra dependencies `lightning` and `ray` are given, but they only ensure that the corresponding package is installed.\nFor example\n\n```bash\npip3 install 'wandb-osh[lightning]'\n```\n\nalso installs pytorch lightning if it is not already present, but has no other effect.\n\nFor development, make sure also to include the `testing` extra requirement.\n\n```bash\npip3 install --editable '.[testing]'\n```\n\n## 🔥 Running it!\n\nTwo steps: Set up the hook, then run the script from your head node.\n\n### Step 1: Setting up the hook\n\n\u003cdetails\u003e\n\u003csummary\u003eWith pure wandb\u003c/summary\u003e\n\nLet's adapt the [simple pytorch example](https://docs.wandb.ai/guides/integrations/pytorch) from the wandb docs (it only takes 3 lines!):\n\n```python\nimport wandb\nfrom wandb_osh.hooks import TriggerWandbSyncHook  # \u003c-- New!\n\n\ntrigger_sync = TriggerWandbSyncHook()  # \u003c--- New!\n\nwandb.init(config=args, mode=\"offline\")\n\nmodel = ... # set up your model\n\n# Magic\nwandb.watch(model, log_freq=100)\n\nmodel.train()\nfor batch_idx, (data, target) in enumerate(train_loader):\n    output = model(data)\n    loss = F.nll_loss(output, target)\n    loss.backward()\n    optimizer.step()\n    if batch_idx % args.log_interval == 0:\n        wandb.log({\"loss\": loss})\n        trigger_sync()  # \u003c-- New!\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eWith pytorch lightning\u003c/summary\u003e\n\nSimply add the `TriggerWandbSyncLightningCallback` to your list of callbacks and you're good to go!\n\n```python\nfrom wandb_osh.lightning_hooks import TriggerWandbSyncLightningCallback  # \u003c-- New!\nfrom pytorch_lightning.loggers import WandbLogger\nfrom pytorch_lightning import Trainer\n\nlogger = WandbLogger(\n    project=\"project\",\n    group=\"group\",\n    offline=True,\n)\n\nmodel = MyLightningModule()\ntrainer = Trainer(\n    logger=logger,\n    callbacks=[TriggerWandbSyncLightningCallback()]  # \u003c-- New!\n)\ntrainer.fit(model, train_dataloader, val_dataloader)\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eWith ray tune\u003c/summary\u003e\n\n\u003e **Note**\n\u003e With ray tune, you might not need this package! While the approach suggested in the\n\u003e [ray tune SLURM docs][ray-tune-slurm-docs] deploys the ray head on a worker node as well (so it doesn't\n\u003e have internet), this actually isn't needed. Instead, you can run the ray head and the\n\u003e tuning script on the head node and only submit batch jobs for your workers.\n\u003e In this way, `wandb` will be called from the head node and internet access is no\n\u003e problem there.\n\u003e For more information on this approach, take a look at my [demo repository][ray-tune-slurm-test].\n\nYou probably already use the `WandbLoggerCallback` callback. We simply add a second callback for `wandb-osh` (it only takes two new lines!):\n\n```python\nimport os\nfrom wandb_osh.ray_hooks import TriggerWandbSyncRayHook  # \u003c-- New!\n\n\nos.environ[\"WANDB_MODE\"] = \"offline\"\n\ncallbacks = [\n    WandbLoggerCallback(...),  # \u003c-- ray tune documentation tells you about this\n    TriggerWandbSyncRayHook(),  # \u003c-- New!\n]\n\ntuner = tune.Tuner(\n    trainable,\n    tune_config=...,\n    run_config=RunConfig(\n        ...,\n        callbacks=callbacks,\n    ),\n)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWith anything else\u003c/summary\u003e\n\nSimply take the `TriggerWandbSyncHook` class and use it as a callback in your training\nloop (as in the `wandb` example above), passing the directory that `wandb` is syncing\nto as an argument.\n\n\u003c/details\u003e\n\n### Step 2: Running the script on the head node\n\nAfter installation, you should have a `wandb-osh` script in your `$PATH`. Simply call it like this:\n\n```\nwandb-osh\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eThe output will look something like this\u003c/summary\u003e\n\n```\nINFO: Starting to watch /home/kl5675/.wandb_osh_command_dir\nINFO: Syncing /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_b1f60706_4_attr_pt_thld=0.0273,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-08-42\nFind logs at: /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_b1f60706_4_attr_pt_thld=0.0273,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-08-42/wandb/debug-cli.kl5675.log\nSyncing: https://wandb.ai/gnn_tracking/gnn_tracking/runs/b1f60706 ... done.\nINFO: Finished syncing /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_b1f60706_4_attr_pt_thld=0.0273,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-08-42\nINFO: Syncing /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_92a3ef1b_1_attr_pt_thld=0.0225,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-07-49\nFind logs at: /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_92a3ef1b_1_attr_pt_thld=0.0225,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-07-49/wandb/debug-cli.kl5675.log\nSyncing: https://wandb.ai/gnn_tracking/gnn_tracking/runs/92a3ef1b ... done.\nINFO: Finished syncing /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_92a3ef1b_1_attr_pt_thld=0.0225,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-07-49\nINFO: Syncing /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_a2caa9c0_2_attr_pt_thld=0.0092,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-08-17\nFind logs at: /home/kl5675/ray_results/tcn-perfect-test-sync/DynamicTCNTrainable_a2caa9c0_2_attr_pt_thld=0.0092,batch_size=1,focal_alpha=0.2500,focal_gamma=2.0000,gnn_tracking_experiments_has_2022-11-03_17-08-17/wandb/debug-cli.kl5675.log\nSyncing: https://wandb.ai/gnn_tracking/gnn_tracking/runs/a2caa9c0 ... done.\n```\n\n\u003c/details\u003e\n\nTake a look at `wandb-osh --help` or check [the documentation](https://wandb-offline-sync-hook.readthedocs.io/en/latest/cli.html) for all command line options.\nYou can add options to the `wandb sync` call by placing them after `--`. For example\n\n```bash\nwandb-osh -- --sync-all\n```\n\n## ❓ Q \u0026 A\n\n\u003e I get the warning \"wandb: NOTE: use wandb sync --sync-all to sync 1 unsynced runs from local directory.\"\n\nYou can start `wandb-osh` with `wandb-osh -- --sync-all` to always synchronize\nall available runs.\n\n\u003e How can I suppress logging messages (e.g., warnings about the syncing not being fast enough)\n\n```python3\nimport wandb_osh\n\n# for wandb_osh.__version__ \u003e= 1.2.0\nwandb_osh.set_log_level(\"ERROR\")\n```\n\n## 🧰 Development setup\n\n```bash\npip3 install pre-commit\npre-commit install\n```\n\n## 💖 Contributing\n\nYour help is greatly appreciated! Suggestions, bug reports and feature requests are best opened as [github issues][github-issues]. You are also very welcome to submit a [pull request][pulls]!\n\nBug reports and pull requests are credited with the help of the [allcontributors bot](https://allcontributors.org/).\n\n\u003c!-- ## ✨ Contributors --\u003e\n\u003c!--  --\u003e\n\u003c!-- Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)): --\u003e\n\u003c!--  --\u003e\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/barthelemymp\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/36533835?v=4?s=100\" width=\"100px;\" alt=\"Barthelemy Meynard-Piganeau\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eBarthelemy Meynard-Piganeau\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/klieret/wandb-offline-sync-hook/issues?q=author%3Abarthelemymp\" title=\"Bug reports\"\u003e🐛\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/MoH-assan\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/123608862?v=4?s=100\" width=\"100px;\" alt=\"MoH-assan\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eMoH-assan\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/klieret/wandb-offline-sync-hook/issues?q=author%3AMoH-assan\" title=\"Bug reports\"\u003e🐛\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/CedricLeon\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/51703091?v=4?s=100\" width=\"100px;\" alt=\"Cedric Leonard\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eCedric Leonard\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/klieret/wandb-offline-sync-hook/commits?author=CedricLeon\" title=\"Code\"\u003e💻\u003c/a\u003e \u003ca href=\"https://github.com/klieret/wandb-offline-sync-hook/issues?q=author%3ACedricLeon\" title=\"Bug reports\"\u003e🐛\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\u003c!--  --\u003e\n\u003c!-- This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome! --\u003e\n\n[github-issues]: https://github.com/klieret/wandb-offline-sync-hook/issues\n[pulls]: https://github.com/klieret/wandb-offline-sync-hook/pulls\n[ray-tune-slurm-docs]: https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html\n[ray-tune-slurm-test]: https://github.com/klieret/ray-tune-slurm-test/\n[ray-tune]: https://docs.ray.io/en/latest/tune/index.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklieret%2Fwandb-offline-sync-hook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fklieret%2Fwandb-offline-sync-hook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fklieret%2Fwandb-offline-sync-hook/lists"}