{"id":29656544,"url":"https://github.com/zincware/laufband","last_synced_at":"2025-07-22T08:35:59.362Z","repository":{"id":290077670,"uuid":"973301709","full_name":"zincware/laufband","owner":"zincware","description":"Parallel Iteration with File-Based Coordination","archived":false,"fork":false,"pushed_at":"2025-07-21T19:41:14.000Z","size":255,"stargazers_count":7,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-21T21:36:09.402Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zincware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-26T17:38:39.000Z","updated_at":"2025-06-16T15:12:29.000Z","dependencies_parsed_at":"2025-05-20T15:31:19.185Z","dependency_job_id":"e1339bf0-daa7-4725-a6f5-5ac68083ec3a","html_url":"https://github.com/zincware/laufband","commit_stats":null,"previous_names":["zincware/laufband"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/zincware/laufband","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zincware%2Flaufband","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zincware%2Flaufband/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zincware%2Flaufband/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zincware%2Flaufband/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zincware","download_url":"https://codeload.github.com/zincware/laufband/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zincware%2Flaufband/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266456406,"owners_count":23931404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-22T08:35:53.952Z","updated_at":"2025-07-22T08:35:59.335Z","avatar_url":"https://github.com/zincware.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n![Logo](https://github.com/user-attachments/assets/c8d2a3f9-284b-474d-b46d-98612c9d266b)\n\u003c/div\u003e\n\n# Laufband: Embarrassingly parallel, embarrassingly simple!\n[![codecov](https://codecov.io/gh/zincware/laufband/graph/badge.svg?token=9DJ3YZGTBA)](https://codecov.io/gh/zincware/laufband)\n\nLaufband enables parallel iteration over a dataset from multiple processes, utilizing file-based locking and communication to ensure each item is processed exactly once.\n\n## Installation\n\nInstall Laufband using pip:\n\n```bash\npip install laufband\n```\n\n## Usage\n\nUsing Laufband is similar to the familiar `tqdm` progress bar for sequential iteration.\n\n```python\nfrom laufband import Laufband\n\ndata = list(range(100))\nfor item in Laufband(data):\n    # Process each item in the dataset\n    pass\n```\n\nThe true power of Laufband emerges when you run your script in parallel. Multiple processes will coordinate using file-based locking to ensure that each item in the dataset is processed by only one process.\n\nHere's a typical example demonstrating parallel processing with Laufband and file-based locking for shared resource access:\n\n```python\nimport json\nimport time\nfrom pathlib import Path\nfrom laufband import Laufband\n\noutput_file = Path(\"data.json\")\noutput_file.write_text(json.dumps({\"processed_data\": []}))\ndata = list(range(100))\n\nworker = Laufband(data, desc=\"using Laufband\")\n\nfor item in worker:\n    # Simulate some computationally intensive task\n    time.sleep(0.1)\n    with worker.lock:\n        # Access and modify a shared resource (e.g., a file) safely using the lock\n        file_content = json.loads(output_file.read_text())\n        file_content[\"processed_data\"].append(item)\n        output_file.write_text(json.dumps(file_content))\n```\n\nTo execute this script (`main.py`) in parallel, you can use a command like the following in your terminal (this example launches 10 background processes):\n\n```bash\nfor i in {1..10} ; do python main.py \u0026 done\n```\n\n\u003e [!IMPORTANT]\n\u003e The different processes may finish at different times. Therefore, the order of items in `file_content` is not guaranteed.\n\u003e If the order is important, you will need to implement sorting logic afterwards.\n\n### Failure Policy\n\nIn Laufband, a job will be automatically marked as failed if the iteration is interrupted by:\n- an unhandled Exception\n- or an explicit break.\n\n```python\nfrom laufband import Laufband\n\ndata = list(range(100))\n\n# Example 1: break\nfor item in Laufband(data):\n    if item == 50:\n        break  # Job 50 will be marked as failed\n\n# Example 2: Exception\nfor item in Laufband(data):\n    if item == 70:\n        raise ValueError(\"Something went wrong\")  # Job 70 will be marked as failed\n```\n\nIf you want to exit early but still mark the job as successfully completed,\nyou should use `Laufband.close()` instead of `break`:\n\n```python\nfrom laufband import Laufband\n\ndata = list(range(100))\n\nworker = Laufband(data)\n\nfor item in worker:\n    if item == 50:\n        worker.close()  # Job 50 will be marked as completed, and iteration will stop cleanly\n```\n\n\n# Examples\n\n## ASE Calculator\nFor atomistic data, the ASE package is widely used to calculate energies and forces of atomic configurations using either _ab initio_ methods or machine-learned interatomic potentials (MLIPs).\n\nYou can use Laufband to parallelize these calculations easily without duplication or manual bookkeeping and automatic checkpointing.\n\nThe following example uses a MACE foundation model to compute energies and forces on the ASE S22 dataset.\n\n\u003e [!TIP]\n\u003e You can safely run this script multiple times — even across multiple SLURM jobs — without any modifications.\n\u003e Laufband will automatically coordinate which configurations are processed.\n\u003e For local parallelization, you can use bash: `for i in {1..10} ; do python main.py \u0026 done`\n\n```python\nimport ase.io\nfrom ase.collections import s22\nfrom laufband import Laufband\nfrom mace.calculators import mace_mp\n\n# Initialize calculator\ncalc = mace_mp(model=\"medium\", dispersion=False, default_dtype=\"float32\")\n\nworker = Laufband(list(s22))\n\nfor atoms in worker:\n    atoms.calc = calc\n    atoms.get_potential_energy()\n    with worker.lock:\n        ase.io.write(\"frames.xyz\", atoms, append=True)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzincware%2Flaufband","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzincware%2Flaufband","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzincware%2Flaufband/lists"}