{"id":23478156,"url":"https://github.com/mtli/parscript","last_synced_at":"2025-04-14T21:31:32.973Z","repository":{"id":41165397,"uuid":"186507099","full_name":"mtli/parscript","owner":"mtli","description":"Parallel or distributed execution of jobs :parking:","archived":false,"fork":false,"pushed_at":"2022-08-09T00:58:04.000Z","size":18,"stargazers_count":5,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-28T09:36:28.239Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mtli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-13T22:56:59.000Z","updated_at":"2024-03-05T18:00:31.000Z","dependencies_parsed_at":"2022-07-25T21:30:05.754Z","dependency_job_id":null,"html_url":"https://github.com/mtli/parscript","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtli%2Fparscript","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtli%2Fparscript/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtli%2Fparscript/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mtli%2Fparscript/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mtli","download_url":"https://codeload.github.com/mtli/parscript/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248643043,"owners_count":21138353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-24T19:16:37.085Z","updated_at":"2025-04-14T21:31:32.956Z","avatar_url":"https://github.com/mtli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parscript: Parallel or distributed execution of jobs\n\nSimple concept: write your jobs (shell, python, batch or anything!) in a txt file, one job one line!\n```\npython train.py --name=exp1 --param=1\npython train.py --name=exp2 --param=2\npython train.py --name=exp3 --param=3\n```\n\nAnd use `python -m parscript.worker job-list.txt` to run the jobs sequentially, or `python -m parscript.dispatcher job-list.txt` to run them in parallel or distributed fashion.\n\nUsage:\n- Specify the number of GPUs through `-g` (default 1)\n- Specify the number of workers (per GPU) through `-w` (default 1)\n- Reset the job counter through `-r`\n- Directly edit `job-list.txt` after launched to add more jobs to it\n- If one job fails, it will not affect other jobs and failed jobs will be recorded and reported at the end.\n- Use `-s` to shutdown the machine after all jobs are finished (useful for running jobs on AWS).\n- Specify which GPUs using `CUDA_VISIBLE_DEVICES`. You still need to set `-g` correspondingly for the dispatcher.\n\nFor example:\n`python -m parscript.dispatcher job-list.txt -g 4 -w 2` means running 8 jobs at a time from the job list using 4 GPUs with 2 jobs on each GPU.\n\nHint: for complicated jobs, you might want to write a job/script generator.\n\n\n## Installation\n```\npip install parscript\n```\n[![PyPI version](https://badge.fury.io/py/parscript.svg)](https://badge.fury.io/py/parscript)\n\nUseful bash aliases (add them to `~/.bash_aliases`):\n```\nalias parworker='python -m parscript.worker'\nalias pardispatch='python -m parscript.dispatcher'\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtli%2Fparscript","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmtli%2Fparscript","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtli%2Fparscript/lists"}