{"id":18875390,"url":"https://github.com/contextualist/grain","last_synced_at":"2025-04-14T17:31:28.562Z","repository":{"id":57435723,"uuid":"270009862","full_name":"Contextualist/grain","owner":"Contextualist","description":"A scheduler for resource-aware parallel external computing on clusters","archived":false,"fork":false,"pushed_at":"2024-08-09T18:46:59.000Z","size":404,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-08-09T20:17:19.458Z","etag":null,"topics":["async","subprocess","supercomputing","trio"],"latest_commit_sha":null,"homepage":"https://grain.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Contextualist.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-06T14:44:02.000Z","updated_at":"2024-08-09T18:46:54.000Z","dependencies_parsed_at":"2024-05-04T00:35:42.091Z","dependency_job_id":"2d457444-a1f9-49da-b9ad-300f36c807cf","html_url":"https://github.com/Contextualist/grain","commit_stats":{"total_commits":141,"total_committers":2,"mean_commits":70.5,"dds":"0.21276595744680848","last_synced_commit":"278bdf5db7e3b314224cee7335aab494e384fca0"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Contextualist%2Fgrain","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Contextualist%2Fgrain/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Contextualist%2Fgrain/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Contextualist%2Fgrain/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Contextualist","download_url":"https://codeload.github.com/Contextualist/grain/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223639397,"owners_count":17177816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","subprocess","supercomputing","trio"],"created_at":"2024-11-08T06:07:15.025Z","updated_at":"2024-11-08T06:07:15.626Z","avatar_url":"https://github.com/Contextualist.png","language":"Python","readme":"# Grain\n\n[![Docs](https://img.shields.io/badge/docs-read%20now-blue.svg)](https://grain.readthedocs.io)\n[![PyPI version](https://img.shields.io/pypi/v/grain-scheduler.svg)](https://pypi.org/project/grain-scheduler)\n\nA scheduler for resource-aware parallel external computing on clusters.\n\n### Install\n\n```Bash\npip install grain-scheduler\n```\n\n### Overview\n\n[Dask-like](https://docs.dask.org/en/latest/delayed.html) async-native delayed objects for you to run jobs in an arbitary mix of parallel and sequential manner.\n\n```python\nfrom grain.delayed import delayed\nfrom grain.resource import Cores\n\n@delayed\nasync def identity(x):\n    # Access what CPU(s) have been allocated for us\n    n_cpu, cpus = GVAR.res.N, ','.join(map(str,GVAR.res.c))\n    # Pretend that we are doing something seriously ...\n    await trio.run_process(f'mpirun -np {n_cpu} --bind-to cpulist:ordered --cpu-set {cpu} sleep 1')\n    return x\n\n@delayed\nasync def weighted_sum():\n    # Run the expensive function **remotely** (on a worker) by demanding resource 1 CPU core\n    # Note that `r_` is a Future, or a placeholder of the return value\n    r_ = (identity @ Cores(1))(0)\n\n    # Futures are composable\n    # Composition implies dependency / parallelization opportunity\n    r_ += (identity @ Cores(1))(1) * 1 + (identity @ Cores(1))(2) * 2\n    # Block until all dependencies finish, then return the composed value\n    return await r_\n\n    # Or do the same in a fancier way\n    #return await sum(\n    #    (identity @ Cores(1))(i) * i for i in range(3)\n    #)\n\n# Run the cheap, orchestrating function **locally**\nprint(await (weighted_sum() + weighted_sum()))\n# Output: 10\n```\n\nCheck out [tutorial](https://grain.readthedocs.io/en/latest/tutorial_delayed.html) for complete demos and how to set up workers.\n\n### Resource-awareness\n\nEvery job in the job queue has a resource request infomation along with the job to run. Before the executor run each job, it inspects each worker for resource availability. If resource is insufficient, the job queue is suspended until completed jobs return resources. Resources can be CPU cores, virtual memory, both, (or anything user defined following interface `grain.resource.Resource`).\n\nEvery time a job function runs, it has access to `GVAR.res`, a [context-local variable](https://trio.readthedocs.io/en/stable/reference-core.html#task-local-storage) giving the information of specific resource dedicated to the job. (e.g. if a job is submitted with `Cores(3)`, asking for 3 CPU cores, it might receive allocation like `Cores([6,7,9])`.)\n\n### Ergonomic user interface\n\nAsync-native API introduces minimal changes to the serial code, while enabling access to the entire Python async ecosystem accommodating complex workflows.\n\nMinimal configuration is needed for migrating to a new supercomputing cluster (See sample configs in the [`example/`](example) dir). When running, the `grain` commandline helper provides easy access to dashboard and worker scaling.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontextualist%2Fgrain","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontextualist%2Fgrain","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontextualist%2Fgrain/lists"}