{"id":21202025,"url":"https://github.com/radiantone/blazer","last_synced_at":"2025-07-10T06:32:25.164Z","repository":{"id":43318084,"uuid":"456524470","full_name":"radiantone/blazer","owner":"radiantone","description":"An HPC abstraction over MPI with built-in parallel compute primitives","archived":false,"fork":false,"pushed_at":"2024-07-25T18:06:44.000Z","size":248,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-08T10:28:52.589Z","etag":null,"topics":["hpc","mpi","mpi4py","parallel-computing","pipeline","python","supercomputing","workflows"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/radiantone.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-02-07T13:48:13.000Z","updated_at":"2024-09-17T07:50:47.000Z","dependencies_parsed_at":"2022-08-27T16:52:39.482Z","dependency_job_id":null,"html_url":"https://github.com/radiantone/blazer","commit_stats":{"total_commits":311,"total_committers":2,"mean_commits":155.5,"dds":0.009646302250803873,"last_synced_commit":"5d2e51fc60dd2fa0cb9e963dfb2e6a73accaf8e1"},"previous_names":[],"tags_count":45,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radiantone%2Fblazer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radiantone%2Fblazer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radiantone%2Fblazer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/radiantone%2Fblazer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/radiantone","download_url":"https://codeload.github.com/radiantone/blazer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225622846,"owners_count":17498168,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","mpi","mpi4py","parallel-computing","pipeline","python","supercomputing","workflows"],"created_at":"2024-11-20T20:12:30.583Z","updated_at":"2024-11-20T20:12:31.434Z","avatar_url":"https://github.com/radiantone.png","language":"Python","readme":"![Blazer Logo](./img/blazer-logo-tiny.svg)\n\n\nAn HPC abstraction over MPI that uses pipes and pydash primitives. Blazer will handle all the MPI orchestration behind the scenes for you. You just work strictly with data and functions. Easy!\n\n[![Documentation Status](https://readthedocs.org/projects/blazerhpc/badge/?version=latest)](https://blazerhpc.readthedocs.io/en/latest/?badge=latest)\n\n### Install\nFrom pypi\n```bash\n$ pip install blazer\n```\n\nFrom clone\n```bash\n$ git clone https://github.com/radiantone/blazer\n$ cd blazer\n$ make init install\n```\n\nNOTE: For some tests ensure you have slurm configured properly (single or muli-machine). However, using slurm is not required to use blazer.\n\n### Linting\n```bash\n$ make lint\n```\n\n### Tests\n```bash\n(venv) $ mpirun -n 2 python setup.py test\nblazer/tests/test_parallel.py::test_parallel PASSED                      [ 50%]\nblazer/tests/test_pipeline.py::test_pipeline PASSED                      [100%]\n\n============================== 2 passed in 0.48s ===============================\nctrl-c\n```\n\nor\n\n```bash\n$ ./bin/tests.sh\n```\n\n### Examples\n```python\nimport blazer\nfrom blazer.hpc.mpi import parallel, pipeline, partial as p, scatter, where, select, filter, rank, size\n\ndef calc_some(value, *args):\n    \"\"\" Do some calculations \"\"\"\n    result = { 'some': value }\n    return result\n\ndef calc_stuff(value, *args):\n    \"\"\" Do some calculations \"\"\"\n    result = { 'this': value }\n    return result\n\ndef add_date(result):\n    from datetime import datetime\n    if type(result) is dict:\n        result['date'] = str(datetime.now())\n    return result\n\ndef calc_more_stuff(result):\n    \"\"\" Do some more calculations \"\"\"\n    if type(result) is list:\n        result += [{'more':'stuff'}]\n    elif type(result) is dict:\n        result['more'] = 'stuff'\n    return result\n\nINPUT_DATA = 'that'\n\nwith blazer.begin():\n    \n    result1=parallel([ \n        p(calc_stuff, 1),\n        p(calc_stuff, 2),\n        p(calc_stuff, 3),\n        p(calc_stuff, 4),\n        p(calc_stuff, 5)\n    ])\n    blazer.print(\"PARALLEL1:\",result1)\n\n    if blazer.ROOT:\n        r = list(\n            result1\n            | where(lambda g: where(lambda g: g['this'] \u003e 1))\n            | select(lambda g: p(calc_stuff, g['this']*2))\n        )\n        # Run the composed computation in parallel, wait for result\n        result = parallel(r)\n        blazer.print(\"PARALLEL2:\",result)\n\n    r=pipeline([\n        p(calc_stuff, 'DATA'),\n        p(pipeline, [\n            calc_some,\n            add_date\n        ]),\n        calc_stuff\n    ])\n    blazer.print(\"PIPELINE:\",r)\n\n    scatter_data = scatter(list(range(0,(size*2)+2)), calc_some)\n    blazer.print(\"SCATTER_DATA:\",scatter_data)\n\n    result = pipeline([\n        p(calc_stuff, INPUT_DATA), \n        add_date,\n        scatter_data,\n        p(parallel,[ \n            calc_some,\n            p(pipeline,[\n                calc_stuff,\n                p(parallel, [\n                    calc_some,\n                    calc_some\n                ]),\n                calc_stuff\n            ]),\n            calc_some\n        ]),\n        calc_more_stuff\n    ])\n\n    blazer.print(\"PIPELINE RESULT:\",result)\n\n    def get_data():\n        \"\"\" Data generator \"\"\"\n        for i in range(0,(size*2)):\n            yield i\n\n    result = scatter(get_data(), calc_some)\n    blazer.print(\"SCATTER:\",result)\n\n```\n\nTo run:\n```\n(venv) $ export PYTHONPATH=.\n(venv) $ mpirun -n 4 python blazer/examples/example1.py \nPARALLEL1: [{'this': 1}, {'this': 2}, {'this': 3}, {'this': 4}, {'this': 5}]\nPARALLEL2: [{'this': 4}, {'this': 6}, {'this': 2}, {'this': 8},; {'this': 10}]\nPIPELINE: {'this': {'some': ({'this': 'DATA'},), 'date': '2022-02-11 02:47:23.356461'}}\nSCATTER_DATA: [{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}, {'some': 8}, {'some': 9}, {'some': None}, {'some': None}]\nPIPELINE RESULT: [{'this': [{'this': ([{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}, {'some': 8}, {'some': 9}, {'some': None}, {'some': None}],)}, {'some': {'some': [{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}, {'some': 8}, {'some': 9}, {'some': None}, {'some': None}]}}]}, {'some': 'some'}, {'more': 'stuff'}]\n[0, 1, 2, 3, 4, 5, 6, 7]\nSCATTER: [{'some': 0}, {'some': 1}, {'some': 2}, {'some': 3}, {'some': 4}, {'some': 5}, {'some': 6}, {'some': 7}]\n```\n\nA map/reduce example\n\n```python\nimport blazer\nfrom blazer.hpc.mpi import map, reduce\n\ndef sqr(x):\n    return x * x\n\ndef add(x, y=0):\n    return x+y\n\nwith blazer.begin():\n    result = map(sqr, [1, 2, 3, 4])\n\n    blazer.print(result)\n    result = reduce(add, result)\n\n    blazer.print(result)\n```\n\nTo run:\n```\n(venv) $ export PYTHONPATH=.\n(venv) $ mpirun -n 4 python blazer/examples/example3.py \n[1, 4, 9, 16]\n30\n```\n\n#### Streaming Compute\n\nBlazer supports the notion of `streaming compute` to handle jobs where the data can't fit into memory on a single machine.\nIn this example we implement a map/reduce computation where everything is streaming from the source data through the results without loading all the data into memory.\n\n```python\n\"\"\" Streaming map/reduce example \"\"\"\nfrom itertools import groupby\nfrom random import randrange\nfrom typing import Generator\n\nimport blazer\nfrom blazer.hpc.mpi import stream\n\ndef datagen() -\u003e Generator:\n    for i in range(0, 1000):\n        r = randrange(2)\n        v = randrange(100)\n        if r:\n            yield {\"one\": 1, \"value\": v}\n        else:\n            yield {\"zero\": 0, \"value\": v}\n\ndef key_func(k):\n    return k[\"key\"]\n\ndef map(datum):\n    datum[\"key\"] = list(datum.keys())[0]\n    return datum\n\ndef reduce(datalist):\n    from blazer.hpc.mpi import rank\n\n    _list = sorted(datalist, key=key_func)\n    grouped = groupby(_list, key_func)\n    return [{\"rank\": rank, key: list(group)} for key, group in grouped]\n\nwith blazer.begin():\n    import json\n\n    mapper = stream(datagen(), map, results=True)\n    reducer = stream(mapper, reduce, results=True)\n    if blazer.ROOT:\n        for result in reducer:\n            blazer.print(\"RESULT\", json.dumps(result, indent=4))\n```\n\n\u003e NOTE: blazer has (currently) only been tested on `mpirun (Open MPI) 4.1.0`\n\n## Overview\n\nBlazer is a _high-performance computing_ (HPC) library that hides the complexities of a super computer's _message-passing interface_ or (MPI).\nUsers want to focus on their code and their data and not fuss with low-level API's for orchestrating results, building pipelines and running fast, parallel code. This is why blazer exists!\n\nWith blazer, a user only needs to work with simple, straightforward python. No cumbersome API's, idioms, or decorators are needed.\nThis means they can get started quicker, run faster code, and get their jobs done _faster_!\n\n### General Design\n\nBlazer is designed around the concept of computing _primitives_ or operations. Some of the primitives include:\n\n- **parallel** - For computing a list of tasks in parallel\n- **pipeline** - For computing a list of tasks in sequence, passing the results along\n- **map** - For mapping a task to a dataset\n- **reduce** - For mapping a task to a data list and computing a single result\n\nIn addition there are other primitives to help manipulate lists of tasks or data, such as:\n\n- **where** - Filter a list of tasks or data elements based on a function or lambda\n- **select** - Apply a function to each list element and return the result\n\n### Context Handlers\n\nBlazer uses convenient context handlers to control blocks of code that need to be scheduled to MPI processes behind the scenes.\nThere are two types of context handlers currently. \n\n#### MPI Context Handler\n\n`blazer.begin()` is a mandatory context that enables the MPI scheduler behind the various primitives to operate correctly.\n\n```python\n\nimport blazer\n\nblazer.begin():\n    def get_data():\n        \"\"\" Data generator \"\"\"\n        for i in range(0, (size * 2)):\n            yield i\n\n    result = scatter(get_data(), calc_some)\n    blazer.print(\"SCATTER:\", result)\n\n```\n#### GPU Context Handler\n\n`blazer.gpu()` is a context that requests (from the invisible MPI scheduler) dedicated access to a specific GPU on your MPI node fabric.\n\n```python\nimport logging\nimport blazer\nimport numpy as np\n\nfrom blazer.hpc.mpi.primitives import host, rank\nfrom numba import vectorize\nfrom timeit import default_timer as timer\n\ndef dovectors():\n\n    @vectorize(['float32(float32, float32)'], target='cuda')\n    def dopow(a, b):\n        return a ** b\n\n    vec_size = 100\n\n    a = b = np.array(np.random.sample(vec_size), dtype=np.float32)\n    c = np.zeros(vec_size, dtype=np.float32)\n\n    start = timer()\n    dopow(a, b)\n    duration = timer() - start\n    return duration\n\nwith blazer.begin(gpu=True):  # on-fabric MPI scheduler\n    with blazer.gpu() as gpu:  # on-metal GPU scheduler\n        # gpu object contains metadata about the GPU assigned\n        print(dovectors())\n```\n\n\n#### Shared Data Context Handler\n\nBlazer supports synchronizing shared data across ranks seamlessly. Here is an example of sharing a dictionary where each rank adds its own data to the dictionary and it is available to all other ranks magically!\n\n```python\nfrom random import randrange\n\nimport blazer\nfrom blazer.hpc.mpi.primitives import rank\n\nwith blazer.environment() as vars:\n    rv = randrange(10)\n    vars[\"rank\" + str(rank)] = [\n        {\"key\": randrange(10)},\n        randrange(10),\n        randrange(10),\n        randrange(10),\n    ]\n\n    print(\"RANK:\", rank, \"DATA\", vars.vars)\n\nblazer.print(vars[\"rank1\"])\n```\n\n### Cross-Cluster Supercomputing\n\nBlazer comes with a built-in design pattern for performing cross-cluster HPC. This is useful if you want to allocate compute resources on different super-computers and then build a pipeline of jobs across them. Here is a simple example using ALCF's Cooley and Theta systems (which are built into blazer).\n\n```python\nfrom blazer.hpc.alcf import cooley, thetagpu\nfrom blazer.hpc.local import parallel, pipeline, partial as p\n\n# Log into each cluster using MFA password from MobilePASS\ncooleyjob   = cooley.job(user='dgovoni', n=1, q=\"debug\", A=\"datascience\", password=True, script=\"/home/dgovoni/git/blazer/testcooley.sh\").login()       \nthetajob    = thetagpu.job(user='dgovoni', n=1, q=\"single-gpu\", A=\"datascience\", password=True, script=\"/home/dgovoni/git/blazer/testthetagpu.sh\").login()\n\ndef hello(data, *args):\n    return \"Hello \"+str(data)\n\n# Mix and match cluster compute jobs with local code tasks\n# in serial chaining\ncooleyjob(\"some data\").then(hello).then(thetajob).then(hello)\n\n# Run a cross cluster compute job\nresult = pipeline([\n    p(thetajob,\"some data2\"),\n    p(cooleyjob,\"some data1\")\n])\n\nprint(\"Done\")\n```\n\nWhen each job `.login()` method is run, it will gather the MFA login credentials for that system and then use that to schedule jobs on that system via ssh. \n\nNotice the use of the `pipeline` primitive above. It's the same primitive you would use to build your compute workflows! Composable tasks and composable super-computers.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fradiantone%2Fblazer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fradiantone%2Fblazer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fradiantone%2Fblazer/lists"}