{"id":18654405,"url":"https://github.com/pycroscopy/distusid","last_synced_at":"2026-07-14T09:32:05.353Z","repository":{"id":104715147,"uuid":"140432937","full_name":"pycroscopy/distUSID","owner":"pycroscopy","description":"Development of distributed framework for USID processing","archived":false,"fork":false,"pushed_at":"2019-06-10T16:19:10.000Z","size":1534,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-10-27T07:27:37.665Z","etag":null,"topics":["cloud-computing","framework","high-performance-computing","mpi4py","parallel-computing"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pycroscopy.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-10T13:02:23.000Z","updated_at":"2024-11-21T11:03:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"c9179b4b-5508-4986-9aba-3c5ba81732a2","html_url":"https://github.com/pycroscopy/distUSID","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pycroscopy/distUSID","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pycroscopy%2FdistUSID","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pycroscopy%2FdistUSID/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pycroscopy%2FdistUSID/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pycroscopy%2FdistUSID/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pycroscopy","download_url":"https://codeload.github.com/pycroscopy/distUSID/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pycroscopy%2FdistUSID/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35455901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-14T02:00:06.603Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-computing","framework","high-performance-computing","mpi4py","parallel-computing"],"created_at":"2024-11-07T07:15:18.111Z","updated_at":"2026-07-14T09:32:05.348Z","avatar_url":"https://github.com/pycroscopy.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"distUSID\n=======\n\nDistributed versions of the embarrassingly-parallel ``Process`` classes in pycroscopy and pyUSID.\n\n* The emphasis is to develop the ``pyUSID.Process`` class such that it continues to work for laptops but also works on HPC / cloud clusters with minimal modifications to children classes (in pycroscopy).\n  Once certified to work well for a handful of examples, the changes in the class(es) will be rolled back into ``pyUSID`` and even ``pycroscopy``.\n* Code here is developed and tested on ORNL CADES SHPC OR Condo only for the time being. The code should, in theory, be portable to OLCF or other machines.\n\nLimitations\n-----------\nBefore diving in to the many strategies for going about how to solve this problem, it is important to be cognizant of the restrictions:\n\n* The user-facing sub-classes should see minimal changes to enable distributed computing and not require expert knowledge of MPI or other paradigms\n* There is a finite bandwidth available for file I/O even on high-performance-computing resources. This means that the number of parallel file writing\n  operations should be minimized. Even with a single node with 36 cores, we do not want 36 processes / ranks waiting on each other to write data to the file.\n  The majority of the time should be spent on the computation which is the main problem that has necessitated distributed computing.\n\nStrategies\n----------\n#. Dask\n#. Pure mpi4py\n#. mpi4py + existing Joblib\n#. pySpark\n#. Workflows such as Swift\n\n2. Pure mpi4py\n--------------\nUse one rank per logical core. All ranks read and write to the file. Code available on the `pure_mpi \u003chttps://github.com/pycroscopy/distUSID/tree/pure_mpi\u003e`_ branch\n\n**Pros**:\n\n* Circumvents ``joblib`` related problems since it obviates the need for ``joblib``\n* (Potentially) less restrictive PBS script\n\n**Cons**:\n\n* If a node has fewer ranks than the number of logical cores, those cores are wasted. This minor problem can be fixed\n\nStatus\n~~~~~~\n* Works very well for both the ``SignalFilter`` and the ``GIVBayesian`` class in addition to Chris' success on the ``image windowing``\n* This same code **had** been `generalized \u003chttps://github.com/pycroscopy/distUSID/commit/4e4e367230c9a85540828b7d8e56cc261f135fae\u003e`_\n  to capture the two sub-cases of mpi4py+joblib below . However, this causes ``GIVBayesian`` to fail - just does not compute anything at all. No errors observed.\n\n  * If a fix is discovered, this capability can be enabled with just `2 lines \u003chttps://github.com/pycroscopy/distUSID/commit/3d43614e8bd1ae722c26e72d7d1a95dbeac4cee8\u003e`_.\n  * This may be related to some complication in the `math libraries \u003chttps://github.com/pycroscopy/distUSID/commit/3930df86c6119226702628145090726ad1f00312\u003e`_\n* Have not yet seen any problems with regards to the bottleneck on up to 4 nodes (36 cores each). Benchmarking will be necessary for identify bottlenecks\n* Comprehensive checkpointing / resuming capability has also been incorporated within the ``Process`` class\n* The ``Process`` class has been made even more robust against accidental damage from user-side by moving more underlying code into private variables.\n* Minimal changes are required for the children classes of ``pyUSID.Process``:\n\n  * mainly in verbose print statements - need to check for ``rank == 0``\n  * ``Process`` completely handles all check-pointing (legacy + new) and flushing the file after each batch. The user-side code literally only needs to write to the HDF5 datasets\n\n* Plenty of documentation about the thought process included within the ``Process`` class file.\n* The ``Process`` class from this branch will be rolled into pyUSID after some checks\n\nTips and Gotchas\n~~~~~~~~~~~~~~~~\n* First test the dataset creation step with the computation disabled to speed up debugging time. Most of the challenges are in the dataset creation portion.\n* ``h5py`` (parallel) results in **segmentation faults** for the following situations:\n\n  * If ``compression`` is specified when creating datasets. Known issue with no workaround\n  * ``if rank == 0: write_simple_attrs(....)`` \u003c-- Make all ranks write attributes\n* Environment variables need to be set in the PBS script to minimize conflicts between LAPACK's preference to use threading and MPI / multiprocessing.\n  Two `environment variables \u003chttps://github.com/pycroscopy/distUSID/commit/72d8ac086ee974a4ed644fbe55738d198b7265ec\u003e`_ made a night-and-day difference\n  in the `pure_mpi \u003chttps://github.com/pycroscopy/distUSID/tree/pure_mpi\u003e`_ branch.\n\n  * Setting these variables within ``parallel_compute()`` had the `same effect \u003chttps://github.com/pycroscopy/distUSID/commit/3ccdacfa32ac97af7eb9994a1562ea9c0caf51e5\u003e`_ as not setting these environment variables at all.\n\n3. mpi4py+joblib\n----------------\n#. **1 rank / node**: Use an MPI + OpenMP paradigm where each rank is in charge of one node and computes via ``joblib`` within the node just as in pyUSID / pycroscopy. See the `mpi_plus_joblib \u003chttps://github.com/pycroscopy/distUSID/tree/mpi_plus_joblib)\u003e`_ branch\n\n   **Pros**:\n\n   * Easy to understand and implement since each node continues to do whatever a laptop would do / has been doing\n\n   **Cons**:\n\n   * ``joblib`` sometimes does not like to work with ``numpy`` and ``mpi4py``\n\n   **Status**:\n\n   * Worked for the ``SignalFilter`` but not for the ``GIVBayesian`` class.\n\n#. **Arbitrary MPI ranks / node**: Use a combination of joblib and MPI and pose no restrictions whatsoever on the number of ranks or configuration\n\n   **Pros**:\n\n   * Probably the programmatically \"proper\" way to do this\n   * PBS script and ``mpiexec`` call can be configured in any way\n\n   **Cons**:\n\n   * Has nearly all the major cons of the two above approaches\n   * ``joblib`` sometimes does not like to work with ``numpy`` and ``mpi4py``\n   * Noticeably more complicated in that additional book-keeping would be required for the relationships (master) within each node\n   * The rank that collects all the results may not have sufficient memory. This may limit how much each rank can compute at a given time\n\n   **Status**:\n\n   * As mentioned above, the ``Process`` class in the `pure_mpi \u003chttps://github.com/pycroscopy/distUSID/tree/pure_mpi\u003e`_ branch already\n     captures this use-case but this refuses to work for ``GIVBayesian`` just like in the `mpi_plus_joblib \u003chttps://github.com/pycroscopy/distUSID/tree/mpi_plus_joblib)\u003e`_ branch\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpycroscopy%2Fdistusid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpycroscopy%2Fdistusid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpycroscopy%2Fdistusid/lists"}