{"id":18685173,"url":"https://github.com/open-atmos/pympdata-mpi","last_synced_at":"2025-04-12T04:32:53.115Z","repository":{"id":103766460,"uuid":"531184793","full_name":"open-atmos/PyMPDATA-MPI","owner":"open-atmos","description":"PyMPDATA + numba-mpi coupler sandbox","archived":false,"fork":false,"pushed_at":"2025-04-04T13:59:08.000Z","size":416,"stargazers_count":5,"open_issues_count":18,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-04T14:24:49.508Z","etag":null,"topics":["advection-equation","distributed-computing","mpi","numba","pde-solver","python","simulation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-atmos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-31T17:16:34.000Z","updated_at":"2025-04-04T13:57:37.000Z","dependencies_parsed_at":"2024-04-19T14:39:53.906Z","dependency_job_id":"dadf2e00-0245-4d7e-977a-08e6277cb376","html_url":"https://github.com/open-atmos/PyMPDATA-MPI","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-atmos%2FPyMPDATA-MPI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-atmos%2FPyMPDATA-MPI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-atmos%2FPyMPDATA-MPI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-atmos%2FPyMPDATA-MPI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-atmos","download_url":"https://codeload.github.com/open-atmos/PyMPDATA-MPI/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248517381,"owners_count":21117442,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["advection-equation","distributed-computing","mpi","numba","pde-solver","python","simulation"],"created_at":"2024-11-07T10:21:04.694Z","updated_at":"2025-04-12T04:32:53.088Z","avatar_url":"https://github.com/open-atmos.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyMPDATA-MPI\n\n[![Python 3](https://img.shields.io/static/v1?label=Python\u0026logo=Python\u0026color=3776AB\u0026message=3)](https://www.python.org/)\n[![LLVM](https://img.shields.io/static/v1?label=LLVM\u0026logo=LLVM\u0026color=gold\u0026message=Numba)](https://numba.pydata.org)\n[![Linux OK](https://img.shields.io/static/v1?label=Linux\u0026logo=Linux\u0026color=yellow\u0026message=%E2%9C%93)](https://en.wikipedia.org/wiki/Linux)\n[![macOS OK](https://img.shields.io/static/v1?label=macOS\u0026logo=Apple\u0026color=silver\u0026message=%E2%9C%93)](https://en.wikipedia.org/wiki/macOS)\n[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/open-atmos/PyMPDATA-MPI/graphs/commit-activity)\n\n[![PL Funding](https://img.shields.io/static/v1?label=PL%20Funding%20by\u0026color=d21132\u0026message=NCN\u0026logoWidth=25\u0026logo=image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAABmJLR0QA/wD/AP+gvaeTAAAAKUlEQVQ4jWP8////fwYqAiZqGjZqIHUAy4dJS6lqIOMdEZvRZDPcDQQAb3cIaY1Sbi4AAAAASUVORK5CYII=)](https://www.ncn.gov.pl/?language=en)\n[![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0.html)\n[![Copyright](https://img.shields.io/static/v1?label=Copyright\u0026color=249fe2\u0026message=Jagiellonian%20University\u0026)](https://en.uj.edu.pl/)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10866521.svg)](https://doi.org/10.5281/zenodo.10866521)    \n\n[![GitHub issues](https://img.shields.io/github/issues-pr/open-atmos/PyMPDATA-MPI.svg?logo=github\u0026logoColor=white)](https://github.com/open-atmos/PyMPDATA-MPI/pulls?q=)\n[![GitHub issues](https://img.shields.io/github/issues-pr-closed/open-atmos/PyMPDATA-MPI.svg?logo=github\u0026logoColor=white)](https://github.com/open-atmos/PyMPDATA-MPI/pulls?q=is:closed)    \n[![GitHub issues](https://img.shields.io/github/issues/open-atmos/PyMPDATA-MPI.svg?logo=github\u0026logoColor=white)](https://github.com/open-atmos/PyMPDATA-MPI/issues?q=)\n[![GitHub issues](https://img.shields.io/github/issues-closed/open-atmos/PyMPDATA-MPI.svg?logo=github\u0026logoColor=white)](https://github.com/open-atmos/PyMPDATA-MPI/issues?q=is:closed)   \n[![Github Actions Build Status](https://github.com/open-atmos/PyMPDATA-MPI/workflows/tests+pypi/badge.svg?branch=main)](https://github.com/open-atmos/PyMPDATA-MPI/actions)\n[![PyPI version](https://badge.fury.io/py/PyMPDATA-MPI.svg)](https://pypi.org/project/PyMPDATA-MPI)\n[![API docs](https://shields.mitmproxy.org/badge/docs-pdoc.dev-brightgreen.svg)](https://open-atmos.github.io/PyMPDATA-MPI/)\n[![Coverage Status](https://codecov.io/gh/open-atmos/PyMPDATA-MPI/branch/main/graph/badge.svg)](https://app.codecov.io/gh/open-atmos/PyMPDATA-MPI)    \n\nPyMPDATA-MPI constitutes a [PyMPDATA](https://github.com/open-atmos/PyMPDATA) +\n[numba-mpi](https://github.com/numba-mpi/numba-mpi) coupler enabling numerical solutions\nof transport equations with the MPDATA numerical scheme in a\nhybrid parallelisation model with both multi-threading and MPI distributed memory communication.\nPyMPDATA-MPI adapts to API of PyMPDATA offering domain decomposition logic.\n\n## Hello world examples\n\nIn a minimal setup, PyMPDATA-MPI can be used to solve the following transport equation: \n$$\\partial_t (G \\psi) + \\nabla \\cdot (Gu \\psi)= 0$$\nin an environment with multiple nodes.\nEvery node (process) is responsible for computing its part of the decomposed domain.\n\n### Spherical scenario (2D)\n\nIn spherical geometry, the $G$ factor represents the Jacobian of coordinate transformation.\nIn this example (based on a test case from [Williamson \u0026 Rasch 1989](https://doi.org/10.1175/1520-0493(1989)117\u003c0102:TDSLTW\u003e2.0.CO;2)),\n  domain decomposition is done cutting the sphere along meridians.\nThe inner dimension uses the [`MPIPolar`](https://open-atmos.github.io/PyMPDATA-MPI/mpi_polar.html) \n  boundary condition class, while the outer dimension uses\n  [`MPIPeriodic`](https://open-atmos.github.io/PyMPDATA-MPI/mpi_periodic.html).\nNote that the spherical animations below depict simulations without MPDATA corrective iterations,\n  i.e. only plain first-order upwind scheme is used (FIX ME).\n\n### 1 worker (n_threads = 1)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.1_rank_0_size_1_c_field_.0.5.0.25._mpi_dim_0_n_threads_1-SphericalScenario-anim.gif\" width=\"49%\" /\u003e \n\u003c/p\u003e\n\n### 2 workers (MPI_DIM = 0, n_threads = 1)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.1_rank_1_size_2_c_field_.0.5.0.25._mpi_dim_0_n_threads_1-SphericalScenario-anim.gif\" width=\"49%\" /\u003e \n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.1_rank_0_size_2_c_field_.0.5.0.25._mpi_dim_0_n_threads_1-SphericalScenario-anim.gif\" width=\"49%\" /\u003e\n\u003c/p\u003e\n\n### Cartesian scenario (2D)\n\nIn the cartesian example below (based on a test case from [Arabas et al. 2014](https://doi.org/10.3233/SPR-140379)),\n  a constant advector field $u$ is used (and $G=1$).\nMPI (Message Passing Interface) is used \n  for handling data transfers and synchronisation with the domain decomposition\n  across MPI workers done in either inner or in the outer dimension (user setting).\nMulti-threading (using, e.g., OpenMP via Numba) is used for shared-memory parallelisation \n  within subdomains (indicated by dotted lines in the animations below) with threading subdomain\n  split done across the inner dimension (internal PyMPDATA logic).\nIn this example, two corrective MPDATA iterations are employed.\n\n### 1 worker (n_threads=3)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_0_size_1_c_field_.0.5.0.25._mpi_dim_0_n_threads_3-CartesianScenario-anim.gif\" width=\"49%\" /\u003e \n\u003c/p\u003e\n\n### 2 workers (MPI_DIM = 0, n_threads = 3)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_0_size_2_c_field_.0.5.0.25._mpi_dim_0_n_threads_3-CartesianScenario-anim.gif\" width=\"49%\" /\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_1_size_2_c_field_.0.5.0.25._mpi_dim_0_n_threads_3-CartesianScenario-anim.gif\" width=\"49%\" /\u003e \n\u003c/p\u003e\n\n### 2 workers (MPI_DIM = -1, n_threads = 3)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_0_size_2_c_field_.0.5.0.25._mpi_dim_-1_n_threads_3-CartesianScenario-anim.gif\" width=\"49%\" /\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_1_size_2_c_field_.0.5.0.25._mpi_dim_-1_n_threads_3-CartesianScenario-anim.gif\" width=\"49%\" /\u003e \n\u003c/p\u003e\n\n### 3 workers (MPI_DIM = 0, n_threads = 3)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_0_size_3_c_field_.0.5.0.25._mpi_dim_0_n_threads_3-CartesianScenario-anim.gif\" width=\"32%\" /\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_1_size_3_c_field_.0.5.0.25._mpi_dim_0_n_threads_3-CartesianScenario-anim.gif\" width=\"32%\" /\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_2_size_3_c_field_.0.5.0.25._mpi_dim_0_n_threads_3-CartesianScenario-anim.gif\" width=\"32%\" /\u003e\n\u003c/p\u003e\n\n### 3 workers (MPI_DIM = -1, n_threads = 3)\n\u003cp align=\"middle\"\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_0_size_3_c_field_.0.5.0.25._mpi_dim_-1_n_threads_3-CartesianScenario-anim.gif\" width=\"32%\" /\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_1_size_3_c_field_.0.5.0.25._mpi_dim_-1_n_threads_3-CartesianScenario-anim.gif\" width=\"32%\" /\u003e\n  \u003cimg src=\"https://github.com/open-atmos/PyMPDATA-MPI/releases/download/latest-generated-plots/n_iters.3_rank_2_size_3_c_field_.0.5.0.25._mpi_dim_-1_n_threads_3-CartesianScenario-anim.gif\" width=\"32%\" /\u003e\n\u003c/p\u003e\n\n## Package architecture\n\n```mermaid\n    flowchart BT\n\n    H5PY ---\u003e HDF{{HDF5}}\n    subgraph pythonic-dependencies [Python]\n      TESTS --\u003e H[pytest-mpi]\n      subgraph PyMPDATA-MPI [\"PyMPDATA-MPI\"]\n        TESTS[\"PyMPDATA-MPI[tests]\"] --\u003e CASES(simulation scenarios)\n        A1[\"PyMPDATA-MPI[examples]\"] --\u003e CASES\n        CASES --\u003e D[PyMPDATA-MPI]\n      end\n      A1 ---\u003e C[py-modelrunner]\n      CASES ---\u003e H5PY[h5py]\n      D --\u003e E[numba-mpi]\n      H --\u003e X[pytest]\n      E --\u003e N\n      F --\u003e N[Numba]\n      D --\u003e F[PyMPDATA]\n    end\n    H ---\u003e MPI\n    C ---\u003e slurm{{slurm}}\n    N --\u003e OMPI{{OpenMP}}\n    N --\u003e L{{LLVM}}\n    E ---\u003e MPI{{MPI}}\n    HDF --\u003e MPI\n    slurm --\u003e MPI\n\nstyle D fill:#7ae7ff,stroke-width:2px,color:#2B2B2B\n\nclick H \"https://pypi.org/p/pytest-mpi\"\nclick X \"https://pypi.org/p/pytest\"\nclick F \"https://pypi.org/p/PyMPDATA\"\nclick N \"https://pypi.org/p/numba\"\nclick C \"https://pypi.org/p/py-modelrunner\"\nclick H5PY \"https://pypi.org/p/h5py\"\nclick E \"https://pypi.org/p/numba-mpi\"\nclick A1 \"https://pypi.org/p/PyMPDATA-MPI\"\nclick D \"https://pypi.org/p/PyMPDATA-MPI\"\nclick TESTS \"https://pypi.org/p/PyMPDATA-MPI\"\n```\nRectangular boxes indicate pip-installable Python packages (click to go to pypi.org package site).\n\n## Credits \u0026 acknowledgments:\n\nPyMPDATA-MPI started as an [MSc project of Kacper Derlatka](https://www.ap.uj.edu.pl/diplomas/166883) ([@Delcior](https://github.com/Delcior)) mentored by [@slayoo](https://github.com/slayoo).\n\nDevelopment of PyMPDATA-MPI has been supported by the [Poland's National Science Centre](https://www.ncn.gov.pl/?language=en) \n(grant no. 2020/39/D/ST10/01220).\n\nWe acknowledge Poland’s high-performance computing infrastructure [PLGrid](https://plgrid.pl/) (HPC Centers: [ACK Cyfronet AGH](https://www.cyfronet.pl/en/)) \nfor providing computer facilities and support within computational grant no. PLG/2023/016369\n\ncopyright: [Jagiellonian University](https://en.uj.edu.pl/en) \u0026 [AGH University of Krakow](https://agh.edu.pl/en)   \nlicence: [GPL v3](https://www.gnu.org/licenses/gpl-3.0.html)\n\n## Design goals\n\n- MPI support for [PyMPDATA](https://pypi.org/project/PyMPDATA/) implemented externally (i.e., not incurring any overhead or additional dependencies for PyMPDATA users)\n- MPI calls within [Numba njitted code](https://numba.pydata.org/numba-doc/dev/reference/jit-compilation.html) (hence not using [`mpi4py`](https://mpi4py.readthedocs.io/), but rather [`numba-mpi`](https://pypi.org/p/numba-mpi/))\n- hybrid domain-decomposition parallelism: threading (internal in PyMPDATA, in the inner dimension) + MPI (either inner or outer dimension)\n- example simulation scenarios featuring HDF5/MPI-IO output storage (using [h5py](https://www.h5py.org/))\n- [py-modelrunner](https://github.com/zwicker-group/py-modelrunner) simulation orchestration\n- portability across Linux \u0026 macOS (no Windows support as of now due to [challenges in getting HDF5/MPI-IO to work there](https://docs.h5py.org/en/stable/build.html#source-installation-on-windows))\n- Continuous Integration (CI) with different OSes and different MPI implementations (leveraging to mpi4py's [setup-mpi Github Action](https://github.com/mpi4py/setup-mpi/))\n- full test coverage including CI builds asserting on same results with multi-node vs. single-node computations (with help of [pytest-mpi](https://pypi.org/p/pytest-mpi/))\n- ships as a [pip-installable package](https://pypi.org/project/PyMPDATA-MPI) - aimed to be a dependency of domain-specific packages \n\n## Related resources\n\n### open-source Large-Eddy-Simulation and related software\n\n#### Julia\n- https://github.com/CliMA/ClimateMachine.jl/\n#### C++\n- https://github.com/microhh/microhh\n- https://github.com/igfuw/UWLCM\n- https://github.com/mrnorman/portUrb\n#### C/CUDA\n- https://github.com/NCAR/FastEddy-model\n#### FORTRAN\n- https://github.com/dalesteam/dales\n- https://github.com/uclales/uclales\n- https://github.com/UCLALES-SALSA/UCLALES-SALSA\n- https://github.com/igfuw/bE_SDs\n- https://github.com/pencil-code/pencil-code\n- https://github.com/AtmosFOAM/AtmosFOAM\n- https://github.com/scale-met/scale\n#### Python (incl. Cython) \n- https://github.com/CliMA/pycles\n- https://github.com/pnnl/pinacles\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-atmos%2Fpympdata-mpi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-atmos%2Fpympdata-mpi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-atmos%2Fpympdata-mpi/lists"}