{"id":27918759,"url":"https://github.com/juliaparallel/mpiclustermanagers.jl","last_synced_at":"2025-07-19T12:32:38.018Z","repository":{"id":43214889,"uuid":"200172731","full_name":"JuliaParallel/MPIClusterManagers.jl","owner":"JuliaParallel","description":"Julia parallel constructs over MPI","archived":false,"fork":false,"pushed_at":"2025-04-29T06:24:27.000Z","size":73,"stargazers_count":45,"open_issues_count":14,"forks_count":7,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-29T06:26:03.499Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JuliaParallel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-08-02T05:44:33.000Z","updated_at":"2025-04-29T06:05:24.000Z","dependencies_parsed_at":"2025-04-29T06:33:44.365Z","dependency_job_id":null,"html_url":"https://github.com/JuliaParallel/MPIClusterManagers.jl","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FMPIClusterManagers.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FMPIClusterManagers.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FMPIClusterManagers.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JuliaParallel%2FMPIClusterManagers.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JuliaParallel","download_url":"https://codeload.github.com/JuliaParallel/MPIClusterManagers.jl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252742631,"owners_count":21797283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-06T18:25:19.913Z","updated_at":"2025-05-06T18:25:20.590Z","avatar_url":"https://github.com/JuliaParallel.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MPIClusterManagers.jl\n\n[![CI](https://github.com/JuliaParallel/MPIClusterManagers.jl/actions/workflows/CI.yml/badge.svg)](https://github.com/JuliaParallel/MPIClusterManagers.jl/actions/workflows/CI.yml) [![codecov](https://codecov.io/github/JuliaParallel/MPIClusterManagers.jl/graph/badge.svg?token=0G39r3DkCn)](https://codecov.io/github/JuliaParallel/MPIClusterManagers.jl)\n\n## MPI and Julia parallel constructs together\n\nIn order for MPI calls to be made from a Julia cluster, it requires the use of\n`MPIManager`, a cluster manager that will start the julia workers using `mpirun`\n\nIt has three modes of operation\n\n- Only worker processes execute MPI code. The Julia master process executes outside of and\n  is not part of the MPI cluster. Free bi-directional TCP/IP connectivity is required\n  between all processes\n\n- All processes (including Julia master) are part of both the MPI as well as Julia cluster.\n  Free bi-directional TCP/IP connectivity is required between all processes.\n\n- All processes are part of both the MPI as well as Julia cluster. MPI is used as the transport\n  for julia messages. This is useful on environments which do not allow TCP/IP connectivity\n  between worker processes Note: This capability works with Julia 1.0, 1.1 and 1.2 and releases\n  after 1.4.2. It is broken for Julia 1.3, 1.4.0, and 1.4.1.\n\n### MPIManager: only workers execute MPI code\n\nAn example is provided in `examples/juliacman.jl`.\nThe julia master process is NOT part of the MPI cluster. The main script should be\nlaunched directly, `MPIManager` internally calls `mpirun` to launch julia/MPI workers.\nAll the workers started via `MPIManager` will be part of the MPI cluster.\n\n```\nMPIManager(;np=Sys.CPU_THREADS, mpi_cmd=false, launch_timeout=60.0)\n```\n\nIf not specified, `mpi_cmd` defaults to `mpirun -np $np`\n`stdout` from the launched workers is redirected back to the julia session calling `addprocs` via a TCP connection.\nThus the workers must be able to freely connect via TCP to the host session.\nThe following lines will be typically required on the julia master process to support both julia and MPI:\n\n```julia\n# to import MPIManager\nusing MPIClusterManagers\n\n# need to also import Distributed to use addprocs()\nusing Distributed\n\n# specify, number of mpi workers, launch cmd, etc.\nmanager=MPIManager(np=4)\n\n# start mpi workers and add them as julia workers too.\naddprocs(manager)\n```\n\nTo execute code with MPI calls on all workers, use `@mpi_do`.\n\n`@mpi_do manager expr` executes `expr` on all processes that are part of `manager`.\n\nFor example:\n```julia\n@mpi_do manager begin\n  using MPI\n  comm=MPI.COMM_WORLD\n  println(\"Hello world, I am $(MPI.Comm_rank(comm)) of $(MPI.Comm_size(comm))\")\nend\n```\nexecutes on all MPI workers belonging to `manager` only\n\n[`examples/juliacman.jl`](https://github.com/JuliaParallel/MPIClusterManagers.jl/blob/master/examples/juliacman.jl) is a simple example of calling MPI functions on all workers interspersed with Julia parallel methods.\n\nThis should be run _without_ `mpirun`:\n```\njulia juliacman.jl\n```\n\nA single instation of `MPIManager` can be used only once to launch MPI workers (via `addprocs`).\nTo create multiple sets of MPI clusters, use separate, distinct `MPIManager` objects.\n\n`procs(manager::MPIManager)` returns a list of julia pids belonging to `manager`\n`mpiprocs(manager::MPIManager)` returns a list of MPI ranks belonging to `manager`\n\nFields `j2mpi` and `mpi2j` of `MPIManager` are associative collections mapping julia pids to MPI ranks and vice-versa.\n\n### MPIManager: TCP/IP transport - all processes execute MPI code\n\nUseful on environments which do not allow TCP connections outside of the cluster\n\nAn example is in [`examples/cman-transport.jl`](https://github.com/JuliaParallel/MPIClusterManagers.jl/blob/master/examples/cman-transport.jl):\n```\nmpirun -np 5 julia cman-transport.jl TCP\n```\n\nThis launches a total of 5 processes, mpi rank 0 is the julia pid 1. mpi rank 1 is julia pid 2 and so on.\n\nThe program must call `MPIClusterManagers.start_main_loop(TCP_TRANSPORT_ALL)` with argument `TCP_TRANSPORT_ALL`.\nOn mpi rank 0, it returns a `manager` which can be used with `@mpi_do`\nOn other processes (i.e., the workers) the function does not return\n\n\n### MPIManager: MPI transport - all processes execute MPI code\n\n`MPIClusterManagers.start_main_loop` must be called with option `MPI_TRANSPORT_ALL` to use MPI as transport.\n```\nmpirun -np 5 julia cman-transport.jl MPI\n```\nwill run the example using MPI as transport.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliaparallel%2Fmpiclustermanagers.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuliaparallel%2Fmpiclustermanagers.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuliaparallel%2Fmpiclustermanagers.jl/lists"}