{"id":21627538,"url":"https://github.com/billsioros/mpisuite","last_synced_at":"2025-09-05T02:05:33.051Z","repository":{"id":105137724,"uuid":"223028906","full_name":"billsioros/MPIsuite","owner":"billsioros","description":"Using bash to automate the compilation, scheduling and profiling of MPI / OpenMP applications","archived":false,"fork":false,"pushed_at":"2023-12-15T17:42:32.000Z","size":650,"stargazers_count":7,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-25T09:04:16.816Z","etag":null,"topics":["automated","automated-build","automated-deployment","automated-testing","automated-tests","automation-framework","bash","bash-script","bash-scripting","openmp","openmp-parallelization","openmp-support","openmpi","openmpi-cpu-clusters"],"latest_commit_sha":null,"homepage":"http://www.di.uoa.gr/eng/node/1451","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/billsioros.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-11-20T21:15:27.000Z","updated_at":"2022-02-27T19:23:24.000Z","dependencies_parsed_at":"2023-03-13T14:48:40.864Z","dependency_job_id":null,"html_url":"https://github.com/billsioros/MPIsuite","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billsioros%2FMPIsuite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billsioros%2FMPIsuite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billsioros%2FMPIsuite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billsioros%2FMPIsuite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/billsioros","download_url":"https://codeload.github.com/billsioros/MPIsuite/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248402545,"owners_count":21097331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automated","automated-build","automated-deployment","automated-testing","automated-tests","automation-framework","bash","bash-script","bash-scripting","openmp","openmp-parallelization","openmp-support","openmpi","openmpi-cpu-clusters"],"created_at":"2024-11-25T01:16:54.138Z","updated_at":"2025-04-11T12:47:11.869Z","avatar_url":"https://github.com/billsioros.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# **MPIsuite** - MPI / OpenMP Task Automation\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n_Important Notice: This project is in early development. Features are sparse and bugs may arise._\n\n## **The Suite**\n\n[mpis-compile](./MPIs/mpis-compile) is responsible for compiling the supplied MPI, OpenMP or Hybrid source file.\n\nIt is expecting at least one arguement, which should be the source file. Any other arguements are grouped and form pairs of keys and values. Each pair represents a macro, named **key** with a value of **value**, that is going to be defined at compile time.\n\n[mpis-schedule](./MPIs/mpis-schedule) is responsible for generating a job and submitting it to the PBS queue.\n\nIt is expecting exactly two arguements, the executable and the number of processes that should be created.\n\n[mpis-profile](./MPIs/mpis-profile) is responsible for compiling our program with various (key, value) macro pairs, running it with different numbers of processes and collecting our measurements.\n\nIt is expecting exactly two arguements, the source file and a [profiling description](#profiling-descriptions).\n\n## **Installation**\n\n    chmod +x setup.sh\n\n    ./setup.sh\n\nUninstalling **MPIs** can be achieved simply by running\n\n    ./setup.sh --uninstall\n\n## **Configuration**\n\n* **MPIS_OUTPUT_ROOT** indicates the directory, under which the files and directories generated by our scheduler and profiler are going to be saved.\n* **MPIS_USER_ID** is being used when querying the job queue to check if a job is done running.\n* **MPIS_EDITOR** and **MPIS_EDITOR_ARGS** are optional and are used by the profiler to open files in your favorite editor.\n\nYou are more than welcome to check _~/.mpisrc_ out, for additional configuration settings.\n\n## **Profiling Descriptions**\n\n[mpis-profile](./MPIs/mpis-profile) is expecting a profiling description, which **must** define:\n\n* a regular expression named **TIME_PATTERN**, which indicates the format of our timer's output.\n* a string named **MACRO**, containing the name of the macro that should be defined by [mpis-compile](./MPIs/mpis-compile).\n* an array named **VALUES**, containing the values that macro **MACRO** should receive in different runs.\n\nYou can optionally define **MPIS_ENABLE_PROFILING** and/or **MPIS_LINK_OPENMP**,\nin order to bypass confirmation on the corresponding compilation step. These can be globally defined, as well, in _~/.mpisrc_.\n\n## **Example**\n\nLet' s assume we would like to estimate the integral from a to b of an equation f(x) using the trapezoidal rule.\n\nLet' s also assume that we have already developed an **MPI** program, named **mpi_trap1.c** to do so and this program defines a macro named **nTraps**, which corresponds to the **number of trapezoids** that are going to be used in the calculation of the integral.\n\n### **Compiling**\n\nWe now need to compile our source file using [mpis-compile](./MPIs/mpis-compile) as follows\n\n    mpis-compile mpi_trap1.c nTraps 512\n\n    [mpis-compile] enable profiling: y\n    [mpis-compile] link OpenMP: n\n\nThis results in the creation of an executable file named **mpi_trap1.x**, inside which the macro **nTraps** has been assigned the value **512**.\n\n**ATTENTION:** If your source code contains a _#define_ statement corresponding to the macro provided, the supplied value will be **overriden**. Remove the _#define_ statement or surround it with a _#ifndef MACRO ... #endif_ preprocessor block to resolve this issue.\n\nExecuting [mpis-compile](./MPIs/mpis-compile) with the _--clean_ option deletes the executable.\n\n### **Scheduling**\n\nScheduling the executable can be achieved like so\n\n    mpis-schedule mpi_trap1.x 16\n\n    [mpis-schedule] name='mpi_trap1_16_argo059_job', id='14524.argo', ps=16, ns=2, ppn=8\n\n    Job id            Name             User              Time Use S Queue\n    ----------------  ---------------- ----------------  -------- - -----\n    12507.argo        myJob            argo081                  0 Q workq\n    14524.argo        mpi_trap1_16_ar  argo059           00:00:00 R workq\n\n* **ps** stands for the **number of processes**\n* **ns** stands for the **number of nodes**\n* **ppn** stands for the **number of processes per node**\n\nThe following files and directories are generated\n\n    find ./out\n\n    ./out/\n    ./out/16\n    ./out/16/21_11_2019\n    ./out/16/21_11_2019/15_06_30\n    ./out/16/21_11_2019/15_06_30/mpi_trap1_16_argo059_job.stderr\n    ./out/16/21_11_2019/15_06_30/mpi_trap1_16_argo059_job.stdout\n\nExecuting [mpis-schedule](./MPIs/mpis-schedule) with the _--clean_ option removes any mpiP and job associated files and removes any **MPIS_USER_ID** associated job from the queue.\n\n### **Profiling**\n\nWe firstly need to define a [profiling description](#profiling-descriptions) like so\n\n```bash\n#!/bin/bash\n\nexport MPIS_ENABLE_PROFILING=true\nexport MPIS_LINK_OPENMP=false\n\nTIME_PATTERN=\"Elapsed time: \\K([0-9]+\\.[0-9]+)\"\n\nMACRO=\"nTraps\"\n\nVALUES=()\n\nfor ((power = 20; power \u003c= 28; power += 2))\ndo\n    VALUES+=( \"$(( 2 \u003c\u003c ($power - 1) ))\" )\ndone\n```\n\nWe can compile the source file with different numbers of trapezoids defined and schedule it with different numbers of processes in a single command as follows\n\n    mpis-profile ./mpi_trap1.c ./description.sh\n\nThe following files and directories are generated\n\n    find ./out\n\n    ./out/\n    ./out/21_11_2019\n    ./out/21_11_2019/15_13_08\n    ./out/21_11_2019/15_13_08/67108864\n    ./out/21_11_2019/15_13_08/67108864/4\n    ./out/21_11_2019/15_13_08/67108864/4/mpi_trap1_4_argo059_job.stderr\n    ./out/21_11_2019/15_13_08/67108864/4/mpi_trap1_4_argo059_job.stdout\n    ...\n    ./out/21_11_2019/15_13_08/268435456/32\n    ./out/21_11_2019/15_13_08/268435456/32/mpi_trap1_32_argo059_job.stderr\n    ./out/21_11_2019/15_13_08/268435456/32/mpi_trap1_32_argo059_job.stdout\n    ./out/21_11_2019/15_13_08/268435456/16\n    ./out/21_11_2019/15_13_08/268435456/16/mpi_trap1_16_argo059_job.stderr\n    ./out/21_11_2019/15_13_08/268435456/16/mpi_trap1_16_argo059_job.stdout\n    ./out/21_11_2019/15_13_08/results.csv\n\nHere is how our measurements look like\n\n    head -5 ./out/21_11_2019/15_13_08/results.csv\n\n    nTraps   , Processes, Time    , Speed Up       , Εfficiency\n    1048576  , 1        , 0.000925, 1.0            , 1.0\n    1048576  , 2        , 0.00082 , 1.12804878049  , 0.564024390245\n    1048576  , 4        , 0.000841, 1.09988109394  , 0.274970273485\n    1048576  , 8        , 0.001022, 0.905088062622 , 0.113136007828\n\n## **Licence**\n\nThis project is licensed under the [MIT License](./LICENCE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbillsioros%2Fmpisuite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbillsioros%2Fmpisuite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbillsioros%2Fmpisuite/lists"}