{"id":27643405,"url":"https://github.com/rrze-hpc/discostic-sim","last_synced_at":"2025-07-26T15:15:40.918Z","repository":{"id":218761719,"uuid":"746934051","full_name":"RRZE-HPC/DisCostiC-Sim","owner":"RRZE-HPC","description":"A cross-architecture resource-based parallel simulation framework that can efficiently predict the performance of real or hypothetical massively parallel MPI programs on current and future heterogeneous systems.","archived":false,"fork":false,"pushed_at":"2024-02-13T12:35:25.000Z","size":20157,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-24T00:13:57.600Z","etag":null,"topics":["current-clusters","future-clusters","open-source","parallel-computing","performance-analysis","performance-engineering","performance-simulation"],"latest_commit_sha":null,"homepage":"https://github.com/RRZE-HPC/DisCostiC-Sim/wiki/DisCostiC","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RRZE-HPC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":".zenodo.json"}},"created_at":"2024-01-22T23:45:23.000Z","updated_at":"2025-04-13T14:33:01.000Z","dependencies_parsed_at":"2025-04-24T00:23:51.575Z","dependency_job_id":null,"html_url":"https://github.com/RRZE-HPC/DisCostiC-Sim","commit_stats":null,"previous_names":["rrze-hpc/discostic-sim"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/RRZE-HPC/DisCostiC-Sim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RRZE-HPC%2FDisCostiC-Sim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RRZE-HPC%2FDisCostiC-Sim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RRZE-HPC%2FDisCostiC-Sim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RRZE-HPC%2FDisCostiC-Sim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RRZE-HPC","download_url":"https://codeload.github.com/RRZE-HPC/DisCostiC-Sim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RRZE-HPC%2FDisCostiC-Sim/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267184146,"owners_count":24049127,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["current-clusters","future-clusters","open-source","parallel-computing","performance-analysis","performance-engineering","performance-simulation"],"created_at":"2025-04-24T00:13:45.634Z","updated_at":"2025-07-26T15:15:40.852Z","avatar_url":"https://github.com/RRZE-HPC.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DisCostiC: Distributed Cost in Cluster\n\n  \u003c/a\u003e\u003ca href=\"https://rrze-hpc.github.io/DisCostiC-Sim/doc/html/index.html\"\u003e\n  \u003cimg alt=\"HTML Doxygen doc\" src=\"https://img.shields.io/badge/HTML%20Doxygen%20Doc%20-Read%20now!-red\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://github.com/RRZE-HPC/DisCostiC-Sim/tree/main/doc/DisCostiC.pdf\"\u003e\n  \u003cimg alt=\"LaTeX Doxygen doc\" src=\"https://img.shields.io/badge/LaTeX%20Doxygen%20Doc%20-Read%20now!-green\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://github.com/RRZE-HPC/DisCostiC-Sim/wiki/DisCostiC\"\u003e\n  \u003cimg alt=\"Wiki Page\" src=\"https://img.shields.io/badge/Wiki%20Page%20-Read%20now!-pink\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://matrix.to/#/!OeNPngEkrqRRkBJjwz:gwaaf.rrze.fau.de?via=gwaaf.rrze.fau.de\"\u003e\n  \u003cimg alt=\"Join the discussion on Matrix\" src=\"https://img.shields.io/badge/Discussions%20%26%20Support-Chat%20now!-yellow\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://www.linkedin.com/in/ayeshaafzal-/\" target=\"\\_parent\"\u003e\n  \u003cimg alt=\"linkedin\" src=\"https://i.stack.imgur.com/gVE0j.png\" /\u003e\n  \u003ca href=\"https://github.com/AyeshaAfzal91\" target=\"\\_parent\"\u003e\n  \u003cimg alt=\"\" src=\"https://img.shields.io/github/stars/tanstack/react-table.svg?style=social\u0026label=Star\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://twitter.com/AyeshaHamad4\" target=\"\\_parent\"\u003e\n  \u003cimg alt=\"\" src=\"https://img.shields.io/twitter/follow/AyeshaHamad4.svg?style=social\u0026label=Follow\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://doi.org/10.5281/zenodo.10606291\"\u003e\n  \u003cimg alt=\"DisCostiC Zenodo release\" src=\"https://zenodo.org/badge/DOI/Zenodo%20Release.svg\" /\u003e\n  \u003c/a\u003e\u003ca href=\"https://github.com/RRZE-HPC/DisCostiC-Sim/releases\"\u003e\n  \u003cimg alt=\"HTML Doxygen doc\" src=\"https://img.shields.io/badge/GitHub%20Release%20-See%20now!-purple\" /\u003e\n\n## Table of Contents\n\n  * [💡Description](#description)\n    * [⚙️ Framework workflow](#workflow)\n    * [🖍 Advantages over existing tools](#advantages)\n  * [:octocat: Compilation and build](#compilation-and-build)\n    * [⚡ Installation](#installation)\n    * [⏱️ Configuration settings](#configuration)\n    * [🥅 Compilation](#compilation)\n    * [:green_circle: Run](#run)\n    * [🔁 Clean and uninstall](#clean)\n  * [:signal_strength: DisCostiC output](#output)\n    * [🌐 1. Web interface with Google Chrome browser (DMS file format)](#chrome)\n    * [📊 2. Graphical user interface with Vampir (OTF2 file format)](#vampir)\n    * [📊 3. Graphical user interface with ITAC (STF file format)](#itac)\n    * [🗄️ 4. Statistical or log data](#stat)\n  * [💻 System model](#cluster-configuration)\n    * [🔌 Cluster model](#cluster)\n    * [🔌 Interconnect model](#interconnect)\n    * [🔌 Node model](#node)\n  * [👩‍💻 Automating DSEL generation through static analysis](#automating-dsel-generation-through-static-analysis)\n    * [📝 Essential DisCostiC routines](#essential-routines)\n    * [📝 Code blueprint as DisCostiC input](#DSEL)\n    * [📝 Code documentation: suite of C++ data structures and enumerated types](#documentation)\n  * [🧐 MPI-parallelized DisCostiC implementation](#mpi-parallelized-implementation)\n  * [🚀 Planned features for further development](#library-limitations)\n  * [📚 References about the theory of potential application scenarios for DisCostiC](#references)\n  * [🔒 License](#license)\n  * [:warning: Disclaimer](#disclaimer)\n  * [🔗 Acknowledgement](#acknowledgement)\n  * [📱Contact](#contact)\n\n\n\u003ca name=\"description\"\u003e\u003c/a\u003e\n## 💡 Description\n\nA `cross-architecture resource-based parallel simulation framework` that can efficiently predict the performance of real or hypothetical massively parallel MPI programs on current and future highly parallel heterogeneous systems.✨\n\nThe runtime predictions are carried out in a controlled environment through _`straightforward, portable, scalable, and efficient simulations`_.⭐\n\nThe simulation framework is an automated version of `analytical first-principle performance models at a full-scale scope`, including cores, chips, nodes, networks, clusters, and their mutual interactions plus inherent bottlenecks such as memory bandwidth.🌟\n\nThe application blueprint created using `Domain Specific Embedded Language (DSEL)` enables key concepts and elements embedded in the modern C++ language, enhancing `readability for domain experts and accurately encoding inter-process dependencies without introducing unknown consequences from the actual systems`. 💥\n\n\n\u003ca name=\"workflow\"\u003e\u003c/a\u003e\n⚙️ **Framework workflow**💬\n\n```diff\n- Application model\n\n    * Domain knowledge of the application expressed in the DisCostiC DSEL language provides information on call dependencies and properties for computation (data access pattern, flop count) and communication (volume, mode and protocol)\n\n- Analytic first-principle performance models\n    \n    * Computation models: Fundamental analytic Roofline and refined ECM models including memory bandwidth bottleneck concept at the system level\n    * Communication models: Fundamental latency-bandwidth and refined LogGP models\n\n- System models\n    \n    * Cluster, network and node models for the full topology of the target system, including multiple chips, nodes, network interfaces and clusters\n \n+ DisCostiC combines all of them to simulate the runtime cost of distributed applications.\n```\n\n\u003ca name=\"advantages\"\u003e\u003c/a\u003e\n🖍 **Advantages over existing tools**💬\n\n```diff\n- DSEL approach for creating application blueprint \n\n    * Accurately encoding inter-process dependencies without introducing unknown consequences from the actual systems \n\n- A full-scale first-principle-model-based simulator \n\n    * Automating analytical first-principle performance models on the entire hierarchy of parallel systems\n\n- Efficient speed \n\n    * No intermediate tracing files requirement like any offline, trace-driven tools \n    * No high memory requirement like any online tools that use the host architectures to execute code\n\t\n+ Last but not least, an open-source low-entry cost lightweight simulator enabling model-based design-space exploration \n```\n\n\u003ca name=\"compilation-and-build\"\u003e\u003c/a\u003e\n## :octocat: Compilation and build\n\nFirst, use the following command to clone the git repository:\n\n```\ngit clone git@github.com:RRZE-HPC/DisCostiC-Sim.git \u0026\u0026 cd DisCostiC-Sim\n```\n\n\u003ca name=\"installation\"\u003e\u003c/a\u003e\n⚡ **Installation**:thought_balloon:\n\nBefore proceeding, make sure the environment is prepared for the compilation.\nThe installation steps are listed below:\n```\nmodule load python git intel intelmpi cmake itac vampir\nconda create --name XYZ\nconda activate XYZ\nconda install pip \ncmake -DCMAKE_INSTALL_PREFIX=~/.local . \u0026\u0026 make all install\n```\nOne way to check the installation is to print the version and the help of the DisCostiC using `./discostic --version` and `./discostic --help`.\n\n\u003ca name=\"configuration\"\u003e\u003c/a\u003e\n⏱️ **Configuration settings**:thought_balloon:\n\nThe [test](test) folder in DisCostiC offers multiple MPI-parallelized programs (`benchmark_kernel`) in distinct functionality for computation (`kernel_mode`) and communication (`exchange_mode`). For illustration, a few examples are given below:\n\n|`benchmark_kernel`                 | Description   |\n|--------------------- | ------------- |\n|`HEAT`          |       Two-dimensional five-point Jacobi kernel with communication |\n|`SOR`          |    Gauss-Seidel Successive Over-Relaxation solver with communication |\n|`DMMM`               | Dense Matrix Matrix Multiplication kernel with communication |\n|`DMVM`               | Dense Matrix Vector Multiplication kernel with communication |\n|`DMVM-TRANSPOSE`     | Dense Matrix Transpose Vector Multiplication kernel with communication |\n|`HEATHEAT`                | Back-to-back two HEAT kernels with communication |\n|`HEATSOR`                | Back-to-back HEAT and SOR kernels with communication |\n|`HEATDIVIDE`                | Back-to-back HEAT and DIVIDE kernels with communication |\n|`HPCG`               | High Performance Conjugate Gradient |\n|`STENCIL-3D-7PT`                | Three-dimensional seven point stencil kernel with communication |\n|`STENCIL-3D-27PT`                | Three-dimensional twenty seven point stencil kernel with communication |\n|`STENCIL-UXX`                | UXX stencil kernel with communication |\n|`STENCIL-3D-LONGRANGE`                | 3D long range stencil kernel with communication |\n|`STENCIL-1D-3PT`                | One-dimensional three point stencil kernel with communication |\n|`STREAM`                | STREAM Triad kernel with communication |\n|`SCHOENAUER`                | SCHOENAUER Triad kernel with communication |\n|`SCHOENAUER-DIV`                | SCHOENAUER divide Triad kernel with communication |\n|`WAXPY`                | WAXPY kernel with communication |\n|`DAXPY`                | DAXPY kernel with communication |\n|`SUM`                | SUM kernel with communication |\n|`VECTOR-SUM`                | Vector SUM kernel with communication |\n|`ADD`                | ADD kernel with communication |\n|`DIVIDE`                | DIVIDE kernel with communication |\n|`SCALE`                | SCALE kernel with communication |\n|`COPY`                | COPY kernel with communication |\n|`KAHAN-DOT`                | KAHAN-DOT kernel with communication |\n|`SCALAR-PRODUCT`                | Scalar Product kernel with communication |\n\n\nThe following `kernel_mode` is exclusively offered for the flexibility of the framework; the simulator's prediction remains unaffected by the selection of the `kernel_mode`. Further explanation of this mode can be found in the [Essential DisCostiC routines](#essential-routines) section.\n\n| `kernel_mode`                 | Integrated tool | Description |\n| --------------------- | ------------- | ------------- |\n| `COMP`          |   no external tool integrated |  This directly embeds the single-core pre-recorded ECM performance model data of the computational kernel into the simulator. |\n| `LBL`           |   no external tool integrated |  This reads the pre-recorded ECM performance model data for the computational kernel from an external file located at [nodelevel/configs](nodelevel/configs) folder. |\n| `SRC`           |   [Kerncraft](https://github.com/RRZE-HPC/kerncraft) integrated      |  This directly embeds the source code of the computational kernel into the simulator. |\n| `FILE`          |   [Kerncraft](https://github.com/RRZE-HPC/kerncraft) integrated      |  This reads the source code for the computational kernel from an external file located at [nodelevel/kernels](nodelevel/kernels) folder. |\n\nThe following `exchange_mode` is provided in LBL mode to enable model-based exploration through experimenting with various communication patterns in MPI applications. \n\n| `exchange_mode`                  | Description |\n| --------------------- | ------------- |\n| `message_size`          |       It specifies the size of the message to be exchanged in bytes. |\n| `step_size`          |       It describes the step size of the message exchange as an int (1: distance one communication, 2: distance two communication, ...). |\n| `direction`          |       It specifies the direction of message exchange as an int (1: uni-directional upwards shift in only positive direction, 2: bi-directional upwards and downwards shift in both positive and negative directions). |\n| `periodic`          |       It enables or disables the communication periodicity as a bool (0: false, 1: true). |\n\nThe [config.cfg](config.cfg) file can be edited to select the `benchmark_kernel` and `kernel_mode`. More information about this `config.cfg` file is available in the [DisCostiC help](#help) documentation.\n\n\u003ca name=\"compilation\"\u003e\u003c/a\u003e\n🥅 **Compilation**:thought_balloon:\n\nThe compilation offers the following choices to enable the output report's data format generation and to enable the tracing of the simulator's own implementation. An executable will be generated after compilation. \n\n| Command                  | Description |\n| --------------------- | ------------- |\n| `make`          | This enables JSON data format without ITAC profiling of simulator implementation. |\n| `make otf2`          | This enables both JSON and OTF2 data format without ITAC profiling of simulator implementation. |\n| `make trace_MPI`          | This enables JSON data format and the standard ITAC tracing mode of simulator implementation with the information about MPI call functions (enabled flag: `-trace`). |\n| `make trace_all`               | This enables JSON data format and the verbose ITAC tracing mode of simulator implementation with the information on both user-defined and MPI call functions (enabled flag: `-trace -tcollect flag`). |\n\n\n\u003ca name=\"run\"\u003e\u003c/a\u003e\n:green_circle: **Run**:thought_balloon:\n\nIn the batch script, the number of the simulator processes is configured as the number of simulated processes plus one. To run the batch script on any system, ITAC profiling can be enabled or disabled, which will or will not dump the simulator's own trace in ITAC:\n\n| Command                  | Description |\n| --------------------- | ------------- |\n| `sbatch Run_Simulation.sh`          | This performs the simulation without tracing the own implementation of the simulator using ITAC. |\n| `sbatch Run_Simulation_ITAC.sh`          | This dumps the simulator's own trace into ITAC to investigate the implementation of the simulator. |\n\n\nThe [Run_Simulation_ITAC.sh](Run_Simulation_ITAC.sh) script only generates a single STF file (`simulation.single.stf`) in the main directory due to the export of the following variables:\n\n```\nexport VT_LOGFILE_FORMAT=SINGLESTF\nexport VT_LOGFILE_NAME=simulation\nexport VT_LOGFILE_PREFIX=$SLURM_SUBMIT_DIR\nexport VT_FLUSH_PREFIX=/tmp\n```\nThe formats, names and locations of output files in these environmental variables can be adjusted as desired.\n\n\u003ca name=\"clean\"\u003e\u003c/a\u003e\n🔁 **Clean and uninstall**:thought_balloon:\n\n| Command                  | Description |\n| --------------------- | ------------- |\n| `make clear`            |   This cleans up the working directory by removing all unnecessary DisCostiC files, such as *.dms, *.otf, *.csv files. |\n| `make uninstall`        |   This uninstalls the DisCostiC framework, including installed files and CMake-specific files. |\n\n\n\u003ca name=\"output\"\u003e\u003c/a\u003e\n## :signal_strength: DisCostiC output\n\n\u003ca name=\"chrome\"\u003e\u003c/a\u003e\n🌐 **1. Web interface with Google Chrome browser (DMS file format)**:speech_balloon:\n\nUpon completion of the run, DisCostiC generates a report referred to as `DisCostiC.dms`.\n`DisCostiC.dms` is a straightforward [JSON object data format file](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit) that can be viewed using the [Google Chrome browser](chrome://tracing).\nUse the following Google Chrome web browser to load the generated JSON file:\n\n```\nchrome://tracing\n```\n\n\u003ca name=\"vampir\"\u003e\u003c/a\u003e\n📊 **2. Graphical user interface with Vampir (OTF2 file format)**:speech_balloon:\n\nUpon completion of a run that was compiled with `make otf2`, DisCostiC generates a report called `DisCostiC/traces.otf2`. The `DisCostiC/traces.otf2` file is an [OTF2 object data format](https://perftools.pages.jsc.fz-juelich.de/cicd/otf2/tags/otf2-3.0.2/html/) file and can be viewed using third-party tools like [ITAC](https://www.intel.com/content/www/us/en/developer/tools/oneapi/trace-analyzer.html) and [Vampir](https://vampir.eu).\nThis format is produced by the [ChromeTrace2Otf2](https://profilerpedia.markhansen.co.nz/converters/otf2-cli-chrome-trace-converter/) converter, which converts the JSON object data format file `(DisCostiC.dms)` to the OTF2 object data format file `(DisCostiC/traces.otf2)`.\n\nUse the following commond to open an OTF2 small trace file (`DisCostiC/traces.ot2`) in [Vampir](https://vampir.eu):\n\n```\nvampir DisCostiC/traces.otf2\n```\n\n\u003ca name=\"itac\"\u003e\u003c/a\u003e\n📊 **3. Graphical user interface with ITAC (STF file format)**:speech_balloon:\n\nTo convert an OTF2 file to a single Structured Trace File (STF) file format `DisCostiC.single.stf` and to open it in [ITAC](https://www.intel.com/content/www/us/en/developer/tools/oneapi/trace-analyzer.html), invoke the Intel Trace Analyzer GUI and follow the below steps:\n\n```\n(1) traceanalyzer \u0026\n(2) Go to File \u003e Open; from the Files of a type field, select Open Trace Format, navigate to the `DisCostiC/traces.otf2` file, and double-click to open it\n(3) The OTF2 to STF conversion dialog appears. Review the available fields and checkboxes, and click Start to start the conversion. As a result, the OTF2 file will be converted to STF (DisCostiC/traces.otf2.single.stf), and you will be able to view it in the Intel Trace Analyzer.\n```\n\n\u003ca name=\"stat\"\u003e\u003c/a\u003e\n🗄️ **4. Statistical or log data**:thought_balloon:\n\n\t    ----------------------------------------------------------------\n        DisCostiC\n        ----------------------------------------------------------------\n        Full form: Distributed Cost in Cluster\n        Version: v1.0.0\n        Timestamp: Tue Jan 20 11:16:42 2024\n        Author: Ayesha Afzal \u003cayesha.afzal@fau.de\u003e\n        Copyright: © 2024 HPC, FAU Erlangen-Nuremberg. All rights reserved\n\n        HEAT program in FILE mode of kernel is simulated for 100000 iterations and 72 processes\n\n        ----------------------------------------------------------------\n        COMPUTATIONAL PERFORMANCE MODEL\n        ----------------------------------------------------------------\n        Code simulating on IcelakeSP_Platinum-8360Y chip of cores: {1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 }\n        Single-precision maximum chip performance [GF/s]: {614.4 , 1228.8 , 1843.2 , 2457.6 , 3072 , 3686.4 , 4300.8 , 4915.2 , 5529.6 , 6144 , 6758.4 , 7372.8 , 7987.2 , 8601.6 , 9216 , 9830.4 , 10444.8 , 11059.2 , 11673.6 , 12288 , 12902.4 , 13516.8 , 14131.2 , 14745.6 , 15360 , 15974.4 , 16588.8 , 17203.2 , 17817.6 , 18432 , 19046.4 , 19660.8 , 20275.2 , 20889.6 , 21504 , 22118.4 }\n        Double-precision maximum chip performance [GF/s]: {307.2 , 614.4 , 921.6 , 1228.8 , 1536 , 1843.2 , 2150.4 , 2457.6 , 2764.8 , 3072 , 3379.2 , 3686.4 , 3993.6 , 4300.8 , 4608 , 4915.2 , 5222.4 , 5529.6 , 5836.8 , 6144 , 6451.2 , 6758.4 , 7065.6 , 7372.8 , 7680 , 7987.2 , 8294.4 , 8601.6 , 8908.8 , 9216 , 9523.2 , 9830.4 , 10137.6 , 10444.8 , 10752 , 11059.2 }\n\n        ----------------------------------------------------------------\n        COMMUNICATION PERFORMANCE MODEL\n        ----------------------------------------------------------------\n        Code simulating on InfiniBand with IntelMPI library\n        Hockney model parameters (intra chip): 41185.7 [ns] latency, 0.079065 [s/GB] inverse bandwidth \n        Hockney model parameters (inter chip): 64567.6 [ns] latency, 0.124122 [s/GB] inverse bandwidth \n        Hockney model parameters (inter node): 47628.7 [ns] latency, 0.091516 [s/GB] inverse bandwidth \n        Hockney model parameters (inter cluster): 47628.7 [ns] latency, 0.091516 [s/GB] inverse bandwidth \n        LogGP model additional parameters (intra chip): 24429.9 [ns] o, 199.008 [ns] g, 0.065 [ns] O\n        LogGP model additional parameters (inter chip): 39748.3 [ns] o, 338.249 [ns] g, 0.085 [ns] O\n        LogGP model additional parameters (inter node): 47938.9 [ns] o, 4980.51 [ns] g, 0.084 [ns] O\n        LogGP model additional parameters (inter cluster): 47938.9 [ns] o, 4980.51 [ns] g, 0.084 [ns] O\n        Eager threshold: 65535 bytes\n\n        ----------------------------------------------------------------\n        Simulation runtime in DisCosTiC: 15.7239 [s]\n        ----------------------------------------------------------------\n\n        ----------------------------------------------------------------\n        FULL APPLICATION PERFORMANCE (for all MPI processes)\n        rank                         runtime [s]\n        ----------------------------------------------------------------\n        0                              2154.83\n        1                              2154.83\n        .                              .......\n        .                              .......\n        .                              .......\n        71                             2154.67\n        ----------------------------------------------------------------\n\n\u003ca name=\"cluster-configuration\"\u003e\u003c/a\u003e\n## 💻 System model\n\nThe system parameters, either hypothetical or actual, can be tuned by editing the [config.cfg](config.cfg) file available in the current directory.\nFor the detailed documentation of the system model, please take a look at the [config.cfg](config.cfg) file and the `YAML files` available in the [machine files](nodelevel/machine-files) and [network files](interconnectlevel/network-files) directories.\n\n\u003ca name=\"cluster\"\u003e\u003c/a\u003e    \n🔌 **Cluster model**:thought_balloon:\n\nThe resource allocation, intercluster characteristics, and runtime modalities are all included in the cluster model.\n\n| Metadata information                  | Description |\n| --------------------- | ------------- |\n| `number_of_nodes`             \t\t    | Number of utilizing nodes on a cluster            |\n| `task_per_node`\t\t\t                | Number of utilizing cores on the node of a cluster| \n| `number_of_processes`                   | Number of running processes on the cluster          |\n| `inter_cluster or heterogeneous`\t\t    | Communication across clusters (0: inter cluster disabled; 1: inter cluster enabled) |\n| `number_of_iterations`                   | Number of iterations for the program run |\n| `dim_x, dim_y, dim_z`                   | Problem size for the program run; high-dimensional parameters will be disregarded for low-dimensional problems. |\n\n\u003ca name=\"interconnect\"\u003e\u003c/a\u003e    \n🔌 **Interconnect model**:thought_balloon:\n\nThe YAML formatted network files and the choice of performance model and mode of communication are included in the network model.\n\n| Metadata information                  | Description | \n| ------------------------------------- | ------------- | \n| `name`                                      | Name of the YAML formatted interconnect file | \n| `intra_chip`\t\t\t\t\t\t\t    | Communication inside chip                      | \n| `inter_chip`\t\t\t\t\t\t\t    | Communication across chips                     | \n| `inter_node`\t\t\t\t\t\t\t    | Communication across nodes                        | \n| `latency`\t\t\t\t\t\t\t        | Latency in sec for various kinds of interconnects | \n| `bandwidth`\t\t\t\t\t\t\t        | Bandwidth in GB/s for various kind of interconnects | \n| `eager_limit`\t\t\t\t\t\t\t    | Data size in bytes at which communication mode changes from eager to rendezvous protocol | \n| `waitio_mode`                               | Mode of the WaitIO MPI library (socket, file or hybrid) | \n| `comm_model`                                | Performance model for communication (0: LogGP, 1: HOCKNEY) | \n\nFor illustration, a few examples of available [network files](interconnectlevel/network-files) are given below:\n\n| Network files (YAML format)                 | Description | \n| --------------------- | ------------- | \n| [InfiniBand_WaitIO_intercluster](interconnectlevel/network-files/A64FX_InfiniBand_WaitIO_socket_intercluster.cfg)          |       Communication across clusters for WaitIO MPI library and InfiniBand interconnect | \n| [InfiniBand_WaitIO_internode](interconnectlevel/network-files/IcelakeSP_Platinum-8360Y_InfiniBand_WaitIO_socket_internode.cfg)          |       Communication across nodes for WaitIO MPI library and InfiniBand interconnect | \n| [InfiniBand_WaitIO_interchip](interconnectlevel/network-files/IcelakeSP_Platinum-8360Y_InfiniBand_WaitIO_socket_interchip.cfg)          |       Communication across chips for WaitIO MPI library and InfiniBand interconnect | \n| [InfiniBand_WaitIO_intrachip](interconnectlevel/network-files/IcelakeSP_Platinum-8360Y_InfiniBand_WaitIO_socket_intrachip.cfg)          |       Communication inside the chip for WaitIO MPI library and InfiniBand interconnect | \n| [Tofu-D_WaitIO_internode](interconnectlevel/network-files/A64FX_Tofu-D_WaitIO_socket_internode.cfg)          |       Communication across nodes for WaitIO MPI library and Tofu-D interconnect    | \n| [Tofu-D_WaitIO_interchip](interconnectlevel/network-files/A64FX_Tofu-D_WaitIO_socket_interchip.cfg)          |       Communication across chips for WaitIO MPI library and Tofu-D interconnect    | \n| [Tofu-D_WaitIO_intrachip](interconnectlevel/network-files/A64FX_Tofu-D_WaitIO_socket_intrachip.cfg)          |       Communication inside the chip for WaitIO MPI library and Tofu-D interconnect | \n| [InfiniBand_IntelMPI_internode](interconnectlevel/network-files/IcelakeSP_Platinum-8360Y_InfiniBand_IntelMPI_internode.cfg)          |       Communication across nodes for Intel MPI library and InfiniBand interconnect | \n| [InfiniBand_IntelMPI_interchip](interconnectlevel/network-files/IcelakeSP_Platinum-8360Y_InfiniBand_IntelMPI_interchip.cfg)          |       Communication across chips for Intel MPI library and InfiniBand interconnect | \n| [InfiniBand_IntelMPI_intrachip](interconnectlevel/network-files/IcelakeSP_Platinum-8360Y_InfiniBand_IntelMPI_intrachip.cfg)          |       Communication inside the chip for Intel MPI library and InfiniBand interconnect  | \n| [OmniPath_IntelMPI_internode](interconnectlevel/network-files/BroadwellEP_E5-2630v4_OmniPath_IntelMPI_internode.cfg)          |       Communication across nodes for Intel MPI library and Omni-Path interconnect     | \n| [OmniPath_IntelMPI_interchip](interconnectlevel/network-files/BroadwellEP_E5-2630v4_OmniPath_IntelMPI_interchip.cfg)          |       Communication across chips for Intel MPI library and Omni-Path interconnect     | \n| [OmniPath_IntelMPI_intrachip](interconnectlevel/network-files/BroadwellEP_E5-2630v4_OmniPath_IntelMPI_intrachip.cfg)          |       Communication inside the chip for Intel MPI library and Omni-Path interconnect | \n\n\n\u003ca name=\"node\"\u003e\u003c/a\u003e    \n🔌 **Node model**:thought_balloon:\n\nThe machine files in YAML format, the selection of compiler settings and the performance model of computation are included in the node model.\n\n| Metadata information                  | Description | \n| --------------------- | ------------- | \n| `name`                                      | Name of the YAML formatted processor file | \n| `sockets_per_node`                          | Number of sockets in one node | \n| `ccNUMA_domains_per_socket`                 | Number of ccNUMA domains in one socket | \n| `cores_per_ccNUMA_domain`                   | Number of cores in one ccNUMA domain | \n| `FP_instructions_per_cycle`\t\t\t\t\t| Floating point instructions (ADD, MUL) per cycle | \n| `FP_operations_per_instruction_(SP/DP)`\t\t| Single or double precision floating point operations per instruction | \n| `clock_frequency`\t\t\t\t\t\t\t| Clock frequency in GHz | \n| `memory_bandwidth`\t\t\t\t\t\t\t| Memory bandwidth in GB/s | \n| `compiler-flags`                            | STD and SIMD optimizations of the compiler | \n| `pmodel`                                | Performance model for computation (Roofline, ECM) | \n\nFor illustration, a few examples of available [machine files](nodelevel/machine-files) are given below:\n\n| Machine files (YAML format)                 | Description | \n| --------------------- | ------------- | \n| [A64FX](nodelevel/machine-files/A64FX.yml)           |       48 core Fujitsu A64FX FX1000 CPU @ 1.8 GHz   | \n| [IcelakeSP_Platinum-8360Y](nodelevel/machine-files/IcelakeSP_Platinum-8360Y.yml)          |      36 core Intel(R) Xeon(R) Icelake SP Platinum 8360Y CPU @ 2.4 GHz         | \n| [CascadelakeSP_Gold-6248_SNC](nodelevel/machine-files/CascadelakeSP_Gold-6248_SNC.yml)           |   20 core Intel(R) Xeon(R) Cascadelake SP Gold 6248 CPU @ 2.5 GHz      | \n| [SkylakeSP_Gold-5122](nodelevel/machine-files/SkylakeSP_Gold-5122.yml)           |       4 core Intel(R) Xeon(R) Skylake SP Gold 5122 CPU @ 3.6 GHz  |  \n| [SkylakeSP_Gold-6148](nodelevel/machine-files/SkylakeSP_Gold-6148.yml)           |       20 core Intel(R) Xeon(R) Skylake SP Gold 6148 CPU @ 2.4 GHz  | \n| [SkylakeSP_Gold-6148_SNC](nodelevel/machine-files/SkylakeSP_Gold-6148_SNC.yml)           |    20 core Intel(R) Xeon(R) Skylake SP Gold 6148 CPU with SNC enabled @ 2.4 GHz   |   \n| [SkylakeSP_Platinum-8147](nodelevel/machine-files/SkylakeSP_Platinum-8147_2.7GHz.yml)           |      24 core Intel(R) Xeon(R) Skylake SP Platinum 8174 CPU @ 3.1 GHz (2.7 GHz used)     | \n| [BroadwellEP_E5-2630v4](nodelevel/machine-files/BroadwellEP_E5-2630v4.yml)           |       18 core Intel(R) Xeon(R) Broadwell EN/EP/EX E5-2697 v4 CPU @ 2.3 GHz | \n| [BroadwellEP_E5-2697v4_CoD](nodelevel/machine-files/BroadwellEP_E5-2697v4_CoD.yml)           |       18 core Intel(R) Xeon(R) Broadwell EN/EP/EX E5-2697 v4 CPU with CoD enabled @ 2.3 GHz | \n| [HaswellEP_E5-2695v3](nodelevel/machine-files/HaswellEP_E5-2695v3.yml)           |       14 core Intel(R) Xeon(R) Haswell EN/EP/EX E5-2695 v3 CPU @ 2.3 GHz | \n| [HaswellEP_E5-2695v3_CoD](nodelevel/machine-files/HaswellEP_E5-2695v3_CoD.yml)           |       14 core Intel(R) Xeon(R) Haswell EN/EP/EX E5-2695 v3 CPU with CoD enabled @ 2.3 GHz  | \n| [IvyBridgeEP_E5-2660v2](nodelevel/machine-files/IvyBridgeEP_E5-2660v2.yml)           |       10 core Intel(R) Xeon(R) IvyBridge EN/EP/EX E5-2660 v2 CPU @ 2.2 GHz | \n| [IvyBridgeEP_E5-2690v2](nodelevel/machine-files/IvyBridgeEP_E5-2690v2.yml)           |       10 core Intel(R) Xeon(R) IvyBridge EN/EP/EX E5-2690 v2 CPU @ 3.0 GHz | \n| [SandyBridgeEP_E5-2680](nodelevel/machine-files/SandyBridgeEP_E5-2680.yml)           |    8 core Intel(R) Xeon(R) SandyBridge EN/EP E5-2680 CPU @ 2.7 GHz   | \n\n\u003ca name=\"help\"\u003e\u003c/a\u003e    \n**DisCostiC help**:thought_balloon:\n\nThe help of DisCostiC (`./discostic --help`) lists as:\n\n```\nDistributed Cost in Cluster (DisCostiC)\nVersion : v1.0.0\nAuthor : Ayesha Afzal \u003cayesha.afzal@fau.de\u003e\nCopyright : 2024 HPC, FAU Erlangen-Nuremberg. All rights reserved\n\n------------------------------------\n Arguments for ./discostic\n------------------------------------\n     --version, -v          show simulator's version information and exit\n     --help, -h             show this help message and exit\n\n------------------------------------\n Application model for config.cfg\n------------------------------------\n     benchmark_kernel       name of the kernel used in the program\n     kernel_mode            mode of the kernel (FILE, SRC, LBL, COMP)\n\n------------------------------------\n Cluster model for config.cfg\n------------------------------------\n     heteregeneous          a bool flag to enable or disable the second system (1: enabled, 0: disabled)\n     number_of_iterations   number of iterations of the program\n     dim_x, dim_y, dim_z    problem size; high-dimensional parameters will be disregarded for low-dimensional problems.\n     task_per_node          number of running processes on one node\n     number_of_processes    total number of running processes\n\n------------------------------------\n Interconnect model for config.cfg\n------------------------------------\n     interconnect           name of the interconnect\n     MPI_library            name of the MPI library for the first system (IntelMPI, WaitIO)\n     comm_model             performance model of communication (0: LogGP, 1: HOCKNEY)\n     waitio_mode            mode of the WaitIO MPI library (socket, file or hybrid)\n\n------------------------------------\n Node model for config.cfg\n------------------------------------\n     micro_architecture     name of the YAML machine file\n     compiler-flags         STD and SIMD optimizations (-03 -xCORE-AVX512 -qopt-zmm-usage=high, -03 -xHost -xAVX, -Kfast -DTOFU); If not set, flags are taken from the YAML formatted machine file.\n     pmodel                 performance model of computation (ECM, Roofline)\n     vvv                    a bool flag to enable or disable the verbose node output (0: disabled, 1: enabled)\n     cache-predictor        cache prediction with layer conditions or cache simulation with pycachesim (LC, SIM)\n     penalty                runtime penalty for the computation model in nanoseconds, used only in LBL or COMP mode\n\n------------------------------------\n Delay injection mode for config.cfg\n------------------------------------\n     delay                  a bool flag to enable or disable the delay injection (0: disabled, 1: enabled)\n     delay_intensity        intensity of delay as a multiple of computation time of one iteration\n     delay_rank             process rank of the injected delay\n     delay_timestep         iteration for the occurrence of the injected delay\n\n------------------------------------\n Noise injection mode for config.cfg\n------------------------------------\n     noise                  a bool flag to enable or disable the noise injection (0: disabled, 1: enabled)\n     noise_intensity        intensity of random noise, i.e., rand() % noise_intensity \n     noise_start_timestep   starting iteration for the injected noise\n     noise_stop_timestep    ending iteration for the injected noise\n\n------------------------------------\n Output for config.cfg\n------------------------------------\n     filename               output file name that contains details for the debug purpose\n     chromevizfilename      output file name that contains all time-rank tracing data for visualization with Chrome tracing browser\n     Verbose                a bool flag to enable or disable the verbose output (0: disabled, 1: enabled)\n```\n\n\u003ca name=\"automating-dsel-generation-through-static-analysis\"\u003e\u003c/a\u003e\n## 👩‍💻 Automating DSEL generation through static analysis\n\nTo perform the static analysis, the following procedure must be followed before running the DisCostiC batch script:\n```\npip install -r staticanalysis/requirements.txt\npython staticanalysis/Convert-\u003cbenchmark_kernel\u003e.py\n```\n\n| Files                  | Description | \n| --------------------- | ------------- | \n| `Convert-\u003cbenchmark_kernel\u003e.py`          | A helper script that takes the original code and, through annotation, locates code loops and communication and identifies user-defined variables, such as dim_x and dim_y in the Cartesian stencil heat.c code, and ultimately generates DSEL code. | \n| `requirements.txt`               | Each (sub)dependency is listed and pinned using \"==\" to specify a particular package version; see [requirements.txt](staticanalysis/requirements.txt). This project makes use of the lightweight `Python tree data structure` [anytree==2.8.0](https://pypi.org/project/anytree) and `type hints for Python 3.7+` [typing_extensions==4.4.0](https://pypi.org/project/typing-extensions). These dependencies are later installed (normally in a virtual environment) through pip using the `pip install -r staticanalysis/requirements.txt` command. The generated tree's syntax is the same as the original C/C++ code. | \n\nFor the specified program, this will produce the following files:\n\n| Files                  | Location       |Description | \n| --------------------- | ------------- | ------------- | \n| `\u003cbenchmark_kernel\u003e.c`          |  [nodelevel/kernels](nodelevel/kernels)  | It comprises only the generated computational loop kernel of the MPI parallel program. | \n| `\u003cbenchmark_kernel\u003e_\u003ckernel_mode\u003e.hpp`   |  [test](test)               | It includes the entire generated code expressed in DisCostiC DSEL language. | \n\n\u003ca name=\"essential-routines\"\u003e\u003c/a\u003e\n📝 **Essential DisCostiC routines**:thought_balloon:\n\nThe goal is to offer convenient, compact and practically usable application programming interfaces (APIs) with appropriate abstractions. It provides information about the call tree and attributes for communication (volume, mode, and protocol) and computation (data access pattern, flop count).\n \n\n```cpp\n1. DisCostiC-\u003eRank_Init(DisCostiC::Indextype rank);\n\n2. DisCostiC-\u003eSetNumRanks(DisCosTiC::Datatype numrank);\n\n3. DisCostiC-\u003eExec(\"LBL:STREAM_TRIAD\", \n\t\t\t\t\tDisCostiC::Event depending_operations);\n   DisCostiC-\u003eExec(\"COMP:TOL=2.0||TnOL=1.0|TL1L2=3.0|TL2L3=6.0|TL3Mem=14.2\", \n   \t\t\t\t\tDisCostiC::Event depending_operations);\n   DisCostiC-\u003eExec(\"FILE:copy.c//BREAK:COPY//./BroadwellEP_E5-2697v4_CoD.yml//18//-D N 100000\", \n   \t\t\t\t\tDisCostiC::Event depending_operations);\n   DisCostiC-\u003eExec(\"SRC:double s, a[N],b[N],c[N];\\n\\nfor(int i=0; i\u003cN; i++)\\n\\ta[i]=b[i]+s*c[i];\\n//BREAK:DAXPY//./BroadwellEP_E5-2697v4_CoD.yml//18//-D N 100000\",\n   \t\t\t\t\tDisCostiC::Event depending_operations);\n\n4. DisCostiC-\u003eSend(const void *message, \n                    DisCosTiC::Datatype sending_message_size_in_bytes, \n                    DisCostiC::Datatype sending_message_datatype, \n                    DisCostiC::Indextype destination_rank,\n                    DisCostiC::Commtype communicator, \n                    DisCosTiC::Event depending_operations);\n\n5. DisCostiC-\u003eIsend(const void *message, \n                    DisCosTiC::Datatype sending_message_size_in_bytes, \n                    DisCostiC::Datatype sending_message_datatype, \n                    DisCostiC::Indextype destination_rank,\n                    DisCostiC::Commtype communicator,\n                    DisCostiC::Request *request,\n                    DisCosTiC::Event depending_operations);\n\n6. DisCostiC-\u003eRecv(const void *message, \n                    DisCosTiC::Datatype receiving_message_size_in_bytes, \n                    DisCostiC::Datatype receiving_message_datatype, \n                    DisCostiC::Indextype source_rank,\n                    DisCostiC::Commtype communicator, \n                    DisCosTiC::Event depending_operations);\n\n7. DisCostiC-\u003eIrecv(const void *message,\n                    DisCosTiC::Datatype receiving_message_size_in_bytes, \n                    DisCostiC::Datatype receiving_message_datatype, \n                    DisCostiC::Indextype source_rank,\n                    DisCostiC::Commtype communicator,\n                    DisCostiC::Request *request,\n                    DisCosTiC::Event depending_operations);\n\t\t\t\t\t\n8. DisCostiC-\u003eRank_Finalize(); \n\n```\n\n\u003ca name=\"DSEL\"\u003e\u003c/a\u003e\n📝 **Code blueprint as DisCostiC input**:thought_balloon:\n\nThe following DSEL code example shows the simplest illustration of how the domain knowledge of the applications is expressed in the DisCostiC language:\n\n```cpp\nDisCostiC::Event recv, send, comp;  \nfor (auto rank : DisCostiC::getRange(systemsize))\n{\n   DisCostiC-\u003eRank_Init(rank);\n   DisCosTiC-\u003eSetNumRanks(systemsize);\n   /* initialization */\n   comp = DisCostiC-\u003eExec(\"STREAM_TRIAD\", recv); \n   send = DisCostiC-\u003eSend(res, 8, ((rank + 1) % NP), MPI_COMM_WORLD, comp); \n      if(rank != 0)\n      {\n         recv = DisCostiC-\u003eRecv(res, 8, MPI_DOUBLE, rank - 1, MPI_COMM_WORLD, comp);\n      } \n      else \n      {\n         recv = DisCostiC-\u003eRecv(res, 8, MPI_DOUBLE, systemsize - 1, MPI_COMM_WORLD, comp);\n      }\t\n   DisCostiC-\u003eRank_Finalize(); \n} \n```\n\n\u003ca name=\"documentation\"\u003e\u003c/a\u003e    \n📝 **Code documentation: suite of C++ data structures and enumerated types**:thought_balloon:\n\nThe HTML and LaTeX documentation can be generated by [doxygen](http://www.doxygen.nl). To build the documentation, navigate to the [Doxyfile](Doxyfile) file available in the current directory and run the command:\n\n```\ndoxygen Doxyfile\n```\nTo read the HTML and LaTeX documentation, point to a web browser at [doc/html/index.html](https://github.com/RRZE-HPC/DisCostiC-Sim/tree/main/doc/html/index.html) and read [doc/DisCostiC.pdf](doc/DisCostiC.pdf), respectively.\n\n* **Single Operation**: accessors for local operations and their individual information at a certain grid point\n\n| Metadata information                  | Description   | \n| ------------------------------------- | ------------- | \n| `bufSize`\t\t\t\t\t\t| Number of bytes (data size) transmitted in the communication operation (no real buffer size for comp, just added for completeness)        | \n| `DepOperations`\t\t\t\t| Dependencies for blocking routines, i.e., other operations that depend on this current operation      | \n| `IdepOperations`\t\t\t\t| Dependencies for non-blocking routines, i.e., other operations that depend on the current operation   | \n| `depCount`\t\t\t\t\t| Number of dependencies for this current operation             | \n| `label`\t\t\t\t\t\t| Index/identifier of this current operation for each rack      | \n| `target` \t\t\t\t\t\t| Rank of target/partner (source for recv or destination for send or no real target for comp but added for completeness)    | \n| `rank`      \t\t\t\t\t| Owning rank of this current operation     | \n| `tag`\t\t\t\t\t\t\t| Tag of this current operation\t(no real tag for comp, just added for completeness) | \n| `node`\t\t\t\t\t\t| Node or processing element for this current operation\t\t    | \t\t\t\t\n| `network`\t\t\t\t\t\t| Type of network for this current operation    | \n| `time`\t\t\t\t\t\t|\tEnding time of this current operation   | \n| `starttime`\t\t\t\t\t|\tStarting time of this current operation | \n| `type`\t\t\t\t\t\t\t\t| Type of this current operation    | \n\n```cpp\nenum Operation_t {\t\t| Operation_t defines different operation types of entities\n\tSEND = 1,         \t| Send operation type\n        RECV = 2,        \t| Recv operation type\n        COMP = 3,        \t| Computation operation type\n        MSG = 4         \t| Message operation type\n}; \n```\n\n  \t\t\t\t\n| Metadata information | Description | \n| --------------------- | -------------  |  \n| `mode`\t|  mode of this current calling operation |  \n\n```cpp\nenum Mode_t {\t\t| Mode_t defines the operation type of communication\n        NONBLOCKING, \t| Routines returning with the start of an operation so the next operation doesnot execute before starting the previous one; such as Isend, Irecv.\n        BLOCKING     \t| Routines returning only on completion of operation so the next operation doesnot execute before finishing the previous one; such as Send, Recv.\n};\n```\n\n* **Performance model**\n\n| Metadata information     | Description | \n| --------------------- | ------------- | \n| `model`\t\t\t\t| Analytic first-principle performance model for computation and communication | \n\n```cpp\nenum Model_t {\t\t| Mode_t defines the used performance model \n        Roofline,\t| Simple computation model type\n        ECM,\t\t| Advanced computation model type\n        LOGGP,\t\t| Simple communication model type\n        HOCKNEY\t\t| Advanced communication model type\n};\n```\n\n* **Custom data types, keywords and high-level classes functionality**:thought_balloon:\n\n    * **accessors** for local vectors and matrices and their individual components without the need for index accesses\n    * **abstract base classes** for the AST-generated grid to access the operations and their associated features\n    * **solver-specific data types**, e.g., time steps, operations, identifiers, etc., are declared globally (or part of a special global declaration block)\n```cpp\n    * DisCostiC::Timetype\n    * DisCostiC::Datatype\n    * DisCostiC::Indextype\n    * DisCostiC::Networktype\n```\t\n\n\n\u003ca name=\"mpi-parallelized-implementation\"\u003e\u003c/a\u003e\n## 🧐 MPI-parallelized DisCostiC implementation\n\nThe underlying principle of parallel simulation is that each operation's entire data is transmitted via blocking or non-blocking MPI routines to the processes it communicates with.\n\n| Terms                  | Description | \n| --------------------- | ------------- | \n| `simulated processes P_i`    | processes from the application point of view | \n| `simulator processes Q_i`    | processes from the parallel simulation framework point of view | \n| `master simulator process Q_0`  | master process from the parallel simulation framework point of view | \n\n\n**Initialization**:thought_balloon:\nIn the MPI implementation, only the master process of the default communicator makes any print calls for debug and verbose output reporting, and all other processes of the new communicator initialize root operations only once in each run. \n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\nint ierr, process_Rank, size_Of_Cluster, sucesss, N_process_Rank, N_size_Of_Cluster;\nMPI_Comm newcomm;\nierr = MPI_Init(NULL, NULL);\nierr = MPI_Comm_rank(MPI_COMM_WORLD, \u0026process_Rank);\n\nMPI_Comm_split(MPI_COMM_WORLD, (process_Rank) \u003c 1 ? 1 : 2, 1, \u0026newcomm);\nierr = MPI_Comm_rank(newcomm, \u0026N_process_Rank);\nierr = MPI_Comm_size(newcomm, \u0026N_size_Of_Cluster);\nierr = MPI_Comm_rank(MPI_COMM_WORLD, \u0026process_Rank);\nierr = MPI_Comm_size(MPI_COMM_WORLD, \u0026size_Of_Cluster);\nstd::queue\u003cint\u003e test;\nMPI_Status status;\nMPI_Request request;\n    \nDisCostiC_Timetype starttimeProcess;\nDisCostiC::DisCostiC_OP operation, operation_og;\t\t\nint iter = 0;\n\niter = ((rankLocalIt-\u003egetNumOps()) / 3);\n\nif (rank == N_process_Rank)\n{\n operation_og = copy(op, operation_og);\n operation = op;\n starttimeProcess = operation.starttime;\n}\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\nop.numOpsInQueue = numOpsInQueue++;\nqueue.emplace(op);\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n**Start of simulation**:thought_balloon:\nDo loop over the number of iterations and the number of steps per iteration. Processes connected to the new communicator, or \"newcomm,\" are only used in the main simulation.\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\nrank = N_process_Rank;\nbool cycle = true;\nint iter_num = 1;\ndo{ //!\u003c over the number of iterations\n  do{ //!\u003c over the number of steps per iteration\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\ndo{\n  queue.pop();\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\nWithin each ccNUMA domain, runtime corrections are made by monitoring the number of processes that are concurrently engaged in each computation and communication step. \n\n**SEND operation**:thought_balloon:\nThe currently active process sends the operation object or array to the process listed as \"operation.target,\" which is specific to the communication pattern simulation program.\n \n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\noperation.type = DisCostiC::Operation_t::RECV;\noperation.starttime = operation.time;\n\nint temp = oSuccessor[operation.rank][operation.node];\n                        \ndouble arr[13];                   \n*serialize(operation, temp, arr);\nif (operation.mode == DisCostiC::Mode_t::NONBLOCKING){\n  MPI_Isend(\u0026arr, 13, MPI_DOUBLE, operation.target, operation.tag, newcomm, \u0026request);\n}else{\n  MPI_Send(\u0026arr, 13, MPI_DOUBLE, operation.target, operation.tag, newcomm);\n}\n\nfinishedRankList.push_back(operation.rank);\noperation.type = DisCostiC::Operation_t::RECV;\ncycle = true;\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\nstd::swap(operation.rank,operation.target);\nqueue.emplace(operation);\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n**RECV operation**:thought_balloon:\nThe currently running process receives the operation object or array from the process listed as \"operation.target\".\n \n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\ndouble brr[13];\nif (operation.mode == DisCostiC::Mode_t::NONBLOCKING){\n  MPI_Irecv(\u0026brr, 13, MPI_DOUBLE, MPI_ANY_SOURCE, operation.tag, newcomm, \u0026request);\n}else{\n  MPI_Irecv(\u0026brr, 13, MPI_DOUBLE, MPI_ANY_SOURCE, operation.tag, newcomm, \u0026request);\n}\n*deserialize(operation, temp, brr);                   \noSuccessor[operation.rank][operation.node] = brr[12];\ncycle=true;\noperation.numOpsInQueue = numOpsInQueue++;\nfinishedRankList.push_back(operation.rank);\n\noperation.type = DisCostiC::Operation_t::MSG;\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\nfinishedRankList.push_back(operation.rank);\nDisCostiC::DisCostiC_queueOP matchedOP;\nif(M.listmatch(operation, \u0026UMQ[operation.rank], \u0026matchedOP))\n\nelse { \n DisCostiC::DisCostiC_queueOP nOp;\n nOp.bufSize = operation.bufSize;\n nOp.src = operation.target;\n nOp.tag = operation.tag;\n nOp.label = operation.label;\n recvQueue[operation.rank].push_back(nOp);\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n**MSG operation**:thought_balloon:\nIf MSG is not found, the cycle will continue (cycle = true) until it is found.\n \n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\nif ((earliestFinish = operation.time) \u003c= ...){\n ...\n operation.numOpsInQueue = numOpsInQueue++;\n ...\n iter_num++;\t\t\n cycle = false;\t\t        \t\t\n}else{\n ...\n cycle = true;\n}\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\nif ((earliestFinish = operation.time) \u003c= ...){\n ...\n DisCostiC::DisCostiC_queueOP matchedOP;\n if(M.listmatch(operation, \u0026recvQueue[operation.rank], \u0026matchedOP)) { \n ...\n }else { \n #ifdef USE_VERBOSE \n std::cout \u003c\u003c \"[MSG ==\u003e NOT FOUND] in receive queue - add to unexpected queue\"\u003c\u003c std::endl;\n #endif\n DisCostiC::DisCostiC_queueOP nOp;\n nOp.bufSize = operation.bufSize;\n nOp.src = operation.target;\n nOp.tag = operation.tag;\n nOp.label = operation.label;\n nOp.starttime = operation.time; // when it was started\n UMQ[operation.rank].push_back(nOp);\n }\n}else{\n ...\n queue.emplace(operation);\n}\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n**Adding new sorted operations in queue in order**:thought_balloon:\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\n\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\nnewOps=false;\n\nnewOps = true;\nop.numOpsInQueue=numOpsInQueue++; \n\nqueue.emplace(op);\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n**End of while statement**:thought_balloon:\nThe end of the do loops over the number of iterations and the number of steps per iteration.\nThe necessary information gathered from all processes of the new communicator is communicated to the default communicator's master process.\n \n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003eMPI-parallelized\u003c/th\u003e\n\u003cth\u003eSerialized\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n  \n```cpp\nwhile (cycle);\n cycle = true;\n operation_og.time = oSuccessor[operation.rank][operation.node];\n operation = copy(operation_og, operation);\n } while (iter_num \u003c= iter);\n double arr[13];\n double brr[13];\n *serialize(operation, oSuccessor[rank][operation.node], arr);\n *finalize(oSuccessor[rank][operation.node], numRanks, numOpsInQueue, rank, operation.node, brr);\n MPI_Send(\u0026brr, 13, MPI_DOUBLE, 0, 1, MPI_COMM_WORLD);\n }\n else if (process_Rank == 0)\n {\n bool cycle = true;\n double brr[13];\n while (test.size() \u003c numRanks)\n {\n   MPI_Recv(\u0026brr, 13, MPI_DOUBLE, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, \u0026status);\n   if (status.MPI_TAG == 1)\n   {\n     test.emplace(status.MPI_SOURCE);\n     oSuccessor[brr[3]][brr[4]] = brr[0];\n     numOpsInQueue = brr[2];\n   }\n }\n}\nierr = MPI_Comm_rank(MPI_COMM_WORLD, \u0026process_Rank);\nMPI_Barrier(MPI_COMM_WORLD);\n```\n  \n\u003c/td\u003e\n\u003ctd\u003e\n\n```cpp\nwhile(!queue.empty() || newOps)\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n\u003ca name=\"library-limitations\"\u003e\u003c/a\u003e\n## 🚀 Planned features for further development\n\n* Threading model beyond message passing\n* Networking-level contention model\n* Energy consumption model\n\n\u003ca name=\"references\"\u003e\u003c/a\u003e\n## 📚 References about the theory of potential application scenarios for DisCostiC\n\n[1]: A. Afzal et al.: Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study. [DOI:10.1109/CLUSTER.2019.8890995](https://doi.org/10.1109/CLUSTER.2019.8890995)\n\n[2]: A. Afzal et al.: Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs. [DOI:10.1007/978-3-030-50743-5_20](https://doi.org/10.1007/978-3-030-50743-5_20)\n\n[3]: A. Afzal et al.: Analytic Modeling of Idle Waves in Parallel Programs: Communication, Cluster Topology, and Noise Impact. [DOI:10.1007/978-3-030-78713-4_19](https://doi.org/10.1007/978-3-030-78713-4_19)\n\n[4]: A. Afzal et al.: The Role of Idle Waves, Desynchronization, and Bottleneck Evasion in the Performance of Parallel Programs. [DOI:10.1109/TPDS.2022.3221085](https://doi.org/10.1109/TPDS.2022.3221085)\n\n[5]: A. Afzal et al.: Analytic performance model for parallel overlapping memory-bound kernels. [DOI:10.1002/cpe.6816](https://doi.org/10.1002/cpe.6816)\n\n[6]: A. Afzal et al.: Exploring Techniques for the Analysis of Spontaneous Asynchronicity in MPI-Parallel Applications. [DOI:10.1007/978-3-031-30442-2_12](https://doi.org/10.1007/978-3-031-30442-2_12)\n\n[7]: A. Afzal et al.: Making applications faster by asynchronous execution: Slowing down processes or relaxing MPI collectives. [DOI:10.1016/j.future.2023.06.017](https://10.1016/j.future.2023.06.017)\n\n\u003ca name=\"license\"\u003e\u003c/a\u003e\n## 🔒 License\n \n[AGPL-3.0](LICENSE)\n \n\u003ca name=\"disclaimer\"\u003e\u003c/a\u003e\n## :warning: Disclaimer\n\n\u003e [!NOTE]\n\u003e A note to the reader: Please report any bugs to the issue tracker or contact [ayesha.afzal@fau.de](ayesha.afzal@fau.de).\n\n\u003ca name=\"acknowledgement\"\u003e\u003c/a\u003e\n## 🔗 Acknowledgement\n\nThis work is funded by the **[KONWHIR](https://www.konwihr.de)** project **OMI4PAPPS**.\n\n\u003ca name=\"contact\"\u003e\u003c/a\u003e\n## 📱Contact\n\n[\u003cimg src=\"https://github.com/AyeshaAfzal91.png?size=115\" width=115\u003e\u003cbr\u003e\u003csub\u003e@AyeshaAfzal91\u003c/sub\u003e](https://github.com/AyeshaAfzal91) \u003cbr\u003e\u003cbr\u003e \nAyesha Afzal, Erlangen National High Performance Computing Center (NHR@FAU)\n\nmailto: [ayesha.afzal@fau.de](mailto:ayesha.afzal@fau.de)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrze-hpc%2Fdiscostic-sim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frrze-hpc%2Fdiscostic-sim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrze-hpc%2Fdiscostic-sim/lists"}