{"id":20027416,"url":"https://github.com/huidaecho/midas","last_synced_at":"2025-10-04T06:57:34.002Z","repository":{"id":233644013,"uuid":"787624479","full_name":"HuidaeCho/midas","owner":"HuidaeCho","description":"Memory-Efficient I/O-Improved Drainage Analysis System (MIDAS)","archived":false,"fork":false,"pushed_at":"2025-08-23T13:25:22.000Z","size":83881,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-24T04:18:38.705Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HuidaeCho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-16T21:56:13.000Z","updated_at":"2025-08-23T13:25:25.000Z","dependencies_parsed_at":"2025-07-30T16:14:03.829Z","dependency_job_id":null,"html_url":"https://github.com/HuidaeCho/midas","commit_stats":null,"previous_names":["huidaecho/meidas"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HuidaeCho/midas","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuidaeCho%2Fmidas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuidaeCho%2Fmidas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuidaeCho%2Fmidas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuidaeCho%2Fmidas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HuidaeCho","download_url":"https://codeload.github.com/HuidaeCho/midas/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HuidaeCho%2Fmidas/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278277873,"owners_count":25960430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T09:10:11.408Z","updated_at":"2025-10-04T06:57:33.997Z","avatar_url":"https://github.com/HuidaeCho.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Memory-Efficient I/O-Improved Drainage Analysis System (MIDAS)\n\n\u003c!--ts--\u003e\n  * [Related projects](#related-projects)\n  * [References](#references)\n  * [Supported flow direction encodings](#supported-flow-direction-encodings)\n  * [Installing on Windows from MSVC binaries](#installing-on-windows-from-msvc-binaries)\n  * [Installing on Windows from MinGW GCC binaries](#installing-on-windows-from-mingw-gcc-binaries)\n  * [Building on Windows using MSVC](#building-on-windows-using-msvc)\n  * [Building on Windows using MinGW GCC](#building-on-windows-using-mingw-gcc)\n  * [Building on Linux](#building-on-linux)\n  * [Testing MSVC binaries on Windows using TX data](#testing-msvc-binaries-on-windows-using-tx-data)\n  * [Testing MinGW GCC binaries on Windows using TX data](#testing-mingw-gcc-binaries-on-windows-using-tx-data)\n  * [Benchmark results of MSVC vs. MinGW GCC vs. WSL GCC vs. Linux GCC binaries using CONUS data](#benchmark-results-of-msvc-vs-mingw-gcc-vs-wsl-gcc-vs-linux-gcc-binaries-using-conus-data)\n  * [Benchmark results](#benchmark-results)\n    * [MEFA](#mefa)\n    * [MESHED](#meshed)\n    * [MELFP](#melfp)\n    * [MEFLEN](#meflen)\n\n\u003c!-- Created by https://github.com/ekalinin/github-markdown-toc --\u003e\n\u003c!-- Added by: hcho, at: Mon Aug  4 08:46:58 MDT 2025 --\u003e\n\n\u003c!--te--\u003e\n\n## Related projects\n\n[MIDAS](https://github.com/HuidaeCho/midas) is the core C library and executables, and is required for all Python, R, and QGIS interfaces. The following projects are thin wrappers or interfaces to MIDAS's shared library (executables for the QGIS plugin). I designed the QGIS plugin to use subprocesses to isolate long-running or error-prone operations from the main QGIS process.\n* [MIDASFlow](https://github.com/HuidaeCho/midasflow): Python package\n* [MIDASFlow-R](https://github.com/HuidaeCho/midasflow-r): R package\n* [MIDAS-QGIS](https://github.com/HuidaeCho/midas-qgis): QGIS plugin\n\n## References\n\n* [MEFA](https://github.com/HuidaeCho/mefa) (Flow Accumulation): Huidae Cho, July 2023. Memory-Efficient Flow Accumulation Using a Look-Around Approach and Its OpenMP Parallelization. Environmental Modelling \u0026 Software 167, 105771. [doi:10.1016/j.envsoft.2023.105771](https://doi.org/10.1016/j.envsoft.2023.105771). [Author's Version](https://idea.isnew.info/publications/Memory-efficient%20flow%20accumulation%20using%20a%20look-around%20approach%20and%20its%20OpenMP%20parallelization%20-%20Cho.2023.pdf).\n* [MESHED](https://github.com/HuidaeCho/meshed) (Watershed Delineation): Huidae Cho, January 2025. Avoid Backtracking and Burn Your Inputs: CONUS-Scale Watershed Delineation Using OpenMP. Environmental Modelling \u0026 Software 183, 106244. [doi:10.1016/j.envsoft.2024.106244](https://doi.org/10.1016/j.envsoft.2024.106244). [Author's Version](https://idea.isnew.info/publications/Avoid%20backtracking%20and%20burn%20your%20inputs:%20CONUS-scale%20watershed%20delineation%20using%20OpenMP%20-%20Cho.2025.pdf).\n* [MELFP](https://github.com/HuidaeCho/melfp) (Longest Flow Path): Huidae Cho, September 2025. Loop Then Task: Hybridizing OpenMP Parallelism to Improve Load Balancing and Memory Efficiency in Continental-Scale Longest Flow Path Computation. Environmental Modelling \u0026 Software 193, 106630. [doi:10.1016/j.envsoft.2025.106630](https://doi.org/10.1016/j.envsoft.2025.106630). [Author's Version](https://idea.isnew.info/publications/Loop%20then%20task%20-%20Hybridizing%20OpenMP%20parallelism%20to%20improve%20load%20balancing%20and%20memory%20efficiency%20in%20continental-scale%20longest%20flow%20path%20computation%20-%20Cho.2025.pdf).\n\n## Supported flow direction encodings\n\nPredefined flow direction encodings in GeoTIFF: power2 (default, r.terraflow, ArcGIS), taudem (d8flowdir), 45degree (r.watershed), degree\u003cbr\u003e\n![image](https://github.com/user-attachments/assets/990f0530-fded-4ee5-bfbb-85056a50ca1c)\n![image](https://github.com/user-attachments/assets/a02dfc15-a825-4210-82c4-4c9296dafadc)\n![image](https://github.com/user-attachments/assets/64f5c65a-c7cc-4e06-a69f-6fccd6435426)\n![image](https://github.com/user-attachments/assets/fafef436-a5f2-464a-89a8-9f50a877932c)\n\nCustom flow direction encoding is also possible by passing `-e E,SE,S,SW,W,NW,N,NE` (e.g., 1,8,7,6,5,4,3,2 for taudem).\n\n## Installing on Windows from MSVC binaries\n\n1. Install [Git for Windows](https://gitforwindows.org/)\n2. Install [Miniconda](https://www.anaconda.com/download/success)\n```cmd\ncurl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe\nmkdir C:\\opt\nMiniconda3-latest-Windows-x86_64.exe /S /D=C:\\opt\\miniconda\nC:\\opt\\miniconda\\condabin\\conda.bat init\n```\n3. Setup Conda for MIDAS run\n```cmd\nconda config --add channels conda-forge\nconda config --set channel_priority strict\nconda create -n midas_msvc libgdal\nconda activate midas_msvc\n```\n4. Download the source code and binaries\n```cmd\ncd \\opt\ngit clone https://github.com/HuidaeCho/midas.git\n```\n5. Run MIDAS (add `C:\\opt\\midas\\windows\\msvc` to `PATH`)\n```cmd\nset PATH=%PATH%;C:\\opt\\midas\\windows\\msvc\nmefa\n```\n\n## Installing on Windows from MinGW GCC binaries\n\n1. Install [Git for Windows](https://gitforwindows.org/)\n2. Install [Miniconda](https://www.anaconda.com/download/success)\n```cmd\ncurl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe\nmkdir C:\\opt\nMiniconda3-latest-Windows-x86_64.exe /S /D=C:\\opt\\miniconda\nC:\\opt\\miniconda\\condabin\\conda.bat init\n```\n3. Setup Conda for MIDAS run\n```cmd\nconda config --add channels conda-forge\nconda config --set channel_priority strict\nconda create -n midas_mingw libgcc libgdal\nconda activate midas_mingw\n```\n4. Download the source code and binaries\n```cmd\ncd \\opt\ngit clone https://github.com/HuidaeCho/midas.git\n```\n5. Run MIDAS (add `C:\\opt\\midas\\windows\\mingw` to `PATH`)\n```cmd\nset PATH=%PATH%;C:\\opt\\midas\\windows\\mingw\nmefa\n```\n\n## Building on Windows using MSVC\n\n1. Install [Visual Studio Community Edition](https://visualstudio.microsoft.com/vs/community/). Select these two components:\n   * MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest)\n   * Windows 11 SDK (10.0.26100.0)\n2. Install [Git for Windows](https://gitforwindows.org/)\n3. Install [Miniconda](https://www.anaconda.com/download/success)\n```cmd\ncurl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe\nmkdir C:\\opt\nMiniconda3-latest-Windows-x86_64.exe /S /D=C:\\opt\\miniconda\nC:\\opt\\miniconda\\condabin\\conda.bat init\n```\n4. Start Developer Command Prompt for VS 2022\n5. Setup Conda for MIDAS build\n```cmd\nconda config --add channels conda-forge\nconda config --set channel_priority strict\nconda create -n midas cmake libgdal\nconda activate midas\n```\n6. Download the source code\n```cmd\ncd \\opt\ngit clone https://github.com/HuidaeCho/midas.git\ncd midas/src\n```\n7. Build MIDAS\n```cmd\nmkdir build\ncd build\ncmake .. \u003e cmake.log 2\u003e\u00261\nmsbuild midas.sln -p:Configuration=Release \u003e msbuild.log 2\u003e\u00261\n```\n8. Run MIDAS (add `C:\\opt\\midas\\src\\build\\dist` to `PATH`)\n```cmd\nset PATH=%PATH%;C:\\opt\\midas\\src\\build\\dist\nmefa\n```\n\n## Building on Windows using MinGW GCC\n\n1. Install [Git for Windows](https://gitforwindows.org/)\n2. Install [Miniconda](https://www.anaconda.com/download/success)\n```cmd\ncurl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe\nmkdir C:\\opt\nMiniconda3-latest-Windows-x86_64.exe /S /D=C:\\opt\\miniconda\nC:\\opt\\miniconda\\condabin\\conda.bat init\n```\n3. Setup Conda for MIDAS build\n```cmd\nconda config --add channels conda-forge\nconda config --set channel_priority strict\nconda create -n midas_mingw cmake make gcc binutils libgdal\nconda activate midas_mingw\n```\n4. Download the source code\n```cmd\ncd \\opt\ngit clone https://github.com/HuidaeCho/midas.git\ncd midas/src\n```\n5. Build MIDAS\n```cmd\nmkdir build\ncd build\ncmake .. -G \"MinGW Makefiles\" \u003e cmake.log 2\u003e\u00261\nmake \u003e make.log 2\u003e\u00261\n```\n6. Run MIDAS (add `C:\\opt\\midas\\src\\build\\dist` to `PATH`)\n```cmd\nset PATH=%PATH%;C:\\opt\\midas\\src\\build\\dist\nmefa\n```\n\n## Building on Linux\n\n1. Install Miniconda\n```bash\nwget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\nchmod a+x Miniconda3-latest-Linux-x86_64.sh\nmkdir ~/opt\n./Miniconda3-latest-Linux-x86_64.sh -b -u -p ~/opt/miniconda\n~/opt/miniconda/bin/conda init\n. ~/.bashrc\n```\n2. Setup Conda for MIDAS build\n```bash\nconda config --add channels conda-forge\nconda config --set channel_priority strict\nconda create -n midas git cmake make gcc gdal\nconda activate midas\n```\n3. Download the source code\n```bash\ncd ~/opt\ngit clone https://github.com/HuidaeCho/midas.git\ncd midas/src\n```\n4. Build MIDAS\n```bash\nmkdir build\ncd build\n\n# LD_LIBRARY_PATH=$CONDA_PREFIX/lib and -DCMAKE_PREFIX_PATH=$CONDA_PREFIX to avoid system libraries\n(\nLD_LIBRARY_PATH=$CONDA_PREFIX/lib \\\ncmake .. \\\n  -DCMAKE_PREFIX_PATH=$CONDA_PREFIX \\\n  -DCMAKE_INSTALL_PREFIX=$HOME/usr/local \u0026\u0026\ncmake --build . \u0026\u0026\ncmake --install .\n) \u0026\u003e build.log\n```\n5. Run MIDAS (add `$HOME/usr/local/bin` to `PATH`)\n```bash\nexport PATH=\"$PATH:$HOME/usr/local/bin\"\nmefa\n```\n\n## Testing MSVC binaries on Windows using TX data\n\n```dos\ngit clone https://github.com/HuidaeCho/midas.git\ncd midas\\windows\\test\npretest.bat\ntest_msvc.bat\n```\n\n## Testing MinGW GCC binaries on Windows using TX data\n\n```dos\ngit clone https://github.com/HuidaeCho/midas.git\ncd midas\\windows\\test\npretest.bat\ntest_mingw.bat\n```\n\n## Benchmark results of MSVC vs. MinGW GCC vs. WSL GCC vs. Linux GCC binaries using CONUS data\n\nSystem specifications\n* System model and OS\n  * ThinkPad X1 Carbon Gen 11\n    * Windows 11 for MSVC and MinGW\n    * WSL Linux 5.15.167.4-microsoft-standard-WSL2 for WSL GCC\n  * ThinkPad X1 Yoga Gen 8 (identical hardware specifications)\n    * Slackware CURRENT Linux 6.12.17 for Linux GCC\n* CPU: Intel Core i7-1370P @ 5.20 GHz\n* Cores: 14\n* Logical processors: 20\n* Memory: 64 GiB\n\nMSVC\n```\n\u003e melfp.exe inputs\\fdr.tif inputs\\outlets515152.shp cat msvc.gpkg lfpid -c msvc.csv\nNo output vector layers specified; Not creating msvc.gpkg\nUsing 20 threads...\nReading flow direction raster \u003cinputs\\fdr.tif\u003e...\nInput time for flow direction: 9198000 microsec\nReading outlets \u003cinputs\\outlets515152.shp\u003e...\nInput time for outlets: 17007000 microsec\nNumber of cells: 14998630400\nNumber of outlets: 515152\nTracing stack size for loop-then-task: 3072\nFinding longest flow paths...\nComputation time for longest flow paths: 622248000 microsec\nNumber of longest flow paths found: 521946\nWriting longest flow path head coordinates \u003cmsvc.csv\u003e...\nOutput time for longest flow path head coordinates: 1980000 microsec\nTotal elapsed time: 657635000 microsec\n```\n* Computation time: 622.248000 sec\n* I/O time: 35.387000 sec\n* Total time: 657.635000 sec\n\nMinGW GCC\n```\n\u003e melfp.exe inputs\\fdr.tif inputs\\outlets515152.shp cat mingw.gpkg lfpid -c mingw.csv\nNo output vector layers specified; Not creating mingw.gpkg\nUsing 20 threads...\nReading flow direction raster \u003cinputs\\fdr.tif\u003e...\nInput time for flow direction: 6823994 microsec\nReading outlets \u003cinputs\\outlets515152.shp\u003e...\nInput time for outlets: 16805443 microsec\nNumber of cells: 14998630400\nNumber of outlets: 515152\nTracing stack size for loop-then-task: 3072\nFinding longest flow paths...\nComputation time for longest flow paths: 120324302 microsec\nNumber of longest flow paths found: 521946\nWriting longest flow path head coordinates \u003cmingw.csv\u003e...\nOutput time for longest flow path head coordinates: 2097205 microsec\nTotal elapsed time: 153459228 microsec\n```\n* Computation time: 120.324302 sec\n* I/O time: 33.13493 sec\n* Total time: 153.459228 sec\n\nWSL GCC\n```\n$ melfp inputs/fdr.tif inputs/outlets515152.shp cat wsl.gpkg lfpid -c wsl.csv\nNo output vector layers specified; Not creating wsl.gpkg\nUsing 20 threads...\nReading flow direction raster \u003cinputs/fdr.tif\u003e...\nInput time for flow direction: 7537860 microsec\nReading outlets \u003cinputs/outlets515152.shp\u003e...\nInput time for outlets: 223358 microsec\nNumber of cells: 14998630400\nNumber of outlets: 515152\nTracing stack size for loop-then-task: 3072\nFinding longest flow paths...\nComputation time for longest flow paths: 38734223 microsec\nNumber of longest flow paths found: 521946\nWriting longest flow path head coordinates \u003cwsl.csv\u003e...\nOutput time for longest flow path head coordinates: 426007 microsec\nTotal elapsed time: 47034415 microsec\n```\n* Computation time: 38.734223 sec\n* I/O time: 8.300192 sec\n* Total time: 47.034415 sec\n\nLinux GCC\n```\n$ melfp inputs/fdr.tif inputs/outlets515152.shp cat linux.gpkg lfpid -c linux.csv\nNo output vector layers specified; Not creating linux.gpkg\nUsing 20 threads...\nReading flow direction raster \u003cinputs/fdr.tif\u003e...\nInput time for flow direction: 1310629 microsec\nReading outlets \u003cinputs/outlets515152.shp\u003e...\nInput time for outlets: 174350 microsec\nNumber of cells: 14998630400\nNumber of outlets: 515152\nTracing stack size for loop-then-task: 3072\nFinding longest flow paths...\nComputation time for longest flow paths: 21766454 microsec\nNumber of longest flow paths found: 521946\nWriting longest flow path head coordinates \u003clinux.csv\u003e...\nOutput time for longest flow path head coordinates: 398142 microsec\nTotal elapsed time: 24136021 microsec\n```\n* Computation time: 21.766454 sec\n* I/O time: 2.369567 sec\n* Total time: 24.136021 sec\n\nComputation time ranking\n1. Linux GCC: fastest baseline\n2. WSL GCC: 1.78x slower than Linux GCC\n3. MinGW GCC: 5.53x slower than Linux GCC\n4. MSVC: 28.59x slower than Linux GCC\n\nBased on these results, Linux offers the best performance and should be used if possible. If you must use Windows, prefer WSL for better performance. If WSL is not available, MinGW GCC binaries provide the most reasonable performance among native Windows options.\n\n## Benchmark results\n\n### MEFA\n\n![MEFA Graphical Abstract](https://idea.isnew.info/publications/Memory-efficient%20flow%20accumulation%20using%20a%20look-around%20approach%20and%20its%20OpenMP%20parallelization%20-%20Graphical%20abstract.png)\n\nSystem specifications\n* CPU: Intel Core i9-12900 @ 2.40GHz\n* Cores: 16\n* Logical processors: 24\n* Memory: 64 GiB\n* OS: Windows 11\n\nCitation\n* Huidae Cho, July 2023. Memory-Efficient Flow Accumulation Using a Look-Aroun d Approach and Its OpenMP Parallelization. Environmental Modelling \u0026 Software 167, 105771. [doi:10.1016/j.envsoft.2023.105771](https://doi.org/10.1016/j.envsoft.2023.105771).\n\n### MESHED\n\n![image](https://clawrim.isnew.info/wp-content/uploads/2024/07/meshed-flyer.png)\n\n### MELFP\n\n![image](https://github.com/HuidaeCho/midas/assets/7456117/0c1fdd15-b900-407c-b89b-503a6c0b2dc2)\n\n### MEFLEN\n\n![image](https://github.com/HuidaeCho/midas/assets/7456117/3a67d9ec-8649-4d24-97a7-b5d3dd5115c7)\n\nSystem specifications\n* CPU: Intel Core i9-12900 @ 2.40GHz\n* Cores: 16\n* Logical processors: 24\n* Memory: 64 GiB\n* OS: Windows 11\n  * Downstream flow length (computation time only)\n    * Texas results\n      * 8.2 s (24 threads)\n      * 19.6 s (1 thread)\n    * CONUS results\n      * 4.7 m (24 threads)\n      * 7.3 m (1 thread)\n* OS: Linux 6.6.30\n  * Downstream flow length (computation time only)\n    * Texas results\n      * 9.4 s (24 threads)\n      * 20.0 s (1 thread)\n    * CONUS results\n      * 2.0 m (24 threads)\n      * 4.0 m (1 thread)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuidaecho%2Fmidas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuidaecho%2Fmidas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuidaecho%2Fmidas/lists"}