{"id":34984233,"url":"https://github.com/erdc/spt_compute","last_synced_at":"2025-12-27T01:24:30.364Z","repository":{"id":36858126,"uuid":"41165084","full_name":"erdc/spt_compute","owner":"erdc","description":"Runs streamflow forecasts using ECMWF predicted runoff and RAPID (Forked from: https://github.com/CI-WATER/erfp_data_process_ubuntu_aws).","archived":false,"fork":false,"pushed_at":"2024-03-13T21:16:46.000Z","size":24811,"stargazers_count":8,"open_issues_count":5,"forks_count":16,"subscribers_count":21,"default_branch":"master","last_synced_at":"2024-05-02T04:00:56.826Z","etag":null,"topics":["ecmwf","forecast-climate","forecasting","forecasting-models","hydraulics","hydrology","rapid"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erdc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2015-08-21T16:07:30.000Z","updated_at":"2023-10-01T19:09:07.000Z","dependencies_parsed_at":"2024-01-07T01:54:25.613Z","dependency_job_id":null,"html_url":"https://github.com/erdc/spt_compute","commit_stats":{"total_commits":284,"total_committers":7,"mean_commits":40.57142857142857,"dds":0.3802816901408451,"last_synced_commit":"cda9469723a22fa83e9d541b365ba31110e78dda"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/erdc/spt_compute","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erdc%2Fspt_compute","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erdc%2Fspt_compute/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erdc%2Fspt_compute/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erdc%2Fspt_compute/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erdc","download_url":"https://codeload.github.com/erdc/spt_compute/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erdc%2Fspt_compute/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28067576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-26T02:00:06.189Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ecmwf","forecast-climate","forecasting","forecasting-models","hydraulics","hydrology","rapid"],"created_at":"2025-12-27T01:24:29.627Z","updated_at":"2025-12-27T01:24:30.355Z","avatar_url":"https://github.com/erdc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spt_compute\n(Previously spt_ecmwf_autorapid_process)\n\nComputational framework to ingest ECMWF ensemble runoff forcasts or other Land Surface Model input; generate input for and run the RAPID (rapid-hub.org) program using HTCondor or Python's Multiprocessing; and upload to CKAN in order to be used by the Streamflow Prediction Tool (SPT). There is also an experimental option to use the AutoRoute program for flood inundation mapping.\n\n[![License (3-Clause BSD)](https://img.shields.io/badge/license-BSD%203--Clause-yellow.svg)](https://github.com/erdc/spt_compute/blob/master/LICENSE)\n\n[![Build Status](https://travis-ci.org/erdc/spt_compute.svg?branch=master)](https://travis-ci.org/erdc/spt_compute)\n\n[![DOI](https://zenodo.org/badge/19918/erdc/spt_ecmwf_autorapid_process.svg)](https://zenodo.org/badge/latestdoi/19918/erdc/spt_ecmwf_autorapid_process)\n\n## How it works:\n\nSnow, Alan D., Scott D. Christensen, Nathan R. Swain, E. James Nelson, Daniel P. Ames, Norman L. Jones,\nDeng Ding, Nawajish S. Noman, Cedric H. David, Florian Pappenberger, and Ervin Zsoter, 2016. A High-Resolution\nNational-Scale Hydrologic Forecast System from a Global Ensemble Land Surface Model. *Journal of the\nAmerican Water Resources Association (JAWRA)* 1-15, DOI: 10.1111/1752-1688.12434\n\nSnow, Alan Dee, \"A New Global Forecasting Model to Produce High-Resolution Stream Forecasts\" (2015). All Theses and Dissertations. Paper 5272. http://scholarsarchive.byu.edu/etd/5272\n\n# Installation\n\n## Step 1: Install RAPID and RAPIDpy\nSee: https://github.com/erdc/RAPIDpy\n\n## Step 2: Install HTCondor (if not using Amazon Web Services and StarCluster or not using Multiprocessing mode)\n### On Ubuntu\n```\napt-get install -y libvirt0 libdate-manip-perl vim\nwget http://ciwckan.chpc.utah.edu/dataset/be272798-f2a7-4b27-9dc8-4a131f0bb3f0/resource/86aa16c9-0575-44f7-a143-a050cd72f4c8/download/condor8.2.8312769ubuntu14.04amd64.deb\ndpkg -i condor8.2.8312769ubuntu14.04amd64.deb\n```\n### On RedHat/CentOS 7\nSee: https://research.cs.wisc.edu/htcondor/yum/\n### After Installation:\n```\n#if master node uncomment CONDOR_HOST and comment out CONDOR_HOST and DAEMON_LIST lines\n#echo CONDOR_HOST = \\$\\(IP_ADDRESS\\) \u003e\u003e /etc/condor/condor_config.local\necho CONDOR_HOST = 10.8.123.71 \u003e\u003e /etc/condor/condor_config.local\necho DAEMON_LIST = MASTER, SCHEDD, STARTD \u003e\u003e /etc/condor/condor_config.local\necho ALLOW_ADMINISTRATOR = \\$\\(CONDOR_HOST\\), 10.8.123.* \u003e\u003e /etc/condor/condor_config.local\necho ALLOW_OWNER = \\$\\(FULL_HOSTNAME\\), \\$\\(ALLOW_ADMINISTRATOR\\), \\$\\(CONDOR_HOST\\), 10.8.123.* \u003e\u003e /etc/condor/condor_config.local\necho ALLOW_READ = \\$\\(FULL_HOSTNAME\\), \\$\\(CONDOR_HOST\\), 10.8.123.* \u003e\u003e /etc/condor/condor_config.local\necho ALLOW_WRITE = \\$\\(FULL_HOSTNAME\\), \\$\\(CONDOR_HOST\\), 10.8.123.* \u003e\u003e /etc/condor/condor_config.local\necho START = True \u003e\u003e /etc/condor/condor_config.local\necho SUSPEND = False \u003e\u003e /etc/condor/condor_config.local\necho CONTINUE = True \u003e\u003e /etc/condor/condor_config.local\necho PREEMPT = False \u003e\u003e /etc/condor/condor_config.local\necho KILL = False \u003e\u003e /etc/condor/condor_config.local\necho WANT_SUSPEND = False \u003e\u003e /etc/condor/condor_config.local\necho WANT_VACATE = False \u003e\u003e /etc/condor/condor_config.local\n```\nNOTE: if you forgot to change lines for master node, change CONDOR_HOST = $(IP_ADDRESS)\nand restart condor as ROOT\n\nIf Ubuntu:\n```\n# . /etc/init.d/condor stop\n# . /etc/init.d/condor start\n```\nIf RedHat:\n```\n# systemctl stop condor\n# systemctl start condor\n```\n\n## Step 3: Install Prerequisite Packages\n### On Ubuntu:\n```\n$ apt-get install libssl-dev libffi-dev\n$ sudo su\n$ pip install requests_toolbelt tethys_dataset_services condorpy\n$ exit\n```\n### On RedHat/CentOS 7:\n```\n$ yum install libffi-devel openssl-devel\n$ sudo su\n$ pip install requests_toolbelt tethys_dataset_services condorpy\n$ exit\n```\nIf you are on RHEL 7 and having troubles, add the epel repo:\n```\n$ wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm\n$ sudo rpm -Uvh epel-release-7*.rpm\n```\nIf you are on CentOS 7 and having troubles, add the epel repo:\n```\n$ sudo yum install epel-release\n```\nThen install packages listed above.\n\n## Step 4: (Optional) Install AutoRoute and AutoRoutePy\nIf you want to try out the forecasted AutoRoute flood inundation (BETA), you will need to complete this section.\n\nFollow the instructions here: https://github.com/erdc/AutoRoutePy\n\n## Step 5: Install Submodule Dependencies\nSee: https://github.com/erdc/spt_dataset_manager\n\n## Step 6: Download and install the source code\n```\n$ cd /path/to/your/scripts/\n$ git clone https://github.com/erdc/spt_ecmwf_autorapid_process.git\n$ cd spt_ecmwf_autorapid_process\n$ python setup.py install\n```\n\n## Step 7: Create folders for RAPID input and for downloading ECMWF\n```\n$ cd /your/working/directory\n$ mkdir -p rapid-io/input rapid-io/output ecmwf logs subprocess_logs era_interim_watershed mp_execute\n```\n## Step 8: Change the locations in the files\nCreate a file *run_ecmwf_rapid.py* and change these variables for your instance. See below for different configurations.\n\n```python\n# -*- coding: utf-8 -*-\nfrom spt_compute import run_ecmwf_forecast_process\n#------------------------------------------------------------------------------\n#main process\n#------------------------------------------------------------------------------\nif __name__ == \"__main__\":\n    run_ecmwf_forecast_process(\n        rapid_executable_location='/home/alan/scripts/rapid/src/rapid',\n        rapid_io_files_location='/home/alan/rapid-io',\n        ecmwf_forecast_location =\"/home/alan/ecmwf\",\n        era_interim_data_location=\"/home/alan/era_interim_watershed\",\n        subprocess_log_directory='/home/alan/subprocess_logs',\n        main_log_directory='/home/alan/logs',\n        data_store_url='http://your-ckan/api/3/action',\n        data_store_api_key='your-ckan-api-key',\n        data_store_owner_org=\"your-organization\",\n        app_instance_id='your-streamflow_prediction_tool-app-id',\n        #sync_rapid_input_with_ckan=False,\n        download_ecmwf=True,\n        ftp_host=\"ftp.ecmwf.int\",\n        ftp_login=\"\",\n        ftp_passwd=\"\",\n        ftp_directory=\"\",\n        upload_output_to_ckan=True,\n        initialize_flows=True,\n        create_warning_points=True,\n        delete_output_when_done=True,\n        mp_mode='htcondor',\n        #mp_execute_directory='',\n    )\n```\n### run_ecmwf_rapid_process Function Variables\n\n|Variable|Data Type|Description|Default|\n|---|:---:|---|:---:|\n|*rapid_executable_location*|String|Path to RAPID executable.||\n|*rapid_io_files_location*|String|Path to RAPID input/output directory.||\n|*ecmwf_forecast_location*|String|Path to ECMWF forecasts.||\n|*main_log_directory*|String|Path to store HTCondor/multiprocess logs.||\n|*data_store_url*|String|(Optional) CKAN API url (e.g. http://your-ckan/api/3/action)|\"\"|\n|*data_store_api_key*|String|(Optional) CKAN API Key (e.g. abcd-1234-defr-3345)|\"\"|\n|*data_store_owner_org*|String|(Optional) CKAN owner organization (e.g. erdc).|\"\"|\n|*app_instance_id*|String|(Optional) Streamflow Prediction tool instance ID. |\"\"|\n|*sync_rapid_input_with_ckan*|Boolean|(Optional) If set to true, this will download ECMWF-RAPID input cooresponding to your instance of the Streamflow Prediction Tool. |False|\n|*download_ecmwf*|Boolean|(Optional) If set to true, this will download the most recent ECMWF forecasts for today before runnning the process. |True|\n|*date_string*|String|(Optional) This string will be used to modify the date of the forecasts downloaded and/or the forecasts ran. It is in the format yyyymmdd (e.g. 20160808). |None|\n|*ftp_host*|String|(Optional) ECMWF ftp site path (e.g. ftp.ecmwf.int). |\"\"|\n|*ftp_login*|String|(Optional) ECMWF ftp login name. |\"\"|\n|*ftp_passwd*|String|(Optional) ECMWF ftp password. |\"\"|\n|*ftp_directory*|String|(Optional) ECMWF ftp directory. |\"\"|\n|*delete_past_ecmwf_forecasts*|Boolean|(Optional) If True, it deletes all past forecasts before the next download. |True|\n|*upload_output_to_ckan*|Boolean|(Optional) If true, this will upload the output to CKAN for the Streamflow Prediction Tool to download. |False|\n|*delete_output_when_done*|String|(Optional) If true, all output will be deleted when the process completes. It is used when using operationally with *upload_output_to_ckan* set to true. |False|\n|*initialize_flows*|String|(Optional) If true, this will initialize flows from all avaialble methods (e.g. Past forecasts, historical data, streamgage data). |False|\n|*warning_flows_threshold*|Float|(Optional) Minimum value for return period in m3/s to generate warning. |10|\n|*era_interim_data_location*|String|(Optional) Path to ERA Interim based historical streamflow, return period data, and seasonal average data. |\"\"|\n|*create_warning_points*|Boolean|(Optional) Generate waring points for Streamflow Prediction Tool. This requires return period data to be located in the *era_interim_data_location*. |False|\n|*autoroute_executable_location*|String|(Optional/Beta) Path to AutoRoute executable. |\"\"|\n|*autoroute_io_files_location*|String|(Optional/Beta) Path to AutoRoute input/output directory. |\"\"|\n|*geoserver_url*|String|(Optional/Beta) Url to API endpoint ending in geoserver/rest. |\"\"|\n|*geoserver_username*|String|(Optional/Beta) Username for geoserver. |\"\"|\n|*geoserver_password*|String|(Optional/Beta) Password for geoserver. |\"\"|\n|*mp_mode*|String|(Optional) This defines how the process is run (HTCondor or Python's Multiprocessing). Valid options are htcondor and multiprocess. |htcondor|\n|*mp_execute_directory*|String|(Optional/Required if using multiprocess mode) Directory used in multiprocessing mode to temporarily store files begin generated.  |\"\"|\n\n### Possible run configurations\nThere are many different configurations. Here are some examples.\n\n#### Mode 1: Run ECMWF-RAPID for Streamflow Prediction Tool using HTCondor to run and CKAN to upload\n```python\nrun_ecmwf_forecast_process(\n    rapid_executable_location='/home/alan/scripts/rapid/src/rapid',\n    rapid_io_files_location='/home/alan/rapid-io',\n    ecmwf_forecast_location =\"/home/alan/ecmwf\",\n    era_interim_data_location=\"/home/alan/era_interim_watershed\",\n    subprocess_log_directory='/home/alan/subprocess_logs',\n    main_log_directory='/home/alan/logs',\n    data_store_url='http://your-ckan/api/3/action',\n    data_store_api_key='your-ckan-api-key',\n    data_store_owner_org=\"your-organization\",\n    app_instance_id='your-streamflow_prediction_tool-app-id',\n    download_ecmwf=True,\n    ftp_host=\"ftp.ecmwf.int\",\n    ftp_login=\"\",\n    ftp_passwd=\"\",\n    ftp_directory=\"\",\n    upload_output_to_ckan=True,\n    initialize_flows=True,\n    create_warning_points=True,\n    delete_output_when_done=True,\n)\n```\n\n#### Mode 2: Run ECMWF-RAPID for Streamflow Prediction Tool using HTCondor to run and CKAN to upload \u0026 to download model files\n```python\nrun_ecmwf_forecast_process(\n    rapid_executable_location='/home/alan/scripts/rapid/src/rapid',\n    rapid_io_files_location='/home/alan/rapid-io',\n    ecmwf_forecast_location =\"/home/alan/ecmwf\",\n    era_interim_data_location=\"/home/alan/era_interim_watershed\",\n    subprocess_log_directory='/home/alan/subprocess_logs',\n    main_log_directory='/home/alan/logs',\n    data_store_url='http://your-ckan/api/3/action',\n    data_store_api_key='your-ckan-api-key',\n    data_store_owner_org=\"your-organization\",\n    app_instance_id='your-streamflow_prediction_tool-app-id',\n    sync_rapid_input_with_ckan=True,\n    download_ecmwf=True,\n    ftp_host=\"ftp.ecmwf.int\",\n    ftp_login=\"\",\n    ftp_passwd=\"\",\n    ftp_directory=\"\",\n    upload_output_to_ckan=True,\n    initialize_flows=True,\n    create_warning_points=True,\n    delete_output_when_done=True,\n)\n```\n#### Mode 3: Run ECMWF-RAPID for Streamflow Prediction Tool using Multiprocessing to run and CKAN to upload\n```python\nrun_ecmwf_forecast_process(\n    rapid_executable_location='/home/alan/scripts/rapid/src/rapid',\n    rapid_io_files_location='/home/alan/rapid-io',\n    ecmwf_forecast_location =\"/home/alan/ecmwf\",\n    era_interim_data_location=\"/home/alan/era_interim_watershed\",\n    subprocess_log_directory='/home/alan/subprocess_logs',\n    main_log_directory='/home/alan/logs',\n    data_store_url='http://your-ckan/api/3/action',\n    data_store_api_key='your-ckan-api-key',\n    data_store_owner_org=\"your-organization\",\n    app_instance_id='your-streamflow_prediction_tool-app-id',\n    download_ecmwf=True,\n    ftp_host=\"ftp.ecmwf.int\",\n    ftp_login=\"\",\n    ftp_passwd=\"\",\n    ftp_directory=\"\",\n    upload_output_to_ckan=True,\n    initialize_flows=True,\n    create_warning_points=True,\n    delete_output_when_done=True,\n    mp_mode='multiprocess',\n    mp_execute_directory='/home/alan/mp_execute',\n)\n```\n#### Mode 4: (BETA) Run ECMWF-RAPID for Streamflow Prediction Tool with AutoRoute using Multiprocessing to run\nNote that in this example, CKAN was not used. However, you can still add CKAN back in to this example with the parameters shown in the previous examples.\n\n```python\nrun_ecmwf_forecast_process(\n    rapid_executable_location='/home/alan/rapid/src/rapid',\n    rapid_io_files_location='/home/alan/rapid-io',\n    ecmwf_forecast_location =\"/home/alan/ecmwf\",\n    era_interim_data_location=\"/home/alan/era_interim_watershed\",\n    subprocess_log_directory='/home/alan/subprocess_logs', #path to store HTCondor/multiprocess logs\n    main_log_directory='/home/alan/logs',\n    download_ecmwf=True,\n    ftp_host=\"ftp.ecmwf.int\",\n    ftp_login=\"\",\n    ftp_passwd=\"\",\n    ftp_directory=\"\",\n    upload_output_to_ckan=True,\n    initialize_flows=True,\n    create_warning_points=True,\n    delete_output_when_done=False,\n    autoroute_executable_location='/home/alan/scripts/AutoRoute/src/autoroute',\n    autoroute_io_files_location='/home/alan/autoroute-io',\n    geoserver_url='http://localhost:8181/geoserver/rest',\n    geoserver_username='admin',\n    geoserver_password='password',\n    mp_mode='multiprocess',\n    mp_execute_directory='/home/alan/mp_execute',\n)\n```\n\n## Step 9: Make sure permissions are correct for these files and any directories the script will use\n\nExample:\n```\n$ chmod u+x run_ecmwf_rapid.py\n```\n\n## Step 10: Add RAPID files to the rapid-io/input directory\nTo generate these files see: https://github.com/erdc/RAPIDpy/wiki/GIS-Tools. If you are using the *sync_rapid_input_with_ckan* option, then you would upload these files through the Streamflow Prediction Tool web interface and this step is unnecessary.\n\nMake sure the directory is in the format [watershed_name]-[subbasin_name]\nwith lowercase letters, numbers, and underscores only. No spaces!\n\n\nExample:\n```\n$ ls /rapid/input\nnfie_texas_gulf_region-huc_2_12\n$ ls /rapid/input/nfie_texas_gulf_region-huc_2_12\ncomid_lat_lon_z.csv\nk.csv\nrapid_connect.csv\nriv_bas_id.csv\nweight_ecmwf_t1279.csv\nweight_ecmwf_tco639.csv\nx.csv\n```\n\n## Step 11: Create CRON job to run the scripts hourly\nTo run this automatically, it is necessary to generate cron jobs to run the script. There are many ways to do this and two are presented here.\n\n### Method 1: In terminal using crontab command\n```\n$ crontab -e\n```\nThen add:\n```\n@hourly /usr/bin/env python /path/to/run_ecmwf_rapid.py # ECMWF RAPID PROCESS\n```\n\n### Method 2: Use *create_cron.py* to create the CRON jobs:\n\n1) Install crontab Python package.\n```\n$ pip install python-crontab\n```\n2) Create and run a script to initialize cron job *create_cron.py*.\n\n```python\nfrom spt_compute.setup import create_cron\n\ncreate_cron(execute_command='/usr/bin/env python /path/to/run_ecmwf_rapid.py')\n```\n\n## Step 12: Create CRON job to release lock on script\nIf the server is killed in the middle of a process, the lock with persist.\nTo prevent this, add a cron job to release the lock on bootup.\n\n### Create Script\nCreate a script to reset the lock info file. Example path: /path/to/ecmwf_rapid_server_reset.py\nThen, change the path to the lock info file. To do this, add *spt_compute_ecmwf_run_info_lock.txt*\nto your *main_log_directory* from the *run_ecmwf_rapid.py* script.\n\n```python\n#! /usr/bin/env python\n\nfrom spt_compute import reset_lock_info_file\n\nif __name__ == \"__main__\":\n    LOCK_INFO_FILE = '/logs/spt_compute_ecmwf_run_info_lock.txt'\n    reset_lock_info_file(LOCK_INFO_FILE)\n```\n### Create Cron Job\n\n```\n$ crontab -e\n```\nThen add:\n```\n@reboot /usr/bin/env python /path/to/ecmwf_rapid_server_reset.py # RESET ECMWF RAPID PROCESS LOCK\n```\n\n# Troubleshooting\nIf you see this error:\nImportError: No module named packages.urllib3.poolmanager\n```\n$ pip install pip --upgrade\n```\nRestart your terminal\n```\n$ pip install requests --upgrade\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferdc%2Fspt_compute","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferdc%2Fspt_compute","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferdc%2Fspt_compute/lists"}