{"id":22798153,"url":"https://github.com/doi-usgs/lake-temperature-process-models-old","last_synced_at":"2025-04-19T15:16:40.801Z","repository":{"id":151437080,"uuid":"148850938","full_name":"DOI-USGS/lake-temperature-process-models-old","owner":"DOI-USGS","description":"Pipeline #2","archived":false,"fork":false,"pushed_at":"2023-06-09T20:06:15.000Z","size":48376,"stargazers_count":0,"open_issues_count":6,"forks_count":6,"subscribers_count":12,"default_branch":"main","last_synced_at":"2024-01-26T18:09:17.172Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DOI-USGS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-09-14T23:22:11.000Z","updated_at":"2023-06-16T13:41:07.000Z","dependencies_parsed_at":"2023-09-12T11:52:29.795Z","dependency_job_id":null,"html_url":"https://github.com/DOI-USGS/lake-temperature-process-models-old","commit_stats":null,"previous_names":["doi-usgs/lake-temperature-process-models-old","usgs-r/lake-temperature-process-models-old"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DOI-USGS%2Flake-temperature-process-models-old","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DOI-USGS%2Flake-temperature-process-models-old/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DOI-USGS%2Flake-temperature-process-models-old/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DOI-USGS%2Flake-temperature-process-models-old/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DOI-USGS","download_url":"https://codeload.github.com/DOI-USGS/lake-temperature-process-models-old/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229338855,"owners_count":18057539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-12T06:08:36.216Z","updated_at":"2024-12-12T06:08:36.805Z","avatar_url":"https://github.com/DOI-USGS.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lake-temperature-process-models\n\nDependent files from pipeline #1 lake-temperature-model-prep:\n  -  feature_crosswalk.rds and .ind \n  -  lakes_sf.rds and .ind\n  -  NLDAS_grid coords and cell resolution  \n\n The following files are dependencies in pipeline #3 lake-temperature-neural-networks:\n  -  [`feature_nldas_coords.rds`](https://drive.google.com/drive/u/1/folders/1pbhIjfYUPZ4lEICm5zwJFjIjGYEz1qwi) and [`.ind`](https://github.com/USGS-R/lake-temperature-neural-networks/tree/master/in)\n\nIf any of these files are changed / updated in this pipeline, remember to: \n  1. copy the update .rds file to the dependent pipeline's drive (which is hyperlinked above) and to _ALSO_\n  2. copy the updated .ind file to the dependent pipeline's github repository (which is also hyperlinked above)\n\n### end around on the batch jobs\nIn order to get around the issue with UV being down, my VM being slow, and normal being even slower in batch mode, I set up a single node in normal for interactive mode, and ran scmake in R.\n\nI used rsync to get the new task table up to Yeti:\n```\nrsync -avz 3_pb0_src_trg_tasks.yml jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_pb0_src_trg_tasks.yml\n\n#or\nrsync -avz 3_pb0_hyperscales_tasks.yml jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_pb0_hyperscales_tasks.yml\n```\nUsed salloc to get 7 hours in interactive (in the end, I needed 5.5hrs to run 450 models):\n```\nsalloc -A cida -n 1 -p normal -t 7:00:00\n```\nthen ssh'd into the node I was given, and from there, got into the working directory\n```\nssh n3-98\ncd /cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models\n```\n\nAs a test of loop_tasks in parallel, I asked for 4 cores\n```\nsalloc -A cida -n 4 -p normal -t 7:00:00\n\nrsync -avz pb0_src_trg_plan.rds jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/pb0_src_trg_plan.rds\nrsync -avz hyperscales_pb0_plan.rds  jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/hyperscales_pb0_plan.rds\n```\nloaded modules on the node and then scipiper::loop_tasks() after installing `foreach` and `doParallel` on Yeti:\n```r\nloop_tasks(task_plan = readRDS('pb0_src_trg_plan.rds'), task_makefile = '3_pb0_src_trg_tasks.yml', n_cores = 4)\n#or\nloop_tasks(task_plan = readRDS('hyperscales_pb0_plan.rds'), task_makefile = '3_pb0_hyperscales_tasks.yml', n_cores = 8)\n```\n\n### setting up Yeti\n\nThis project is in `/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models`\n\nI have set up the R library dir in `/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/Rlib` and used `.Renviron` file in the project root to specify `R_LIBS=...`\n\nFor installing packages, I did\n\n```\nmodule avail #list of all avail\nmodule purge #DON'T DO THIS???\nmodule load legacy # had to do this w/ Yeti refresh\nmodule load R/3.6.3\nmodule load tools/nco-4.7.8-gnu \nmodule load tools/netcdf-c-4.3.2-intel #tools/netcdf-c-4.6.2-gnu need this because there is a Yeti error where \n#module load gdal/2.2.0-gcc\n#module load proj/5.2.0-gcc-7.1.0\n\n\n#module load tools/nco-4.4.4-gnu ??tools/nco-4.7.8-gnu \n#module load tools/netcdf-4.3.2-gnu ??tools/netcdf-c-4.6.2-gnu\n\n module list\nCurrently Loaded Modulefiles:\n  1) tools/nco-4.4.4-gnu      3) tools/hdf5-1.8.13-gnu    5) tools/netcdf-4.3.2-gnu\n  2) tools/hdf-4.2.10-gnu     4) tools/szip-2.1-gnu\n  \n  WORKED!!\ninstall.packages(\"GLMr\", repos=c(\"https://owi.usgs.gov/R\",getOption(\"repos\")))\n\n??\nmodule load hdf5/1.8.18-gcc6.1.0\n[jread@yeti-login20 lake-temperature-process-models] module load netcdf/4.4.1.1-gcc6.1.0\n\n```\n\nthen \n`R`\n\nfrom R:\n```r\ninstall.packages(c('stringi','stringr','dplyr','tidyr','rLakeAnalyzer','lubridate','remotes','ncdf4','readr','feather'))\n\nlibrary(remoates)\ninstall_github('tidyverse/dplyr@v0.8.5')\ninstall_version(\"vctrs\", version = \"0.2.4\", repos = \"http://cran.us.r-project.org\")\ninstall_version(\"tidyr\", version = \"1.0.0\", repos = \"http://cran.us.r-project.org\")\n\n#remotes::install_github('GLEON/GLM3r')\n#remotes::install_github('GLEON/GLMr')\nremotes::install_github('usgs-r/glmtools')\n#remotes::install_github('mrc-ide/syncr')\n```\nThis project uses rsync and ssh to pull/push data/predictions from Yeti in the `1_get_lake_attr` and `2_model` phase. For this to work, SSH keys must be set up for communication with Yeti:\n\n  -  If you don't yet have a local ssh key pair, use `ssh-keygen -t rsa` from within a local terminal.\n  -  Copy the public key to Yeti with `ssh-copy-id username@yeti.cr.usgs.gov` (also from within your local terminal). You can then check that you're set up by running `ssh username@yeti.cr.usgs.gov` from a terminal - it should log you in without a password.\n  -  On Windows with RStudio, there will be a problem in that SSH/rsync assume your `.ssh` folder is at `~/.ssh`, but `~` means `C:/Users/username` within a terminal but `C:/Users/username/Documents` within RStudio. Therefore you should create a symlink for the `.ssh` folder by calling `ln -s ~/.ssh ~/Documents/.ssh` in a bash shell.\n\n\n\n### my workflow\nI build things locally and some (not all) sync to Yeti as part of the process. \nSeems rsync is faster than the cross-platform ssh package\nI need to sync nmls and drivers to yeti, then use the \"out\" job list (rds) to tell yeti what to do\nWhen the job list is on yeti, I need to modify the batch file with the same number of array jobs that appear in the job table\nWhen the jobs are done, I sync _from_ yeti (see below) the feather files. \nThen I run a failed job array (same process as above) which picks up maybe a few more lakes when they run again. I don't know why they'd fail the first time but not the second. Doesn't make sense to me and I haven't dug into the lobs. \n\nSo, this is clunky and seems that if I used `drake` instead on Yeti, I would be happier. \n\nStill some pain points with moving files around to and from Yeti. Kinda slow and maybe I should zip files or something (although rsync must be compressing, right?)\n\nsync to yeti\n```\ncd 2_prep/sync\nrsync -avz .  jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/2_prep/sync\n\n# sync the job lists to yeti:\ncd 2_prep/out\nrsync -avz .  jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/2_prep/out\n\n# sync task table to yeti:\nrsync -avz 3_pb0_src_trg_tasks.yml jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_pb0_src_trg_tasks.yml\n\nrsync -avz 3_pb0_hyperscales_tasks.yml jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_pb0_hyperscales_tasks.yml\n```\n\nsync from yeti\n```\ncd 3_run/sync\nrsync -avz jread@yeti-dtn1.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_run/sync/. .\n\ncd 3_run/out\nrsync -avz jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_run/out/pb0_src_trg_tasks.rds.ind pb0_src_trg_tasks.rds.ind \n\nrsync -avz jread@yeti.cr.usgs.gov:/cxfs/projects/usgs/water/iidd/data-sci/lake-temp/lake-temperature-process-models/3_run/out/pb0_hyperscales_tasks.ind pb0_hyperscales_tasks.ind\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoi-usgs%2Flake-temperature-process-models-old","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoi-usgs%2Flake-temperature-process-models-old","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoi-usgs%2Flake-temperature-process-models-old/lists"}