{"id":21441561,"url":"https://github.com/atreyagaurav/locus-code-usace","last_synced_at":"2026-02-04T00:32:11.641Z","repository":{"id":159509508,"uuid":"618662195","full_name":"Atreyagaurav/locus-code-usace","owner":"Atreyagaurav","description":"Code from locus-usace without the input files","archived":false,"fork":false,"pushed_at":"2024-07-18T20:28:22.000Z","size":3490,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-16T12:39:53.070Z","etag":null,"topics":["dataset","netcdf4","precipitation","precipitation-analysis"],"latest_commit_sha":null,"homepage":"https://atreyagaurav.github.io/locus/index.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Atreyagaurav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-25T01:29:12.000Z","updated_at":"2024-07-18T20:28:25.000Z","dependencies_parsed_at":null,"dependency_job_id":"e713b85c-71f6-41c8-8670-f342eda2e1de","html_url":"https://github.com/Atreyagaurav/locus-code-usace","commit_stats":{"total_commits":40,"total_committers":3,"mean_commits":"13.333333333333334","dds":"0.32499999999999996","last_synced_commit":"6b4636b69bf92848d2f2e54eba11b26cc118d355"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Atreyagaurav/locus-code-usace","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Atreyagaurav%2Flocus-code-usace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Atreyagaurav%2Flocus-code-usace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Atreyagaurav%2Flocus-code-usace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Atreyagaurav%2Flocus-code-usace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Atreyagaurav","download_url":"https://codeload.github.com/Atreyagaurav/locus-code-usace/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Atreyagaurav%2Flocus-code-usace/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263594623,"owners_count":23485877,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","netcdf4","precipitation","precipitation-analysis"],"created_at":"2024-11-23T01:38:27.591Z","updated_at":"2026-02-04T00:32:11.595Z","avatar_url":"https://github.com/Atreyagaurav.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# locus\n\nIdentifies spatial patterns in extreme precipitation events.\n\n[Interactive Map](http://ronin-river.github.io/locus/map.html)\n\n[Interactive Data Table](http://ronin-river.github.io/locus/netcdfs-index.html)\n\n\n## Preparing Datasets\n### Precipitation Data\nLivneh datasets can be downloaded from: https://psl.noaa.gov/thredds/catalog/Datasets/livneh/metvars/catalog.html\n\nExample link: https://psl.noaa.gov/thredds/fileServer/Datasets/livneh/metvars/prec.1915.nc\nData is available from 1915 to 2011.\n\nThere is a download script (bash) that'll download the files for you in `data` directory, you need gnu parallel and curl for downloads, if you don't then it'll generate the links in the file `data/livneh-files.txt`, you can download it using your favourite download manager.\n\n### Water Boundary Data\nDownload the watershed boundaries from [USGS TNM Download (v2.0)](https://apps.nationalmap.gov), [direct link](https://prd-tnm.s3.amazonaws.com/StagedProducts/Hydrography/WBD/National/GPKG/WBD_National_GPKG.zip).\n\nIt's 4.8 GB to download and 7.1 GB after you unzip it.\n\n## Running the program\nThe program has all the source codes inside the `src` directory, you can run it with `python`.\n\nFirst install the requirements in `requirements.txt`\n\n    pip install -r requirements.txt\n\nThen download the input files for Water Boundary Dataset and Livneh Dataset. See instructions above.\n\nRun the CLI program with python\n\n    python -m src.locus -h\n\nWill give you the usage instructions\n\n    usage: locus [-h] [-n NUM_DAYS] [-l] [-d] [-D] [-s {ams,pds,both}] [-c] [-e] [-a] [-p] [-b] HUCode\n\n    Analyse extreme precipitation pattern\n\n    positional arguments:\n      HUCode\n\n    options:\n      -h, --help            show this help message and exit\n      -n NUM_DAYS, --num-days NUM_DAYS\n                            Number of Days for ams/pds event\n      -l, --list-hucs       List HUCodes in the given category and exit\n      -d, --details         print basin details before processing\n      -D, --details-only    print basin details and exit\n      -s {ams,pds,both}, --series {ams,pds,both}\n                            ams or pds event to plot\n      -c, --calculate-weights\n                            run the calculate_weights function\n      -e, --extract-annual-timeseries\n                            run the extract_annual_timeseries function\n      -a, --ams-and-pds     run the ams_and_pds function\n      -p, --plot-clusters   run the plot_clusters function\n      -b, --batch-process   run the batch_process function\n\n    https://github.com/Atreyagaurav/locus-code-usace\n\nFor example:\n\nto run all the process for HUCode 02, for single day precipitation event:\n\n    python -m src.locus 02 -d -b -n 1\n\nto list all HUC2 basins:\n\n    python -m src.locus 2 -l\n\n\n## Example Output\nFor interactive Results visit: https://atreyagaurav.github.io/locus/index.html\n\nFor Ohio Region (HUC 05)\n\nClusters from AMS:\n\n![AMS Clusters](images/05/ams_1dy.png)\n\nClusters from PDS:\n\n![PDS Clusters](images/05/pds_1dy.png)\n\n## Output Files\n\nThe program creates a directory with the HUCode as the name inside the `data/output` for data files (`.csv`, `.nc`, etc), and makes a directory with HUCode as name in the `images` for the plots (`.png`).\n\nThe files generated by each functions are discussed below.\n\n### calculate-weights\nFiles: `ids-and-weights.nc,ids.csv`\n\nIds and Weights for each cells in the Livneh Dataset cropped to the given basin.\n\n### extract-annual-timeseries\nFiles: `prec.YYYY.csv` (where YYYY is year from 1915-2011)\n\nDaily timeseries of the basin averaged precipitation data for each year.\n\n### amd-and-pds\nFiles: `{ams,pds}_Ndy_{series,grids}.csv` where N is the `num_days` (duration of event considered)\n\nAMS and PDS events for that basin for all the years. `_series.csv` has the basin averaged precipitation, while `_grids.csv` has the values of the precipitation in the grids inside the basin.\n\nPDS is calculated with threhold value of minimum precipitation value of AMS events.\n\n### plot-clusters\nFiles: `{ams,pds}_Ndy.png` where N is the `num_days` (duration of event considered)\n\nPlot of the clusters made from the events data for the AMS list or PDS list. The images show clusters with individual scale on bottom, and global scale on top for comparision.\n\n## Run Times\nThis is the runtime on Trinity (HUCode: 1203) on my laptop\n- CPU: AMD Ryzen 7 6800U with Radeon Graphics (16) @ 2.700GHz,\n- Storage: PCIe Gen 4 NVMe SSD Up to Seq Read: 4000MB/s, Write: 2000MB/s,\n- RAM:  32 GB Dual Channel LPDDR5 up to 5500 MHz\n\nOn the first run, where it has to calculate everything:\n\n    *** CALCULATE_WEIGHTS\n        Time taken:  2.290 seconds ( 0.038 minutes)\n    *** EXTRACT_ANNUAL_TIMESERIES\n        Time taken:  86.598 seconds ( 1.4 minutes)\n    *** AMS_AND_PDS\n        Time taken:  23.125 seconds ( 0.39 minutes)\n\t*** PLOT_CLUSTERS\n\t    Time taken:  3.401 seconds ( 0.057 minutes)\n\nAnd then on the subsequent runs:\n\n    *** CALCULATE_WEIGHTS\n        Time taken:  0.002 seconds ( 2.9e-05 minutes)\n    *** EXTRACT_ANNUAL_TIMESERIES\n        Time taken:  0.075 seconds ( 0.0013 minutes)\n    *** AMS_AND_PDS\n        Time taken:  0.192 seconds ( 0.0032 minutes)\n    *** PLOT_CLUSTERS\n        Time taken:  3.503 seconds ( 0.058 minutes)\n\nFor North Branch Potomac (HUCode: 02070002) first runtime are as follows:\n\n    *** CALCULATE_WEIGHTS\n        Time taken:  0.104 seconds ( 0.0017 minutes)\n    *** EXTRACT_ANNUAL_TIMESERIES\n        Time taken:  83.067 seconds ( 1.4 minutes)\n    *** AMS_AND_PDS\n        Time taken:  9.011 seconds ( 0.15 minutes)\n    *** PLOT_CLUSTERS\n        Time taken:  2.308 seconds ( 0.038 minutes)\n\nFor Mid Atlantic Region \u003cHUC 02\u003e\n\n    *** CALCULATE_WEIGHTS\n        Time taken:  64.774 seconds ( 1.1 minutes)\n    *** EXTRACT_ANNUAL_TIMESERIES\n        Time taken:  91.243 seconds ( 1.5 minutes)\n    *** AMS_AND_PDS\n        Time taken:  22.159 seconds ( 0.37 minutes)\n\t*** PLOT_CLUSTERS\n        Time taken:  22.557 seconds ( 0.38 minutes)\n\nFor Ohio Region \u003cHUC 05\u003e\n\n    *** CALCULATE_WEIGHTS\n        Time taken:  128.020 seconds ( 2.1 minutes)\n    *** EXTRACT_ANNUAL_TIMESERIES\n        Time taken:  91.441 seconds ( 1.5 minutes)\n    *** AMS_AND_PDS\n        Time taken:  18.734 seconds ( 0.31 minutes)\n    *** PLOT_CLUSTERS\n        Time taken:  32.160 seconds ( 0.54 minutes)\n\nSeems like the extraction part is similar for all the basins, but the other ones vary by size. But it's reasonable time if you want to process a lot of basins.\n\n# Tips and tricks\n## Running in parallel\nThe program is written to run everything without multiprocessing in the module itself, so multiple basins can be processed in parallel.\n\nUse the `--list-hucs` command to get a list of codes for all basins, and then run the batch processing in parallel for them. You may want to filter the basins to only the ones that you want to process.\n\nThere are some basins that seem to be troublesome, you can add the `--timeout` flag on `gnu parallel` to kill processes that take too much time.\n\nHere the first line makes the list of huc2 basins, edit it to the basins you want. Then run the second line that'll get the codes from that file and run locus on parallel, `-j 4` will run 4 process in parallel at a given time, `--timeout 200%` will kill any process that takes more than double the median time to complete.\n\n    python -m src.locus -l 2 \u003e basin-lists/huc2.txt\n\tawk -F ' :' '{print $1}' \u003c basin-lists/huc4.txt | parallel --timeout 200% -j 4 --bar 'python -m src.locus -d -b {0} \u003e\u003e reports/{0}.org'\n\nYou can also log the progress and resume from the where you left off. For more usage refer to `gnu parallel` manual.\n\n## Remore extra paddings for plots\nThe plots are a little weird, and I'm not that familiar with matplotlib, it seems to have a lot of extra border specially since we don't know how many clusters will be there, so I use `imagemagick` to trim the borders.\n\nHere this example command will remove the excess padding and then add 20x20 padding on all sides\n\n    mogrify -trim +repage -bordercolor white -border 20x20  images/02/*\n## Generating reports\nThe output of the command can be piped, only the reporting texts from the main function of the `src/locus.py` is output to the `stdout`. The piped contents are in `emacs-org` format, which can be exported to any other formats (html,latex,pdf,etc) with emacs, if you do not have emacs you can use `pandoc` to convert it. Or you can modify the code to print markdown syntax instead, the output is simple so you only have to change all `\"*\"` in headings to `\"#\"`.\n\nIf you need to print any debugging information that's not supposed to goto the report, print it in the `stderr`.\n\n# Server\nThe locus server files are in server directory. The server data directory structure are as follows:\n\n    download/\n    thumbnails/\n    map.html\n    netcdfs-index.html\n    netcdfs-index.json\n\nThe generated `netCDF` files are in `download` directory, one file per grid value (cluster), and the netcdf files are converted into PNG images and saved in `thumbnails`. Following command can be used to convert the files in parallel using [`gnu parallel`](https://www.gnu.org/software/parallel/) and [`gdal`](https://gdal.org/):\n\n    ls -1 download/*.nc | parallel --silent --bar echo 'gdaldem color-relief -alpha netcdf:{1} ../precip-color.txt thumbnails/{1.}.png'\n\nAs for the files:\n- map.html: Interactive map showing the HUC boundaries and the clusters\n- netcdfs-index.html: interactive page with tabulated data for the clusters\n- netcdfs-index.json: tablulated data for the clusters in computer understandable languages for scripts\n\n# Note\n\n## Livneh Data Range\nThe Livneh Data Range is approximately from lat 25 to 53, and lon -125 to -67, which means it doesn't cover the following regions:\n- Alaska Region \u003cHUC 19\u003e\n  (-179.230, 51.157, 179.857, 71.440)\n- Hawaii Region \u003cHUC 20\u003e\n  (-178.431, 18.860, -154.753, 28.507)\n- Caribbean Region \u003cHUC 21\u003e\n  (-68.004, 17.624, -64.513, 18.567)\n- South Pacific Region \u003cHUC 22\u003e\n  (-176.674, -14.610, 166.711, 20.604)\n\n## Reporting Times\nThe time reported can include other steps than the one you asked it to do, if that is needed. For example, if you asked for extracting the timeseries, then it'll calculate the weights if there is no weights, or if you ask for `ams/pds` it'll extract the timeseries.\n\n## Weird behaviours\nHUC 09 takes a really long time (2.1 minutes in my laptop), I don't know why. Could be due to geometry? Need to check it out. Same with Great Lakes Region.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatreyagaurav%2Flocus-code-usace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatreyagaurav%2Flocus-code-usace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatreyagaurav%2Flocus-code-usace/lists"}