{"id":18368938,"url":"https://github.com/cheginit/camels_netcdf","last_synced_at":"2025-04-06T17:31:56.398Z","repository":{"id":109631295,"uuid":"464631820","full_name":"cheginit/camels_netcdf","owner":"cheginit","description":"CAMELS dataset in NetCDF/Feather formats","archived":false,"fork":false,"pushed_at":"2022-03-02T04:56:55.000Z","size":26120,"stargazers_count":5,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-22T04:03:08.923Z","etag":null,"topics":["camels","camels-dataset","hydrology"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cheginit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-02-28T20:20:59.000Z","updated_at":"2024-01-31T06:25:24.000Z","dependencies_parsed_at":"2023-03-10T04:45:22.497Z","dependency_job_id":null,"html_url":"https://github.com/cheginit/camels_netcdf","commit_stats":{"total_commits":15,"total_committers":2,"mean_commits":7.5,"dds":0.1333333333333333,"last_synced_commit":"3158b3c14426142ab226de1b7742001eab3cffe5"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Fcamels_netcdf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Fcamels_netcdf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Fcamels_netcdf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cheginit%2Fcamels_netcdf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cheginit","download_url":"https://codeload.github.com/cheginit/camels_netcdf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247522438,"owners_count":20952552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camels","camels-dataset","hydrology"],"created_at":"2024-11-05T23:27:55.540Z","updated_at":"2025-04-06T17:31:54.500Z","avatar_url":"https://github.com/cheginit.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CAMELS dataset in NetCDF/Feather formats\n\n## Motivation\n\nThe [CAMELS](https://ral.ucar.edu/solutions/products/camels) datasets\nare not provided in an ideal format and takes\na bit of data processing to convert them to useful and convenient\nforms for geospatial analyses. So, I decided to use the beloved `netcdf` and `feather`\nformats to make the dataset more accessible while taking care of some\nsmall annoyances! The generated files are hosted on\n[Hydroshare](https://www.hydroshare.org/resource/658c359b8c83494aac0f58145b1b04e6/).\n\n## Usage\n\nFirst make sure that `h5netcdf`, `geopandas`, `xarray`, and `requests`\nPython packages are installed, then load the files directly like so:\n\n```python\nimport io\n\nimport geopandas as gpd\nimport requests\nimport xarray as xr\n\nr = requests.get(\"https://www.hydroshare.org/resource/658c359b8c83494aac0f58145b1b04e6/data/contents/camels_attributes_v2.0.feather\")\nattrs = gpd.read_feather(io.BytesIO(r.content))\n\nr = requests.get(\"https://www.hydroshare.org/resource/658c359b8c83494aac0f58145b1b04e6/data/contents/camels_attrs_v2_streamflow_v1p2.nc\")\nqobs = xr.open_dataset(io.BytesIO(r.content), engine=\"h5netcdf\")\n```\n\n## Methodology\n\nThis repo contains the code that I used to generate the datasets.\nThree data sources are available from the CAMELS dataset:\n\n- [Observed Flow](https://ral.ucar.edu/sites/default/files/public/product-tool/camels-catchment-attributes-and-meteorology-for-large-sample-studies-dataset-downloads/basin_timeseries_v1p2_metForcing_obsFlow.zip):\n  Streamflow observations for all 671 stations.\n- [Basin Geometries](https://ral.ucar.edu/sites/default/files/public/product-tool/camels-catchment-attributes-and-meteorology-for-large-sample-studies-dataset-downloads/basin_set_full_res.zip):\n  Polygons representing basins' boundaries for all 671 stations.\n- [Basin Attributes](https://ral.ucar.edu/sites/default/files/public/product-tool/camels-catchment-attributes-and-meteorology-for-large-sample-studies-dataset-downloads/camels_attributes_v2.0.zip): 60 Basin-level attributes for all 671 stations.\n\nThe `camel_netcdf.py` generates two files:\n\n- `camels_attributes_v2.0.feather`: Includes basin geometries and 60\n  basin-level attributes that are available in CAMELS.\n- `camels_attrs_v2_streamflow_v1p2.nc`: Includes observed flows\n  for all 671 stations, as well as the 60 basin-level attributes. It has\n  two dimensions (`station_id` and `time`) and 60 data variables.\n\nAdditionally, the script takes care of some small annoyances:\n\n- Station names didn't have a consistent format and there were some missing\n  commas and extra periods! Now, the names have a consistent format (`title`)\n  and there is comma before the states.\n- Station IDs and HUC 02 are strings with leading zeros if needed.\n\nYou can recreate the generated files locally using\n[`mambaforge`](https://github.com/conda-forge/miniforge/) (or `conda`) like so:\n\n```bash\ngit clone https://github.com/cheginit/camels_netcdf\ncd camels_netcdf\nmamba env create -f environment.yml\nconda activate camels\nchmod +x ./camels_netcdf.py\n./camels_netcdf.py\n```\n\n## Example Plots\n\nSnow fraction using `camels_attributes_v2.0.feather`:\n![camels_snow_fraction](plots/camels_snow_fraction.png)\n\nThe dataset `camels_attrs_v2_streamflow_v1p2.nc`:\n![dataset](plots/dataset.png)\n\nStreamflow observations for USGS-01013500:\n![qobs_01013500](plots/qobs_01013500.png)\n\n## Contributions\n\nContributions are welcome! Please feel free to open an issue/PR if you\nhave any suggestions that can improve the database.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheginit%2Fcamels_netcdf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheginit%2Fcamels_netcdf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheginit%2Fcamels_netcdf/lists"}