{"id":26131849,"url":"https://github.com/mdsumner/ngdal","last_synced_at":"2025-08-27T03:18:00.951Z","repository":{"id":280165876,"uuid":"938476488","full_name":"mdsumner/ngdal","owner":"mdsumner","description":"'GDAL' Multi-dimensional Array Model","archived":false,"fork":false,"pushed_at":"2025-02-25T02:41:00.000Z","size":9,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-18T14:55:44.302Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mdsumner.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-25T02:40:54.000Z","updated_at":"2025-02-26T02:16:36.000Z","dependencies_parsed_at":"2025-03-01T18:09:15.460Z","dependency_job_id":"773c27c8-5589-47f7-a7b5-1a8556743bfd","html_url":"https://github.com/mdsumner/ngdal","commit_stats":null,"previous_names":["mdsumner/ngdal"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mdsumner/ngdal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fngdal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fngdal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fngdal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fngdal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mdsumner","download_url":"https://codeload.github.com/mdsumner/ngdal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fngdal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272284348,"owners_count":24906871,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-27T02:00:09.397Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-10T22:21:07.335Z","updated_at":"2025-08-27T03:18:00.915Z","avatar_url":"https://github.com/mdsumner.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\neditor_options: \n  chunk_output_type: console\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n# ngdal\n\n\u003c!-- badges: start --\u003e\n\u003c!-- badges: end --\u003e\n\nThe goal of ngdal is to explore a multidimensional array model based on the [tidync package](https://docs.ropensci.org/tidync/). \n\nVery very unstable, we're relying on a dev-branch of the gdalraster package at [mdsumner/gdalraster](https://github.com/mdsumner/gdalraster/). \n\nProbably won't keep this name. \n\nThe value compared to tidync is that we can already hit single-end-point datacubes with GDAL, we don't have to go through the virtualization schemes used by xarray and co to append NetDF files. And we can\n\n- explore GDAL's virtualization in multidim VRT\n- use GDAL for a very wide range of sources and access schemes (earthdata, google cloud, netcdf, hdf5, grib, and Zarr)\n\nThere's no modelling of groups yet. \n\n```r\nlibrary(gdalraster)\nlibrary(ngdal)\nSys.setenv(GS_NO_SIGN_REQUEST=\"YES\")\nSys.setenv(\"GS)\")\nm \u003c- ngdal:::model(ngdal:::source(\"/vsigs/gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3\"))\nm@variables\n```\n\n```\n# A tibble: 277 × 7\n   group    id name                            ndims natts dim_coord active\n   \u003cchr\u003e \u003cdbl\u003e \u003cchr\u003e                           \u003cint\u003e \u003cint\u003e \u003clgl\u003e     \u003clgl\u003e \n 1 /         0 latitude                            1     1 NA        NA    \n 2 /         0 level                               1     1 NA        NA    \n 3 /         0 longitude                           1     1 NA        NA    \n 4 /         0 time                                1     1 NA        NA    \n 5 /         0 100m_u_component_of_wind            3     2 NA        NA    \n 6 /         0 100m_v_component_of_wind            3     2 NA        NA    \n 7 /         0 10m_u_component_of_neutral_wind     3     2 NA        NA    \n 8 /         0 10m_u_component_of_wind             3     2 NA        NA    \n 9 /         0 10m_v_component_of_neutral_wind     3     2 NA        NA    \n10 /         0 10m_v_component_of_wind             3     2 NA        NA    \n# ℹ 267 more rows\n# ℹ Use `print(n = ...)` to see more rows\n```\n\n```r\n## that thing NetCDF doesn't define, but tidync does (I think xaray calls these Datasets within a tree each unique 'grid' a group of same-shape variables )\ndplyr::distinct(m@grids, grid, rank)\n```\n\n```\n# A tibble: 6 × 2\n  grid     rank\n  \u003cchr\u003e   \u003cint\u003e\n1 1           1\n2 2           1\n3 3           1\n4 4           1\n5 4,1,3       3\n6 4,2,1,3     4\n```\n\n```r\nm@dimensions  ## sets of dimindex define a \"grid\"\n\nstr(m@coordinates)\n```\n\n```\n# A tibble: 4 × 3\n  name         size dimindex\n  \u003cchr\u003e       \u003cint\u003e    \u003cint\u003e\n1 latitude      721        1\n2 level          37        2\n3 longitude    1440        3\n4 time      1323648        4\n\nList of 4\n $ latitude : tibble [721 × 1] (S3: tbl_df/tbl/data.frame)\n  ..$ value: num [1:721] 90 89.8 89.5 89.2 89 ...\n $ level    : tibble [37 × 1] (S3: tbl_df/tbl/data.frame)\n  ..$ value: num [1:37] 1 2 3 5 7 10 20 30 50 70 ...\n $ longitude: tibble [1,440 × 1] (S3: tbl_df/tbl/data.frame)\n  ..$ value: num [1:1440] 0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 ...\n $ time     : tibble [1,323,648 × 1] (S3: tbl_df/tbl/data.frame)\n  ..$ value: num [1:1323648] 0 1 2 3 4 5 6 7 8 9 ...\n```\n\nTry a file on earthdata (we have set \"GDAL_HTTP_HEADER_FILE\")\n\n\n```r\ndsn \u003c- \"/vsicurl/https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20250223090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc\"\nngdal:::model(ngdal:::source(dsn))\n\n```\n\n```\n\u003cmodel\u003e\n @ source     : \u003csource\u003e\n .. @ description: chr \"/vsicurl/https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/2025022309\"| __truncated__\n .. @ family     : chr(0) \n @ rawinfo    : chr \"{  \\\"type\\\": \\\"group\\\",  \\\"driver\\\": \\\"netCDF\\\",  \\\"name\\\": \\\"/\\\",  \\\"attributes\\\": {    \\\"Conventions\\\": \\\"CF-\"| __truncated__\n @ variables  : tibble [9 × 7] (S3: tbl_df/tbl/data.frame)\n $ group    : chr [1:9] \"/\" \"/\" \"/\" \"/\" ...\n $ id       : num [1:9] 0 0 0 0 0 0 0 0 0\n $ name     : chr [1:9] \"time\" \"lat\" \"lon\" \"analysed_sst\" ...\n $ ndims    : int [1:9] 1 1 1 3 3 3 3 3 3\n $ natts    : int [1:9] 4 6 6 7 5 8 7 6 5\n $ dim_coord: logi [1:9] NA NA NA NA NA NA ...\n $ active   : logi [1:9] NA NA NA NA NA NA ...\n @ coordmeta  : tibble [3 × 7] (S3: tbl_df/tbl/data.frame)\n $ group    : chr [1:3] \"/\" \"/\" \"/\"\n $ id       : num [1:3] 0 0 0\n $ name     : chr [1:3] \"time\" \"lat\" \"lon\"\n $ ndims    : int [1:3] 1 1 1\n $ natts    : int [1:3] 4 6 6\n $ dim_coord: logi [1:3] NA NA NA\n $ active   : logi [1:3] NA NA NA\n @ axes       : tibble [21 × 3] (S3: tbl_df/tbl/data.frame)\n $ variable : chr [1:21] \"time\" \"lat\" \"lon\" \"analysed_sst\" ...\n $ dimension: Named chr [1:21] \"/time\" \"/lat\" \"/lon\" \"/time\" ...\n  ..- attr(*, \"names\")= chr [1:21] \"time\" \"lat\" \"lon\" \"analysed_sst1\" ...\n $ dimindex : int [1:21] 1 2 3 1 2 3 1 2 3 1 ...\n @ grids      : tibble [6 × 3] (S3: tbl_df/tbl/data.frame)\n $ dimension: Named chr [1:6] \"/time\" \"/lat\" \"/lon\" \"/time\" ...\n  ..- attr(*, \"names\")= chr [1:6] \"time\" \"lat\" \"lon\" \"analysed_sst1\" ...\n $ grid     : chr [1:6] \"1\" \"2\" \"3\" \"1,2,3\" ...\n $ rank     : int [1:6] 1 1 1 3 3 3\n @ coordinates:List of 3\n .. $ time: tibble [1 × 1] (S3: tbl_df/tbl/data.frame)\n ..  ..$ value: num 1.39e+09\n .. $ lat : tibble [17,999 × 1] (S3: tbl_df/tbl/data.frame)\n ..  ..$ value: num [1:17999] -90 -90 -90 -90 -89.9 ...\n .. $ lon : tibble [36,000 × 1] (S3: tbl_df/tbl/data.frame)\n ..  ..$ value: num [1:36000] -180 -180 -180 -180 -180 ...\n @ dimensions : tibble [3 × 3] (S3: tbl_df/tbl/data.frame)\n $ name    : chr [1:3] \"time\" \"lat\" \"lon\"\n $ size    : int [1:3] 1 17999 36000\n $ dimindex: int [1:3] 1 2 3\n \n ```\n\n## Code of Conduct\n  \nPlease note that the ngdal project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/1/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdsumner%2Fngdal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmdsumner%2Fngdal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdsumner%2Fngdal/lists"}