{"id":39854269,"url":"https://github.com/mdsumner/gdx","last_synced_at":"2026-01-18T13:48:12.525Z","repository":{"id":289018029,"uuid":"969302127","full_name":"mdsumner/gdx","owner":"mdsumner","description":null,"archived":false,"fork":false,"pushed_at":"2025-11-25T12:47:47.000Z","size":46,"stargazers_count":11,"open_issues_count":5,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-01-12T02:48:09.726Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mdsumner.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-19T20:53:34.000Z","updated_at":"2025-12-25T12:44:00.000Z","dependencies_parsed_at":"2025-10-20T08:16:55.506Z","dependency_job_id":"4baab3f2-0603-47f3-9a87-91aede5017e1","html_url":"https://github.com/mdsumner/gdx","commit_stats":null,"previous_names":["mdsumner/gdx"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mdsumner/gdx","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fgdx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fgdx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fgdx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fgdx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mdsumner","download_url":"https://codeload.github.com/mdsumner/gdx/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mdsumner%2Fgdx/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28536884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T13:04:05.990Z","status":"ssl_error","status_checked_at":"2026-01-18T13:01:44.092Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T13:48:11.766Z","updated_at":"2026-01-18T13:48:12.509Z","avatar_url":"https://github.com/mdsumner.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\"\n)\nreticulate::use_python(\"~/workenv/bin/python3\")\n#reticulate::repl_python()\n\n```\n\n# gdx\n\n\u003c!-- badges: start --\u003e\n\n\u003c!-- badges: end --\u003e\n\nThe goal of gdx is to integrate GDAL with xarray, especially for the multidimensional API which is still relatively underutilized.\n\n## Todo\n\n- [ ] apply xarray indexes when relevant in Raster and Multidim (see [issue](https://github.com/mdsumner/gdx/issues/3 for some discussion))\n- [ ] explore when we need to control driver choice\n- [ ] compare to opening with GDAL itself after `mdim mosaic`\n\n\n\n\n\nHere's a basic example, this could be registered as an xarray backend *engine*.\n\n```{python basic, eval = F}\nfrom gdx import GDALBackendEntrypoint\nbackend = GDALBackendEntrypoint()\ndsn =  \"/vsicurl/https://projects.pawsey.org.au/idea-sealevel-glo-phy-l4-nrt-008-046/data.marine.copernicus.eu/SEALEVEL_GLO_PHY_L4_NRT_008_046/cmems_obs-sl_glo_phy-ssh_nrt_allsat-l4-duacs-0.125deg_P1D_202506/2025/08/nrt_global_allsat_phy_l4_20250825_20250825.nc\"\nds = backend.open_dataset(f'vrt://{dsn}?sd_name=vgos', chunks = {})\nds1 = backend.open_dataset(dsn, multidim = True, chunks = {}) \n```\n\nWe have a Raster xarray: \n  \n```{python r,eval=F}\nds\n\n\u003cxarray.Dataset\u003e Size: 17MB\nDimensions:  (x: 2880, y: 1440)\nCoordinates:\n  * x        (x) float64 23kB -180.0 -179.9 -179.8 -179.6 ... 179.6 179.8 179.9\n  * y        (y) float64 12kB 90.0 89.88 89.75 89.62 ... -89.62 -89.75 -89.88\nData variables:\n    band_1   (y, x) int32 17MB dask.array\u003cchunksize=(1440, 2880), meta=np.ndarray\u003e\nAttributes:\n    crs:           GEOGCS[\"unknown\",DATUM[\"unnamed\",SPHEROID[\"Spheroid\",63781...\n    geotransform:  (-180.0, 0.125, 0.0, 90.0, 0.0, -0.125)\n```\n\nand a Multidim xarray:\n\n```{python m,eval=FALSE}\nds1\n\n\u003cxarray.Dataset\u003e Size: 166MB\nDimensions:    (latitude: 1440, nv: 2, longitude: 2880, time: 1)\nCoordinates:\n  * latitude   (latitude) float32 6kB -89.94 -89.81 -89.69 ... 89.69 89.81 89.94\n  * nv         (nv) int32 8B 0 1\n  * longitude  (longitude) float32 12kB -179.9 -179.8 -179.7 ... 179.8 179.9\n  * time       (time) float32 4B 2.763e+04\nData variables:\n    lat_bnds   (latitude, nv) float32 12kB dask.array\u003cchunksize=(1440, 2), meta=np.ndarray\u003e\n    lon_bnds   (longitude, nv) float32 23kB dask.array\u003cchunksize=(2880, 2), meta=np.ndarray\u003e\n    sla        (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    err_sla    (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    ugosa      (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    err_ugosa  (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    vgosa      (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    err_vgosa  (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    adt        (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    ugos       (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    vgos       (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\n    flag_ice   (time, latitude, longitude) int32 17MB dask.array\u003cchunksize=(1, 1440, 2880), meta=np.ndarray\u003e\nAttributes: (12/44)\n    Conventions:                     CF-1.6\n    Metadata_Conventions:            Unidata Dataset Discovery v1.0\n    cdm_data_type:                   Grid\n    comment:                         Sea Surface Height measured by Altimetry...\n    contact:                         servicedesk.cmems@mercator-ocean.eu\n    creator_email:                   servicedesk.cmems@mercator-ocean.eu\n    ...                              ...\n    summary:                         DUACS Near-Real-Time Level-4 sea surface...\n    time_coverage_duration:          P1D\n    time_coverage_end:               2025-08-25T12:00:00Z\n    time_coverage_resolution:        P1D\n    time_coverage_start:             2025-08-24T12:00:00Z\n    title:                           NRT merged all satellites Global Ocean G...\n```\n\nThere's one variable called 'band_1' for the raster: \n\n```{python eval = FALSE}\nds.band_1.isel(x = 0)\n# \u003cxarray.DataArray 'band_1' (y: 1440)\u003e Size: 6kB\n# dask.array\u003cgetitem, shape=(1440,), dtype=int32, chunksize=(1440,), chunktype=numpy.ndarray\u003e\n# Coordinates:\n#     x        float64 8B -180.0\n#   * y        (y) float64 12kB 90.0 89.88 89.75 89.62 ... -89.62 -89.75 -89.88\n# Attributes:\n#     nodata:   -2147483647.0\n#     scale:    0.0001\n#     offset:   0.0\n```\n\nwe can access actual values\n\n```{python, eval = F}\n## the raw values for now\nds.band_1.sel(x = 100, y = -50).values\n# array(441, dtype=int32)\n\n\nds1.sla.isel(longitude = 0, latitude = 1000).values\n#array([2404], dtype=int32)\n\n```\n\nThis example is a virtualized mosaic of NetCDF in multidim VRT. \n\n```{python mdim, eval=F}\nbig_virtual_mdim = \"/vsicurl/https://gist.githubusercontent.com/mdsumner/18c5d302d00b9a456bb73d30ac758764/raw/f26e1b2e202f759d6aace4d7deb3e04ea3c85f15/mdim.vrt\"\n\nbvm = backend.open_dataset(big_virtual_mdim, multidim = True, chunks = {})\n# \u003cxarray.Dataset\u003e Size: 3TB\n# Dimensions:   (Time: 5479, st_ocean: 51, yt_ocean: 1500, xt_ocean: 3600)\n# Coordinates:\n#   * Time      (Time) float64 44kB 1.132e+04 1.132e+04 ... 1.68e+04 1.68e+04\n#   * st_ocean  (st_ocean) float64 408B 2.5 7.5 12.5 ... 3.603e+03 4.509e+03\n#   * yt_ocean  (yt_ocean) float64 12kB -74.95 -74.85 -74.75 ... 74.75 74.85 74.95\n#   * xt_ocean  (xt_ocean) float64 29kB 0.05 0.15 0.25 0.35 ... 359.8 359.9 360.0\n# Data variables:\n#     temp      (Time, st_ocean, yt_ocean, xt_ocean) int16 3TB dask.array\u003cchunksize=(5479, 51, 1500, 3600), meta=np.ndarray\u003e\n    \n\nbvm.sel(xt_ocean = slice(140, 150), yt_ocean = slice(-55, -45), st_ocean = slice(8, 13)).isel(Time = -1).temp.values\n\n# array([[[-30770, -30784, -30799, ..., -30418, -30424, -30445],\n#         [-30755, -30771, -30788, ..., -30418, -30425, -30446],\n#         [-30744, -30764, -30788, ..., -30417, -30426, -30448],\n#         ...,\n#         [-29852, -29868, -29889, ..., -29413, -29338, -29325],\n#         [-29835, -29851, -29883, ..., -29385, -29327, -29324],\n#         [-29821, -29840, -29879, ..., -29353, -29319, -29322]]],\n#       shape=(1, 100, 100), dtype=int16)\n\n\n```\n\nThere's a lot more to do,  scaling works but I turned that off to test for now. .\n\nTemplate a list of netcdf files and mosaic them to VRT, then open with this xarray backend.  (Note this requires GDAL\u003e=3.12.0 ). \n\n```{python eval = FALSE}\nmonth = \"202501\"\nurl = [f\"/vsicurl/https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/{month}/oisst-avhrr-v02r01.{month}{(day+1):02d}.nc\" for day in range(31)]\ngdal.Run(\"mdim mosaic\", input = url, output =  \"oisst.vrt\", array = \"sst\")\nfrom gdx import GDALBackendEntrypoint\nbackend = GDALBackendEntrypoint()\n\nbackend.open_dataset(\"oisst.vrt\", multidim = True)\n\n# \u003cxarray.Dataset\u003e Size: 64MB\n# Dimensions:  (lat: 720, lon: 1440, time: 31, zlev: 1)\n# Coordinates:\n#   * lat      (lat) float64 6kB -89.88 -89.62 -89.38 -89.12 ... 89.38 89.62 89.88\n#   * lon      (lon) float64 12kB 0.125 0.375 0.625 0.875 ... 359.4 359.6 359.9\n#   * time     (time) float64 248B 1.717e+04 1.72e+04 ... 1.717e+04 1.717e+04\n#   * zlev     (zlev) float64 8B 0.0\n# Data variables:\n#     sst      (time, zlev, lat, lon) int16 64MB ...\n# \n\n```\n\n### Open questions \n\n- I set `chunks = {}` by default, is that ok\n- dask will very happily throw fsspec byte range requests at Thredds, more than 10x will get a 104 error but GDAL multidim is better behaved when you use it on its own: can we leverage this (GDAL connection pooling?) within xarray??\n- I saw errors from HDF5, but is that via NetCDF or is my driver select going wrong\n\n\n\n## Code of Conduct\n\nPlease note that the gdx project is released with a [Contributor Code of Conduct](https://contributor-covenant.org/version/2/1/CODE_OF_CONDUCT.html). By contributing to this project, you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdsumner%2Fgdx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmdsumner%2Fgdx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmdsumner%2Fgdx/lists"}