{"id":13948383,"url":"https://github.com/weiji14/deepbedmap","last_synced_at":"2025-09-13T00:32:32.588Z","repository":{"id":32970741,"uuid":"147595013","full_name":"weiji14/deepbedmap","owner":"weiji14","description":"Going beyond BEDMAP2 using a super resolution deep neural network. Also a convenient flat file data repository for high resolution bed elevation datasets around Antarctica.","archived":false,"fork":false,"pushed_at":"2022-06-17T17:57:39.000Z","size":177452,"stargazers_count":43,"open_issues_count":6,"forks_count":26,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-11-21T08:35:48.726Z","etag":null,"topics":["antarctica","bedmap","binder","chainer","data-science","deep-neural-network","digital-elevation-model","flat-file-db","generative-adversarial-network","glaciology","jupyter-notebook","optuna","pangeo","remote-sensing","super-resolution"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/weiji14.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-09-06T00:28:32.000Z","updated_at":"2024-11-09T01:09:17.000Z","dependencies_parsed_at":"2022-08-31T20:00:36.815Z","dependency_job_id":null,"html_url":"https://github.com/weiji14/deepbedmap","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weiji14%2Fdeepbedmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weiji14%2Fdeepbedmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weiji14%2Fdeepbedmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/weiji14%2Fdeepbedmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/weiji14","download_url":"https://codeload.github.com/weiji14/deepbedmap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226910645,"owners_count":17701848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["antarctica","bedmap","binder","chainer","data-science","deep-neural-network","digital-elevation-model","flat-file-db","generative-adversarial-network","glaciology","jupyter-notebook","optuna","pangeo","remote-sensing","super-resolution"],"created_at":"2024-08-08T05:01:21.888Z","updated_at":"2025-01-07T00:16:09.690Z","avatar_url":"https://github.com/weiji14.png","language":"Jupyter Notebook","funding_links":[],"categories":["Cryosphere","Observation and Conservation of Ecosystems"],"sub_categories":["Glacier and Ice Sheets","Ice and Poles"],"readme":"# DeepBedMap  [[paper]](https://doi.org/10.5194/tc-14-3687-2020) [[poster]](https://github.com/weiji14/deepbedmap/issues/133) [[presentation]](https://hackmd.io/@weiji14/2022ML4Polar)\n\nGoing beyond BEDMAP2 using a super resolution deep neural network.\nAlso a convenient [flat file](https://en.wikipedia.org/wiki/Flat-file_database) data repository for high resolution bed elevation datasets around Antarctica.\n\n[![DOI](https://zenodo.org/badge/147595013.svg)](https://zenodo.org/badge/latestdoi/147595013)\n![GitHub top language](https://img.shields.io/github/languages/top/weiji14/deepbedmap.svg)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)\n[![Comet.ML: experiments](https://img.shields.io/badge/Comet.ml-experiments-orange.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABHNCSVQICAgIfAhkiAAAAhpJREFUOI2Nk01IVGEUhp/zzfW3sUgllUIEzVApS2JyIwStylYmlC3EWvWzcNGyhdAqiGpZy4ja2UYMrBgoatnCTCNRJBxBpNE0bcbx3rlviyycScF3d+B93nMO33eMPEkiRLEs9BRgdYDzUSKCDTl4bWY5/pwqlJoNBkllmjZevWVjYgrzA37V13LgwjnYUzwH9JlZPL8xodQlSWsPHynJES3VnFTiYLvelcQ0QptGiEl+Vpvqz5kglBoMplZvXsP/8J5I+X5WffiWFMpECANHai5Bx2ScaGPdX/asmY14kgAG088eE0y9IdJYSSif+ek03uJP/KQjRJR3tG+FAZ5IqrZAaowQTK5c2gv7GjEzFhPfsdPXqbnchzmPTHKJsqNN/60NdHtBSK99jkNlGoumMGUoPXaFqhu3/7kKa6q2gwHOuyJHbbY0pOjiLbyWNojOUtHVuxOQr1oHECy8xJ+5T6S1haLOu7gSt9sAXChmS07dofDEAwoarpJd/YorrdwtP+s2Ap6yNIqLHiKYeYEyK6yP3gPC3QQM2+Yzjq5/HGj1Ko4TqahHmR+4gkKIHgZvx2kWgGpnZqQDuovrO7HiMswV4pbjkJ2H9Bikxzf92fyAvpy7yAR/vrKmB6TUuLQ8LCWfS6kxKTUh+Qvaov78NAAkNUv6IklaiUtrn6RsaiuYkHRmK2PbhADE/ICeAo86wAEJYIhtzvk3y+cYpafNe/QAAAAASUVORK5CYII=)](https://www.comet.ml/weiji14/deepbedmap/)\n[![Github Actions Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fweiji14%2Fdeepbedmap%2Fbadge\u0026style=flat)](https://github.com/weiji14/deepbedmap/actions)\n[![Dependabot Status](https://api.dependabot.com/badges/status?host=github\u0026repo=weiji14/deepbedmap)](https://dependabot.com)\n\n![DeepBedMap DEM over entire Antarctic continent, EPSG:3031 projection](https://user-images.githubusercontent.com/23487320/94385232-16dee180-01a1-11eb-83ce-8793709079ff.png)\n\n![DeepBedMap Pipeline](https://yuml.me/diagram/scruffy;dir:LR/class/[Data|Highres/Lowres/Misc]-\u003e[Preprocessing|data_prep.ipynb],[Preprocessing]-\u003e[Model-Training|srgan_train.ipynb],[Model-Training]-\u003e[Inference|deepbedmap.ipynb])\n\n\u003cdetails\u003e\n\u003csummary\u003eDirectory structure\u003c/summary\u003e\n\n```\n  deepbedmap/\n    ├── features/ (files describing the high level behaviour of various features)\n    │    ├── *.feature... (easily understandable specifications written using the Given-When-Then gherkin language)\n    │    └── README.md (markdown information on the feature files)\n    ├── highres/ (contains high resolution localized DEMs)\n    │    ├── *.txt/csv/grd/xyz... (input vector file containing the point-based bed elevation data)\n    │    ├── *.json (the pipeline file used to process the xyz point data)\n    │    ├── *.nc (output raster netcdf files)\n    │    └── README.md (markdown information on highres data sources)\n    ├── lowres/ (contains low resolution whole-continent DEMs)\n    │    ├── bedmap2_bed.tif (the low resolution DEM!)\n    │    └── README.md (markdown information on lowres data sources)\n    ├── misc/ (miscellaneous raster datasets)\n    │    ├── *.tif (Surface DEMs, Ice Flow Velocity, etc. See list in Issue #9)\n    │    └── README.md (markdown information on miscellaneous data sources)\n    ├── model/ (*hidden in git, neural network model related files)\n    │    ├── train/ (a place to store the raster tile bounds and model training data)\n    │    └── weights/ (contains the neural network model's architecture and weights)\n    ├── .env (environment variable config file used by pipenv)\n    ├── .\u003csomething\u003eignore (files ignored by a particular piece of software)\n    ├── .\u003csomething else\u003e (stuff to make the code in this repo look and run nicely e.g. linters, CI/CD config files, etc)\n    ├── Dockerfile (set of commands to fully reproduce the software stack here into a docker image, used by binder)\n    ├── LICENSE.md (the license covering this repository)\n    ├── Pipfile (what you want, the summary list of core python dependencies)\n    ├── Pipfile.lock (what you need, all the pinned python dependencies for full reproducibility)\n    ├── README.md (the markdown file you're reading now)\n    ├── data_list.yml (human and machine readable list of the datasets and their metadata)\n    ├── data_prep.ipynb/py (paired jupyter notebook/python script that prepares the data)\n    ├── deepbedmap.ipynb/py (paired jupyter notebook/python script that predicts an Antarctic bed digital elevation model)\n    ├── environment.yml (conda binary packages to install)\n    ├── paper_figures.ipynb/py (paired jupyter notebook/python script to produce figures for DeepBedMap paper\n    ├── srgan_train.ipynb/py (paired jupyter notebook/python script that trains the ESRGAN neural network model)\n    └── test_ipynb.ipynb/py (paired jupyter notebook/python script that runs doctests in the other jupyter notebooks!)\n```\n\u003c/details\u003e\n\n# Getting started\n\n## Quickstart\n\nLaunch in [Binder](https://mybinder.readthedocs.io) (Interactive jupyter notebook/lab environment in the cloud).\n\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/weiji14/deepbedmap/master)\n[![Open All Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/weiji14/deepbedmap)\n\n## Installation\n\n![Installation steps](https://yuml.me/diagram/scruffy/class/[Git|clone-repo]-\u003e[Conda|install-binaries-and-pipenv],[Conda]-\u003e[Pipenv|install-python-libs])\n\nStart by cloning this [repo-url](/../../)\n\n    git clone \u003crepo-url\u003e\n\nThen I recommend [using conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) to install the non-python binaries (e.g. GMT, CUDA, etc).\nThe conda virtual environment will also be created with Python and [pipenv](https://pipenv.readthedocs.io) installed.\n\n    cd deepbedmap\n    conda env create -f environment.yml\n\nActivate the conda environment first.\n\n    conda activate deepbedmap\n\nThen set some environment variables **before** using pipenv to install the necessary python libraries,\notherwise you may encounter some problems (see Common problems below).\nYou may want to ensure that `which pipenv` returns something similar to ~/.conda/envs/deepbedmap/bin/pipenv.\n\n    export HDF5_DIR=$CONDA_PREFIX/\n    export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/\n    pipenv install --python $CONDA_PREFIX/bin/python --dev\n    #or just\n    HDF5_DIR=$CONDA_PREFIX/ LD_LIBRARY_PATH=$CONDA_PREFIX/lib/ pipenv install --python $CONDA_PREFIX/bin/python --dev\n\nFinally, double-check that the libraries have been installed.\n\n    pipenv graph\n\n### Syncing/Updating to new dependencies\n\n    conda env update -f environment.yml\n    pipenv sync --dev\n\n### Common problems\n\nNote that the [.env](https://pipenv.readthedocs.io/en/latest/advanced/#configuration-with-environment-variables) file stores some environment variables.\nSo if you run `conda activate deepbedmap` followed by some other command and get an `...error while loading shared libraries: libpython3.7m.so.1.0...`,\nyou may need to run `pipenv shell` or do `pipenv run \u003ccmd\u003e` to have those environment variables registered properly.\nOr just run this first:\n\n    export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/\n\nAlso, if you get a problem when using `pipenv` to install [netcdf4](https://github.com/Unidata/netcdf4-python), make sure you have done:\n\n    export HDF5_DIR=$CONDA_PREFIX/\n\nand then you can try using `pipenv install` or `pipenv sync` again.\nSee also this [issue](https://github.com/pydata/xarray/issues/3185#issuecomment-520693149) for more information.\n\n## Running jupyter lab\n\n    conda activate deepbedmap\n    pipenv shell\n\n    python -m ipykernel install --user --name deepbedmap  #to install conda env properly\n    jupyter kernelspec list --json                        #see if kernel is installed\n    jupyter lab \u0026\n\n## Citing\n\nThe paper is published at [The Cryosphere](https://www.the-cryosphere.net) and can be referred to using the following BibTeX code:\n\n    @Article{tc-14-3687-2020,\n        AUTHOR = {Leong, W. J. and Horgan, H. J.},\n        TITLE = {DeepBedMap: a deep neural network for resolving the bed topography of Antarctica},\n        JOURNAL = {The Cryosphere},\n        VOLUME = {14},\n        YEAR = {2020},\n        NUMBER = {11},\n        PAGES = {3687--3705},\n        URL = {https://tc.copernicus.org/articles/14/3687/2020/},\n        DOI = {10.5194/tc-14-3687-2020}\n    }\n\nThe DeepBedMap_DEM v1.1.0 dataset is available from Zenodo at https://doi.org/10.5281/zenodo.4054246.\nNeural network model training experiment runs are also recorded at https://www.comet.ml/weiji14/deepbedmap.\nPython code for the DeepBedMap model here on Github is also mirrored on Zenodo at https://doi.org/10.5281/zenodo.3752613.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweiji14%2Fdeepbedmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fweiji14%2Fdeepbedmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fweiji14%2Fdeepbedmap/lists"}