Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/weiji14/deepbedmap
Going beyond BEDMAP2 using a super resolution deep neural network. Also a convenient flat file data repository for high resolution bed elevation datasets around Antarctica.
https://github.com/weiji14/deepbedmap
antarctica bedmap binder chainer data-science deep-neural-network digital-elevation-model flat-file-db generative-adversarial-network glaciology jupyter-notebook optuna pangeo remote-sensing super-resolution
Last synced: about 1 month ago
JSON representation
Going beyond BEDMAP2 using a super resolution deep neural network. Also a convenient flat file data repository for high resolution bed elevation datasets around Antarctica.
- Host: GitHub
- URL: https://github.com/weiji14/deepbedmap
- Owner: weiji14
- License: lgpl-3.0
- Created: 2018-09-06T00:28:32.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-06-17T17:57:39.000Z (over 2 years ago)
- Last Synced: 2024-11-21T08:35:48.726Z (3 months ago)
- Topics: antarctica, bedmap, binder, chainer, data-science, deep-neural-network, digital-elevation-model, flat-file-db, generative-adversarial-network, glaciology, jupyter-notebook, optuna, pangeo, remote-sensing, super-resolution
- Language: Jupyter Notebook
- Homepage:
- Size: 169 MB
- Stars: 43
- Watchers: 5
- Forks: 26
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
- open-sustainable-technology - DeepBedMap - Using a deep neural network to better resolve the bed topography of Antarctica. (Cryosphere / Glacier and Ice Sheets)
README
# DeepBedMap [[paper]](https://doi.org/10.5194/tc-14-3687-2020) [[poster]](https://github.com/weiji14/deepbedmap/issues/133) [[presentation]](https://hackmd.io/@weiji14/2022ML4Polar)
Going beyond BEDMAP2 using a super resolution deep neural network.
Also a convenient [flat file](https://en.wikipedia.org/wiki/Flat-file_database) data repository for high resolution bed elevation datasets around Antarctica.[](https://zenodo.org/badge/latestdoi/147595013)

[](https://github.com/ambv/black)
[](https://www.comet.ml/weiji14/deepbedmap/)
[](https://github.com/weiji14/deepbedmap/actions)
[](https://dependabot.com)

Directory structure
```
deepbedmap/
├── features/ (files describing the high level behaviour of various features)
│ ├── *.feature... (easily understandable specifications written using the Given-When-Then gherkin language)
│ └── README.md (markdown information on the feature files)
├── highres/ (contains high resolution localized DEMs)
│ ├── *.txt/csv/grd/xyz... (input vector file containing the point-based bed elevation data)
│ ├── *.json (the pipeline file used to process the xyz point data)
│ ├── *.nc (output raster netcdf files)
│ └── README.md (markdown information on highres data sources)
├── lowres/ (contains low resolution whole-continent DEMs)
│ ├── bedmap2_bed.tif (the low resolution DEM!)
│ └── README.md (markdown information on lowres data sources)
├── misc/ (miscellaneous raster datasets)
│ ├── *.tif (Surface DEMs, Ice Flow Velocity, etc. See list in Issue #9)
│ └── README.md (markdown information on miscellaneous data sources)
├── model/ (*hidden in git, neural network model related files)
│ ├── train/ (a place to store the raster tile bounds and model training data)
│ └── weights/ (contains the neural network model's architecture and weights)
├── .env (environment variable config file used by pipenv)
├── .ignore (files ignored by a particular piece of software)
├── . (stuff to make the code in this repo look and run nicely e.g. linters, CI/CD config files, etc)
├── Dockerfile (set of commands to fully reproduce the software stack here into a docker image, used by binder)
├── LICENSE.md (the license covering this repository)
├── Pipfile (what you want, the summary list of core python dependencies)
├── Pipfile.lock (what you need, all the pinned python dependencies for full reproducibility)
├── README.md (the markdown file you're reading now)
├── data_list.yml (human and machine readable list of the datasets and their metadata)
├── data_prep.ipynb/py (paired jupyter notebook/python script that prepares the data)
├── deepbedmap.ipynb/py (paired jupyter notebook/python script that predicts an Antarctic bed digital elevation model)
├── environment.yml (conda binary packages to install)
├── paper_figures.ipynb/py (paired jupyter notebook/python script to produce figures for DeepBedMap paper
├── srgan_train.ipynb/py (paired jupyter notebook/python script that trains the ESRGAN neural network model)
└── test_ipynb.ipynb/py (paired jupyter notebook/python script that runs doctests in the other jupyter notebooks!)
```# Getting started
## Quickstart
Launch in [Binder](https://mybinder.readthedocs.io) (Interactive jupyter notebook/lab environment in the cloud).
[](https://mybinder.org/v2/gh/weiji14/deepbedmap/master)
[](https://colab.research.google.com/github/weiji14/deepbedmap)## Installation

Start by cloning this [repo-url](/../../)
git clone
Then I recommend [using conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) to install the non-python binaries (e.g. GMT, CUDA, etc).
The conda virtual environment will also be created with Python and [pipenv](https://pipenv.readthedocs.io) installed.cd deepbedmap
conda env create -f environment.ymlActivate the conda environment first.
conda activate deepbedmap
Then set some environment variables **before** using pipenv to install the necessary python libraries,
otherwise you may encounter some problems (see Common problems below).
You may want to ensure that `which pipenv` returns something similar to ~/.conda/envs/deepbedmap/bin/pipenv.export HDF5_DIR=$CONDA_PREFIX/
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/
pipenv install --python $CONDA_PREFIX/bin/python --dev
#or just
HDF5_DIR=$CONDA_PREFIX/ LD_LIBRARY_PATH=$CONDA_PREFIX/lib/ pipenv install --python $CONDA_PREFIX/bin/python --devFinally, double-check that the libraries have been installed.
pipenv graph
### Syncing/Updating to new dependencies
conda env update -f environment.yml
pipenv sync --dev### Common problems
Note that the [.env](https://pipenv.readthedocs.io/en/latest/advanced/#configuration-with-environment-variables) file stores some environment variables.
So if you run `conda activate deepbedmap` followed by some other command and get an `...error while loading shared libraries: libpython3.7m.so.1.0...`,
you may need to run `pipenv shell` or do `pipenv run ` to have those environment variables registered properly.
Or just run this first:export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/
Also, if you get a problem when using `pipenv` to install [netcdf4](https://github.com/Unidata/netcdf4-python), make sure you have done:
export HDF5_DIR=$CONDA_PREFIX/
and then you can try using `pipenv install` or `pipenv sync` again.
See also this [issue](https://github.com/pydata/xarray/issues/3185#issuecomment-520693149) for more information.## Running jupyter lab
conda activate deepbedmap
pipenv shellpython -m ipykernel install --user --name deepbedmap #to install conda env properly
jupyter kernelspec list --json #see if kernel is installed
jupyter lab &## Citing
The paper is published at [The Cryosphere](https://www.the-cryosphere.net) and can be referred to using the following BibTeX code:
@Article{tc-14-3687-2020,
AUTHOR = {Leong, W. J. and Horgan, H. J.},
TITLE = {DeepBedMap: a deep neural network for resolving the bed topography of Antarctica},
JOURNAL = {The Cryosphere},
VOLUME = {14},
YEAR = {2020},
NUMBER = {11},
PAGES = {3687--3705},
URL = {https://tc.copernicus.org/articles/14/3687/2020/},
DOI = {10.5194/tc-14-3687-2020}
}The DeepBedMap_DEM v1.1.0 dataset is available from Zenodo at https://doi.org/10.5281/zenodo.4054246.
Neural network model training experiment runs are also recorded at https://www.comet.ml/weiji14/deepbedmap.
Python code for the DeepBedMap model here on Github is also mirrored on Zenodo at https://doi.org/10.5281/zenodo.3752613.