{"id":18818952,"url":"https://github.com/geoscienceaustralia/ml-preprocessing","last_synced_at":"2025-04-13T23:32:50.599Z","repository":{"id":145587646,"uuid":"180438971","full_name":"GeoscienceAustralia/ML-preprocessing","owner":"GeoscienceAustralia","description":"2D Wavelet decomposition/reconstruction for raster data","archived":false,"fork":false,"pushed_at":"2024-11-19T08:58:30.000Z","size":4976,"stargazers_count":6,"open_issues_count":0,"forks_count":4,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-27T13:46:06.674Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeoscienceAustralia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-04-09T19:50:41.000Z","updated_at":"2024-11-19T08:58:34.000Z","dependencies_parsed_at":"2023-12-12T03:22:20.264Z","dependency_job_id":"70070808-800e-44f7-9485-a072a5156dfd","html_url":"https://github.com/GeoscienceAustralia/ML-preprocessing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2FML-preprocessing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2FML-preprocessing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2FML-preprocessing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2FML-preprocessing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeoscienceAustralia","download_url":"https://codeload.github.com/GeoscienceAustralia/ML-preprocessing/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248796962,"owners_count":21163058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T00:19:35.307Z","updated_at":"2025-04-13T23:32:50.309Z","avatar_url":"https://github.com/GeoscienceAustralia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# geo-wavelets\n2D Wavelet decomposition/reconstruction for raster data\n\n## Running on Gadi:\n\nLoad modules and define paths (amend as required) as follows for interactive runs:\n\n```\nmodule purge\nmodule load python3/3.7.4\nmodule load gdal/3.0.2\nmodule load openmpi/4.1.4\nmodule load hdf5/1.10.5p\n\nsource /g/data/ge3/ml_preproc_venv/bin/activate\n\nexport GDAL_DATA=/apps/gdal/3.0.2/share/gdal/\nexport PYTHONPATH=/apps/gdal/3.0.2/lib64:/apps/gdal/3.0.2/lib64/python3.7/site-packages\nexport LC_ALL=en_AU.UTF-8\nexport LANG=en_AU.UTF-8\nexport PYTHONPATH=/g/data/ge3/rakib/ML-preprocessing:$PYTHONPATH\n```\n\nAmend the last line above to point to a local copy of ML-preprocessing.\nFor `pbs` batch jobs, check example pbs scripts in the `preprocessing/pbs` folder in this repository.\n\n## Running Tests on Gadi:\n\nNote that if you have a local version of Python3 installed in your environment, running `pytest` in the `tests` folder will likely fail, with `gdal` being reported as missing. In such cases, run `pytest` as follows:\n\n`python3 -m pytest`\n\nThe above will run the system version of `pytest` and the tests should all run to completion.\n\n## User Parameters \n\n`multiscale.py` takes the following input parameters:\n\n```\nUsage: multiscale.py [OPTIONS] INPUT OUTPUT_FOLDER MAX_LEVEL\n\n  INPUT: Path to raster files, or a file containing a list of raster file\n  names (with full path)\n\n  OUTPUT_FOLDER: Output folder\n\n  MAX_LEVEL: Maximum level up to which wavelet reconstructions are to be\n  computed. Each consecutive            level halves the raster resolution\n  (doubles the spatial wavelength); thus the spatial            wavelength\n  at a given level is given by:\n\n             spatial_wavelength = orig_spatial_wavelength * 2^level\n\n  Example usage: mpirun -np 2 python multiscale.py filelist.txt /tmp/output\n  10 --max-search-dist 500\n\n  Running in Cluster Environments: This script requires all raster data to\n  be loaded into memory for processing. Furthermore, each raster being\n  processed requires 4 times as much memory, e.g. a 7 GB raster will require\n  ~28 GB of RAM. Hence, for parallel runs, one must be judicious in terms of\n  allocating processors, bearing in mind the amount of memory available on a\n  given compute node on the cluster.\n\nOptions:\n  --file-extension TEXT           File extension to use (e.g. '.tif') to\n                                  search for input files; only applicableif\n                                  the 'input' argument is a folder.\n\n  --mother-wavelet [bior1.1|bior1.3|bior1.5|bior2.2|bior2.4|bior2.6|bior2.8|bior3.1|bior3.3|bior3.5|bior3.7|bior3.9|bior4.4|bior5.5|bior6.8|coif1|coif2|coif3|coif4|coif5|coif6|coif7|coif8|coif9|coif10|coif11|coif12|coif13|coif14|coif15|coif16|coif17|db1|db2|db3|db4|db5|db6|db7|db8|db9|db10|db11|db12|db13|db14|db15|db16|db17|db18|db19|db20|db21|db22|db23|db24|db25|db26|db27|db28|db29|db30|db31|db32|db33|db34|db35|db36|db37|db38|dmey|haar|rbio1.1|rbio1.3|rbio1.5|rbio2.2|rbio2.4|rbio2.6|rbio2.8|rbio3.1|rbio3.3|rbio3.5|rbio3.7|rbio3.9|rbio4.4|rbio5.5|rbio6.8|sym2|sym3|sym4|sym5|sym6|sym7|sym8|sym9|sym10|sym11|sym12|sym13|sym14|sym15|sym16|sym17|sym18|sym19|sym20]\n                                  Name of the mother wavelet\n  --extension-mode [zero|constant|symmetric|reflect|periodic|smooth|periodization]\n                                  Signal extension mode used for padding\n  --extrapolate BOOLEAN           Extrapolate raster if NO_DATA_VALUES are\n                                  found. 'Ringing' artefacts can result near\n                                  sharp contrasts in image values --\n                                  especially at the edges of NO_DATA_VALUE\n                                  regions. By extrapolating image values to\n                                  regions of NO_DATA_VALUE, 'ringing'\n                                  artefacts can be pushed further outward,\n                                  away from the region of interest in the\n                                  original image. Raster data are extrapolated\n                                  by default and this parameter has no effect\n                                  when the input raster has no masked regions\n\n  --max-search-dist INTEGER       Maximum search distance (in pixels) for\n                                  extrapolating image values to regions of\n                                  NO_DATA_VALUE in input raster; not used if\n                                  raster has no masked regions\n\n  --smoothing-iterations INTEGER  Number of smoothing iterations used for\n                                  smoothing extrapolated values; see option\n                                  --max-search-dist\n\n  --keep-level INTEGER            Level to keep. Note that by default all\n                                  levels up to max-level are saved, which may\n                                  cause disk space issues. This option allows\n                                  users to save only those levels that are of\n                                  interest; e.g. to keep only levels 5 and 6,\n                                  this option must be repeated twice for the\n                                  corresponding levels\n\n  --log-level [DEBUG|INFO|WARN]   Logging verbosity\n  -h, --help                      Show this message and exit.\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fml-preprocessing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeoscienceaustralia%2Fml-preprocessing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fml-preprocessing/lists"}