{"id":18819037,"url":"https://github.com/geoscienceaustralia/dea-cogger","last_synced_at":"2025-08-15T22:11:51.600Z","repository":{"id":50607295,"uuid":"136563960","full_name":"GeoscienceAustralia/dea-cogger","owner":"GeoscienceAustralia","description":"Convert NetCDFs to Cloud Optimized GeoTIFFs","archived":false,"fork":false,"pushed_at":"2021-08-16T00:31:39.000Z","size":386,"stargazers_count":15,"open_issues_count":4,"forks_count":4,"subscribers_count":40,"default_branch":"main","last_synced_at":"2025-04-13T23:49:58.961Z","etag":null,"topics":["cog","cog-conversion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeoscienceAustralia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-08T03:55:48.000Z","updated_at":"2022-07-30T19:54:00.000Z","dependencies_parsed_at":"2022-09-24T15:23:10.895Z","dependency_job_id":null,"html_url":"https://github.com/GeoscienceAustralia/dea-cogger","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/GeoscienceAustralia/dea-cogger","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-cogger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-cogger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-cogger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-cogger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeoscienceAustralia","download_url":"https://codeload.github.com/GeoscienceAustralia/dea-cogger/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-cogger/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270637208,"owners_count":24620425,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-15T02:00:12.559Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cog","cog-conversion"],"created_at":"2024-11-08T00:19:55.799Z","updated_at":"2025-08-15T22:11:51.559Z","avatar_url":"https://github.com/GeoscienceAustralia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NetCDF to COG Conversion Tool\n\nConvert Open Datacube NetCDF files to Cloud Optimised GeoTIFFs.\n\nIt runs prior to uploading from the NCI file systems to AWS S3.\n\nUse `dea-cogger` convert to COG:\n\n## Installation\n\n```\nconda create -n cogger datacube=1.8.3 python=3.8 gdal boto3 boltons structlog mpi4py colorama requests chardet\nconda activate cogger\npip install requests-aws4auth\npip install -e .\n```\n## Usage\n\nExample commands for calculating which datasets require conversion to COG by\ndownloading a subset of an S3 Inventory and comparing it with an ODC Index.\n\n```\ndea-cogger save-s3-inventory -p ls7_fc_albers -o tmp/ \ndea-cogger generate-work-list --product-name ls7_fc_albers -o tmp\n```\n\n### Running Conversion in parallel on Gadi\n\nExample PBS submission script to run in parallel on Gadi.\n\n``` sh\n\n#!/bin/bash\n#PBS -l wd,walltime=5:00:00,mem=190GB,ncpus=48,jobfs=1GB\n#PBS -P v10\n#PBS -q normal\n#PBS -l storage=gdata/v10+gdata/fk4+gdata/rs0+gdata/if87\n#PBS -W umask=33\n#PBS -N cog_ls8_nbar_albers \n\n\nmodule load dea\nmodule load openmpi/3.1.4\n\nmpirun --tag-output dea-cogger mpi-convert --product-name \"{{params.product}}\" --output-dir \"{{work_dir}}/out/\" ls8_nbar_albers_file_list.txt\n                \n```\n\n### Example configuration file\n\n\n```\n    products:\n      product_name:\n       prefix: WOfS/WOFLs/v2.1.5/combined\n       name_template: x_{x}/y_{y}/{time:%Y}/{time:%m}/{time:%d}/LS_WATER_3577_{x}_{y}_{time:%Y%m%d%H%M%S%f}\n       predictor: 2\n       no_overviews: [\"source\", \"observed\"]\n       default_resampling: average\n       white_list: None\n       black_list: None\n```\n\n- **product_name**:              A unique user defined string (required)\n- **prefix**:                    Define the cogs folder structure and name (required)\n- **name_template**:             Define how to decipher the input file names (required)\n- **default_resampling**:        Define the resampling method of pyramid view (default: average)\n- **predictor**:                 Define the predictor in COG convert (default: 2)\n- **no_overviews**:              A list of keywords of bands which don't require resampling (optional)\n- **white_list**:                A list of keywords of bands to be converted (optional)\n- **black_list**:                A list of keywords of bands excluded in cog convert (optional)\n\nNote: `no_overviews` contains the key words of the band names which one doesn't want to generate overviews.\n      This element cannot be used with other products as this 'cause it will match as *source*'.\n      For most products, this element is not needed. So far, only fractional cover percentile use this.\n      \n\n### Command: `save-s3-inventory`\n\nScan through S3 bucket for the specified product and fetch the file path of the uploaded files.\nSave those file into a pickle file for further processing.\nUses a configuration file to define the file naming schema.\n\n\n### Command: `generate-work-list`\n\nCompares ODC URI's against an S3 bucket  and writes the list of datasets\nfor COG conversion into a file.\n\nUses a configuration file to define the file naming schema.\n\n\n### Command: `mpi-convert`\n\nBulk COG Convert netcdf files to COG format using MPI tool.\nIterate over the file list and assign MPI worker for processing.\nSplit the input file by the number of workers, each MPI worker completes every nth task.\nAlso, detect and fail early if not using full resources in an MPI job.\n\nReads the file naming schema from the configuration file.\n\n\n\n### Command: `verify`\n\nVerify converted GeoTIFF files are (Geo)TIFF with cloud optimized compatible structure.\nMandatory Requirement: `validate_cloud_optimized_geotiff.py` gdal file.\n\n\n## COG Creation Settings (What to set for predictor and resampling)\n\n**Predictor**\n\n\u003cint\u003e (1=default, 2=Horizontal differencing, 3 =floating point prediction)\n\n- **Horizontal differencing**\n\n  particularly useful for 16-bit data when the high-order and low-order bytes are changing at different frequencies.\n\n  Predictor=2 option should improve compression on any files greater than 8 bits/resel.\n\n- **Floating Point Prediction**\n  The **floating point** predictor PREDICTOR=3 results in significantly better compression ratios for floating point data.\n\n  There doesn't seem to be a performance penalty either for writing data with the floating point predictor, so it's a\n  pretty safe bet for any Float32 data.\n\n**Raster Resampling**\n\n*default_resampling* \u003cresampling method\u003e (average, nearest, mode)\n\n- **nearest**: Nearest neighbor has a tendency to leave artifacts such as stair-stepping and periodic striping in the\n  data which may not be apparent when viewing the elevation data but might affect derivative products.\n  They are not suitable for continuous data\n\n- **average**: average computes the average of all non-NODATA contributing pixels\n- **mode**: selects the value which appears most often of all the sampled points\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fdea-cogger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeoscienceaustralia%2Fdea-cogger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fdea-cogger/lists"}