{"id":13735424,"url":"https://github.com/rouault/cog_validator","last_synced_at":"2025-06-22T04:38:18.779Z","repository":{"id":142343749,"uuid":"100952334","full_name":"rouault/cog_validator","owner":"rouault","description":"Cloud Optimized GeoTIFF validation service","archived":false,"fork":false,"pushed_at":"2023-11-08T23:09:32.000Z","size":27,"stargazers_count":62,"open_issues_count":1,"forks_count":23,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-18T20:11:37.771Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rouault.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-08-21T12:54:18.000Z","updated_at":"2025-06-10T13:10:59.000Z","dependencies_parsed_at":"2023-11-09T00:23:09.889Z","dependency_job_id":"e81c24bc-6c71-4cf6-ac21-c2d930e34ec5","html_url":"https://github.com/rouault/cog_validator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rouault/cog_validator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rouault%2Fcog_validator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rouault%2Fcog_validator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rouault%2Fcog_validator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rouault%2Fcog_validator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rouault","download_url":"https://codeload.github.com/rouault/cog_validator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rouault%2Fcog_validator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261237812,"owners_count":23128844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T03:01:06.732Z","updated_at":"2025-06-22T04:38:13.760Z","avatar_url":"https://github.com/rouault.png","language":"Python","funding_links":[],"categories":["`Python` processing of optical imagery (non deep learning)","Python"],"sub_categories":["Cloud Native Geospatial"],"readme":"# Cloud Optimized GeoTIFF validator\n\nThis is a standalone (Python / Flask) service that allows users to submit\nGeoTIFF files (preferably by URL) and check their compliance with the\nCloud Optimized GeoTIFF (COG) specification:\nhttps://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF\n\nThis utility is also compatible of being deployed as a AWS Lambda function,\nthrough the AWS API Gateway.\n\n## API endpoint: /api/validate\n\nGET request, with the following query parameters :\n  * url (required): URL to the GeoTIFF file\n  * use_vsicurl=true/false (optional, defaults to true): if true, the file is read using the GDAL /vsicurl/ subsystem (using HTTP GET range requests). If false, the file is locally downloaded in its entirety before being validated (note: when the service run as a AWS Lambda function, only up to 500 MB can be downloaded)\n\nFor example: /api/validate?url=http://path/to/my.tif\n\nPOST request, with a form encoded with multipart/form-data\n  * file: file content as multipart attachment\n\nPOST request, with a form encoded with application/x-www-form-urlencoded\n  * url (exclusive with file): URL to the GeoTIFF file\n  * use_vsicurl=true/false (defaults to true). See above\n  * filename (optional, recommended): file name\n  * file_b64: file content as a Base64 encoded string\n\nThis later interface is mostly needed to overcome a current limitation of the AWS API Gateway interface that does not accept multipart/form-data\n\nFor all the above interfaces, the query will return a JSON document with the following keys:\n  * status (required): 'success' or 'failure'\n  * error (optional): error message. present when the request is invalid, or the file cannot be read\n  * validation_errors (optional): array of errors. Only present if the file is a GeoTIFF file but does not comply with the COG requirements\n  * gdal_info (optional): dictionary with the output of \"gdalinfo -json\". Only present if the file is a GeoTIFF file\n  * details (optional): dictionary with file offsets of IFDs and first data block of each IFD. Only present if the file is a GeoTIFF file\n\n## HTML endpoint: /html\n\nThe service expose a basic HTML page for users to submit their GeoTIFF files\nand display the result of the validation\n\n## AWS Lambda / API Gateway\n\nThe service can be deployed as a AWS Lamba function, accessible through the AWS API Gateway.\n\nRunning \"make\" will generate a cog_validator.zip that contains the Python code of this service, the Python dependencies as well as a GDAL 2.2 build. This requires Docker to be available, to generate the cog_validator_deps.zip (which contains the Python dependencies as well as a GDAL 2.2 build)\n\nAssuming you have a AWS account with initial setup, follow the following steps to deploy the service:\n\n- Role creation\n\n  * Go to the AWS IAM management console\n  * Click on \"Roles\"\n  * Click on \"Create new role\"\n  * Click on the Select button of \"AWS Lambda\"\n  * In the Filter enter \"AWSLambdaBasicExecutionRole\" and check the corresponding checkbox\n  * Click on \"Next Step\"\n  * Enter \"lambda_basic_execution\" as role name\n  * Click on \"Create role\"\n\n- Lambda function creation\n\n  * Go to the AWS Lambda management console\n  * \"Create function\"\n  * In \"Select Blueprint\" step, select \"Author from scratch\"\n  * Skip Add Trigger with \"Next\"\n  * Give a name to the function, for example \"cog_validator\"\n  * Select \"Python 2.7\" as Runtime\n  * Select \"Upload a .ZIP file\" as \"Code entry type\"\n  * In \"Function package\", click on Upload an select the generated cog_validator.zip\n  * Enter \"lambda_main.handle\" in \"Handler\"\n  * In \"Existing role\", select \"lambda_basic_execution\"\n  * Click on Next, and Creation function to proceed on file uploading and lambda function creation\n  * Edit the Configuration / Advanced settings, to increase the timeout to 5 minutes and the memory to 512 MB, and Save\n  * To test everything works, in Actions dropdown list, choose \"Configure test event\" and enter the following payload.\n```\n        {\n            \"headers\": { \"Host\": \"foo\" },\n            \"httpMethod\": \"GET\",\n            \"queryStringParameters\": { \"url\": \"http://svn.osgeo.org/gdal/trunk/autotest/gcore/data/byte.tif\" },\n            \"path\": \"/api/validate\"\n        }\n```\n\n- API Gateway deployment\n\n  * Go to the AWS API Gateway management console\n  * In APIs tab, click on \"Create API\"\n  * Enter \"cog_validator\" as API name\n  * Click on \"Create API\"\n  * In Resources tab, in Actions dropdown list, select \"Create Resource\"\n  * Check the \"Configure as Proxy resource\" checkbox and click on \"Create Resource\"\n  * In the \"/{proxy+} - ANY - Setup\" form that is now displayed, keep the \"Lambda Function Proxy\" integration type\n  * Select the appropriate Lambda region (the one in which you created the Lambda function in the above steps)\n  * In \"Lambda Function\" entry, type \"cog_validator\" \n  * Click on \"Save\" and confirm that you add permission to the API Gateway to invoke your Lambda function\n  * To test everything works, click on the TEST icon\n    * A new form is displayed. Select GET as method\n    * In Path entry, enter \"/api/validate\"\n    * In \"Query strings\" entry, enter \"url=http://svn.osgeo.org/gdal/trunk/autotest/gcore/data/byte.tif\"\n    * In \"Headers\" entry, enter \"Host: foo\"\n    * Click on Test. A JSON document should be displayed (with validation errors)\n  * In Resources tab, in Actions dropdown list, select \"Deploy API\"\n  * In Deployment stage, select \"New stage\"\n  * Enter \"prod\" as stage name\n  * Click on Deploy\n  * A new form is displayed with an invoke URL like https://some_value_here.execute-api.eu-central-1.amazonaws.com/prod\n  * Copy-paste it in your browser and add \"/html\" at the end. A HTML page \"Cloud optimized GeoTIFF validator\" should now be displayed !\n\n## Development\n\nGDAL 2.2 with its Python (2.7) bindings must be installed, as well as the Python\nflask and requests modules.\n\nA basic self test is available with the ./test.sh script\n\n## Credits\n\nThe following resources have served as inspiration for AWS Lamba and API Gateway deployment\n  * https://medium.com/@mojodna/slimming-down-lambda-deployment-zips-b3f6083a1dff\n  * https://github.com/mojodna/marblecutter-tools\n  * http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frouault%2Fcog_validator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frouault%2Fcog_validator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frouault%2Fcog_validator/lists"}