{"id":24493266,"url":"https://github.com/sentinel-hub/byoc-tool","last_synced_at":"2025-04-14T01:41:14.373Z","repository":{"id":53542503,"uuid":"230931177","full_name":"sentinel-hub/byoc-tool","owner":"sentinel-hub","description":"Tool that prepares your data for use in Sentinel Hub","archived":false,"fork":false,"pushed_at":"2024-06-04T12:32:41.000Z","size":1190,"stargazers_count":7,"open_issues_count":2,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-27T15:52:17.046Z","etag":null,"topics":["sentinel-hub"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sentinel-hub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-30T14:39:39.000Z","updated_at":"2024-06-04T12:32:45.000Z","dependencies_parsed_at":"2023-02-08T19:30:28.903Z","dependency_job_id":null,"html_url":"https://github.com/sentinel-hub/byoc-tool","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Fbyoc-tool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Fbyoc-tool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Fbyoc-tool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Fbyoc-tool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sentinel-hub","download_url":"https://codeload.github.com/sentinel-hub/byoc-tool/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248809040,"owners_count":21164893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sentinel-hub"],"created_at":"2025-01-21T19:19:44.638Z","updated_at":"2025-04-14T01:41:13.892Z","avatar_url":"https://github.com/sentinel-hub.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tool for Sentinel Hub BYOC service\n\nThe Sentinel Hub BYOC Tool is software which can be used to prepare your data for use in Sentinel Hub in a few simple steps. It is simple to use for simple cases but is also highly configurable allowing for more complex requirements.\n\nThe tool converts your TIFF and JP2 files to supported Cloud Optimized GeoTIFFs, uploads them to you AWS S3 bucket and registers them in the Sentinel Hub BYOC service. When complete, your data becomes visible in Sentinel Hub. The same steps can be done manually and are detailed in our [documentation](https://docs.sentinel-hub.com/api/latest/#/API/byoc).\n\nThe tool can be run either in Docker or as Java JAR. Instructions on how to install the tool are [here](#installation-instructions).\n\nThe tool works only with AWS S3 buckets in EU-Central-1 and US-West-2 region.\n\n## Prerequisites\n\nBefore using the tool make sure you have the following:\n\n- A Sentinel Hub OAuth client -- for communicating with Sentinel Hub. If you don't have one, create one in your [dashboard](https://apps.sentinel-hub.com/dashboard/#/account/settings). Click [here](https://docs.sentinel-hub.com/api/latest/api/overview/authentication/#registering-oauth-client) for detailed instructions.\n\n- A BYOC collection -- the collection data will be added to. If you don't have one, create one using your [dashboard](https://apps.sentinel-hub.com/dashboard/#/collections) or use the [API](https://docs.sentinel-hub.com/api/latest/reference/?service=byoc#tag/byoc_collection).\n\n- An AWS bucket which is configured so that Sentinel Hub can access data from it -- how to do this is documented [here](https://docs.sentinel-hub.com/api/latest/api/byoc/#aws-bucket-settings). While this is not strictly speaking necessary for the BYOC tool to function, for the data to work on Sentinel Hub it needs read access to the data.\n\n- The AWS credentials with access to your bucket -- Get them from your AWS console. These are only used to upload to or read your data from your bucket (reading may happen during more advanced usage of the tool). These credentials are not shared with Sentinel Hub or any other service and never leave your computer.\n\n- Imagery (of course)\n\nDepending on the installation type, the following are also required:\n\nJava install:\n\n- Java 8 to 11 have been tested and are supported. Newer versions may not work due to gradle (in)compatibility.\n- GDAL (https://gdal.org/) installed, at least v2.3.0, but it is highly recommend that you install a newer version of GDAL (v3.1 or newer), as these versions contain important fixes. Additionally, the `GDAL` and `GDAL_DATA` system environment variables must be set. \n\nDocker install:\n\n- Docker installed on your computer\n\n## Installation instructions\n\nClone/download the contents of this repository. Open a command prompt and navigate to the byoc-tool folder which contains the downloaded repository. Depending on your chosen installation type, run one of the following:\n\n### Building a Docker image\n\nRun `docker build --tag byoctool .`\n\n### Building a Java JAR\n\nRun gradle task \"shadowJar\":\n- on Windows: `gradlew.bat shadowJar`\n- on Linux/Unix: `./gradlew shadowJar`\n\nThe jar will be located in the folder `build/libs`. To run commands, make sure that the command prompt is navigated to this folder, alternatively provide the absolute path for the -jar parameter value.\n\nProvide the Sentinel Hub OAuth client id and client secret in the environment variables `SH_CLIENT_ID` and `SH_CLIENT_SECRET`. How to set the environment variables depends on the operating system you have.\n\nThe AWS client credentials will be read from `~/.aws/credentials`, if present. If not, or you wish to override them, set the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the values required.\n\n## Basic Commands\n\nThe following part of the readme is targeted towards the Java JAR install.\n\nFor a list of commands run: `java -jar byoc-tool.jar --help`\n\nFor a list of ingestion parameters run: `java -jar byoc-tool.jar ingest --help`\n\nGive the tool the BYOC collection id `\u003cMyCollectionId\u003e` you wish to import to and the path to the folder containing imagery:\n\nThe basic import command (see the next chapter for details) is thus: `java -jar byoc-tool.jar ingest \u003cMyCollectionId\u003e \u003cMyFolder\u003e`\n\n## The Simple Default Case\n\nThe tool offers parameters which will allow tuning for various folder/file structures. The default case, which needs no additional parameters is as follows:\nBy default, the tool takes the input folder and looks for folders inside which have tiff or jp2 images. In this case, each such folder found represents a tile and each file represents a band. For example, if you have files at the following locations:\n\n- `\u003cMyFolder\u003e`/\n  - tile_1/\n    - B01.tif\n    - B02.tif\n  - tile_2/\n    - B01.tif\n    - B02.tif\n\nwith `folder/` as the input path, the tool would ingest 2 tiles with names `tile_1` and `tile_2`, and each tile would have two bands named `B01` and `B02`. By default, band names equal the file names without file extensions.\n\nThe command will prepare Cloud Optimized GeoTIFFs and upload them to the S3 bucket associated with the BYOC collection. Finally, it will register the tiles in your BYOC collection. The file `\u003cMyFolder\u003e/tile_1/B01.tiff` will be uploaded to`s3://\u003cMyBucket\u003e/tile_1/B01.tiff`.\n\nFor more elaborate folder, tile, band structures, see the help of the `--file-pattern` and `--file-map` parameters.\n\nNote that in this case neither the tile sensing time nor the tile coverage will be set (see [Tracing Coverage](#tracing-coverage)).\n\n## Advanced Example\n\nThe tool can be quite powerful with the right parameters. This example will attempt to showcase these without being too complicated.\n\nSuppose in this case that the folder structure is as follows:\n\n- folder/\n  - tile_1/\n    - DATA_and_sensing_time_1.tif\n  - tile_2/\n    - DATA_and_sensing_time_2.tif\n\nIn this case lets assume the DATA tiffs are three bands each, containing R,G,B bands.\n\nTo effectively use the tool in this case, the `--file-pattern` and `--file-map` parameters need to be used. The `--file pattern` in this case can look something like this: `(?\u003ctile\u003e.*)\\/.*(?\u003cyear\u003e[0-9]{4})(?\u003cmonth\u003e[0-9]{2})(?\u003cday\u003e[0-9]{2})T(?\u003chour\u003e[0-9]{2})(?\u003cminute\u003e[0-9]{2})(?\u003csecond\u003e[0-9]{02})`. This will find files with the defined sensing time structure and use the name of their parent folder as the tile name. This can be modified to support multiple files per folder or even files in different folders which together represent one tile.\n\nThe `--file-map` parameter allows all bands from the tiff file to be used. In this case since there is only one file per tile only one is needed and it can look something like this: `.*tif;1:R;2:G;3:B`. In words: From a .tif file extract band 1 and name it R, extract band 2 and name it G, extract band 3 and name it B.\n\nTo remember: `--file-pattern` finds files using a regular expression. Files with an equal `tile` capture group value are grouped into that one tile. The `--file-map` pattern is then applied to each file within that tile. You can define as many `--file-map` parameters as are files in a tile so that each file can be mapped.\n\n## Tracing Coverage\n\nInformation about what coverage tracing is and why it is important is available [here](https://docs.sentinel-hub.com/api/latest/api/byoc/#a-note-about-cover-geometries).\n\nTo enable geometry tracing add the flag `--trace-coverage`. See `--distance-tolerance` and `--negative-buffer` for tuning parameters. If not set, the cover geometry will equal the image bounding box.\n\nTo speed up tracing, you can trace coverage from one of image overviews. For example, to trace coverage from the first overview, set the flag `--trace-image-idx 1`.\n\n## S3 Multipart upload\n\nYou can enable multipart upload with the flag: `--multipart-upload`. This is recommended if your files are larger than 100MB or if you have an unstable internet connection.\n\nTo learn about it, check this page https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html, and if you decided to use it, it is highly recommended setting the bucket lifecycle policy for stopping incomplete multipart uploads https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-stop-incomplete-mpu-lifecycle-config. byoc-tool tries to stop incomplete uploads, if it has time to clean up, otherwise uploads remain active.  \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsentinel-hub%2Fbyoc-tool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsentinel-hub%2Fbyoc-tool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsentinel-hub%2Fbyoc-tool/lists"}