Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/samapriya/Planet-GEE-Pipeline-CLI
Planet and Google Earth Engine Pipeline Command Line Interface Tool
https://github.com/samapriya/Planet-GEE-Pipeline-CLI
cli earth-engine planet planet-api
Last synced: 2 months ago
JSON representation
Planet and Google Earth Engine Pipeline Command Line Interface Tool
- Host: GitHub
- URL: https://github.com/samapriya/Planet-GEE-Pipeline-CLI
- Owner: samapriya
- License: apache-2.0
- Created: 2017-05-05T21:20:29.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2023-05-12T20:56:00.000Z (over 1 year ago)
- Last Synced: 2024-07-08T11:26:07.161Z (3 months ago)
- Topics: cli, earth-engine, planet, planet-api
- Language: Python
- Homepage: https://pypi.org/project/ppipe/
- Size: 7.83 MB
- Stars: 42
- Watchers: 6
- Forks: 12
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Planet GEE Pipeline CLI
[](https://badge.fury.io/py/ppipe)
[](https://doi.org/10.5281/zenodo.3479525)
[](https://www.planet.com/markets/education-and-research/)
[](https://saythanks.io/to/samapriya)
This tool is designed to facilitate moving data from Planet's API into Google Earth Engine and using a metadata library. The tool downloads data into a local storage and allows you to process the metadata before uploading into Google Earth Engine. This tool has been updated to account for metadata property type by going through each metadata column and then uses the **upload manifest** function to upload images for ingestion to EE. This tool also draws from an additional tool I created with is the [Google Earth Engine Asset Manager Addon](https://github.com/samapriya/gee_asset_manager_addon) This includes the batch upload feature, but now additional tools such as generating reports of Earth Engine assets and querying quota to name just a few. The ambition is apart from helping users with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as the need arises. I have now released this as a [PyPI package](https://pypi.org/project/ppipe/) for easy installation and this will be updated along with the the GitHub package.
Though this tool is designed to download planet imagery, use [porder](https://github.com/samapriya/porder) to use the new ordersv2 API to download planet imagery. This includes additional filter capabilities along with capability to clip and download.
**Please note: This tool is in no way an official tool or Planet offering, but is a personal project created and maintained by Samapriya Roy**
If you use this tool to download data for your research, and find this tool useful, star and cite it as below
```
Samapriya Roy. (2019, October 10). samapriya/Planet-GEE-Pipeline-CLI: Planet-GEE-Pipeline-CLI (Version 0.4.8). Zenodo.
http://doi.org/10.5281/zenodo.3479525
```
## Table of contents
* [Installation](#installation)
* [Getting started](#getting-started)
* [Usage examples](#usage-examples)
* [Planet Tools](#planet-tools)
* [Planet Key](#planet-key)
* [Planet Quota](#planet-quota)
* [Download Async](#download-async)
* [Download Saved Searches](#download-saved-searches)
* [Metadata Parser](#metadata-parser)
* [Earth Engine Tools](#earth-engine-tools)
* [selenium update](#selenium-update)
* [gee selsetup](#gee-selsetup)
* [EE User](#ee-user)
* [EE Quota](#ee-quota)
* [Create](#create)
* [Batch uploader](#batch-uploader)
* [Asset List](#asset-list)
* [Asset Size](#asset-size)
* [Task Query](#task-query)
* [Assets Access](#assets-access)
* [Cancel all tasks](#cancel-all-tasks)
* [Credits](#credits)
## Installation
This assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying
```python``` and then ```pip list```
If you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have released this as a python 2.7 but can be easily modified for python 3.
**Make sure you install OpenSSL from [this page](https://www.openssl.org/)**
**This toolbox also uses some functionality from GDAL**
For installing GDAL in Ubuntu
```
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install gdal-bin
sudo apt-get install python-gdal
```
For Windows I found this [guide](https://sandbox.idre.ucla.edu/sandbox/tutorials/installing-gdal-for-windows) from UCLA
To install **Planet-GEE-Pipeline-CLI:**
You can install using two methods
```pip install ppipe```
or you can also try
```
git clone https://github.com/samapriya/Planet-GEE-Pipeline-CLI.git
cd Planet-GEE-Pipeline-CLI
python setup.py install
```
For linux use sudo. This release also contains a windows installer which bypasses the need for you to have admin permission, it does however require you to have python in the system path meaning when you open up command prompt you should be able to type python and start it within the command prompt window. Post installation using the installer you can just call ppipe using the command prompt similar to calling python. Give it a go post installation type
```
ppipe -h
```
Installation is an optional step; the application can be also run directly by executing ppipe.py script. The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment. If you don't want to install, browse into the ppipe folder and try ```python ppipe.py``` to get to the same result.
## Getting started
As usual, to print help:
```
> ppipe -h
usage: ppipe [-h]
{
,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,ca
...
Planet Pipeline with Google Earth Engine Batch Addons
positional arguments:
{ ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,cancel}
---------------------------------------
-----Choose from Planet Tools Below-----
---------------------------------------
planetkey Enter your planet API Key
pquota Prints your Planet Quota Details
dasync Uses the Planet Client Async Downloader to download
Planet Assets: Does not require activation
savedsearch Tool to download saved searches from Planet Explorer
metadata Tool to tabulate and convert all metadata files from
Planet or Digital Globe Assets
-------------------------------------------
----Choose from Earth Engine Tools Below----
-------------------------------------------
update Updates Selenium drivers for firefox
selsetup Non headless setup for new google account, use if
upload throws errors
ee_user Get Earth Engine API Key & Paste it back to Command
line/shell to change user
quota Print Earth Engine total quota and used quota
create Allows the user to create an asset collection or
folder in Google Earth Engine
selupload Batch Asset Uploader for Planet Items & Assets using
Selenium
lst List assets in a folder/collection or write as text
file
assetsize Prints collection size in Human Readable form & Number
of assets
tasks Queries current task status
[completed,running,ready,failed,cancelled]
access Sets Permissions for items in folder
cancel Cancel all running tasks
optional arguments:
-h, --help show this help message and exit
```
To obtain help for a specific functionality, simply call it with _help_ switch, e.g.: `ppipe upload -h`. If you didn't install ppipe, then you can run it just by going to _ppipe_ directory and running `python ppipe.py [arguments go here]`
## Usage examples
Usage examples have been segmented into two parts focusing on both planet tools as well as earth engine tools, earth engine tools include additional developments in CLI which allows you to recursively interact with their python API
## Planet Tools
The Planet Toolsets consists of tools required to access control and download planet labs assets (PlanetScope and RapidEye OrthoTiles) as well as parse metadata in a tabular form which maybe required by other applications.
### Planet Key
This tool basically asks you to input your Planet API Key using a password prompt this is then used for all subsequent tools. This tool now includes an option for a quiet authentication using the API key incase it is unable to invoke an interactive environment such as in Google colaboratory. You can also perform ```planet init``` to set the key as the tool can read from it directly.
```
usage: ppipe planetkey [-h] [--type TYPE] [--key KEY]
optional arguments:
-h, --help show this help message and exit
Optional named arguments:
--type TYPE For direct key entry type --type quiet
--key KEY Your Planet API Key
```
If using on a private machine the Key is saved as a csv file for all future runs of the tool.
### Planet Quota
This tool prints your Planet quota including allocation name, Total quota, quota used and quota remaining. Calling it is simple
```
ppipe pquota
```
### Download Async
This tool is built as a wrapper around the Planet Client's own download tool. The tool included in the planet client is multithreaded and allows the user to activate, poll and download at the same time. This will allow you to pass geometry in terms of a geojson files or all filter like start and end date and time using the .json files you created using the ```aoijson``` tool earlier.
```
usage: ppipe dasync [-h] [--infile INFILE] [--item ITEM] [--asset ASSET]
[--local LOCAL] [--start START] [--end END]
[--cmin CMIN] [--cmax CMAX]
optional arguments:
-h, --help show this help message and exit
--infile INFILE Choose a geojson from geojson.io or the aoi-json you
created earlier using ppipe aoijson
--item ITEM Choose from Planet Item types Example: PSScene4Band,
PSOrthoTile, REOrthoTile etc
--asset ASSET Choose an asset type example: anlaytic,
analytic_dn,analytic_sr,analytic_xml etc
--local LOCAL Local Path where Planet Item and asset types are saved
Optional named arguments:
--start START Start date filter format YYYY-MM-DD
--end END End date filter format YYYY-MM-DD
--cmin CMIN Cloud cover minimum between 0-1
--cmax CMAX Cloud cover maximum between 0-1
```
A setup using geojson needs to include other filters too and a typical setup would be
```
ppipe dasync --infile "C:\Users\johndoe\geometry.geojson" --item "PSScene4Band" --asset "analytic" --local "C:\planet" --start "2018-06-01" --end "2018-08-01" --cmin 0 --cmax 0.4
```
Using a stuctured json file that you might have created earlier means you don't have to pass additional filters everytime
```
python ppipe.py dasync --infile "C:\Users\johndoe\geometry.json" --item "PSScene4Band" --asset "analytic_xml" --local "C:\planet_demo"
```
However, you can still decide to pass the filters and the filters you pass will overwrite existing filters
```
python ppipe.py dasync --infile "C:\Users\johndoe\geometry.json" --item "PSScene4Band" --asset "analytic_xml" --local "C:\planet_demo" --start "2018-06-01" --end "2018-08-01" --cmin "0" --cmax 0.4
```
### Download Saved Searches
Download assets from saved searches which are saved in your planet explorer.
```
usage: ppipe savedsearch [-h] [--name NAME] [--asset ASSET] [--local LOCAL]
[--limit LIMIT]
optional arguments:
-h, --help show this help message and exit
--name NAME Name of your saved search(It is case sensitive)
--asset ASSET Choose asset type analytic, analytic_xml, analytic_sr,
analytic_dn etc
--local LOCAL Local Path (full path address) where PlanetAssets are saved
Optional named arguments:
--limit LIMIT Choose number of assets you want to download
```
### Metadata Parser
The metadata tab is a more powerful tool and consists of metadata parsing for All PlanetScope and RapiEye Assets along with Digital Globe MultiSpectral and DigitalGlobe PanChromatic datasets. This was developed as a standalone to process xml metadata files from multiple sources and is important step is the user plans to upload these assets to Google Earth Engine.
```
usage: ppipe metadata [-h] [--asset ASSET] [--mf MF] [--mfile MFILE]
[--errorlog ERRORLOG] [--dir DIR]
optional arguments:
-h, --help show this help message and exit
--asset ASSET Choose PS OrthoTile(PSO)|PS OrthoTile DN(PSO_DN)|PS
OrthoTile Visual(PSO_V)|PS4Band Analytic(PS4B)|PS4Band
DN(PS4B_DN)|PS4Band SR(PS4B_SR)|PS3Band
Analytic(PS3B)|PS3Band DN(PS3B_DN)|PS3Band
Visual(PS3B_V)|RE OrthoTile (REO)|RE OrthoTile
Visual(REO_V)|DigitalGlobe
MultiSpectral(DGMS)|DigitalGlobe
Panchromatic(DGP)|PolarGeospatial CenterDEM
Strip(PGCDEM)?
--mf MF Metadata folder?
--mfile MFILE Metadata filename to be exported along with Path.csv
--errorlog ERRORLOG Errorlog to be exported along with Path.csv
Optional named arguments:
--dir DIR Path to Image Directory to be used to get ImageTags
with metadata. use only with PS4B_SR
```
## Earth Engine Tools
The ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises. This is also a seperate package for earth engine users to use and can be downloaded [here](https://github.com/samapriya/gee_asset_manager_addon)
### selenium update
**This is a key step since all upload function depends on this step, so make sure you run this**. This downloads selenium driver and places to your local directory for windows and Linux subsystems. This is the first step to use selenium supported upload.
``` ppipe update```
### gee selsetup
Once in a while the geckodriver requires manual input before signing into the google earth engine, this tool will allow you to interact with the initialization of Google Earth Engine code editor window. It allows the user to specify the account they want to use, and should only be needed once.
```geeup selsetup```
### EE User
This tool is designed to allow different users to change earth engine authentication credentials. The tool invokes the authentication call and copies the authentication key verification website to the clipboard which can then be pasted onto a browser and the generated key can be pasted back. This command takes in no arguments.
```ppipe ee_user```
### EE Quota
This tool prints earthengine quota tools, used and remaining quota. No argument is required
```ppipe quota```
### Create
This tool allows you to create a collection or folder in your earth engine root directory. The tool uses the system cli to achieve this and this has been included so as to reduce the need to switch between multiple tools and CLI.
```
usage: ppipe.py create [-h] --typ TYP --path PATH
optional arguments:
-h, --help show this help message and exit
--typ TYP Specify type: collection or folder
--path PATH This is the path for the earth engine asset to be created full
path is needsed eg: users/johndoe/collection
```
### Batch uploader
The script creates an Image Collection from GeoTIFFs in your local directory. By default, the image name in the collection is the same as the local directory name; with the optional parameter you can provide a different name.You have to process the metadata for images, which is covered in the next section along with a manifest type for Planet image and asset:
[Metadata parser](#metadata-parser).
```
usage: ppipe selupload [-h] --source SOURCE --dest DEST [-m METADATA]
[--large] [--nodata NODATA] [--bands BANDS]
[-u USER] [-b BUCKET]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--source SOURCE Path to the directory with images for upload.
--dest DEST Destination. Full path for upload to Google Earth
Engine, e.g. users/pinkiepie/myponycollection
-u USER, --user USER Google account name (gmail address).
Optional named arguments:
-m METADATA, --metadata METADATA
Path to CSV with metadata.
--large (Advanced) Use multipart upload. Might help if upload
of large files is failing on some systems. Might cause
other issues.
--nodata NODATA The value to burn into the raster as NoData (missing
data)
--bands BANDS Comma-separated list of names to use for the image
bands. Spacesor other special characters are not
allowed.
-b BUCKET, --bucket BUCKET
Google Cloud Storage bucket name.
```
### Asset List
This tool is designed to either print or output asset lists within folders or collections using earthengine ls tool functions.
```
usage: ppipe lst [-h] --location LOCATION --typ TYP [--items ITEMS]
[--output OUTPUT]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--location LOCATION This it the location of your folder/collection
--typ TYP Whether you want the list to be printed or output as
text[print/report]
Optional named arguments:
--items ITEMS Number of items to list
--output OUTPUT Folder location for report to be exported
```
### Asset Size
This tool allows you to query the size of any Earth Engine asset[Images, Image Collections, Tables and Folders] and prints out the number of assets and total asset size in non-byte encoding meaning KB, MB, GB, TB depending on size.
```
usage: ppipe assetsize [-h] --asset ASSET
optional arguments:
-h, --help show this help message and exit
--asset ASSET Earth Engine Asset for which to get size properties
```
### Task Query
This script counts all currently running,ready,completed,failed and cancelled tasks along with failed tasks. This is linked to the account you initialized with your google earth engine account. This takes no argument.
```
usage: ppipe.py tasks [-h]
optional arguments:
-h, --help show this help message and exit
ppipe.py tasks
```
### Assets Access
This tool allows you to set asset acess for either folder , collection or image recursively meaning you can add collection access properties for multiple assets at the same time.
```
usage: ppipe.py access [-h] --asset ASSET --user USER --role ROLE
optional arguments:
-h, --help show this help message and exit
--asset ASSET This is the path to the earth engine asset whose permission
you are changing folder/collection/image
--user USER Full email address of the user, try using "AllUsers" to make
it public
--role ROLE Choose between reader, writer or delete
```
### Cancel all tasks
This is a simpler tool, can be called directly from the earthengine cli as well
```
earthengine cli command
earthengine task cancel all
```
Usage using this tool
```
usage: ppipe cancel [-h]
optional arguments:
-h, --help show this help message and exit
```
### Credits
[JetStream](https://jetstream-cloud.org/) A portion of the work is suported by JetStream Grant TG-GEO160014.
Also supported by [Planet Labs Ambassador Program](https://www.planet.com/markets/ambassador-signup/)
Original upload function adapted from [Lukasz's asset manager tool](https://github.com/tracek/gee_asset_manager)
# Changelog
### v0.4.9
- fixed issue with selsetup
### v0.4.8
- fixed raw_input vs input for py2 and py3 cases
- updated selenium setup path for windows vs other operating systems
### v0.4.7
- Download selenium drivers for all operating systems
- updated asset size, acl changer and other tools
- fixed metadata tool to work with python3
- overall general improvements to the tool and updates
- added selenium setup tools
### v0.4.6
- added fix for downloading selenium drivers
- improved error handling
### v0.4.4
- Fixed selenium download path
- Streamlined overall functions of the tool
- Handles metadata for clipped assets (use [porder](https://github.com/samapriya/porder) to clip)
- Updated selenium uploader to function with earthengine-api
- Overall improvements and stability check
### v0.4.3
- Added selenium uploader to upload images to Earth Engine after auth issues
- Overall improvements to the tools and added notifications and contribution notices
### v0.4.1
- Major improvements to earth engine tools including better task reporting, batch copy and move
- Improvement to the access tool which allows you to add read/write permissions for entire EE folder and collections for specific users
### v0.3.8
- Now include a tool to print your planet quota details ```pquota```
- Tool includes ```dasync``` which uses the [Planet's Python Client Downloader](https://github.com/planetlabs/planet-client-python) to activate, download using multithreading
### v0.3.7
- Now allows users to download from saved searches in planet explorer
### v0.3.6
- Now handles complex geometry by using a bounding box for GeoJSON files
### v0.3.5
- Fixed issues with space function
- Improved file size parsing
### v0.3.2
- Generalized CLI arguments
- Fixed subprocess shell issue for upload
### v0.3.0
- Allows for quiet authentication for use in Google Colab or non interactive environments
- Improved planet key entry and authentication protocols
### v0.2.91
- Fixed issue with Surface Reflectance metadata and manifest lib
- Improved ingestion support for (PSScene4Band analytic_Sr)[PS4B_SR]
### v0.2.9
- Fixed issues with generating id list
- Improved overall security of command calls
### v0.2.2
- Major improvements to ingestion using manifest ingest in Google Earth Engine
- Contains manifest for all commonly used Planet Data item and asset combinations
- Added additional tool to Earth Engine Enhancement including quota check before upload to GEE
### v0.2.1
- Fixed initialization loop issue
### v0.2.0
- Metadata parser and Uploader Can now handle PlanetScope 4 Band Surface Reflectance Datasets
- General Improvements
### v0.1.9
- Changes made to reflect updated GEE Addon tools
- general improvements
### v0.1.8
- Minor fixes to parser and general improvements
- Planet Key is now stored in a configuration folder which is safer "C:\users\.config\planet"
- Earth Engine now requires you to assign a field type for metadata meaning an alphanumeric column like satID cannot also have numeric values unless specified explicitly . Manifest option has been added to handle this (just use -mf "planetscope")
- Added capability to query download size and local disk capacity before downloading planet assets.
- Added the list function to generate list of collections or folders including reports
- Added the collection size tool which allows you to estimate total size or quota used from your allocated quota.
- ogr2ft feature is removed since Earth Engine now allows vector and table uploading.