{"id":20630646,"url":"https://github.com/brookisme/eeuploader","last_synced_at":"2025-06-14T04:06:17.907Z","repository":{"id":151273994,"uuid":"260595209","full_name":"brookisme/eeuploader","owner":"brookisme","description":"a CLI and python-module for uploading (lots of) images to Google Earth Engine","archived":false,"fork":false,"pushed_at":"2020-05-05T22:33:24.000Z","size":124,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-14T04:06:05.136Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brookisme.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-02T01:55:37.000Z","updated_at":"2024-06-04T23:21:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"65ed9417-dff2-4004-ac8f-eece1d5fcd94","html_url":"https://github.com/brookisme/eeuploader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/brookisme/eeuploader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brookisme%2Feeuploader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brookisme%2Feeuploader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brookisme%2Feeuploader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brookisme%2Feeuploader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brookisme","download_url":"https://codeload.github.com/brookisme/eeuploader/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brookisme%2Feeuploader/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259756872,"owners_count":22906678,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T14:09:07.221Z","updated_at":"2025-06-14T04:06:17.847Z","avatar_url":"https://github.com/brookisme.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### EE Uploader\n\n_a CLI and python-module for uploading (lots of) images to Google Earth Engine_\n\n---\n\n1. [Install](#install)\n2. [Quick Start](#quickstart)\n3. [Project Setup](#setup)\n4. [EEImagesUp Docs](#pydocs)\n5. [Requirements](#requirments)\n\n---\n\n\u003ca name=\"install\"/\u003e\n\n### INSTALL\n\nNote: During these early days this must be installed locally but it will be pushed to PIP soon. Similarly, one of the requirements, [mproc](https://github.com/brookisme/mproc) should also be installed locally.\n\n```bash\ngit clone https://github.com/wri/dl_exporter.git\npushd dl_exporter\npip install -e .\npopd\n```\n\n---\n\n\u003ca name=\"quickstart\"/\u003e\n\n### QUICK START EXAMPLES\n\nNote: these examples use a feature collection file ([fc.geojson](#fcgeojson)) and args file ([upargs.yaml](#upargsyaml)) that are described in detail [below](#setup).\n\n##### CLI\n\n```bash\n# print info before run output includes:\n# - the number-of-features\n# - an example upload manifest (defaults to the first feature)\neeuploader info fc.geojson upargs.yaml\n# or using kwargs instead of arg-config file\neeuploader info fc.geojson user=brookwilliams collection=IM_COLLECTION_NAME\n# - save all upload manifests to a pickle file\neeuploader info fc.geojson upargs.yaml --dest manifests.p --all true\n# - save upload manifest for indices 1,10,100 to a json file\neeuploader info fc.geojson upargs.yaml --dest manifests.json --save_as json --indices 1,10,100\n\n\n# upload images to a collection (as above kwargs can be used instead of an arg-config file)\n# - all images\neeuploader upload fc.geojson upargs.yaml\n# - the first 10 image features\neeuploader upload fc.geojson upargs.yaml --limit 10\n# - image features from 10 to 20\neeuploader upload fc.geojson upargs.yaml --index_range 10,20\n# - image features 3,400,24\neeuploader upload fc.geojson upargs.yaml --index_range 3,400,24\n# - image features 3,400,24 with overwrite=True\neeuploader upload fc.geojson upargs.yaml --index_range 3,400,24 force=True\n```\n\n##### PYTHON\n\n```python\nimport eeuploader.image as eup\n\nup=eup.EEImagesUp(\n    USER,\n    features='fc.geojson',\n    collection=IC,\n    start_time_key='date',\n    no_data=0,\n    force=False)\n\n# print manifest for first feature\npprint(up.manifest(1))\n\n# upload first feature\nup.upload(1)\n\"\"\" ouput (returns without waiting for task to finish)\n{'id': 'XER4J2RLIOWJRYNRMC7EALXH',\n 'name': 'projects/earthengine-legacy/operations/XER4J2RLIOWJRYNRMC7EALXH',\n 'started': 'OK'}\n\"\"\"\n\n# upload_collection\nup.upload_collection()\nprint(up.tasks)\n\"\"\" output (waits for all tasks to complete to return)\n[{'creation_timestamp_ms': 1588615171535,\n  'description': 'Ingest image: '\n                 '\"projects/earthengine-legacy/assets/users/...\"',\n  'destination_uris': ['https://...'],\n  'id': 'VRJ3JFOPHL5ZFKVXTOKB4JEA',\n  'name': 'projects/earthengine-legacy/operations/VRJ3JFOPHL5ZFKVXTOKB4JEA',\n  'start_timestamp_ms': 1588615180457,\n  'state': 'COMPLETED',\n  'task_type': 'INGEST_IMAGE',\n  'update_timestamp_ms': 1588615262051},...]\n\"\"\"\n\n# upload some random thing\nup.upload(\n    uri='gs://bucket/path/to/image.tif',\n    crs='epsg:32717',\n    properties={\n        'property_1': 123\n        'property_2': '2018-01-01',\n        'propery_3': 'important piece of information'\n    },\n    start_time='2019-04-02' )\n```\n\n---\n\n\u003ca name=\"setup\"/\u003e\n\n### PROJECT SETUP\n\n1. features_collection file*: JSON containing a `features`-list, where each feature must has a `properties`-dict.\n2. (optional) args file: a yaml file containing default arguments for `EEImagesUp.__init__`\n\n* NOTE: Technically you can use `EEImagesUp` without the features_collection file, either for single uploads or by passing a feature-collection-python-dict instead of a file path.\n\n\u003ca name=\"fcgeojson\"/\u003e\n\n##### FEATURE COLLECTION EXAMPLE:\n\n```json\n# fc.geojson\n{\n  \"type\": \"FeatureCollection\",\n  \"features\": [\n    {\n      \"type\": \"Feature\",\n      \"id\": \"0\",\n      \"geometry\": {\n        \"type\": \"Polygon\",\n        \"coordinates\": [[[-63.374182,-3.93245],[-63.328243,-3.932469],[-63.328225,-3.886331],[-63.374161,-3.886312],[-63.374182,-3.93245]]]\n      },\n      \"properties\": {\n        \"gcs\": \"v1/data/dev/WH/1/dw_-66.3899849022_-2.5792093972-20190820.tif\",\n        \"crs\": \"epsg:32720\",\n        \"date\": \"2019-08-12\",\n        \"biome\": 1,\n        \"biome_name\": \"Tropical \u0026 Subtropical Moist Broadleaf Forests\",\n        \"NBPixels\": 250596,\n        \"BareGround\": 0.0004948203482896778,\n        \"BuiltArea\": 0.001017574103337643,\n        \"Clouds\": 0,\n        \"Crops\": 0,\n        \"FloodedVegetation\": 0.00739836230426663,\n        \"Grass\": 0.00702724704304937,\n        \"Scrub\": 0,\n        \"Snow Ice\": 0,\n        \"Trees\": 0.4854546760522913,\n        \"Water\": 0.4986073201487654,\n        \"S2_DATASTRIP_ID\": \"S2B_OPER_MSI_L2A_DS_SGS__20190812T165657_S20190812T143758_N02.13\",\n        \"S2_GEE_ID\": \"20190812T143759_20190812T143758_T20MMA\",\n        \"S2_GRID\": \"20MMA\",\n        \"S2_LEVEL\": \"2A\",\n        \"S2_METHOD\": \"firstNonNull\",\n        \"S2_PRODUCT_ID\": \"S2B_MSIL2A_20190812T143759_N0213_R096_T20MMA_20190812T165657\",\n        \"S2_SENSING_ORBIT_DIRECTION\": \"DESCENDING\",\n        \"dw_id\": \"dw_-63.3512901800_-3.9093045532-20190812\",\n        \"eco_region\": \"Purus v�rzea\",\n        \"flipped\": false,\n        \"folder\": \"v1/data/dev/WorkForce/WH/1\",\n        \"lat\": -3.9093045532,\n        \"lon\": -63.35129018,\n        \"map\": \"https://www.google.com/maps/@-3.9093046188354488,-63.35129165649414,14z/data=!3m1!1e3\",\n        \"timestamp\": 1565568000000,\n      }\n    },\n    ...\n  ]\n}\n```\n\n\u003ca name=\"upargsyaml\"/\u003e\n\n##### UPLOAD ARGS EXAMPLE:\n\n```yaml\n# upargs.yaml\nuser: projects/wri-datalab\ncollection: image_collection_name\nbands: null\nband_names: \n    - lulc\npyramiding_policy: mode\nno_data: 0\nexclude: \n    - flipped\n    - map\nstart_time_key: date\nend_time_key: null\ndays_delta: 1\ncrs_key: crs\nuri_key: gcs\nname_key: ee_name\nforce: false\nnoisy: false \nraise_error: false\n```\n \n---\n\n\u003ca name=\"pydocs\"/\u003e\n\n### EEImagesUp DOCS\n\nMETHODS:\n\n1. [Initializer](#up-init)\n2. [manifest](#up-manifest)\n3. [upload](#up-upload)\n4. [upload_collection](#up-upload_collection)\n\n\u003ca name=\"up-init\"/\u003e\n\n##### EEImagesUp.\\_\\_init\\_\\_\n\n```python\n\"\"\"\nArgs:\n\n    user\u003cstr\u003e:\n        gee user or project root\n        * if it begins with \"users\" or \"projects\" the string is unaltered\n        * otherwise it is pre-pended with \"users\"\n    features\u003cdict|list|str|None\u003e:\n        features list or file path to (geo)json feature collection\n        * if dict or loaded from file path the features list is assumed to \n          be under the the key \"features\"\n        * if None feat(s) or feat properties must be passed directly to the\n          public methods.\n        * otherwise feature indices can be used for manifest/upload/upload_collection\n    collection\u003cstr|False\u003e:\n        name of image_collection/folder to upload the images.\n\n        note: since the main purpose of this script is to upload many features\n              it attempts to force you to use specify a collection. if you want\n              to upload to your user/project folder root pass \"False\" \n    bands\u003clist[dict]|None\u003e:\n        ** alternatively specify `band_names` (see below) **\n        a manifest band list as specified here https://developers.google.com/earth-engine/image_manifest#bands\n        \n        note: every image being uploaded must have the same band structure\n    band_names\u003clist[dict]|None\u003e:\n        ** ignored if `bands` is not specified **\n        generates a manifest band list as specified here https://developers.google.com/earth-engine/image_manifest#bands\n        from a list of band_names\n\n        note: every image being uploaded must have the same band structure\n    pyramiding_policy\u003cstr\u003e:\n        one of MEAN, MODE, SAMPLE (upper or lower case is fine). default=MEAN\n\n        note: for band-level control must use `bands` not `band_names` above\n    no_data\u003cint|list|dict\u003e:\n        no_data value(s) or \"missing_data\" object described here https://developers.google.com/earth-engine/image_manifest#bands\n    include\u003clist|None\u003e:\n        feature-property-keys to include as ee.image-properties\n        * if None all the feature-property-keys will be included unless `exclude` list is provided\n    exclude\u003clist|None\u003e:\n        ** ignored if `include` is provided **\n        feature-property-keys to exclude as ee.image-properties\n    start_time_key,end_time_key,crs_key,uri_key,name_key\u003cstr|None\u003e:\n        if start_time/end_time/crs/... not provided at run time the system will\n        attempt to find them in the feature-properties using these keys\n    days_delta\u003cint|False\u003e:\n        if not False, and start_time is provided (or found with start_time_key), and end_time is \n        not provided or found, end_time will be created by adding `days_delta` number of days\n        to the start_time.\n    timeout\u003cint\u003e:\n        how quickly to timeout if `wait` is set to true. defaults to TIMEOUT above.\n    force\u003cbool\u003e:\n        set to true to overwrite existing assets\n    noisy\u003cbool\u003e:\n        print progress during `upload_collection`\n    raise_error\u003cbool\u003e:\n        raise_errors during `upload_collection`\n\nUsage:\n\n    import eeuploader.image as eup\n    import eeuploader.gee_utils as gutils\n\n    up=eup.EEImagesUp(\n        'projects/wri-datalab',\n        features='dw_organized_features.geojson',\n        collection='image_collection_name',\n        start_time_key='date',\n        no_data=0,\n        force=True)\n\n    # print nb-features and manifest for first feature    \n    print('NB FEATURES:',len(up.features))\n    pprint(up.manifest(0))\n\n\n    # upload the first feature / print task status\n    # note: `upload` does not wait for task to complete.\n    #       set `wait=True` to wait for task to complete \n    print(up.upload(0))\n    gutils.task_info(up.task_id)\n\n\n    # upload the first 3 features / print task final task status for each\n    up.upload_collection(limit=3)\n    print(up.tasks)\n\n\"\"\"\n```\n\n\u003ca name=\"up-manifest\"/\u003e\n\n##### EEImagesUp.manifest\n\n```python\n\"\"\" manifest for single upload or manifests list\n\nArgs:\n\n    feat\u003cdict\u003e: \n        a feature dictionary containing a properties dictionary\n        from which it can pull the uri, crs, ee.image-properties, ...\n    uri\u003cstr|None\u003e:\n        google cloud storage uri (with or without the preceding \"gs://\")\n        or gcs url for image asset.\n    name\u003cstr|None\u003e:\n        name of the new ee.image.  if not provided it will create a name\n        from the uri. `.`s will be replaced with `d` due to ee-naming policies.\n    tileset_id\u003cstr|None\u003e:\n        if not provided one will be created from the name\n    crs\u003cstr|None\u003e:\n        crs of image (for example 'epsg:4326')\n    propertie\u003cdict\u003e:\n        updates any features existing in feat['properties']\n    start/end_time\u003cstr|datetime|None\u003e:\n        strings should be in YYYY-MM-DD format\n        \n        if start_time but end_time is None, and self.days_delta end_time\n        will be set start_time+(self.days_delta)days\n    features\u003clist\u003e:\n        list of features or feature-indices. if exists the returned manifest\n        will be a list of upload manifests\n    dest\u003cstr\u003e:\n        if dest: manifest will be saved and dest will be returned\n        otherwise: manifest will be returned\n    save_as:\n        * file-type: one of ['json', 'pickle']\n        * defaults to 'pickle'\n    indent\u003cint|None\u003e:\n        if saving to json: indent pretty printing arg.\n\nReturns\u003cstr|dict\u003e:\n    \n    * \u003cdict\u003e Manifest for a single upload\n    * \u003cstr\u003e Destination of saved file\n\"\"\"\n```\n\n\u003ca name=\"up-upload\"/\u003e\n\n##### EEImagesUp.upload\n\n```python\n\"\"\" single upload\n\nNote: if `wait=False` the upload will not wait for task to complete.\n\nArgs:\n\n    **feat/uri/.../start_time/end_time (see manifest doc-string)**\n\n    manifest\u003cdict\u003e:\n        upload manifest.  if provided ignores all other arguments an upload\n        using this manifest\n    wait\u003cbool\u003e:\n        wait for task to complete\n    noisy\u003cbool\u003e:\n        print progress during upload\n    raise_error\u003cbool\u003e:\n        raise_errors during upload\n\nSets:\n\n    self.task_id\u003cstr\u003e: task id\n    self.task\u003cdict\u003e: task status\n\nReturns:\n    \n    \u003cdict\u003e task status\n\"\"\"\n```\n\n\u003ca name=\"up-upload_collection\"/\u003e\n\n##### EEImagesUp.upload_collection\n\n```python\n\"\"\" upload set of features in batches\n\n* This method will always wait for tasks to complete before returning.\n* `nb_batches` should be understood as the max number of simultaneous \n  requests for ee-image-uploads \n\nArgs:\n\n    features\u003clist|None\u003e:\n        * list of features or feature indices in self.features to upload\n        * if not provided upload all the features in self.features\n    limit\u003cint|None\u003e:\n        * limit features to first `limit`-elements\n    nb_batches:\n        divide uploads into `nb_batches` groups and upload them simultaneously\n\nSets:\n\n    self.tasks\u003clist\u003e: list of task status\n\n\"\"\"\n```\n\n \n---\n\n\u003ca name=\"requirments\"/\u003e\n\n### REQUIRMENTS\n\n* https://github.com/brookisme/mproc\n* https://click.palletsprojects.com/en/7.x/\n* https://pyyaml.org/wiki/PyYAMLDocumentation\n* https://pypi.org/project/geojson/\n* https://pypi.org/project/Unidecode/\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrookisme%2Feeuploader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrookisme%2Feeuploader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrookisme%2Feeuploader/lists"}