Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/remotesensinginfo/eodatadown
Automated download and processing of Earth Observation data to create a monitoring system.
https://github.com/remotesensinginfo/eodatadown
ard database earth earth-observation google-cloud monitoring monitoring-system observation remote remote-sensing sensing
Last synced: 2 months ago
JSON representation
Automated download and processing of Earth Observation data to create a monitoring system.
- Host: GitHub
- URL: https://github.com/remotesensinginfo/eodatadown
- Owner: remotesensinginfo
- License: apache-2.0
- Created: 2020-04-06T20:27:30.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2023-12-05T10:53:16.000Z (about 1 year ago)
- Last Synced: 2024-08-04T03:03:17.241Z (6 months ago)
- Topics: ard, database, earth, earth-observation, google-cloud, monitoring, monitoring-system, observation, remote, remote-sensing, sensing
- Language: Python
- Homepage: https://eodatadown.remotesensing.info
- Size: 11.8 MB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
- awesome-earthobservation-code - eodatadown - The Earth Observation Data Downloader (EODataDown) is a tool for automatically downloading and processing EO data to an analysis ready data product. This software forms a core component of a monitoring system based on EO data. (Remote Sensing.info)
README
# EODataDown
A tool for automatically downloading Earth Observation imagery and products.
## Dependencies
* SQLAlchemy [http://www.sqlalchemy.org]
* RSGISLib [http://www.rsgislib.org]
* ARCSI [http://arcsi.remotesensing.info]
* Requests [http://www.python-requests.org]
* Planet [https://github.com/planetlabs/planet-client-python]
* pycurl [http://pycurl.io]
* wget [https://www.gnu.org/software/wget/]
* postgresql [https://www.postgresql.org]
* psycopg2 [http://initd.org/psycopg/]
* Google Cloud Services [https://github.com/GoogleCloudPlatform/google-cloud-python]
* PyYAML [https://pyyaml.org]
### Google Cloud Services
For google cloud services the following dependencies need to installed.
* google-api-core
* google-cloud-core
* google-auth
* google-cloud-bigquery
* google-cloud-storage
[https://github.com/GoogleCloudPlatform/google-cloud-python]
## Installation
Using conda and conda-forge the dependencies can be installed within the following command:
```bash
conda install -c conda-forge pycurl postgresql psycopg2 sqlalchemy google-cloud-storage google-cloud-bigquery arcsi wget pyyaml planet
```
## Configuration
There are a number of configuration files which need to created for the system to work
### Overall System
An example is within the share directory of the source code.
### Logging
EODataDown uses the python logging library. A general configuration suitable for most users is installed alongside the source `/share/eodatadown`. This can be edited or the `EDD_LOG_CFG` variable used to use a different configuration file.
## Environmental Variables
* `EDD_MAIN_CFG` - specify the location of a JSON file configuring the system.
* `EDD_LOG_CFG` - specify the location of a JSON file configuring the python logging system.
* `EDD_NCORES` - specify the number of cores to use when running jobs which can use multiple cores.
## Notes for Sensors
### Landsat and Sentinel-2 from Google.
To use this sensor you need to set up a google account and project. While the data and download is free an account is still required. See https://cloud.google.com/resource-manager/docs/creating-managing-projects
Details of the account will need to be specified in the JSON config file for the LandsatGOOG sensor.
## Setting up Google Accounts
See the following page for setting up a project:
https://cloud.google.com/resource-manager/docs/creating-managing-projects
And then the following page for setting up your credentials:
https://cloud.google.com/bigquery/docs/reference/libraries
## Setting up Postgresql Database
### New database system
To initialise a database use the `initdb` command.
```bash
initdb -D
```
### Start database
```bash
postgres -D
```
Alternatively, you can start the database as a background service.
```bash
pg_ctl -D -l logfile start
```
### Create database
```bash
createdb eodd_dev_db
```
### Setup User
```bash
psql
```
```SQL
CREATE USER eodduser WITH PASSWORD '';
GRANT ALL PRIVILEGES ON DATABASE eodd_dev_db TO eodduser;
```