{"id":17334379,"url":"https://github.com/alpsayin/s5p-client","last_synced_at":"2025-03-27T07:11:04.795Z","repository":{"id":147700343,"uuid":"305157867","full_name":"alpsayin/s5p-client","owner":"alpsayin","description":"Another Sentinel-5 Precursor data query and downloader tool with automation for daily downloads and a ReactJS app that can visualise the data on a map.","archived":false,"fork":false,"pushed_at":"2021-08-05T10:23:04.000Z","size":2132,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-01T12:09:03.972Z","etag":null,"topics":["climate","climate-change","climate-data","copernicus","environmental-monitoring","esa","geojson","leaflet-map","leaflet-react","novit","s5p","s5p-client","s5p-data","sentinel","sentinel-5","sentinel-5p","sentinel-hub"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alpsayin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":null,"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":["https://www.buymeacoffee.com/alpsayin"]}},"created_at":"2020-10-18T17:28:00.000Z","updated_at":"2024-11-05T11:28:48.000Z","dependencies_parsed_at":"2023-07-04T16:33:09.736Z","dependency_job_id":null,"html_url":"https://github.com/alpsayin/s5p-client","commit_stats":{"total_commits":37,"total_committers":1,"mean_commits":37.0,"dds":0.0,"last_synced_commit":"938afea9390eb80661e6616487e800d9301939d7"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpsayin%2Fs5p-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpsayin%2Fs5p-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpsayin%2Fs5p-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alpsayin%2Fs5p-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alpsayin","download_url":"https://codeload.github.com/alpsayin/s5p-client/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245798359,"owners_count":20673902,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["climate","climate-change","climate-data","copernicus","environmental-monitoring","esa","geojson","leaflet-map","leaflet-react","novit","s5p","s5p-client","s5p-data","sentinel","sentinel-5","sentinel-5p","sentinel-hub"],"created_at":"2024-10-15T15:05:53.483Z","updated_at":"2025-03-27T07:11:04.776Z","avatar_url":"https://github.com/alpsayin.png","language":"Python","funding_links":["https://www.buymeacoffee.com/alpsayin"],"categories":[],"sub_categories":[],"readme":"# s5p-client\n### Another Sentinel-5 Precursor data query and downloader tool\nThis tool is written to automate data download, conversion and aggregation for a daily NO2 levels visualisation project.\n\n### Acknowledgements\nI wrote this tool while working at Novit.AI for https://emissions.novit.ai project. I'm releasing the source under my management with Novit's permission. \n\nI would also like to acknowledge and thank the ESA (European Space Agency) for the access to the Sentinel 5P Hub.\n\n### Advantages \u0026 Disadvantages of this tool\n1. Input areas are lat,lon bboxes in the format of minlat, maxlat, minlon, maxlon. While this makes it a breeze to quickly query some areas, if you need to work with more complex geojson defined areas see below.\n2. The products are downloaded directly from the copernicus s5phub. This is both an advantage because I've noticed AWS host is sometimes broken and it was missing products, but also a disadvantage because it may not be as fast as the AWS host. In fact, be nice to Copernicus, limit your download bandwidth yourself.\n3. If you search for NRT products older than 30 days you'll get nothing. If you search for OFFLINE products newer than 30 days, you'll get nothing. And who knows if there are even any RPRO products. It's just the way s5p data is organised. The tool is not smart enough to tell you if products couldnt be found for a specific reason. It's up to you to send reasonable queries.\n\n### Alternative tool which uses the proper sentinelsat library but a bit more complex\nFor all purposes that require a bit more official touch; you should probably use below tool\nhttps://github.com/bilelomrani1/s5p-tools\n\n### Installation \n##### Requirements\nI'd say you need a Linux, at least a unix-like system. Confirmed working on arch and debian.\n\n##### S5P Client\nIt's highly suggested to use a virtualenv.\n```bash\npip install -r requirements.txt\n# Cartopy needs to be installed after numpy is installed because it's build wheel needs it\npip install Cartopy==0.18.0  \npython3 s5p.py --help\n```\n##### Web UI\n```\ncd reactapp\nnpm install\nnpm run build\n```\n\n### React App (Web UI) Quick Start:\nAssuming you installed the client and built the app as instructed just above:\n```bash\ncp -Rf sample-s5p-data $HOME/s5p-data\nsource run_ui.bash\n```\nThen head to [http://localhost:8080/](http://localhost:8080/).\n\n![screenshot](screenshot.jpg \"what it should look like\")\n\n### S5P Query \u0026 Download Tool Usage\ns5p.py client comes with a couple of main modes query and download. \n - Query mode allows downloading of search results with certain flags. \n - Download mode is just used to download products with their uuids and not very useful on its own. \n - Populate mode is an extension of query mode for a range of dates.\nThe 3 most important command you'll most likely need are given in Quick Start.\n\n#### Quick start:\n*Query only:*\n```bash\npython s5p.py query -c milan -pt NO2 -pm NRT\n```\n*Query and download latest product and convert the netcdf file to geojson and json:*\n```bash\npython s5p.py query -c milan -pt NO2 -pm NRT -rl 1 -d -g -j -tf /tmp/s5p-data/\n```\n*Do the above for a range of dates for a set of cities:*\n```bash\npython s5p.py populate -df 2020-11-01 -dt 2020-11-05 -pt NO2 -c milan -c barcelona -c paris -g -j -d\n```\n\n##### Query Mode\nChances are this is the mode you'll use most to both query and automatically download products and convert those products into more friendly jsons. Download flag picks the first product on the results from query for downloading.\n\nYou can use arbitrary latlon bboxes but you can also use predefined cities. Cities are defined in a *s5p_cities.py*. If there's an area you'll likely query a lot, I suggest you fork the repo and add in your own cities.\n\nType in `python3 s5p.py query -h ` to get all available options.\n\n##### Populate Mode\nPopulate mode is an automated version of download mode for a given range of dates. It's practically an extension to query mode's download flag to query and download the first pick on a range of searches.\n\nType in `python3 s5p.py populate -h ` to get all available options.\n\n### S5P Summary Generator\nS5P data is usually has missing spots or not very reliable on a single measurement. `s5p_summary_generator.py` will take in json files and average them out in the span of a given number of days. And it will even generate summaries for each city too. \n\nFor it to work your data must be structured in accordance to `output_directory_structure.txt` *(more on this later)*.\n\nType in `python3 s5p_summary_generator.py -h ` to get all available options.\n\n### Automation\nAll this downloading, averaging and summarisation is a lot of automatisable work. A bash script is provided to be called from cron unsurprisingly called `cron_script.bash`. Of course you may need to dig in and change a few parameters like product type, target folders, the path to python executable etc.\n\nThe script is also quite useful on its own too and can be called with a date parameter. You can use this script to use much less parameters to download data for a predefined city by only giving it a date parameter in the form of YYYY-MM-DD.\n```bash\n./cron_script.bash 2020-11-01\n```\n1. This will create an `s5p-data` folder in `/tmp` directory and structure it in accordance with `required_download_structure.txt`. \n1. Then it will call `s5p_summary_generator.py` on this folder to generate ready-to-serve jsons in acoordance with `output_directory_structure.txt` in the caller's home directory under `s5p-data` folder.\n1. This will then be ready to be served with the reactapp provided as below.\n```bash\nsource run_ui.bash  # sourced so that we can then fg to kill spawned processes\n```\nThis starts two http servers; one `http-server` instance which requires npm. \nThe other is a simple python web server defined in `web_app.py`. This script is merely a demo and there are definitely better ways to serve both the data and the react app.\n\nWebapp is served on [http://localhost:8080](http://localhost:8080) .\nJson Data is served on http://localhost:8081 .\n\n### Hosting your own version:\n```bash\nmkdir -p \"$HOME/s5p-data/data\"\nmkdir -p \"/tmp/s5p-data/data\"\npython3 s5p.py populate -df 2020-11-01 -dt $(date +%Y-%m-%d) -pt NO2 -j -d\n# wait for it to finish; this will take some time\ntouch -a \"$HOME/s5p-data/excluded.txt\"\nrsync -va --exclude='*.nc' --exclude-from \"$HOME/s5p-data/excluded.txt\" --info=\"progress2\" /tmp/s5p-data/data $HOME/s5p-data/\npython3 s5p_summary_generator.py -tf $HOME/s5p-data/data/\nsource run_ui.bash\n```\n\n### Excluding bad data\nSometimes there'll be outlier data that just doesn't seem right (i.e. saturated to a level that even a 10-day average shoots up to 100x). \n\nYou may not want to include this in your averaging process. Best way to do that is to simply remove the data associated with that data and call `s5p_summary_generator.py` again. \n\nBut to avoid re-including this data within the automation process is to add a line to `$HOME/s5p-data/excluded.txt` mentioned above so it's not rsynced again from temp folder to data hosting folder. \n\nRefer to `rsync`'s man pages' `--exclude-from` section ([https://linux.die.net/man/1/rsync](https://linux.die.net/man/1/rsync)).\n\n### Extras \n\n##### S5P Averager\nS5P summary generator uses the algorithms defined in this script to take a running average of the data. This is an implementation file and even though you can run it, it's mostly for test purposes.\n\nType in `python3 s5p_averager.py -h ` to get all available options.\n\n##### Web_app.py\nThere is a reason a custom python based web server is used. The webapp can navigate to certain cities by looking at the URL. However, for this to work the backend server needs to not send 404 responses but serve the index.html as usual. \n\nTry this by typing [http://localhost:8080/paris](http://localhost:8080/paris) or [http://localhost:8080/paris,milan](http://localhost:8080/paris,milan) directly into the browser.\n\n##### Download Mode\nDownload mode only accepts a product uuid and not very useful on its own. It downloads products in netcdf format with uuid. It cannot tell where and when it really belongs to because it doesn't dive into the downloaded file. \nAt this point this an archaic utility and I guess It's only useful if you want to separate your querying and downloading processes.\nType in `python3 s5p.py download -h ` to get all available options.\n\n### License\nLicense is an MIT License if it's not obvious. TL;DR afaik\n - use as you wish\n - i'm not responsible if something breaks\n - an acknowledgement/reference is kindly asked and would be nice\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpsayin%2Fs5p-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falpsayin%2Fs5p-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falpsayin%2Fs5p-client/lists"}