{"id":20200905,"url":"https://github.com/openclimatefix/satip","last_synced_at":"2025-04-04T18:04:30.384Z","repository":{"id":37445490,"uuid":"309434883","full_name":"openclimatefix/Satip","owner":"openclimatefix","description":"Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data","archived":false,"fork":false,"pushed_at":"2024-10-28T17:55:37.000Z","size":489353,"stargazers_count":41,"open_issues_count":49,"forks_count":28,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-29T22:32:36.770Z","etag":null,"topics":["eumetsat","gis","nowcasting","python","satellite-imagery"],"latest_commit_sha":null,"homepage":"https://satip.readthedocs.io/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openclimatefix.png","metadata":{"funding":{"github":["openclimatefix"],"patreon":null,"open_collective":"openclimatefix","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null},"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-02T16:49:59.000Z","updated_at":"2024-10-10T11:16:06.000Z","dependencies_parsed_at":"2024-02-04T07:44:39.866Z","dependency_job_id":"9db6bb3d-b2bb-467d-92b0-d1857ba3e4bf","html_url":"https://github.com/openclimatefix/Satip","commit_stats":{"total_commits":1213,"total_committers":23,"mean_commits":52.73913043478261,"dds":0.7394888705688376,"last_synced_commit":"38854460b47ad34333e5b6c0235f530bf1422119"},"previous_names":[],"tags_count":346,"template":false,"template_full_name":"fastai/nbdev_template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FSatip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FSatip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FSatip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openclimatefix%2FSatip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openclimatefix","download_url":"https://codeload.github.com/openclimatefix/Satip/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226212,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["eumetsat","gis","nowcasting","python","satellite-imagery"],"created_at":"2024-11-14T04:46:08.094Z","updated_at":"2025-04-04T18:04:30.362Z","avatar_url":"https://github.com/openclimatefix.png","language":"Jupyter Notebook","funding_links":["https://github.com/sponsors/openclimatefix","https://opencollective.com/openclimatefix"],"categories":[],"sub_categories":[],"readme":"# Satip\n\u003c!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section --\u003e\n[![All Contributors](https://img.shields.io/badge/all_contributors-14-orange.svg?style=flat-square)](#contributors-)\n\u003c!-- ALL-CONTRIBUTORS-BADGE:END --\u003e\n\n[![PyPI version](https://badge.fury.io/py/satip.svg)](https://badge.fury.io/py/satip)\n\n[![codecov](https://codecov.io/gh/openclimatefix/Satip/branch/main/graph/badge.svg?token=GTQDR2ZZ2S)](https://codecov.io/gh/openclimatefix/Satip)\n\n\u003e Satip is a library for \u003cb\u003esat\u003c/b\u003eellite \u003cb\u003ei\u003c/b\u003emage \u003cb\u003ep\u003c/b\u003erocessing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data\n\n\u003cbr\u003e\n\n\n### Installation\n\nTo install the `satip` library please run:\n\n```bash\npip install satip\n```\n\nOr if you're working in the development environment you can run the following from the directory root:\n\n```bash\npip install -e .\n```\n\n#### Conda\n\nOr, if you want to use `conda` from a cloned Satip repository:\n\n```bash\nconda env create -f environment.yml\nconda activate satip\npip install -e .\n```\n\nIf you plan to work on the development of Satip then also consider installing these development tools:\n\n```bash\nconda install pytest flake8 jedi mypy black pre-commit\npre-commit install\n```\n\n### Development Environment\n\nIn order to contribute:\n- it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.\n- At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.\n\n## Operation\n\n### Getting your own API key\n\nIn order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:\n\n1. Go to https://eoportal.eumetsat.int and register an account.\n2. You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option \"API key\" or go to https://api.eumetsat.int/api-key/ directly.\n3. Please make sure that you added the key and secret to your user's environment variables.\n\n### Downloading EUMETSAT Data\n\nWe have moved this to [here](https://github.com/openclimatefix/dagster-dags/blob/main/containers/sat/download_process_sat.py)\n\n### Converting Native files to Zarr\n`scripts/convert_native_to_zarr.py` converts EUMETSAT `.nat` files to Zarr datasets, using very mild lossy [JPEG-XL](https://en.wikipedia.org/wiki/JPEG_XL) compression. (JPEG-XL is the \"new kid on the block\" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent `bz2` compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range `[0, 1]`. NaNs are encoded as the value `0.025`. All \"real\" values are in the range `[0.075, 1]`. We leave a gap between \"NaNs\" and \"real values\" because there is very slight \"ringing\" around areas of constant value (see [this comment for more details](https://github.com/openclimatefix/Satip/issues/67#issuecomment-1036456502)). Use `satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs` to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range `[0, 1]`.\n\n\n## Running in Production\n\nThe live service uses `app.py` as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.\n\n`--api-key` or `API_KEY` is the EUMETSAT API key\n\n`--api-secret` or `API_SECRET` is the EUMETSAT API secret\n\n`--save-dir` or `SAVE_DIR` is the top level directory to save the output files, a `latest` subfolder will be added to that directory to contain the latest data\n\n`--history` or `HISTORY` is the amount of history timesteps to use in the `latest.zarr` files\n\n`--db-url` or `DB_URL` is the URL to the database to save to when a run has finished\n\n`--use-rescaler` or `USE_RESCALER` tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.\n\n`--use-iodc` or `USE_IODC` is an option to get the IODC satellite data\n\n## Testing\n\nTo run tests, simply run ```pytest .``` from the root of the repository. To generate the test plots, run ```python scripts/generate_test_plots.py```.\n\n### Environmental Variables\nSome tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:\n - `EUMETSAT_USER_KEY`: the EUMETSAT API key\n - `EUMETSAT_USER_SECRET`: the EUMETSAT API secret\n\n These can be added using the `export` command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. \"~/.bashrc\" if using bash).\n\n## Contributors ✨\n\nThanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://www.jacobbieker.com\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/7170359?v=4?s=100\" width=\"100px;\" alt=\"Jacob Bieker\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJacob Bieker\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=jacobbieker\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://jack-kelly.com\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/460756?v=4?s=100\" width=\"100px;\" alt=\"Jack Kelly\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eJack Kelly\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=JackKelly\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/AyrtonB\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/29051639?v=4?s=100\" width=\"100px;\" alt=\"Ayrton Bourn\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAyrton Bourn\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=AyrtonB\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://laurencewatson.com\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1125376?v=4?s=100\" width=\"100px;\" alt=\"Laurence Watson\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eLaurence Watson\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=Rabscuttler\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/notger\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/1180540?v=4?s=100\" width=\"100px;\" alt=\"Notger Heinz\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNotger Heinz\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=notger\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peterdudfield\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/34686298?v=4?s=100\" width=\"100px;\" alt=\"Peter Dudfield\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ePeter Dudfield\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=peterdudfield\" title=\"Documentation\"\u003e📖\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/norbline\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/39647420?v=4?s=100\" width=\"100px;\" alt=\"Azah Norbline\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eAzah Norbline\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=norbline\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/TomPughe\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/147526382?v=4?s=100\" width=\"100px;\" alt=\"Tom Pughe\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eTom Pughe\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=TomPughe\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://huggingface.co/64bits\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/40121574?v=4?s=100\" width=\"100px;\" alt=\"Zhenbang Feng\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eZhenbang Feng\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=JasonFengGit\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/jsbaasi\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/72830904?v=4?s=100\" width=\"100px;\" alt=\"jsbaasi\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003ejsbaasi\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=jsbaasi\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/suleman1412\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/37236131?v=4?s=100\" width=\"100px;\" alt=\"Suleman Karigar\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eSuleman Karigar\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=suleman1412\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://richasharma.co.in/\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/41283476?v=4?s=100\" width=\"100px;\" alt=\"Richa\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eRicha\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/commits?author=14Richa\" title=\"Code\"\u003e💻\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"http://phinate.github.io\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/49782545?v=4?s=100\" width=\"100px;\" alt=\"Nathan Simpson\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNathan Simpson\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"https://github.com/openclimatefix/Satip/issues?q=author%3Aphinate\" title=\"Bug reports\"\u003e🐛\u003c/a\u003e\u003c/td\u003e\n      \u003ctd align=\"center\" valign=\"top\" width=\"14.28%\"\u003e\u003ca href=\"https://github.com/peach280\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/187241561?v=4?s=100\" width=\"100px;\" alt=\"peach280\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003epeach280\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003cbr /\u003e\u003ca href=\"#maintenance-peach280\" title=\"Maintenance\"\u003e🚧\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fsatip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenclimatefix%2Fsatip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenclimatefix%2Fsatip/lists"}