https://github.com/fotioudim/nwc-safer
An awesome CLI tool simplifying NetCDF data extraction of NWC-SAF products and transformation to different file formats!
https://github.com/fotioudim/nwc-safer
cli-app eumetsat netcdf nowcasting nwcsaf pandas poetry python queues satellite-imagery threads typer watchdog xarray
Last synced: 7 months ago
JSON representation
An awesome CLI tool simplifying NetCDF data extraction of NWC-SAF products and transformation to different file formats!
- Host: GitHub
- URL: https://github.com/fotioudim/nwc-safer
- Owner: fotioudim
- License: mit
- Created: 2023-03-23T23:13:48.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-14T21:16:04.000Z (over 1 year ago)
- Last Synced: 2024-08-11T10:09:01.081Z (about 1 year ago)
- Topics: cli-app, eumetsat, netcdf, nowcasting, nwcsaf, pandas, poetry, python, queues, satellite-imagery, threads, typer, watchdog, xarray
- Language: Python
- Homepage: https://pypi.org/project/nwc-safer/
- Size: 2.21 MB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
NWC-SAFer
An awesome CLI tool simplifying NetCDF data extraction of NWC-SAF products and transformation to different file formats!
[](https://github.com/fotioudim/nwc-safer/commits/main)
[](https://github.com/fotioudim/nwc-safer/issues/)
[](https://github.com/fotioudim/nwc-safer/network/members)
[](https://github.com/fotioudim/nwc-safer/stargazers)
[](https://opensource.org/licenses/MIT)
Report Bug
·
Request Feature
# :notebook_with_decorative_cover: Table of Contents
- [:notebook\_with\_decorative\_cover: Table of Contents](#notebook_with_decorative_cover-table-of-contents)
- [:star2: About the Project](#star2-about-the-project)
- [:books: Tech Stack](#books-tech-stack)
- [:toolbox: Getting Started](#toolbox-getting-started)
- [:wrench: Prerequisites](#wrench-prerequisites)
- [:gear: Installation](#gear-installation)
- [PyPi (recommended)](#pypi-recommended)
- [Source](#source)
- [:running: Run Locally](#running-run-locally)
- [:bookmark\_tabs: Usage](#bookmark_tabs-usage)
- [Watch directory and convert incoming files](#watch-directory-and-convert-incoming-files)
- [Manual file conversion](#manual-file-conversion)
- [Check compatible products](#check-compatible-products)
- [Launch Github repository](#launch-github-repository)
- [:compass: Roadmap](#compass-roadmap)
- [:warning: License](#warning-license)
- [:handshake: Contact](#handshake-contact)
## :star2: About the Project
This is a modern and simplified Command Line Interface tool created to extract info and metadata from NetCDF files and convert them into different file formats such as Csv, Excel, plain text, etc.
Its aim is to simplify data extraction from NWC SAF [products](https://www.nwcsaf.org/web/guest/nwc/geo-geostationary-near-real-time-v2021) and their transformation to different file formats, more readable and easier to access and edit. ΝWC-SAF is an initiative of [EUMETSAT](https://www.eumetsat.int/) to provide operational services ensuring the optimum use of meteorological satellite data in Nowcasting and Very Short Range Forecasting.
### :books: Tech Stack
The tech stack of this project consists of the following main technologies/frameworks/tools:
- [](https://www.python.org/)
- [](https://typer.tiangolo.com/)
- [](https://python-poetry.org/)
- [](https://github.com/gorakhargosh/watchdog)
- _Python Queues and thread handling is applied to enhance user experience_
- _NetCDF4, Pandas, xarray libraries are used for file manipulation_
## :toolbox: Getting Started
### :wrench: Prerequisites
You just need to have installed Python language preferrably > 3.10 version and a Python package/dependency management tool like Pip, Poetry, Conda etc.
The tool is compatible with all popular OS.
_Better used along with CLIs that support emojis and Unicode characters for a better user experience._
### :gear: Installation
#### PyPi (recommended)
- Install it from PyPi using pip or any other python package manager
```bash
pip install --user nwc-safer
```
- Enable auto-completion by executing the following command
```bash
nwc-safer --install-completion
```
#### Source
- Clone the Git repo
```bash
git clone https://github.com/fotioudim/nwc-safer.git
```
- Install using poetry in the corresponding directory
```bash
poetry install
```
### :running: Run Locally
Just run it using `nwc-safer`. Get help for the available commands
```bash
nwc-safer --help
```
## :bookmark_tabs: Usage
Run `nwc-safer` or `nwc-safer -- help` in your CLI to get a better description of the available commands and options.
### Watch directory and convert incoming files
Constantly watch a directory for incoming NetCDF files, in order to extract the desired data and export them in the form of Csv/Excel/Text files.
```bash
nwc-safer watch [OPTIONS] [INPUT_PATH] [OUTPUT_PATH]
```
Arguments:
- `input_path`: The path (relative/absolute) for the directory desired to be watched [default: .]
- `output_path`: The path (relative/absolute) for the output directory [default: .\output\]
Options:
- `--lat` `-la`: The latitude coordinate range in ascending order, given in the form of two int values (min & max)
- `--lon` `-lo`: The longitude coordinate range in ascending order, given in the form of two int values (min & max)
- `--format` `-f`: The output file format [default: csv]
- `--recursive` `-r`: Watch for incoming files recursively in all the subdirectories of the specified directory
- `--existing` `-e`: Convert pre-existing files in the specified directory
### Manual file conversion
Process a single or multiple NetCDF file(s), by extracting the desired data
and exporting them in a new file format (eg. Csv, Excel).
```bash
nwc-safer convert [OPTIONS] [FILE_PATHS]...
```
Arguments:
- `file_paths`: The path(s) (relative/absolute) for the file(s) desired to be converted [default: None] [required]
Options:
- `--lat` `-la`: The latitude coordinate range in ascending order, given in the form of two int values (min & max)
- `--lon` `-lo`: The longitude coordinate range in ascending order, given in the form of two int values (min & max)
- `--output` `-o`: The path (relative/absolute) for the output directory [default: .\output\]
- `--format` `-f`: The output file format [default: csv]
### Check compatible products
Check which NWC-SAF products are currently supported by the tool
```bash
nwc-safer compatibility
```
### Launch Github repository
Launch NWC-SAF NetCDF Data Exporter Github repository in the web browser
```bash
nwc-safer repo
```
## :compass: Roadmap
* [x] Add continuous watching & conversion operation
* [x] Single conversion operation
* [x] OS agnostic functionality
* [x] Multiple conversion operation
* [x] Subset dataset based on coordinates range
* [x] Support 3 output file formats (.csv, .xlsx, txt)
* [ ] Generalize solution for a bigger set of [NWC SAF products](https://www.nwcsaf.org/web/guest/nwc/geo-geostationary-near-real-time-v2021)
## :warning: License
Distributed under the MIT License. See `LICENSE.md` for more information.
## :handshake: Contact
You can reach me at:
- Personal website: https://www.dfotiou.gr
- StackOverflow profile: https://stackoverflow.com/users/11680294/fotiou-d
- Linkedin profile: https://www.linkedin.com/in/dimitris-fotiou-4141a8197/
- Email: [fotioudimitris93@gmail.com](mailto:fotioudimitris93@gmail.com)