https://github.com/observingclouds/ecmwfspec

Fsspec filesystem driver for ECMWF File Storage System
https://github.com/observingclouds/ecmwfspec

ecfs ecmwf fsspec

Last synced: about 1 month ago
JSON representation

Fsspec filesystem driver for ECMWF File Storage System

• Host: GitHub
• URL: https://github.com/observingclouds/ecmwfspec
• Owner: observingClouds
• Created: 2024-09-05T11:13:14.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-10-24T14:29:03.000Z (2 months ago)
• Last Synced: 2024-10-25T17:01:58.045Z (about 2 months ago)
• Topics: ecfs, ecmwf, fsspec
• Language: Python
• Homepage:
• Size: 93.8 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md

Awesome Lists containing this project

README

        
[![CI](https://github.com/observingClouds/ecmwfspec/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/observingClouds/ecmwfspec/actions?query=workflow%3ATests)

# ecmwfspec

This is work in progress! This unofficial repository aims to provide an fsspec driver for the [ECMWF File Storage System](https://confluence.ecmwf.int/display/UDOC/ECFS+user+documentation).

Pull requests (also for additional protocols like MARS and FDB) are welcomed!

```python

import fsspec

with fsspec.open("ec://path/to/file", "r") as f:

    print(f.read())

```

## Loading datasets

```python

import fsspec

import xarray as xr

url = fsspec.open("ec:/arch/project/file.grib").open()

ds = xr.open_dataset(url, engine='cfgrib')  # does not work until https://github.com/ecmwf/cfgrib/issues/326 is solved

```

### Usage in connection with gribscan

This is just an example what ecmwfspec can be used for.

```python

import gribscan

import json

import xarray as xr

file_to_scan = "ec:///path/to/some/grib/file/forecast+0038.grib2"

index_file = "index.json"

reference_file = "reference.json"

gribscan.write_index(gribfile=file_to_scan, idxfile=index_file)  # required currently patch https://github.com/gribscan/gribscan/commit/7a5e595759f48e3118964091358f1b2e9eb32b37 to work with fsspec paths

magician = gribscan.magician.HarmonieMagician()  # use magician fitting the grib file

refs = gribscan.grib_magic(

                        filenames=[index_file],

                        magician=magician,

                        global_prefix="",

                    )

with open(reference_file, "w") as outfile:

    json.dump(refs["heightAboveGround"], outfile)

ds = xr.open_zarr(f"reference::{reference_file}")

ds.u.max()  # with the help of ecmwfspec the data is now fetched if it is not locally cached

```