https://github.com/franckalbinet/marisco
Encoding IAEA MARIS data as NetCDF and others.
https://github.com/franckalbinet/marisco
data iaea marine-radioactivity netcdf4
Last synced: about 1 year ago
JSON representation
Encoding IAEA MARIS data as NetCDF and others.
- Host: GitHub
- URL: https://github.com/franckalbinet/marisco
- Owner: franckalbinet
- License: apache-2.0
- Created: 2022-10-10T10:31:12.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-02-18T10:30:33.000Z (over 1 year ago)
- Last Synced: 2025-03-28T09:04:43.544Z (about 1 year ago)
- Topics: data, iaea, marine-radioactivity, netcdf4
- Language: Jupyter Notebook
- Homepage: https://fr.anckalbi.net/marisco/
- Size: 32.6 MB
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# MARISCO
The [IAEA **M**arine **R**adioactivity **I**nformation **S**ystem
(MARIS)](https://maris.iaea.org) provides open access to radioactivity
measurements in marine environments. Developed by the [IAEA Marine
Environmental
Laboratories](https://www.iaea.org/about/organizational-structure/department-of-nuclear-sciences-and-applications/division-of-iaea-environment-laboratories)
in Monaco, MARIS offers data on seawater, biota, sediment, and suspended
matter.
This Python package includes command-line tools to convert MARIS
datasets into [`NetCDF`](https://www.unidata.ucar.edu/software/netcdf/)
or `.csv` formats, enhancing compatibility with various scientific and
data analysis software.
## Core Concept: Handlers
`marisco` is built around the concept of `handlers` - specialized
modules designed to convert MARIS datasets into NetCDF format. Each
handler is tailored to a specific data provider and implemented as a
dedicated Jupyter notebook.
### Literate Programming Approach
We’ve adopted a Literate Programming approach, which means:
1. **Documentation**: Each handler serves as comprehensive
documentation.
2. **Code Reference**: The notebooks contain the actual implementation
code.
3. **Communication Tool**: They facilitate discussions with data
providers about discrepancies or inconsistencies.
### Powered by nbdev
To achieve this, we leverage [nbdev](https://nbdev.fast.ai), a powerful
tool that allows us to:
1. Write code within Jupyter notebooks
2. Automatically export relevant parts as dedicated Python modules
This approach bridges the gap between documentation and implementation,
ensuring they remain in sync.
### See It in Action
For a concrete example of this approach, check out our [OSPAR dataset
handler
implementation](https://fr.anckalbi.net/marisco/handlers/ospar.html).
### List of currently available handlers
MARISCO includes a suite of specialized data handlers designed to:
- Convert provider-specific data formats into standardized MARIS NetCDF
files
- Ensure data quality and consistency across providers
- Facilitate integration with the MARIS marine radioactivity database
- Support automated data processing workflows
The following handlers are currently implemented:
| Handler | Description | Link to Data Source |
|----|----|----|
| [MARIS Legacy](https://fr.anckalbi.net/marisco/handlers/maris_legacy.html) | All legacy MARIS datasets from the MARIS Master Database | \- |
| [HELCOM](https://fr.anckalbi.net/marisco/handlers/helcom.html) | HELCOM marine environment protection datasets | [HELCOM](https://helcom.fi/about-us) |
| [OSPAR](https://fr.anckalbi.net/marisco/handlers/ospar.html) | OSPAR marine environment datasets | [ODIMS OSPAR](https://odims.ospar.org/en/) |
| [TEPCO](https://fr.anckalbi.net/marisco/handlers/tepco.html) | TEPCO Fukushima monitoring data | [TEPCO Monitoring](https://radioactivity.nsr.go.jp/ja/list/349/list-1.html) |
| [GEOTRACES](https://fr.anckalbi.net/marisco/handlers/geotraces.html) | BODC GEOTRACES oceanographic data | [GEOTRACES IDP2021](https://www.geotraces.org/geotraces-intermediate-data-product-2021/) |
## Install
Now, to install `marisco` simply run
``` console
pip install marisco
```
Once successfully installed, run the following command:
``` console
maris_init
```
This command:
1. creates a `.marisco/` directory containing various
configuration/configurable files ((below)) in your `/home`
directory;
2. creates a `configs.toml` file containing default but configurable
settings (default paths, …);
3. downloads several MARIS DB nomenclature/lookup table into
`.marisco/lut/` directory;
4. downloads `maris-template.nc`, the MARIS NetCDF4 template.
### Zotero API key
Upon conversion, `marisco` will automatically retrieve the bibliographic
metadata of each MARIS dataset from [Zotero](https://www.zotero.org/).
To do so, you need to define the following environment variable
`ZOTERO_API_KEY` containing the MARIS Zotero API key. Please contact the
MARIS team to get your API key.
## Getting started
### Command line utilities
All commands accept a `-h` argument to get access to its documentation.
#### `maris_init`
Download configuration file, NetCDF MARIS template and required lookup
tables (nomenclatures).
#### `maris_to_nc`
Convert `helcom`, `geotraces`, `tepco` or `ospar` marine radioactivity
datasets to MARIS NetCDF4 format.
usage: maris_to_nc [-h] [--src SRC] ds dest
positional arguments:
ds Name of the dataset to encode as NetCDF4
dest Output path for NetCDF file
options:
-h, --help show this help message and exit
--src SRC Optional input data path only required for the 'GEOTRACES' dataset
For instance: `maris_to_nc ospar 191-OSPAR-2024.nc`
#### `maris_db_to_nc`
The MARIS Master Database integrates two types of datasets:
- Historical datasets retrieved from published scientific papers
- Ongoing monitoring data from international programs like `HELCOM`,
`OSPAR`, `TEPCO`, and `GEOTRACES`
This command-line utility converts MARIS datasets from their legacy
format to NetCDF4, making them more accessible for modern data analysis
workflows. Users can either convert the entire database or specify
particular datasets by their reference IDs for selective conversion.
usage: maris_db_to_nc [-h] [--ref_ids REF_IDS] src dest
Convert MARIS legacy database to NetCDF4 format. If ref_ids is provided as comma-separated values, only encodes those subsets.
positional arguments:
src Path to MARIS database dump as `.txt` file
dest Output path for NetCDF file(s)
options:
-h, --help show this help message and exit
--ref_ids REF_IDS Optional comma-separated reference IDs (e.g., "123,456,789") (default: )
For instance:
- `maris_db_to_nc "~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt" ~/pro/tmp/output`
- or
`maris_db_to_nc "~/pro/data/maris/2024-11-20 MARIS_QA_shapetype_id=1.txt" ~/pro/tmp/output --ref_ids="16,30"`
for a subset of the MARIS Master Database.
#### `maris_nc_to_csv`
This utility converts NetCDF files to CSV files that conform to the
MARIS Standard format, originally designed for OpenRefine workflows.
Although MARISCO has now superseded OpenRefine in the data preparation
pipeline, the MARIS master database continues to require CSV inputs in
this legacy format. This command-line utility, built with the MARISCO
library, handles the conversion process.
usage: maris_nc_to_csv [-h] src dest
Converts NetCDF files into CSV files that follow the MARIS Standard format.
positional arguments:
src Input path and filename for NetCDF file
dest Output path and filename (without extension) for CSV file
options:
-h, --help show this help message and exit
For instance:
`maris_nc_to_csv ~/pro/tmp/output/191-OSPAR-2024.nc ~/pro/tmp/output/191-OSPAR-2024`
> [!TIP]
>
> ### Note
>
> When specifying the destination path (e.g.,
> `~/pro/tmp/output/191-OSPAR-2024`), the utility automatically appends
> the MARIS sample type to the filename. For example:
>
> - `191-OSPAR-2024_BIOTA.csv` for biological samples
>
> While this specific example produces only a BIOTA file, the utility
> can generate multiple files (one per sample type) depending on the
> content of the source dataset. This reflects the NetCDF4 file
> structure, where each MARIS sample type is stored as a separate group
> within the file.
## Development
The MARIS NetCDF template is generated from `nbs/files/cdl/maris.cdl`
Common Data Language (CDL) file as defined by
[Unidata](https://docs.unidata.ucar.edu/). To generate the MARIS NetCDF
template `nbs/files/nc/maris-template.nc`, install the
[NetCDF-C](https://pjbartlein.github.io/REarthSysSci/install_netCDF.html)
utilities, once in `Marisco` home directory, run:
``` console
ncgen -4 -o nc/maris-template.nc cdl/maris.cdl
```