https://github.com/ecmwf-projects/dasi
A metadata driven data store
https://github.com/ecmwf-projects/dasi
fdb mars
Last synced: 5 months ago
JSON representation
A metadata driven data store
- Host: GitHub
- URL: https://github.com/ecmwf-projects/dasi
- Owner: ecmwf-projects
- License: apache-2.0
- Created: 2022-01-06T10:31:06.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-12-11T13:44:36.000Z (over 1 year ago)
- Last Synced: 2024-12-29T00:59:14.669Z (over 1 year ago)
- Topics: fdb, mars
- Language: C++
- Homepage:
- Size: 538 KB
- Stars: 0
- Watchers: 5
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
Data Access and Storage Interface (DASI)
========================================
.. image:: https://readthedocs.org/projects/dasi/badge/?version=latest
:target: https://dasi.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
**DASI** is a metadata-driven data store. It is a semantic interface for data, where the data is indexed and uniquely identified by sets of scientifically-meaningful metadata keys.
DASI is modular and is compatible with multiple backends (i.e., object stores or POSIX) through diverse frontends (Python, C++, C).
**Warning**
This project is BETA and will be experimental for the foreseeable future. Interfaces and functionality are likely to change. DO NOT use this software in any project/software that is operational.
Introduction
============
DASI is built on top of FDB [1]_, which has been developed
at ECMWF [2]_ for previous EU projects (see `Acknowledgements `_) and
has been adapted to be highly configurable for different domains.
Using FDB allows DASI to use various backends, such as POSIX, Ceph and Cortx-Motr object-storage, and NVRAM backend.
Configuration
-------------
An example configuration:
.. code-block:: yaml
---
schema: path/to/schema/file
catalogue: toc
store: file
spaces:
- handler: Default
roots:
- path: path/to/data/output1
- path: path/to/data/output2
Schema
------
The schema defines the metadata keys which index and identify the data within a domain.
An example schema describing a hierarchical taxonomy of metadata keys:
.. code-block:: yaml
[ User, Laboratory?, Project
[ DateTime, Processing
[ Type, Object ]]]
Installation
============
DASI Library (C/C++)
--------------------
The supported way to install DASI library is by building from source.
Dependencies
~~~~~~~~~~~~
* C/C++ compiler
* `CMake`_
* `ecbuild`_
* `eckit`_
* `metkit`_
* `fdb`_
Build and Install
~~~~~~~~~~~~~~~~~
.. code-block:: shell
git clone https://github.com/ecmwf-projects/dasi
cd dasi
# Setup environment variables (edit as needed)
SRC_DIR=$(pwd)
BUILD_DIR=build
INSTALL_DIR=$HOME/local
export eckit_DIR=$ECKIT_DIR
export metkit_DIR=$METKIT_DIR
export fdb5_DIR=$FDB_DIR
# Create the the build directory
mkdir $BUILD_DIR
cd $BUILD_DIR
# Run ecbuild (CMake)
ecbuild --prefix=$INSTALL_DIR -- $SRC_DIR
# Build and install
make -j10
make test # optional
make install
# Check installation
$INSTALL_DIR/bin/dasi --version
Install pydasi
--------------
The Python interface to DASI is called **pydasi**.
It uses the `cffi`_ Python package for interfacing with the DASI C API.
Dependencies
~~~~~~~~~~~~
* `DASI Library (C/C++)`_
* `cffi`_
Optional: Python Environment Setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is advised to create a virtual Python environment:
.. code-block:: console
$ cd project_dir
$ python -m venv .venv
$ source .venv/bin/activate
Installation
~~~~~~~~~~~~
**pydasi** can be installed using **pip** command:
.. code-block:: console
$ cd project_dir
$ source .venv/bin/activate
(.venv) $ pip install cffi
(.venv) $ pip install pydasi
.. _`CMake`: https://cmake.org
.. _`ecbuild`: https://github.com/ecmwf/ecbuild
.. _`eckit`: https://github.com/ecmwf/eckit
.. _`metkit`: https://github.com/ecmwf/metkit
.. _`fdb`: https://github.com/ecmwf/fdb
.. _`cffi`: https://pypi.org/project/cffi/
Contributions
=============
Have any feedback / questions / comments / issues ? You can post them `here `_.
The main repository is hosted on GitHub; testing, bug reports and contributions are highly welcomed and appreciated.
See also the `contributors `_ for a more complete list.
Contacts:
- James Hawkes [2]_
- Simon Smart [2]_
- Tiago Quintino [2]_
Acknowledgements
================
Past and current funding and support for this project are listed in the `Acknowledgements `_.
License
=======
This software is licensed under the terms of the Apache License Version 2.0 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
In applying this license, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.
.. |License| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
:target: https://github.com/ecmwf/dasi/blob/develop/LICENSE
:alt: Apache License
Footnotes
=========
.. [1] "Fields DataBase (`FDB `_) is a domain-specific object store"
.. [2] "European Centre for Medium-Range Weather Forecasts (`ECMWF `_)"