https://github.com/lren-chuv/meta-db-setup
Setup for meta-db database which contains the list of variables and groups supported by MIP
https://github.com/lren-chuv/meta-db-setup
database-migrations docker-image reference-data
Last synced: 2 months ago
JSON representation
Setup for meta-db database which contains the list of variables and groups supported by MIP
- Host: GitHub
- URL: https://github.com/lren-chuv/meta-db-setup
- Owner: LREN-CHUV
- License: apache-2.0
- Created: 2017-05-10T15:32:29.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-05-17T09:06:01.000Z (over 6 years ago)
- Last Synced: 2025-03-12T22:21:45.123Z (7 months ago)
- Topics: database-migrations, docker-image, reference-data
- Language: Scala
- Size: 99.6 KB
- Stars: 0
- Watchers: 4
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://www.unil.ch/lren/en/home.html) [](https://github.com/LREN-CHUV/meta-db-setup/blob/master/LICENSE) [](https://hub.docker.com/r/hbpmip/meta-db-setup/) [](https://www.codacy.com/app/hbpmip/meta-db-setup?utm_source=github.com&utm_medium=referral&utm_content=LREN-CHUV/meta-db-setup&utm_campaign=Badge_Grade)
[](https://circleci.com/gh/LREN-CHUV/meta-db-setup)
# Setup for database 'meta-db'
## Introduction
This Docker image manages the database migration scripts for the 'meta-db' database used by MIP.
This database contains the metadata used for reference, including:
* the list of variables and groups for the Common Data Elements (CDE) defined by MIP.
* the list of variables and groups for other datasets available in MIP.
It uses Flyway to perform the versioned database migrations.
## Usage
Run:
```console
$ docker run -i -t --rm -e FLYWAY_HOST=`hostname` hbpmip/meta-db-setup:2.5.0 migrate
```
where the environment variables are:
* FLYWAY_HOST: database host, default to 'db'.
* FLYWAY_PORT: database port, default to 5432.
* FLYWAY_DATABASE_NAME: name of the database or schema, default to 'meta'
* FLYWAY_URL: JDBC url to the database, constructed by default from FLYWAY_DBMS, FLYWAY_HOST, FLYWAY_PORT and FLYWAY_DATABASE_NAME
* FLYWAY_DRIVER: Fully qualified classname of the jdbc driver (autodetected by default based on flyway.url)
* FLYWAY_USER: database user, default to 'meta'.
* FLYWAY_PASSWORD: database password, default to 'meta'.
* FLYWAY_SCHEMAS: Optional, comma-separated list of schemas managed by Flyway
* FLYWAY_TABLE: Optional, name of Flyway's metadata table (default: schema_version)
### Child images
Child images should follow the following procedure to be able to load their metadata:
#### List of Data Elements (aka hierarchy of variables)
Define the environment variable TAXONOMIES defining a space-separated list of data elements definitions, where each element definition is of the form [data elements name]|[target table]|[list of groupings for histograms view].
* __data element name__ is the name of this organisation of variables. It should point to file /src/variables/[data elements name].json located inside the Docker image.
* __target table__ should be the name of the table or view to use to retrieve features for algorithms and data exploration.
* __list of groupings for histograms view__ should be a comma separated list of columns in the target table that defines a default breakdown of a dataset into several histograms with groupings, where each column defined here will create an histogram grouping values on the group by column.
For example,
```
ENV TAXONOMIES=test-set|main_features_table|dataset,gender,agegroup,alzheimerbroadcategory
```
Then for each data element definition, place a file named [data elements name].json into folder /src/variables/ describing the hierarchy of variables and following schema [variables_schema.json](variables_schema.json)
#### List of Json patches to apply to existing Data Elements definitions
Define the environment variable TAXONOMY_PATCHES defining a space-separated list of Json patches to apply to existing data elements, where each patch description has the form [data elements name to patch]|[new data elements name]|[target table]|[list of groupings for histograms view].
* __data elements name to patch__ is the name of the original organisation of variables that we want to update.
* __new data elements name__ is the name of the new organisation of variables. It should point to file /src/patches/[data elements name to patch].patch.json located inside the Docker image and containing the JSON patch definition to apply to existing list of variables for [data elements name to patch].
* __target table__ should be the name of the new table or view to use to retrieve features for algorithms and data exploration.
* __list of groupings for histograms view__ should be a comma separated list of columns in the target table that defines a default breakdown of a dataset into several histograms with groupings, where each column defined here will create an histogram grouping values on the group by column.
For example,
```
ENV TAXONOMY_PATCHES=test-set|test-set-with-custom-vars|main_features_table|dataset,gender,custom_category
```
Each new data elements definition, target table and hierarchy json will be inserted into the meta table. For each new data elements, we generate the json describing the hierarchy of variables by taking the hierarchy json from the original data elements definition and applying to it a [Json patch](http://jsonpatch.com/) loaded from the container folder /src/patches/ and named [new data elements name].json
## Build
Run: `./build.sh`
## Publish on Docker Hub
Run: `./publish.sh`
## License
### Meta-db-setup
(this project)
Copyright (C) 2017 [LREN CHUV](https://www.unil.ch/lren/en/home.html)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
### Flyway
Copyright (C) 2016-2017 [Boxfuse GmbH](https://boxfuse.com)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
## Trademark
Flyway is a registered trademark of [Boxfuse GmbH](https://boxfuse.com).
## Contribution policy ##
Contributions via GitHub pull requests are gladly accepted from their original author. Along with
any pull requests, please state that the contribution is your original work and that you license
the work to the project under the project's open source license. Whether or not you state this
explicitly, by submitting any copyrighted material via pull request, email, or other means you
agree to license the material under the project's open source license and warrant that you have the
legal authority to do so.
# Acknowledgements
This work has been funded by the European Union Seventh Framework Program (FP7/20072013) under grant agreement no. 604102 (HBP)
This work is part of SP8 of the Human Brain Project (SGA1).