Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/thecodeteam/mesos-module-dvdi

Mesos Docker Volume Driver Isolator module
https://github.com/thecodeteam/mesos-module-dvdi

Last synced: about 2 months ago
JSON representation

Mesos Docker Volume Driver Isolator module

Host: GitHub
URL: https://github.com/thecodeteam/mesos-module-dvdi
Owner: thecodeteam
License: apache-2.0
Created: 2015-09-25T03:11:45.000Z (over 9 years ago)
Default Branch: master
Last Pushed: 2017-05-16T00:49:21.000Z (over 7 years ago)
Last Synced: 2024-07-29T01:03:19.686Z (5 months ago)
Language: C++
Size: 4.57 MB
Stars: 77
Watchers: 11
Forks: 16
Open Issues: 5
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# mesos-module-dvdi [![Build Status](https://travis-ci.org/emccode/mesos-module-dvdi.svg)](https://travis-ci.org/emccode/mesos-module-dvdi)

This repo contains the `Docker Volume Driver Isolator Module` for Mesos. The purpose is to create a module that lives on the Mesos Agents (formerly slaves) that enables external storage to be created/mounted/unmounted with each task that is assigned to a agent.

The module leverages [dvdcli](https://github.com/clintonskitson/dvdcli) to enable any existing `Docker Volume Drivers` to be used **without** the Docker containerizer. All Volume Drivers that work with `Docker`, **will also** work with `dvdcli` and thus this Isolator Module.

Currently it targets Mesos 0.23.1, 0.24.2, 0.25.1, 0.26.1, 0.27.2, 0.28.3, 1.0.1, 1.1.0, 1.2.0 and 1.2.1

## Project Summary

This repo is part of a larger project to deliver external storage and introduce cluster wide resources capabilities to the Mesos platform.

The initial Mesos architecture was based on having cluster node agents (aka slaves) determine and report their available resources. This works adequately when workloads consume storage exclusively from direct attached storage on the cluster nodes, but no so well for external storage volumes, shared among Mesos agents. External volume mounts enable enhanced availability and scale.

See the notes, project, and planning information [here](https://github.com/cantbewong/mesos-proposal-externalstorage).

## Functionality

With this module running, the frameworks are now able to leverage the environment variables parameters to determine which `Volume`, from which `Storage Platform` to make available on the `Mesos Agents`. This is without a resource advertisement. Below is an example of `Marathon` specifying an environment variable for `Volume Management`.

In addition, notice how the `VOLUME_OPTS` parameter allows for specifying extra functionality. The `size`, `iops`, and `volumetype` can be requested from the `Storage Platform`, if the `Volume` does not exist yet. In addition, when the `Volume` is then created, a filesystem (EXT4/XFS) can be specified to be used on the `Volume`.

There is one additional option, `overwritefs` which can be used to determine whether to overwrite the filesystem or not. When the `overwritefs` flag is set, and the `Volume` already contains a EXT4/XFS filesystem, it is wiped clean on mount. Otherwise a filesystem will **always** be created if EXT4/XFS is not found.

These options are only available if the specified `Volume Driver` exposes them. The `rexray` volume driver supported these options, but depends support from the `Storage Driver`. See the [dvdcli](https://github.com/emccode/dvdcli) for a full list of options.

## Mesos Agent Configuration

### Volume Driver Endpoint

---
[REX-Ray](https://github.com/emccode/rexray) is a `Docker Volume Driver` endpoint that runs as a service that can be then consumed by `dvdcli`. This isolator can also use any other Docker volume driver in it's place. See the [Docker Plugin List](https://github.com/docker/docker/blob/master/docs/extend/plugins.md).

`REX-Ray` provides visibility and management of external/underlying storage via guest storage introspection.

Below is a one-liner `REX-Ray` install.

```
curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -s -- stable 0.3.3
```

Following install, a configuration file, or environment variables, must be specified. Then Rex-Ray must be started as a service. See the [Rex-Ray project page](https://github.com/emccode/rexray) for more details.

Issuing a `rexray volume` command should return you a list of volumes to test that the configuration is correct for your storage platform.

```
rexray start
rexray volume
```

#### Pre-emptive Volume Mount

The `Docker Volume Driver Isolator Module` can be configured to pre-emptively detach a volume from other agents before attempting a new mount. This will enable availability where another agent has suffered a crash or disconnect. The operation is considered equivalent to a power off of the existing instance for the device.

If this capability is desired, rexray version 0.3.1 or higher needs to be used and a rexray configuration file /etc/rexray/config.yml needs to be created with the addition of the preempt and ignoreUsedCount flags in their respective sections as seen below. More details can be found at [REX-Ray Configuration Guide](http://rexray.readthedocs.org/en/stable/user-guide/config/#configuration-properties) Please check the guide for pre-emptive volume mount compatibility with your backing storage.

```
rexray:
storageDrivers:
- openstack
volume:
mount:
preempt: true
unmount:
ignoreUsedCount: true
openStack:
authUrl: https://authUrl:35357/v2.0/
username: username
password: password
tenantName: tenantName
regionName: regionName
```

#### Volume Containerization

To restrict access to your application's external volumes, across multiple Mesos tasks running on the same agent, you can specify a containerpath which will then provide containerization or isolation of those mounts.

The value specified for the containerpath will affect the behavior of the containerization.

- If no containerpath is provided, the directory will be autocreated and the volume mount will succeed but will provide no containerization
- If a containerpath doesn't start with / (meaning an absolute path is not provided), this is invalid and the task will be reported as FAILURE
- If a containerpath starts with something other than /tmp (meaning it is not destined to the /tmp folder), the directory must pre-exist or you get FAILURE
- If a containerpath starts with /tmp (meaning it is destined to reside within the /tmp folder), the directory will be autocreated if needed and the volume mount will be owned by root:root if it doesn't preexist

**Some examples - pre 1.x Marathon**

The example below will autogenerate the directory because its within the /tmp folder and provide containerization of the volume at /tmp/ebs-auto

```
"env": {
"DVDI_VOLUME_NAME": "VolXYZ",
"DVDI_VOLUME_DRIVER": "rexray",
"DVDI_VOLUME_OPTS": "size=5,iops=150,volumetype=io1,newfstype=xfs,overwritefs=true",
"DVDI_VOLUME_CONTAINERPATH": "/tmp/ebs-auto"
}
```

The next example will fail if the directory /etc/ebs-explicit does not exist. The result of this will provide a mount with containerization at /etc/ebs-explicit

```
"env": {
"DVDI_VOLUME_NAME": "test12345",
"DVDI_VOLUME_DRIVER": "rexray",
"DVDI_VOLUME_OPTS": "size=5,iops=150,volumetype=io1,newfstype=xfs,overwritefs=true",
"DVDI_VOLUME_CONTAINERPATH": "/etc/ebs-explicit"
}
```

Multiple volumes example:

```
"env": {
"DVDI_VOLUME_NAME": "testing",
"DVDI_VOLUME_DRIVER": "platform1",
"DVDI_VOLUME_OPTS": "size=5,iops=150,volumetype=io1,newfstype=ext4,overwritefs=false",
"DVDI_VOLUME_NAME1": "testing2",
"DVDI_VOLUME_DRIVER1": "platform2",
"DVDI_VOLUME_OPTS1": "size=6,volumetype=gp2,newfstype=xfs,overwritefs=true"
}
```

**Example - 1.x Marathon**

```
...
"container": {
"type": "MESOS",
"volumes": [
{
"containerPath": "test-rexray-volume",
"external": {
"size": 100,
"name": "my-test-vol",
"provider": "dvdi",
"options": { "dvdi/driver": "rexray" }
},
"mode": "RW"
}
]
},
```

See [Specifying an External Volume](https://mesosphere.github.io/marathon/docs/external-volumes.html)

### Docker Volume Driver CLI

---

The isolator utilizes a CLI implementation of the `Docker Volume Driver`, called [dvdcli](https://github.com/emccode/dvdcli). Below is a one-liner install for `dvdcli`.

```
curl -sSL https://dl.bintray.com/emccode/dvdcli/install | sh -s stable
```

The `dvdcli` functions exactly as the `Docker` daemon would by looking up spec files from `/etc/docker` or socket files from `/run/docker/plugins` based on the `Volume Driver` name. To make `dvdcli` work, a `Volume Driver` service must be actively running.

The combination of the `mesos-module-dvdi` isolator, `dvdcli`, and the `Docker Volume Driver` must be functioning on each Mesos agent to enable external volumes. The `Docker` daemon installation is not required.

The following command can be used to test the installation and configuration of dvdcli and the Docker volume driver. You should be returned a path to a mounted volume. Following this, perform a `unmount`.

```
dvdcli mount --volumedriver=rexray --volumename=test1
```

### Mesos Docker Volume Driver Isolator

The installation of the isolator is simple. It is a matter of placing the `.so` file, creating a json file, and updating the startup parameters.

1. Copy/Update the `libmesos_dvdi_isolator-.so` to /usr/lib/ on each Mesos Agent node that will offer external storage volumes.

2. Compose or copy a json configuration file tells the agent to load the module and enable the isolator.

3. Create a text file similar to `/usr/lib/dvdi-mod.json` with respective paths set. Replace X.YY.Z to correspond to the version of Mesos and its matching Isolator version.
```
{
"libraries": [
{
"file": "/usr/lib/libmesos_dvdi_isolator-X.YY.Z.so",
"modules": [
{
"name": "com_emccode_mesos_DockerVolumeDriverIsolator"
}
]
}
]
}
```

4. (optional) Mesos slave/agent option flags may be specified in several ways. One common way is to create a text file in `/etc/mesos-slave/modules` and additionally `/etc/mesos-slave/isolation` matching the flags in step 5.

5. (optional) Run slave/agent with explicit `--modules` flag and `--isolation` flags.

```
nohup /usr/sbin/mesos-slave \
--master=zk://172.31.0.11:2181/mesos \
--log_dir=/var/log/mesos \
--containerizers=docker,mesos \
--executor_registration_timeout=5mins \
--ip=172.31.2.11 --work_dir=/tmp/mesos \
--modules=file:///usr/lib/dvdi-mod.json \
--isolation="com_emccode_mesos_DockerVolumeDriverIsolator" &
```

### Example Marathon Call

The following will submit a job, which mounts a volume from an external storage platform.

Up to nine additional volumes may be mounted by appending a digit (1-9)
to the environment variable name. (e.g DVDI_VOLUME_NAME1=).

---

`curl -i -H 'Content-Type: application/json' -d @test.json localhost:8080/v2/apps`

Pre 1.0 Marathon:

```
{
"id": "hello-play",
"cmd": "while [ true ] ; do touch /var/lib/rexray/volumes/test12345/hello ; sleep 5 ; done",
"mem": 32,
"cpus": 0.1,
"instances": 1,
"env": {
"DVDI_VOLUME_NAME": "test12345",
"DVDI_VOLUME_DRIVER": "rexray",
"DVDI_VOLUME_OPTS": "size=5,iops=150,volumetype=io1,newfstype=xfs,overwritefs=true"
}
}
```

Marathon 1.x:

```
{
"id": "hello-play",
"cmd": "while [ true ] ; do touch /var/lib/rexray/volumes/test12345/hello ; sleep 5 ; done",
"mem": 32,
"cpus": 0.1,
"instances": 1,
"container": {
"type": "MESOS",
"volumes": [
{
"containerPath": "test-rexray-volume",
"external": {
"size": 5,
"name": "test12345",
"provider": "dvdi",
"options": { "dvdi/driver": "rexray" }
},
"mode": "RW"
}
]
},
"upgradeStrategy": {
"minimumHealthCapacity": 0,
"maximumOverCapacity": 0
}
```

## Troubleshooting

See the `/var/log/mesos/mesos-slave.INFO` log for details. Troubleshooting via `dvdcli` and `rexray` at the Mesos agent command line is always a great first step to prove that these pre-requisites are properly configured.

## Building Mesos isolator modules with Docker Images

To simplify the process of assembling and configuring a build environment for this docker volume driver isolator, a Docker image for building Mesos modules is offered.

### Build an isolator using our Docker Image

To compile your own customized isolator module. Replace X.Y.Z to correspond to the version of Mesos and its matching Isolator version.
```
git clone https://github.com/emccode/mesos-module-dvdi
cd mesos-module-dvdi/isolator
docker run -ti -v `pwd`:/isolator emccode/mesos-build-module-dev:X.Y.Z /bin/bash -c '/usr/bin/make all && cp -p -v /isolator/build/.libs/libmesos_dvdi_isolator-${ISOLATOR_VERSION}.so /isolator/'
```

Following this, locate the `libmesos_dvdi_isolator-.so` file under `isolator/` and copy it to the `/usr/lib` directory on your Mesos agent node(s).

### (optional) Build a custom Mesos Build Image

If you wish to customize your own Mesos module builder Docker image, modify the Dockerfile and rebuild it like this. Note that this image contains a pre-built Mesos "tree" and is intended to have a unique version for each Mesos release.
```
docker build -t /mesos-build-module-dev:X.Y.Z -f Dockerfile-mesos-build-module-dev .
```

## Release information

Please refer to the [wiki](https://github.com/emccode/mesos-module-dvdi/wiki) for more information relating to the project.

### Licensing

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

### Support

If you have questions relating to the project, please either post [Github Issues](https://github.com/emccode/mesos-module-dvdi/issues), join our Slack channel available by signup through [community.emc.com](https://community.emccode.com) and post questions into the `#mesos` channel, or reach out to the maintainers directly. The code and documentation are released with no warranties or SLAs and are intended to be supported through a community driven process.