Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mesos/kibana

Kibana on Mesos
https://github.com/mesos/kibana

Last synced: 26 days ago
JSON representation

Kibana on Mesos

Awesome Lists containing this project

README

        

# mesos-kibana

[Kibana](https://www.elastic.co/products/kibana) framework for Mesos.

This uses the [Mesos-Framework](https://github.com/ContainerSolutions/mesosframework) project. The framework is generic and only becomes a Kibana framework with the correct configuration.

# Features
(Features come from the upstream [Mesos-Framework](https://github.com/ContainerSolutions/mesosframework) and [Mesos-Starter](https://github.com/ContainerSolutions/mesos-starter) projects)

- [x] State stored in ZooKeeper
- [x] Mesos Authorisation
- [ ] ZooKeeper Authorisation (should work, requires testing)
- [x] Live horizontal scaling via REST endpoint
- [x] Jar mode (no docker)
- [x] Resource specification (including port)
- [x] Import Kibana.yml settings file
- [x] "Spread" orchestration strategy (Spreads instances across distinct hosts)
- [x] Decoupled from Kibana. Use any version.
- [x] Decoupled from Mesos. Use any version 0.25+.
- [x] Single endpoint to check health of all instances

# Quick start
To start the kibana framework, simply start the mesosframework binary or docker container with a properties file containing your settings.

## On minimesos
First install [minimesos](http://minimesos.org), then run:

```
minimesos up --num-agents 3
export MINIMESOS_MARATHON=http://192.168.99.100:8080
docker run -d elasticsearch:latest
curl -sS -o marathon.json https://raw.githubusercontent.com/mesos/kibana/master/manual-tests/marathon-minimesos-docker.json ; curl -XPOST -H 'Content-Type:application/json' -d @marathon.json $MINIMESOS_MARATHON/v2/apps
```

This will start a cluster of Kibana instances running on your local machine connecting to a static elasticsearch instance. Please check out the [Elasticsearch framework](http://www.github.com/mesos/elasticsearch) if you are interested in running Elasticsearch.

## On a real Mesos cluster (docker mode)
If you want to run on your own cluster, you will probably want to customize the installation using a properties file, rather than in the marathon file itself. This marathon file will ask Mesos to download the properties file from a url. First export the locations of your master and first slave, then run these commands:

```
export MASTER=${master ip address}
export SLAVE0=${elasticsearch ip address - see json file}
curl -sS -o marathon.json https://raw.githubusercontent.com/mesos/kibana/master/manual-tests/marathon-docker.json
cat marathon.json | sed -e 's/$MASTER/'"$MASTER"'/' | sed -e 's/$SLAVE0/'"$SLAVE0"'/' | curl -XPOST -H 'Content-Type:application/json' -d @- http://$MASTER:8080/v2/apps
```

See the [marathon file](./manual-tests/marathon-docker.json) and the [properties file](./docs/examples/docker.properties) for further ways to customize the installation.

## On a real Mesos cluster (jar mode)
Sometimes people don't want to run docker. This example will install a Kibana cluster using the java binaries. This marathon file will ask Mesos to download the properties file from a url. First export the locations of your master and first slave, then run these commands:

```
export MASTER=${master ip address}
export SLAVE0=${elasticsearch ip address - see json file}
curl -sS -o marathon.json https://raw.githubusercontent.com/mesos/kibana/master/manual-tests/marathon-jar.json
cat marathon.json | sed -e 's/$MASTER/'"$MASTER"'/' | sed -e 's/$SLAVE0/'"$SLAVE0"'/' | curl -XPOST -H 'Content-Type:application/json' -d @- http://$MASTER:8080/v2/apps
```

See the [marathon file](./manual-tests/marathon-jar.json) and the [properties file](./docs/examples/jar.properties) for further ways to customize the installation.

# Passing options
Because this project uses [Mesos-Framework](https://github.com/ContainerSolutions/mesosframework) there is no Kibana-specific code to compile or download. To run, simply pass a configuration file or options. Example marathon files can be found in the [manual-tests](./manual-tests) directory.

All options can be specified as either:
- A cli parameter: `--mesos.command=pwd`
- A properties file: `mesos.command=pwd`
- Java options: `-Dmesos.command=pwd`
- Environmental variables: `MESOS_COMMAND=pwd`

In that order of preference.

To pass a configuration file, the following property must be set:
- `--spring.config.location=my.properties` (Or the env var `SPRING_CONFIG_LOCATION`, etc.)

# Full list of Kibana related settings
All settings are written in properties or argument format. Remember that these can also be specified in environment or yml format.

| Command | Description |
| --- | --- |
| `spring.application.name` | Required application name for Spring |
| `mesos.framework.name` | Framework name used in Mesos and ZooKeeper |
| `mesos.master` | URL of the master (usually provided by ZooKeeper) |
| `mesos.zookeeper.server` | IP:PORT of the zookeeper server |
| `mesos.resources.cpus` | CPUs allocated to the task |
| `mesos.resources.mem` | RAM allocated to the task |
| `mesos.resources.count` | Number of task instances |
| `mesos.resources.ports.${VAR}.host` | A requested port, where VAR is the name of the port. |
| `mesos.resources.ports.${VAR}.container` | When in bridge mode, the container port to map the host port to. |
| `mesos.docker.image` | Docker image to use |
| `mesos.docker.network` | Type of docker network |
| `mesos.command` | The command to run |
| `mesos.uri[0..]` | Files to download into the Mesos sandbox |
| `logging.level.com.containersolutions.mesos` | Logging level |
| `mesos.principal` | The principal for Mesos Auth. |
| `mesos.secret` | The secret for Mesos Auth. |
| `mesos.healthCheck.command` | The command to run as the Mesos healthcheck |

Note that there are more parameters. See [Mesos-Starter](https://github.com/ContainerSolutions/mesos-starter).

# Passing Kibana settings
First upload the settings file into the task sandbox with the `mesos.uri` property:
```
mesos.uri[0]=https://gist.githubusercontent.com/philwinder/592a1ab2db40431c1b08/raw/kibana.yml
```
The file can be a local file (local to the host that the task is to run on) or a url.

Then copy that file into the kibana config directory, overwriting the default. Kibana does not expose a config location parameter so overwriting is the only way to pass the settings into Kibana. For example, in jar mode:
```
mesos.command= cp $MESOS_SANDBOX/kibana.yml $MESOS_SANDBOX/kibana-*/config/kibana.yml ; cd kibana-* ; bin/kibana --port=$UI_5061 --elasticsearch ${elasticsearch.http}
```
Or when in docker mode:
```
mesos.command=mv $MESOS_SANDBOX/kibana.yml /opt/kibana/config/kibana.yml ; kibana --port=$UI_5061 --elasticsearch ${elasticsearch.http}
```
# Port allocation
Ports are allocated by Mesos and provided to the application as an environmental variable. For example:
```
mesos.resources.ports.UI_5061.host=ANY
mesos.resources.ports.UI_5061.container=5601
```
Assigns an unprivileged port to the environmental variable `UI_5061`. This environmental variable can now be use in the `mesos.command`, if required.

The value can be one of the following types:

| Command | Description |
| --- | --- |
| `ANY` | The next available unprivileged port (>1024) |
| `UNPRIVILEGED` | The next available unprivileged port (>1024) |
| `PRIVILEGED` | The next available privileged port (<=1024) |
| `1234` | A specific port (e.g. 1234) |

# Health checks
[MesosFramework](https://github.com/ContainerSolutions/mesosframework) uses Spring Actuator to provide health and metrics endpoints. To access the health endpoint visit: `http://${SCHEDULER_IP_ADDRESS}:${server.port}/health`. Acuator defaults the `server.port` to 8080, although it is recommended to reserve ports in the marathon command and set this port explicitly. E.g. [jar mode json file](./manual-tests/marathon-jar.json)

See the [Spring documentation](http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready-endpoints) for more information.

# Horizontal scaling
This adds an endpoint at the following location to control the number of instances in the cluster. The endpoint matches the properties file definition of the same name:

`GET /mesos/resources/count` Returns the current number of requested instances. For example to get the current number of instances:

```
$ curl -s http://${SCHEDULER_IP_ADDRESS}:${server.port}/mesos/resources/count
3
```

`POST /mesos/resources/count` with a body of type `Integer` will set the number of requested instances. For example, to set the number of instances to 1:

```
$ curl -XPOST -H 'Content-Type: text/plain' http://${SCHEDULER_IP_ADDRESS}:${server.port}/mesos/resources/count -d 1
```

# Sponsors
This project is sponsored by Cisco Cloud Services.

# License
Apache License 2.0