Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/elodina/go-kafka-client-mesos
Go Kafka Client Mesos Framework
https://github.com/elodina/go-kafka-client-mesos
Last synced: 1 day ago
JSON representation
Go Kafka Client Mesos Framework
- Host: GitHub
- URL: https://github.com/elodina/go-kafka-client-mesos
- Owner: elodina
- License: apache-2.0
- Created: 2016-01-25T16:43:37.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2016-04-06T16:43:15.000Z (over 8 years ago)
- Last Synced: 2024-04-18T16:02:19.931Z (7 months ago)
- Language: Go
- Homepage: http://www.elodina.net
- Size: 3.48 MB
- Stars: 2
- Watchers: 11
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-mesos - Kafka client
README
go_kafka_client Mesos Framework
==============================[![Build Status](https://travis-ci.org/elodina/go-kafka-client-mesos.svg?branch=master)](https://travis-ci.org/elodina/go-kafka-client-mesos)
[Description](#description)
[Installation](#installation)
* [Prerequisites](#prerequisites)
* [Scheduler configuration](#scheduler-configuration)
* [Run the scheduler](#run-the-scheduler)
* [Quick start](#quick-start)
* [Quick start using Vagrant](https://github.com/elodina/go-kafka-client-mesos/tree/master/vagrant/quickstart.md)
* [Running with Marathon](https://github.com/elodina/go-kafka-client-mesos/tree/master/marathon)
* [Running with Docker](https://github.com/elodina/go-kafka-client/tree/master/docker)[Typical operations](#typical-operations)
* [Shutting down framework](#shutting-down-framework)[Navigating the CLI](#navigating-the-cli)
* [Requesting help](#requesting-help)
* [Adding tasks to the cluster](#adding-tasks-to-the-cluster)
* [Configuring tasks](#configuring-tasks-in-the-cluster)
* [Starting tasks](#starting-tasks-in-the-cluster)
* [Stopping tasks](#stopping-tasks-in-the-cluster)
* [Removing tasks](#removing-tasks-from-the-cluster)
* [Cluster status](#cluster-status)Description
-----------This framework aims to simplify running all sorts of software built on top of go_kafka_client on Mesos. Being actively developed right now.
Right now this framework supports running only the mirror maker.
Installation
============
Prerequisites
-------------Install go 1.5 (or higher) http://golang.org/doc/install
Install glide:
```bash
go get -u github.com/Masterminds/glide
```Clone and build the project
# git clone https://github.com/elodina/go-kafka-client-mesos.git
# cd go-kafka-client-mesos
# export GO15VENDOREXPERIMENT=1
# glide install
# go build cli.go
# go build executor.goScheduler configuration
-----------------------```
# ./cli help scheduler
Usage: scheduler [options]Options:
--master: Mesos Master addresses.
--api: API host:port for advertizing.
--user: Mesos user. Defaults to current system user.
--storage: Storage for cluster state. Examples: file:go_kafka_client_mesos.json; zk:master:2181/go-mesos.
--log.level: Log level. trace|debug|info|warn|error|critical. Defaults to info.
--framework.name: Framework name.
--framework.role: Framework role.
--framework.timeout: Framework failover timeout.
```
Quick start
-----------In order not to pass the API url to each CLI call lets export the URL as follows:
# export GM_API=http://master:6666
First lets start 1 mirror maker task with the default settings. Further in the readme you can see how to change these from the defaults.```
# ./cli add mirrormaker 0 --executor=executor
Added tasks 0cluster:
task:
type: mirrormaker
id: 0
state: inactive
cpu: 0.50
mem: 512.00
configs:
num.producers: 1
num.streams: 1
queue.size: 10000
```
You now have a cluster with 1 task that is not started.```
# ./cli status
cluster:
task:
type: mirrormaker
id: 0
state: inactive
cpu: 0.50
mem: 512.00
configs:
queue.size: 10000
num.producers: 1
num.streams: 1
```Each mirror maker task requires some basic configuration.
```
# ./cli update 0 --producer.config producer.config --consumer.config consumer.config --whitelist "^mytopic$"
Configuration updated:cluster:
task:
type: mirrormaker
id: 0
state: inactive
cpu: 0.50
mem: 512.00
configs:
num.producers: 1
num.streams: 1
queue.size: 10000
consumer.config: consumer.config
producer.config: producer.config
whitelist: ^mytopic$
```Now lets start the task. This call to CLI will block until the task is actually started but will wait no more than a configured timeout. Timeout can be passed via --timeout flag and defaults to 30s. If a timeout of 0ms is passed CLI won't wait for tasks to start at all and will reply with "Scheduled tasks ..." message.
```
# ./cli start 0
Started tasks 0cluster:
task:
type: mirrormaker
id: 0
state: running
cpu: 0.50
mem: 512.00
task id: mirrormaker-0-e5ab7581-1c31-d85d-ff14-77b5fb3de4d4
slave id: 20150914-091943-84125888-5050-10717-S1
executor id: mirrormaker-0
attributes:
hostname: slave0
configs:
consumer.config: consumer.config
producer.config: producer.config
whitelist: ^mytopic$
num.producers: 1
num.streams: 1
queue.size: 10000
```By now you should have a single mirror maker instance running. Here's how you stop it:
```
# ./cli stop 0
Stopped tasks 0
```And remove:
```
# ./cli remove 0
Removed tasks 0
```
Typical operations
=================
-----------------------
Shutting down framework
-----------------------While the scheduler has a shutdown hook it doesn't actually finish the framework. To shutdown the framework completely (e.g. unregister it in Mesos) you may shoot a `POST` to `/teardown` specifying the framework id to shutdown:
```
# curl -d frameworkId=20150807-094500-84125888-5050-14187-0005 -X POST http://master:5050/teardown
```Navigating the CLI
==================
Requesting help
--------------```
# ./cli help
Usage:
help: show this message
scheduler: start scheduler
add: add task
update: update configuration
start: start task
status: get current cluster status
stop: stop task
remove: remove task
Get detailed help from ./cli help
```
Adding tasks to the cluster
--------------------------```
# ./cli help add
Usage: add [options]Types:
mirrormakerOptions:
--executor: Executor binary name.
--api: API host:port for advertizing. Optional if GM_API env is set.
--cpu: CPUs per task. Defaults to 0.5.
--mem: Mem per task. Defaults to 512.
--constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).id-expr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in clusterconstraint examples:
like:slave0 - value equals 'slave0'
unlike:slave0 - value is not equal to 'slave0'
like:slave.* - value starts with 'slave'
unique - all values are unique
cluster - all values are the same
cluster:slave0 - value equals 'slave0'
groupBy - all values are the same
groupBy:3 - all values are within 3 different groups
```
Configuring tasks in the cluster
-------------------------------All file-related configs should be set as file names located `.` (e.g. setting --producer.config producer.properties) OR
http/https URLs (e.g. http://192.168.3.1:6666/producer.config). Please note that http URLs MUST end with file name to be downloaded as Mesos does not respect `Content-Disposition` header.
```
# ./cli help update
Usage: update [options]Options:
--api: API host:port for advertizing. Optional if GM_API env is set.
--cpu: CPUs per task.
--mem: Mem per task.
--constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).
--whitelist: Regex pattern for whitelist. Providing both whitelist and blacklist is an error.
--blacklist: Regex pattern for blacklist. Providing both whitelist and blacklist is an error.
--producer.config: Producer config url or file name.
--consumer.config: Consumer config url or file name.
--num.producers: Number of producers.
--num.streams: Number of consumption streams.
--preserve.partitions: Preserve partition number. E.g. if message was read from partition 5 it'll be written to partition 5.
--preserve.order: E.g. message sequence 1, 2, 3, 4, 5 will remain 1, 2, 3, 4, 5 in destination topic.
--prefix: Destination topic prefix.")
--queue.size: Maximum number of messages that are buffered between the consumer and producer.
--options: Additional options for executor, separated by ';'.
id-expr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
constraint examples:
like:slave0 - value equals 'slave0'
unlike:slave0 - value is not equal to 'slave0'
like:slave.* - value starts with 'slave'
unique - all values are unique
cluster - all values are the same
cluster:slave0 - value equals 'slave0'
groupBy - all values are the same
groupBy:3 - all values are within 3 different groups
```
Starting tasks in the cluster
-----------------------------```
# ./cli help start
Usage: start [options]Options:
--api: API host:port for advertizing. Optional if GM_API env is set.
--timeout: Timeout in seconds to wait until the task receives Running status.
id-expr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
```
Stopping tasks in the cluster
-----------------------------```
# ./cli help stop
Usage: stop [options]Options:
--api: API host:port for advertizing. Optional if GM_API env is set.
id-expr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
```
Removing tasks from the cluster
-----------------------------```
# ./cli help remove
Usage: remove [options]Options:
--api: API host:port for advertizing. Optional if GM_API env is set.
id-expr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
```
Cluster status
--------------```
# ./cli help status
Usage: status [options]Options:
--api: API host:port for advertizing. Optional if GM_API env is set.
```