https://github.com/elodina/syscol

Collect local Mesos slave, underlying operating system and machine metrics and produce to Apache Kafka
https://github.com/elodina/syscol

Last synced: about 1 month ago
JSON representation

Collect local Mesos slave, underlying operating system and machine metrics and produce to Apache Kafka

• Host: GitHub
• URL: https://github.com/elodina/syscol
• Owner: elodina
• License: apache-2.0
• Created: 2015-08-18T16:37:37.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2016-01-29T14:54:40.000Z (almost 9 years ago)
• Last Synced: 2024-08-04T04:06:36.175Z (5 months ago)
• Language: Go
• Homepage: http://www.elodina.net
• Size: 1.46 MB
• Stars: 20
• Watchers: 12
• Forks: 5
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-mesos - Mesos slave metrics -> Kafka

README

        
Syscol Mesos Framework

======================

Installation

------------

Install go 1.4 (or higher) http://golang.org/doc/install

Install godep https://github.com/tools/godep

Clone and build the project

    # git clone https://github.com/elodina/syscol.git

    # cd syscol

    # godep restore

    # go build .

    # go build cli.go

    # go build executor.go

Usage

-----

Syscol framework ships with command-line utility to manage schedulers and executors:

    # ./cli help

    Usage:

        help: show this message

        scheduler: configure and start scheduler

        start: start syscol reporters

        stop: stop syscol reporters

        update: update configuration

        status: get current status of cluster

    More help you can get from ./cli  -h

Scheduler Configuration

-----------------------

The scheduler is configured through the command line.

    # ./cli scheduler 

Following options are available:

    -master="": Mesos Master addresses.

    -api="": Binding host:port for http/artifact server. Optional if SM_API env is set.

    -user="": Mesos user. Defaults to current system user.

    -log.level="info": Log level. trace|debug|info|warn|error|critical. Defaults to info.

    -framework.name="syscol": Framework name.

    -framework.role="*": Framework role.

Starting and Stopping a Server

------------------------------

    # ./cli start|stop 

Options available:

    -api="": Binding host:port for http/artifact server. Optional if SM_API env is set.

Updating Server Preferences

---------------------------

    # ./cli update 

Following options are available:

    -api="": Binding host:port for http/artifact server. Optional if SM_API env is set.

    -producer.properties="": Producer.properties file name.

    -topic="": Topic to produce data to.

    -transform="": Transofmation to apply to each metric. none|avro

    -schema.registry.url="": Avro Schema Registry url for transform=avro

    -reporting.

Quick start:

-----------

```

# export SM_API=http://master:6666

# ./cli scheduler --master master:5050

# ./cli update --producer.properties producer.properties --topic metrics

# ./cli start

```

By now you should see metrics from each Mesos slave getting produced to Kafka with reporting interval of 1 second (*TODO* configurable reporting interval)