Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/crccheck/cloudwatch-to-graphite

Helper for pushing AWS CloudWatch metrics to Graphite
https://github.com/crccheck/cloudwatch-to-graphite

Last synced: about 2 months ago
JSON representation

Helper for pushing AWS CloudWatch metrics to Graphite

Awesome Lists containing this project

README 

        
Cloudwatch-to-Graphite

======================



.. image:: https://travis-ci.org/crccheck/cloudwatch-to-graphite.svg

    :target: https://travis-ci.org/crccheck/cloudwatch-to-graphite



Cloudwatch-to-Graphite (leadbutt) is a small utility to take metrics from

CloudWatch to Graphite.



Installation

------------



Install using pip::



    pip install cloudwatch-to-graphite



Configuring ``boto``

~~~~~~~~~~~~~~~~~~~~



Cloudwatch-to-Graphite uses `boto`_, so make sure to follow its `configuration

instructions`_. The easiest way to do this is to set up the

``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` environment variables.



.. _configuration instructions: http://boto.readthedocs.org/en/latest/boto_config_tut.html



Usage

-----



Configuration Files

~~~~~~~~~~~~~~~~~~~



If you have a simple setup, the easiest way to get started is to set up a

config.yaml. You can copy the included config.yaml.example. Then just run::



    leadbutt



If you have several configs you want to switch between, you can specify a

custom configuration file::



    leadbutt --config-file=production.yaml -n 20



You can even generate configs on the fly and send them in via stdin by setting

the config file to '-'::



    generate_config_from_inventory | leadbutt --config-file=-



There's a helper to generate configuration files called ``plumbum``.  Use it like::



    plumbum [-r REGION] [-f FILTER] [--token TOKEN] template namespace



Namespace is the CloudWatch namespace for the resources of interest; for example ``AWS/RDS``.

The template is a Jinja2 template. You can add arbitrary replacement tokens, eg ``{{ replace_me }}``, and then

pass in values on the CLI via ``--token``. For example, if you called::



    plumbum --token replace_me='hello, world' sample_templates/rds.yml.j2 AWS/RDS



You would get all instances of ``{{ replace_me }}`` in the templace replaced with ``hello, world``.



Filters

~~~~~~~



You can pass simple ``key=value`` filters in to ``plumbum``; be aware of the limitations:



* the filters run against whatever the AWS API has returned; if you have a lot of objects of whatever type, expect the API request to take a while.

* they work only against object attributes and tags returned by the API. For example, RDS and ELB objects can be tagged, but as getting the tags is a per-object subrequest; ``plumbum`` does not do those, so you can only filter on the object attributes.



Example: ``plumbum -f Name=my-dev-instance sample_templates/ec2.yml.j2 ec2``



Sending Data to Graphite

~~~~~~~~~~~~~~~~~~~~~~~~



If your graphite server is at graphite.local, you can send metrics by chaining

with netcat::



    leadbutt | nc -q0 graphite.local 2003



Or if you want to use UDP::



    leadbutt | nc -uw0 graphite.local 2003



If you need to namespace your metrics for a hosted Graphite provider, you could

provide a custom formatter, but the easiest way is to just run the output

through awk::



    leadbutt | \

      awk -v namespace="$HOSTEDGRAPHITE_APIKEY" '{print namespace"."$0}' | \

      nc -uw0 my-graphite-provider.xxx 2003



Customizing Your Graphite Metric Names

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Set the ``Formatter`` option to set the template used to generate Graphite

metric names. I wasn't sure what should be default, so I copied

`cloudwatch2graphite`_'s. Here's what it looks like::



    cloudwatch.%(Namespace)s.%(dimension)s.%(MetricName)s.%(statistic)s.%(Unit)s



TitleCased variables come directly from the YAML configuration, while lowercase

variables are derived:



* **statistic** -- the current statistic since ``Statistics`` can be a list

* **dimension** -- the dimension value, e.g. "i-r0b0t" or "my-load-balancer"



The format string is Python's `%-style `_.



config.yaml

-----------



What metrics are pulled is in a YAML configuration file. See the example

config.yaml.example for an idea of what you can do.



Developing

----------



See: : `Contributing `__.



Useful References

-----------------



* `CloudWatch Reference `_

* `boto CloudWatch docs `_



Prior Art

---------



Cloudwatch-to-Graphite was inspired by edasque's `cloudwatch2graphite`_. I was

looking to expand it, but I wanted to use `boto`_.



.. _cloudwatch2graphite: https://github.com/edasque/cloudwatch2graphite

.. _boto: https://boto.readthedocs.org/en/latest/