https://github.com/fluent/fluent-plugin-flume
Flume input and output plugin for Fluentd
https://github.com/fluent/fluent-plugin-flume
fluentd fluentd-plugin flume
Last synced: 5 months ago
JSON representation
Flume input and output plugin for Fluentd
- Host: GitHub
- URL: https://github.com/fluent/fluent-plugin-flume
- Owner: fluent
- Created: 2012-05-02T23:22:08.000Z (almost 14 years ago)
- Default Branch: master
- Last Pushed: 2017-07-06T01:19:30.000Z (over 8 years ago)
- Last Synced: 2025-10-09T11:54:35.717Z (5 months ago)
- Topics: fluentd, fluentd-plugin, flume
- Language: Ruby
- Homepage:
- Size: 43.9 KB
- Stars: 24
- Watchers: 18
- Forks: 13
- Open Issues: 2
-
Metadata Files:
- Readme: README.rdoc
- Changelog: ChangeLog
Awesome Lists containing this project
README
= Flume input/output plugin for Fluentd[http://fluentd.org/]
== Overview
This is a plugin for fluentd[http://fluentd.org/] data collector. This plugin adds the Flume[https://github.com/apache/flume] compatible interface to fluentd.
== What's Flume?
Flume[https://github.com/apache/flume] is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
It uses Thrift[http://thrift.apache.org/], a cross-language RPC framework, to communicate between clients and servers.
== What's Flume plugin for fluent?
The Flume plugin for fluentd, which enables fluentd to talk the Flume protocol. Flume protocol is defined as follows, in Thrift-IDL format:
typedef i64 Timestamp
enum Priority {
FATAL = 0,
ERROR = 1,
WARN = 2,
INFO = 3,
DEBUG = 4,
TRACE = 5
}
enum EventStatus {
ACK = 0,
COMMITED = 1,
ERR = 2
}
struct ThriftFlumeEvent {
1: Timestamp timestamp,
2: Priority priority,
3: binary body,
4: i64 nanos,
5: string host,
6: map fields
}
# Instead of using thrift's serialization, we just assume the contents are serialized already.
struct RawEvent {
1: binary raw
}
service ThriftFlumeEventServer {
oneway void append( 1:ThriftFlumeEvent evt ),
oneway void rawAppend( 1:RawEvent evt),
EventStatus ackedAppend( 1: ThriftFlumeEvent evt ),
void close(),
}
A value that is stored in the ThriftFlumeEvent.fields map is used as fluentd 'tag'. A key of the value enables be specified by users as configuration parameter.
== How to use?
fluent-plugin-flume contains both input and output.
== Flume Input
Please add the following configurations to fluent.conf.
# Flume input
type flume
port 56789
These options are supported.
* port: port number (default: 56789)
* bind: bind address (default: 0.0.0.0)
* server_type: server architecture either in 'simple', 'threaded', 'thread_pool', (default: simple)
* is_framed: use framed protocol or not (default: false)
* tag_field: key name of fluentd 'tag' that is stored in ThriftFlumeEvent.fields (default: nil)
* default_tag: default fluentd 'tag' (default: 'category')
* add_prefix: prefix string, added to the tag (default: nil)
Use flume-ng-fluentd-sink[https://github.com/cosmo0920/flume-ng-fluentd-sink] to send events from flume-ng to fluentd.
== Flume Output
Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Flume server. Note that fluentd conveys semi-structured data while Flume conveys unstructured data. Thus the plugin translates semi-structured data into JSON data by default and conveys it to Flume. The format can be adjusted via formatters.
# Flume output
type flume
host flume-host.local
port 56789
These options are supported.
* host: host name or address (default: localhost)
* port: port number (default: 35863)
* timeout: thrift protocol timeout (default: 30)
* remove_prefix: prefix string, removed from the tag (default: nil)
* format: The format of the thrift body (default: json)
* trim_nl: Trim new line from thrift body (default: true)
== Contributors
== Copyright
Copyright:: Copyright (c) 2012 - 2013 Treasure Data, Inc.
License:: Apache License, Version 2.0