https://github.com/fluent/fluent-plugin-scribe
Scribe input/output plugin for Fluentd data collector
https://github.com/fluent/fluent-plugin-scribe
Last synced: 8 months ago
JSON representation
Scribe input/output plugin for Fluentd data collector
- Host: GitHub
- URL: https://github.com/fluent/fluent-plugin-scribe
- Owner: fluent
- Created: 2011-09-02T20:48:33.000Z (over 14 years ago)
- Default Branch: master
- Last Pushed: 2018-12-12T02:07:29.000Z (about 7 years ago)
- Last Synced: 2025-06-05T22:05:37.076Z (9 months ago)
- Language: Ruby
- Homepage: http://docs.fluentd.org/articles/in_scribe
- Size: 77.1 KB
- Stars: 22
- Watchers: 15
- Forks: 18
- Open Issues: 3
-
Metadata Files:
- Readme: README.rdoc
- Changelog: ChangeLog
Awesome Lists containing this project
README
= Scribe input plugin for Fluentd[http://fluentd.org/]
== Overview
This is a plugin for fluentd[http://fluentd.org] data collector. This plugin adds the Scribe[https://github.com/facebook/scribe] compatible interface to fluentd.
== What's Scribe?
Scribe[https://github.com/facebook/scribe] is a server for aggregating log data streamed in real time from a large number of servers, developed at Facebook.
It uses Thrift[http://thrift.apache.org/], a cross-language RPC framework, to communicate between clients and servers.
== What's Scribe plugin for fluent?
The Scribe plugin for fluentd, which enables fluentd to talk the Scribe protocol. Scribe protocol is defined as follows, in Thrift-IDL format:
enum ResultCode
{
OK,
TRY_LATER
}
struct LogEntry
{
1: string category,
2: string message
}
service scribe extends fb303.FacebookService
{
ResultCode Log(1: list messages);
}
The category field is used as fluentd 'tag'.
== How to use?
fluent-plugin-scribe contains both input and output.
=== Scribe Input
Please add the following configurations to fluent.conf. This allows your Scribe client to import logs through port 1463.
# Scribe input
type scribe
port 1463
These options are supported.
* port: port number (default: 1463)
* bind: bind address (default: 0.0.0.0)
* server_type: server architecture either in 'simple', 'threaded', 'thread_pool', 'nonblocking' (default: nonblocking)
* is_framed: use framed protocol or not (default: true)
* add_prefix: prefix string, added to the tag (default: nil)
* msg_format: format of the messages either in 'text', 'json', 'url_param' (default: text)
=== Scribe Output
Please add the following configurations to fluent.conf. This allows fluentd to output its logs into another Scribe server. Note that fluentd conveys semi-structured data while Scribe conveys unstructured data, thus, 'field_ref' parameter is prepared to specify which field will be transferred.
# Scribe output
type scribe
host scribed-host.local
port 1463
field_ref message
These options are supported.
* host: host name or address (default: localhost)
* port: port number (default: 1463)
* field_ref: field name which sent as scribe log message (default: message)
* timeout: thrift protocol timeout (default: 30)
* format_to_json: if true/yes, format entire record as json, and send as message (default: false)
* remove_prefix: prefix string, removed from the tag (default: nil)
== For Developers
To run fluentd with this plugin on chaging,
$ bundle # (or 'bundle update')
$ bundle exec fluentd -v -v -v -c example.conf
Then please execute the sample client.
$ bundle exec bin/fluent-scribe-remote
== Contributors
* {Satoshi Tagomori}[https://github.com/tagomoris]
* {Sadayuki Furuhashi}[https://github.com/frsyuki]
* {Ken Robertson}[https://github.com/krobertson]
== Copyright
Copyright:: Copyright (c) 2011 Treasure Data, Inc.
License:: Apache License, Version 2.0