https://github.com/joker1007/fluent-plugin-arrow
https://github.com/joker1007/fluent-plugin-arrow
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/joker1007/fluent-plugin-arrow
- Owner: joker1007
- License: apache-2.0
- Created: 2018-11-28T10:58:11.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-02-01T11:39:28.000Z (over 6 years ago)
- Last Synced: 2025-07-27T17:57:21.542Z (3 months ago)
- Language: Ruby
- Size: 24.4 KB
- Stars: 0
- Watchers: 2
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# fluent-plugin-arrow
[Fluentd](https://fluentd.org/) buffer plugin to output Apache Arrow and Parquet format.
## Prerequisite
- [Apache Arrow c++](https://github.com/apache/arrow/tree/master/cpp) (with -DARROW_PARQUET=ON)
- [Apache Arrow c_glib](https://github.com/apache/arrow/tree/master/c_glib)
- [red-arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow)
- [red-parquet](https://github.com/apache/arrow/tree/master/ruby/red-parquet)
## Installation
### RubyGems
```
$ gem install fluent-plugin-arrow
```
### Bundler
Add following line to your Gemfile:
```ruby
gem "fluent-plugin-arrow"
```
And then execute:
```
$ bundle
```
## Configuration
You can generate configuration template:
```
@type file
path arrow_test
@type arrow_memory
arrow_format arrow # or parquet
schema [
{"name": "key1", "type": "string"},
{"name": "key2", "type": "uint64"},
{"name": "key3", "type": "timestamp", "unit": "milli"},
{"name": "key4", "type": "list", "field": {"name": "value", "type": "uint64"}},
{"name": "key5", "type": "struct", "fields": [
{"name": "bar1", "type": "uint64"},
{"name": "bar2", "type": "list", "field": {"name": "value", "type": "string"}}
]}
]
@type arrow
```
You can copy and paste generated documents here.
## Copyright
* Copyright(c) 2018- joker1007
* License
* Apache License, Version 2.0