https://github.com/joker1007/fluent-plugin-arrow

https://github.com/joker1007/fluent-plugin-arrow

Last synced: 2 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/joker1007/fluent-plugin-arrow
• Owner: joker1007
• License: apache-2.0
• Created: 2018-11-28T10:58:11.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2019-02-01T11:39:28.000Z (almost 6 years ago)
• Last Synced: 2024-10-04T14:17:23.595Z (about 1 month ago)
• Language: Ruby
• Size: 24.4 KB
• Stars: 0
• Watchers: 3
• Forks: 3
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# fluent-plugin-arrow

[Fluentd](https://fluentd.org/) buffer plugin to output Apache Arrow and Parquet format.

## Prerequisite

- [Apache Arrow c++](https://github.com/apache/arrow/tree/master/cpp) (with -DARROW_PARQUET=ON)

- [Apache Arrow c_glib](https://github.com/apache/arrow/tree/master/c_glib)

- [red-arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow)

- [red-parquet](https://github.com/apache/arrow/tree/master/ruby/red-parquet)

## Installation

### RubyGems

```

$ gem install fluent-plugin-arrow

```

### Bundler

Add following line to your Gemfile:

```ruby

gem "fluent-plugin-arrow"

```

And then execute:

```

$ bundle

```

## Configuration

You can generate configuration template:

```

  @type file

  path arrow_test

  

    @type arrow_memory

    arrow_format arrow # or parquet

    schema [

      {"name": "key1", "type": "string"},

      {"name": "key2", "type": "uint64"},

      {"name": "key3", "type": "timestamp", "unit": "milli"},

      {"name": "key4", "type": "list", "field": {"name": "value", "type": "uint64"}},

      {"name": "key5", "type": "struct", "fields": [

        {"name": "bar1", "type": "uint64"},

        {"name": "bar2", "type": "list", "field": {"name": "value", "type": "string"}}

      ]}

    ]

  

  

    @type arrow

  

```

You can copy and paste generated documents here.

## Copyright

* Copyright(c) 2018- joker1007

* License

  * Apache License, Version 2.0