https://github.com/rollulus/kafka-streams-csv2avro

A generic csv to Avro mapper
https://github.com/rollulus/kafka-streams-csv2avro

Last synced: 2 months ago
JSON representation

A generic csv to Avro mapper

• Host: GitHub
• URL: https://github.com/rollulus/kafka-streams-csv2avro
• Owner: rollulus
• Created: 2016-04-25T14:14:05.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2016-04-29T16:20:20.000Z (almost 10 years ago)
• Last Synced: 2025-05-20T15:47:24.277Z (9 months ago)
• Language: Scala
• Size: 18.6 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
Generic Csv to Avro mapper

==========================

[![Build Status](https://travis-ci.org/rollulus/kafka-streams-csv2avro.svg?branch=master)](https://travis-ci.org/rollulus/kafka-streams-csv2avro)

Maps csv records to Avro records.

Proof of concept.

Work in progress. 

Input

-----

A stream with String values, each String is a csv record.

Output

------

A stream with Avro records. The fields of the record correspond to the csv values.

Quick Example

-------------

For example, provided that this Avro schema is given in the configuration:

```json

{"type": "record", "name": "Person", "fields" : [

    {"name": "firstName", "type": "string"},

    {"name": "lastName", "type": "string"},

    {"name": "length", "type": "float"},

    {"name": "awesome", "type": "boolean"}]}

```

And that the .csv layout is described like:

```properties

csv.separator=,

csv.columns=awesome,length,,firstName,lastName

```

Then the stream processor will map inputs like this:

    true,1.96,male,Rollulus,Rouloul

To an Avro record whose JSON representation is:

    {"firstName":"Rollulus","lastName":"Rouloul","length":1.96,"awesome":true}