https://github.com/chezou/sparkavro

Load Avro data into Spark with sparklyr
https://github.com/chezou/sparkavro

avro r spark sparklyr

Last synced: 3 months ago
JSON representation

Load Avro data into Spark with sparklyr

• Host: GitHub
• URL: https://github.com/chezou/sparkavro
• Owner: chezou
• License: apache-2.0
• Archived: true
• Created: 2017-03-15T14:16:55.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2020-06-04T04:32:32.000Z (over 4 years ago)
• Last Synced: 2024-05-01T23:58:46.690Z (6 months ago)
• Topics: avro, r, spark, sparklyr
• Language: R
• Size: 298 KB
• Stars: 12
• Watchers: 3
• Forks: 7
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-sparklyr - sparkavro: Load Avro data into Spark with sparklyr

README

        
[![Travis-CI Build Status](https://travis-ci.org/chezou/sparkavro.svg?branch=master)](https://travis-ci.org/chezou/sparkavro)

# sparkavro

Load Avro data into Spark with sparklyr. It is a wrapper of [spark-avro](https://github.com/databricks/spark-avro)

## Installation

Install using `{devtools}` as follows:

```r

devtools::install_github("chezou/sparkavro")

```

## Usage

```r

library(sparklyr)

library(sparkavro)

sc <- spark_connect(master = "spark://HOST:PORT")

df <- spark_read_avro(sc, "test_table", "/user/foo/test.avro")

spark_write_avro(df, "/tmp/output")

```

Example data are from https://github.com/miguno/avro-cli-examples