https://github.com/neuw84/spark-continuous-streaming

Spark 2.3. End to End Avro Continous Structured Streaming Kafka demo using Twitter´s bijection in Java.
https://github.com/neuw84/spark-continuous-streaming

avro bijection kafka spark sparksql structured-streaming

Last synced: 3 months ago
JSON representation

Spark 2.3. End to End Avro Continous Structured Streaming Kafka demo using Twitter´s bijection in Java.

• Host: GitHub
• URL: https://github.com/neuw84/spark-continuous-streaming
• Owner: Neuw84
• License: bsd-3-clause
• Created: 2018-03-09T09:57:39.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2018-03-09T10:09:53.000Z (over 8 years ago)
• Last Synced: 2025-04-02T16:52:17.321Z (about 1 year ago)
• Topics: avro, bijection, kafka, spark, sparksql, structured-streaming
• Language: Java
• Size: 60.5 KB
• Stars: 0
• Watchers: 2
• Forks: 6
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Continous Structured Streaming Avro demo

Spark 2.3. End to End Avro Continous Structured Streaming Kafka demo using Twitter´s bijection in Java. 

## Requirements 

* Kafka: 1.0.0

* Java 8

* Maven

## Architecture

![Architecture](https://raw.githubusercontent.com/Neuw84/spark-continuous-streaming/master/architecture.png)

 

## Usage

Configure kafka hosts and execute both the ```ProducerConsumerDemo``` and the ```StructuredDemo``` classes.

Latency measures can be seen on ```ProducerConsumerDemo```'s output.