https://github.com/pierrekieffer/sparkstreaming_kafkaconsumer

Kafka consumer example based on spark streaming with message formatting to spark dataframe
https://github.com/pierrekieffer/sparkstreaming_kafkaconsumer

kafka kafka-consumer scala spark spark-streaming

Last synced: 5 days ago
JSON representation

Kafka consumer example based on spark streaming with message formatting to spark dataframe

• Host: GitHub
• URL: https://github.com/pierrekieffer/sparkstreaming_kafkaconsumer
• Owner: PierreKieffer
• License: apache-2.0
• Created: 2019-09-10T13:00:52.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2019-09-10T13:18:14.000Z (over 5 years ago)
• Last Synced: 2023-03-07T01:31:42.017Z (almost 2 years ago)
• Topics: kafka, kafka-consumer, scala, spark, spark-streaming
• Language: Scala
• Homepage:
• Size: 8.79 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# KafkaConsumer

Kafka consumer example based on spark streaming  with message formatting to spark dataframe. 

- Input (From Kafka) : 

```

Row01,Value1,value2,value3,value4

Row1,Value11,value12,value13,value14

Row2,Value21,value22,value23,value24

Row3,Value31,value32,value33,value34

```

- Output :

```

+-----+-------+-------+-------+-------+

|  ROW|   VAL1|   VAL2|   VAL3|   VAL4|

+-----+-------+-------+-------+-------+

|Row01| Value1| value2| value3| value4|

| Row1|Value11|value12|value13|value14|

| Row2|Value21|value22|value23|value24|

| Row3|Value31|value32|value33|value34|

+-----+-------+-------+-------+-------+

```