https://github.com/spektom/spark-http-stream

Spark streaming HTTP direct stream implementation and usage example
https://github.com/spektom/spark-http-stream

http spark spark-streaming

Last synced: 3 months ago
JSON representation

Spark streaming HTTP direct stream implementation and usage example

• Host: GitHub
• URL: https://github.com/spektom/spark-http-stream
• Owner: spektom
• Created: 2017-07-27T19:22:38.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2017-07-27T19:29:04.000Z (almost 8 years ago)
• Last Synced: 2025-01-20T16:53:27.040Z (4 months ago)
• Topics: http, spark, spark-streaming
• Language: Scala
• Size: 244 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
spark-http-stream

==================

The following example shows Spark streaming application, which reads

from HTTP sources periodically, and processes downloaded content.

The input stream adapts automatically to the processing rate due to

Spark streaming backpressure feature. The feature can be seen in action on this picture:

![](bp.png)

## Building

To build Spark application jar, run:

    ~$ mvn package

    

## Running

First, run some dummy multi-threaded Web server that listens on port 8000.

Then, submit the application jar using the following command:

 

    ~$ spark-submit --class com.github.spektom.spark.TestJob target/spark-http-stream_2.11-0.0.1.jar

    

To view Spark streaming statistics, open this URL in your browser: [http://localhost:4040/streaming](http://localhost:4040/streaming)