https://github.com/streamr-dev/streamr-spark
Streamr Integration library for Apache Spark
https://github.com/streamr-dev/streamr-spark
Last synced: 7 months ago
JSON representation
Streamr Integration library for Apache Spark
- Host: GitHub
- URL: https://github.com/streamr-dev/streamr-spark
- Owner: streamr-dev
- License: apache-2.0
- Created: 2019-09-11T12:57:48.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-10-08T11:48:39.000Z (over 6 years ago)
- Last Synced: 2025-08-09T23:48:43.156Z (7 months ago)
- Language: Java
- Size: 18.6 KB
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Streamr - Apache Spark integration library
# How to use (maven):
Add this to your pom.xml file:
```
com.streamr.labs
streamr_spark
0.2
```
Then import the library with:
```
import com.streamr.labs.streamr_spark.StreamrReceiver;
```
## Spark Streaming example
```
public class Streamrspark {
public static void main(String[] args) {
SparkConf conf = new SparkConf();
conf.setAppName("Streamrspark");
JavaStreamingContext jssc =
new JavaStreamingContext(conf, Durations.seconds(1));
JavaDStream streamrReceiverStream = jssc.receiverStream(
new StreamrReceiver("STREAMR_API_KEY","STREAM_ID"));
JavaDStream filtered = streamrReceiverStream.filter(
new Function() {
@Override
public Boolean call(String s) throws Exception {
return s.contains("6T");
}
});
filtered.count().print();
jssc.start();
try {
jssc.awaitTermination();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
```
## Running
You have to add all these packages to your spark-submit script or shade the package. The requirement manual imports will be fixed in the future.
```
--packages com.streamr.labs:streamr_spark:0.2,com.streamr:client:1.1.0,org.apache.logging.log4j:log4j-core:2.9.0,org.apache.logging.log4j:log4j-api:2.9.0,org.apache.logging.log4j:log4j-slf4j-impl:2.9.0
```
For examples go to https://github.com/streamr-dev/streamr-spark-integrations