{"id":21399300,"url":"https://github.com/streamr-dev/streamr-spark","last_synced_at":"2025-08-13T07:14:06.081Z","repository":{"id":57727310,"uuid":"207813078","full_name":"streamr-dev/streamr-spark","owner":"streamr-dev","description":"Streamr Integration library for Apache Spark","archived":false,"fork":false,"pushed_at":"2019-10-08T11:48:39.000Z","size":19,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-08-09T23:48:43.156Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamr-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-09-11T12:57:48.000Z","updated_at":"2019-09-16T10:41:02.000Z","dependencies_parsed_at":"2022-09-26T21:51:21.576Z","dependency_job_id":null,"html_url":"https://github.com/streamr-dev/streamr-spark","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/streamr-dev/streamr-spark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamr-dev%2Fstreamr-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamr-dev%2Fstreamr-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamr-dev%2Fstreamr-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamr-dev%2Fstreamr-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamr-dev","download_url":"https://codeload.github.com/streamr-dev/streamr-spark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamr-dev%2Fstreamr-spark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270201764,"owners_count":24544197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T15:14:02.239Z","updated_at":"2025-08-13T07:14:06.006Z","avatar_url":"https://github.com/streamr-dev.png","language":"Java","readme":"# Streamr - Apache Spark integration library\n\n# How to use (maven):\n\nAdd this to your pom.xml file:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.streamr.labs\u003c/groupId\u003e\n  \u003cartifactId\u003estreamr_spark\u003c/artifactId\u003e\n  \u003cversion\u003e0.2\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThen import the library with:\n\n```\nimport com.streamr.labs.streamr_spark.StreamrReceiver;\n```\n\n## Spark Streaming example\n\n```\npublic class Streamrspark {\n\n      public static void main(String[] args) {\n      SparkConf conf = new SparkConf();\n      conf.setAppName(\"Streamrspark\");\n      JavaStreamingContext jssc = \n        new JavaStreamingContext(conf, Durations.seconds(1));\n      JavaDStream\u003cString\u003e streamrReceiverStream = jssc.receiverStream(\n        new StreamrReceiver(\"STREAMR_API_KEY\",\"STREAM_ID\"));\n\n      JavaDStream\u003cString\u003e filtered = streamrReceiverStream.filter(\n          new Function\u003cString, Boolean\u003e() {\n            @Override\n            public Boolean call(String s) throws Exception {\n                return s.contains(\"6T\");\n            }\n        });\n      filtered.count().print();\n\n      jssc.start();\n      try {\n          jssc.awaitTermination();\n      } catch (InterruptedException e) {\n          e.printStackTrace();\n      }\n  }\n\n}\n\n```\n\n## Running\n\nYou have to add all these packages to your spark-submit script or shade the package. The requirement manual imports will be fixed in the future. \n\n```\n--packages com.streamr.labs:streamr_spark:0.2,com.streamr:client:1.1.0,org.apache.logging.log4j:log4j-core:2.9.0,org.apache.logging.log4j:log4j-api:2.9.0,org.apache.logging.log4j:log4j-slf4j-impl:2.9.0\n```\nFor examples go to https://github.com/streamr-dev/streamr-spark-integrations\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamr-dev%2Fstreamr-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamr-dev%2Fstreamr-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamr-dev%2Fstreamr-spark/lists"}