{"id":24455797,"url":"https://github.com/nashtech-labs/spark-streaming-gnip","last_synced_at":"2025-04-13T02:29:32.548Z","repository":{"id":34310195,"uuid":"38227202","full_name":"NashTech-Labs/spark-streaming-gnip","owner":"NashTech-Labs","description":"An Apache Spark utility for pulling Tweets from Gnip's PowerTrack in realtime","archived":false,"fork":false,"pushed_at":"2015-07-27T11:07:58.000Z","size":133,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-26T20:01:35.252Z","etag":null,"topics":["gnip","gnip-powertrack","knoldus","pulling-tweets","realtime","scala","spark","spark-streaming","spark-utility","sparkconf","sparkcontext","tweets"],"latest_commit_sha":null,"homepage":"http://spark-packages.org/package/knoldus/spark-streaming-gnip","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NashTech-Labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-29T04:09:20.000Z","updated_at":"2023-08-04T08:57:40.000Z","dependencies_parsed_at":"2022-09-03T01:05:00.306Z","dependency_job_id":null,"html_url":"https://github.com/NashTech-Labs/spark-streaming-gnip","commit_stats":null,"previous_names":["nashtech-labs/spark-streaming-gnip"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NashTech-Labs%2Fspark-streaming-gnip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NashTech-Labs%2Fspark-streaming-gnip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NashTech-Labs%2Fspark-streaming-gnip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NashTech-Labs%2Fspark-streaming-gnip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NashTech-Labs","download_url":"https://codeload.github.com/NashTech-Labs/spark-streaming-gnip/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248657732,"owners_count":21140839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gnip","gnip-powertrack","knoldus","pulling-tweets","realtime","scala","spark","spark-streaming","spark-utility","sparkconf","sparkcontext","tweets"],"created_at":"2025-01-21T02:14:22.123Z","updated_at":"2025-04-13T02:29:32.527Z","avatar_url":"https://github.com/NashTech-Labs.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark-Streaming-Gnip\n========================\n\nAn Apache Spark utility for pulling Tweets from Gnip's PowerTrack in realtime.\n\n-----------------------------------------------------------------------\n### Discription\n-----------------------------------------------------------------------\n\nThis utility will help to pull tweets from Gnip using Spark Streaming. The Gnip, Inc. provides data from dozens of social media websites via a single API. It is also known as the Grand Central Station for social media web.\n\nThis code have implemented a Custom Receiver which uses Spark Streaming to fetch tweets from Gnip and 'store' it in Spark BlockManager.\n\nThe logic will automatically store tweets from Gnip for the interval of time specified in Spark Streaming Context.\n\nPlease see Scala code example on how to use this Spark Streaming Gnip utility.\n\n-----------------------------------------------------------------------\n### Scala Example\n-----------------------------------------------------------------------\n\n\timport org.apache.spark.SparkConf\n\timport org.apache.spark.SparkContext\n\timport org.apache.spark.streaming.Seconds\n\timport org.apache.spark.streaming.StreamingContext\n\timport com.knoldus.spark.streaming.gnip.GnipUtils\n \n\tobject GnipStreaming extends App {\n \n \t\tval sparkConf: SparkConf = new SparkConf().setAppName(\"spark-streaming-gnip\").setMaster(\"local[*]\")\n \t\tval sc: SparkContext = new SparkContext(sparkConf)\n \t\tval ssc: StreamingContext = new StreamingContext(sc, Seconds(10))\n \t\tval gnipDstream = GnipUtils.createStream(ssc, \"https://stream.gnip.com:443/accounts/Knoldus/publishers/twitter/streams/track/prod.json\", \"knoldus@knoldus.com\", \"knoldus\")\n \n \t\tgnipDstream.print()\n \n \t\tssc.start()\n \t\tssc.awaitTermination()\n \n\t}\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnashtech-labs%2Fspark-streaming-gnip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnashtech-labs%2Fspark-streaming-gnip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnashtech-labs%2Fspark-streaming-gnip/lists"}