{"id":18842814,"url":"https://github.com/aamend/spark-gdelt","last_synced_at":"2025-04-14T07:31:58.762Z","repository":{"id":44662389,"uuid":"126642140","full_name":"aamend/spark-gdelt","owner":"aamend","description":"Binding the GDELT universe in a Spark environment","archived":false,"fork":false,"pushed_at":"2023-04-21T20:43:58.000Z","size":5882,"stargazers_count":23,"open_issues_count":3,"forks_count":10,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-27T21:21:55.043Z","etag":null,"topics":["analytics","gdelt","news","parser","spark","structured-streaming"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aamend.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-24T21:34:41.000Z","updated_at":"2024-12-16T08:36:11.000Z","dependencies_parsed_at":"2022-08-23T14:31:09.091Z","dependency_job_id":null,"html_url":"https://github.com/aamend/spark-gdelt","commit_stats":null,"previous_names":["aamend/gdelt-spark"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aamend%2Fspark-gdelt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aamend%2Fspark-gdelt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aamend%2Fspark-gdelt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aamend%2Fspark-gdelt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aamend","download_url":"https://codeload.github.com/aamend/spark-gdelt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248839466,"owners_count":21169817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","gdelt","news","parser","spark","structured-streaming"],"created_at":"2024-11-08T02:55:46.968Z","updated_at":"2025-04-14T07:31:57.750Z","avatar_url":"https://github.com/aamend.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gdelt Spark\n\n[![Build Status](https://travis-ci.org/aamend/gdelt-spark.svg?branch=master)](https://travis-ci.org/aamend/spark-gdelt) \n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.aamend.spark/spark-gdelt/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.aamend.spark.spark-gdelt)\n\n## GDELT (global database of events language and tone)\n\n![gdelt-spark](/images/gdelt.png)\n\n[GDELT Project](https://www.gdeltproject.org/): \n*The GDELT project monitors the world's broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day, creating a free open platform for computing on the entire world.*\n\nThe GDELT universe being quite large, its data format is by essence complex and convoluted. Official documentation can be found here:\n* [http://data.gdeltproject.org/documentation/GDELT-Data_Format_Codebook.pdf](http://data.gdeltproject.org/documentation/GDELT-Data_Format_Codebook.pdf)\n* [http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook.pdf](http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook.pdf)\n* [http://data.gdeltproject.org/documentation/CAMEO.Manual.1.1b3.pdf](http://data.gdeltproject.org/documentation/CAMEO.Manual.1.1b3.pdf)\n\n## gdelt-spark\n\nThis project has been built to make [GDELT v2](https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/) environment easy to load on a Spark based environment. \nWhile the `v1.0` version only came with a GDELT data model (case classes), a set of parsers and all its reference data, `v2.0` embeds the [Goose](https://github.com/GravityLabs/goose/wiki) library packaged for `Scala 2.11`. \nIt is planned to later enrich this library with advance analytics and libraries I gathered / created over the past few years. \n\n### Getting Started\n\n_spark-gdelt_ project has been built for __Scala 2.11__ and __Spark 2.1.0__ and is available on Maven central.\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.aamend.spark\u003c/groupId\u003e\n  \u003cartifactId\u003espark-gdelt\u003c/artifactId\u003e\n  \u003cversion\u003ex.y.z\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nAlso available as a [spark package](https://spark-packages.org/package/aamend/gdelt-spark), include this package in your Spark Applications as follows\n\n```bash\nspark-shell --packages com.aamend.spark:spark-gdelt:x.y.z\n```\n\n### Usage\n\n### GDELT core feeds\nLoading core GDELT files (unzipped) as `Dataset[_]`. \nNote we support both english and translingual files, V2 only. \n\n```scala\nimport com.aamend.spark.gdelt._\n\nval gdeltEventDS: Dataset[Event] = spark.read.gdeltEvent(\"/path/to/event.csv\")\nval gdeltGkgDS: Dataset[GKGEvent] = spark.read.gdeltGkg(\"/path/to/gkg.csv\")\nval gdeltMention: Dataset[Mention] = spark.read.gdeltMention(\"/path/to/mention.csv\")\n```\n\nHere is an example of `Dataset[Event]`\n\n```\n+---------+----------+--------------------+--------------------+------+--------------+------------------+------------------+--------------------+---------+-----------+----------+-----------+-----------+--------------------+--------------------+--------------------+----------+--------------------+\n|  eventId|  eventDay|          actor1Code|          actor2Code|isRoot|cameoEventCode|cameoEventBaseCode|cameoEventRootCode|           quadClass|goldstein|numMentions|numSources|numArticles|    avgTone|           actor1Geo|           actor2Geo|            eventGeo| dateAdded|           sourceUrl|\n+---------+----------+--------------------+--------------------+------+--------------+------------------+------------------+--------------------+---------+-----------+----------+-----------+-----------+--------------------+--------------------+--------------------+----------+--------------------+\n|741462369|2008-03-25|[USA,UNITED STATE...|         [,,,,,,,,,]| false|           050|               050|                05|  VERBAL_COOPERATION|      3.5|          4|         1|          4|  1.1655011|[COUNTRY,United S...|[UNKNOWN,,,,,[NaN...|[COUNTRY,United S...|2018-03-23|https://www.citiz...|\n|741462370|2017-03-23|         [,,,,,,,,,]|[USA,CALIFORNIA,U...|  true|           100|               100|                10|     VERBAL_CONFLICT|     -5.0|          3|         1|          3| -1.2836186|[UNKNOWN,,,,,[NaN...|[USCITY,San Franc...|[USCITY,San Franc...|2018-03-23|https://www.nytim...|\n|741462371|2017-03-23|         [,,,,,,,,,]|[USACVL,NORTH CAR...|  true|           043|               043|                04|  VERBAL_COOPERATION|      2.8|          8|         1|          8|  0.9041591|[UNKNOWN,,,,,[NaN...|[USSTATE,North Ca...|[USSTATE,North Ca...|2018-03-23|http://www.charlo...|\n|741462372|2017-03-23|[EDU,INTELLECTUAL...|[CHN,CHINA,CHN,,,...|  true|           111|               111|                11|     VERBAL_CONFLICT|     -2.0|         10|         1|         10| -2.2959185|[WORLDCITY,Guangz...|[WORLDCITY,Guangz...|[WORLDCITY,Guangz...|2018-03-23|https://www.teleg...|\n|741462373|2017-03-23|[GOV,GOVERNMENT,,...|         [,,,,,,,,,]| false|           014|               014|                01|  VERBAL_COOPERATION|      0.0|         10|         1|         10|  2.5117738|[WORLDSTATE,Bay O...|[UNKNOWN,,,,,[NaN...|[WORLDSTATE,Bay O...|2018-03-23|http://www.nzhera...|\n|741462374|2017-03-23|[USA,UNITED STATE...|         [,,,,,,,,,]| false|          0831|               083|                08|MATERIAL_COOPERATION|      5.0|          4|         1|          4|  -2.259887|[USCITY,Willow Cr...|[UNKNOWN,,,,,[NaN...|[USCITY,Willow Cr...|2018-03-23|https://www.relig...|\n|741462375|2017-03-23|[USA,UNITED STATE...|[GOV,PRESIDENT,,,...|  true|           110|               110|                11|     VERBAL_CONFLICT|     -2.0|          1|         1|          1|-0.63897765|[COUNTRY,Russia,R...|[COUNTRY,Russia,R...|[COUNTRY,Russia,R...|2018-03-23|http://www.tribun...|\n|741462376|2017-03-23|[USA,UNITED STATE...|[RUSGOV,RUSSIA,RU...|  true|           110|               110|                11|     VERBAL_CONFLICT|     -2.0|          1|         1|          1|-0.63897765|[COUNTRY,North Ko...|[COUNTRY,North Ko...|[COUNTRY,North Ko...|2018-03-23|http://www.tribun...|\n|741462377|2017-03-23|[USA,NORTH CAROLI...|[USACVL,UNITED ST...|  true|           042|               042|                04|  VERBAL_COOPERATION|      1.9|          2|         1|          2|  0.9041591|[USCITY,Chapel Hi...|[USCITY,Chapel Hi...|[USCITY,Chapel Hi...|2018-03-23|http://www.charlo...|\n|741462378|2017-03-23|[USACVL,NORTH CAR...|         [,,,,,,,,,]|  true|           042|               042|                04|  VERBAL_COOPERATION|      1.9|          8|         1|          8|  0.9041591|[USCITY,Chapel Hi...|[UNKNOWN,,,,,[NaN...|[USCITY,Chapel Hi...|2018-03-23|http://www.charlo...|\n|741462379|2017-03-23|[USACVL,UNITED ST...|[USA,NORTH CAROLI...|  true|           043|               043|                04|  VERBAL_COOPERATION|      2.8|          2|         1|          2|  0.9041591|[USSTATE,North Ca...|[USSTATE,North Ca...|[USSTATE,North Ca...|2018-03-23|http://www.charlo...|\n|741462380|2017-03-23|[USAGOV,UNITED ST...|[USAMED,UNITED ST...| false|           036|               036|                03|  VERBAL_COOPERATION|      4.0|          1|         1|          1|-0.75376886|[USCITY,White Hou...|[USCITY,White Hou...|[USCITY,White Hou...|2018-03-23|http://www.breitb...|\n|741462381|2017-03-23|[chr,CHEROKEE,,,c...|         [,,,,,,,,,]|  true|           193|               193|                19|   MATERIAL_CONFLICT|    -10.0|          5|         1|          5| -1.4614646|[USSTATE,Michigan...|[UNKNOWN,,,,,[NaN...|[USSTATE,Michigan...|2018-03-23|https://jalopnik....|\n|741462382|2018-02-21|[CAN,CANADA,CAN,,...|[VNM,VIETNAM,VNM,...| false|           043|               043|                04|  VERBAL_COOPERATION|      2.8|          2|         1|          2| -12.418301|[COUNTRY,Canada,C...|[COUNTRY,Canada,C...|[COUNTRY,Canada,C...|2018-03-23|http://calgarysun...|\n|741462383|2018-02-21|[CAN,CANADA,CAN,,...|[VNM,VIETNAM,VNM,...| false|           043|               043|                04|  VERBAL_COOPERATION|      2.8|          8|         1|          8| -12.418301|[COUNTRY,Canada,C...|[COUNTRY,Vietnam ...|[COUNTRY,Canada,C...|2018-03-23|http://calgarysun...|\n|741462384|2018-02-21|[DEU,GERMANY,DEU,...|[GOV,GOVERNMENT,,...| false|           036|               036|                03|  VERBAL_COOPERATION|      4.0|         14|         7|         14| -0.9155244|[WORLDCITY,Berlin...|[COUNTRY,Mali,ML,...|[WORLDCITY,Berlin...|2018-03-23|https://www.barri...|\n|741462385|2018-02-21|[DEU,GERMANY,DEU,...|[GOV,GOVERNMENT,,...| false|           036|               036|                03|  VERBAL_COOPERATION|      4.0|          4|         2|          4|-0.18219218|[WORLDCITY,Berlin...|[COUNTRY,Mali,ML,...|[COUNTRY,Mali,ML,...|2018-03-23|http://lethbridge...|\n|741462386|2018-02-21|[EDU,STUDENTS AND...|[UAF,GUNMAN,,,,,,...| false|           180|               180|                18|   MATERIAL_CONFLICT|     -9.0|          1|         1|          1| -1.7881706|[USCITY,Baltimore...|[USCITY,Baltimore...|[USCITY,Baltimore...|2018-03-23|http://www.watert...|\n|741462387|2018-02-21|[GOV,GOVERNMENT,,...|[DEU,GERMANY,DEU,...| false|           036|               036|                03|  VERBAL_COOPERATION|      4.0|         14|         7|         14| -0.9155244|[COUNTRY,Mali,ML,...|[WORLDCITY,Berlin...|[WORLDCITY,Berlin...|2018-03-23|https://www.barri...|\n|741462388|2018-02-21|[GOV,GOVERNMENT,,...|[DEU,GERMANY,DEU,...| false|           036|               036|                03|  VERBAL_COOPERATION|      4.0|          4|         2|          4|-0.18219218|[COUNTRY,Mali,ML,...|[WORLDCITY,Berlin...|[COUNTRY,Mali,ML,...|2018-03-23|http://lethbridge...|\n+---------+----------+--------------------+--------------------+------+--------------+------------------+------------------+--------------------+---------+-----------+----------+-----------+-----------+--------------------+--------------------+--------------------+----------+--------------------+\n```\n\n#### GDELT reference data\n\n```scala\nimport com.aamend.spark.gdelt._\n\nval countryCodes: Dataset[CountryCode] = spark.loadCountryCodes\nval gcam: Dataset[GcamCode] = spark.loadGcams\nval cameoEvent: Dataset[CameoCode] = spark.loadCameoEventCodes\nval cameoType: Dataset[CameoCode] = spark.loadCameoTypeCodes\nval cameoGroup: Dataset[CameoCode] = spark.loadCameoGroupCodes\nval cameoEthnic: Dataset[CameoCode] = spark.loadCameoEthnicCodes\nval cameoReligion: Dataset[CameoCode] = spark.loadCameoReligionCodes\nval cameoCountry: Dataset[CameoCode] = spark.loadCameoCountryCodes\n```\n\nHere is an example of `Dataset[CameoCode]`\n\n```\n+---------+--------------------+\n|cameoCode|          cameoValue|\n+---------+--------------------+\n|      COP|       police forces|\n|      GOV|          government|\n|      INS|          insurgents|\n|      JUD|           judiciary|\n|      MIL|            military|\n|      OPP|political opposition|\n|      REB|              rebels|\n|      SEP|   separatist rebels|\n|      SPY|  state intelligence|\n|      UAF|unaligned armed f...|\n|      AGR|         agriculture|\n|      BUS|            business|\n|      CRM|            criminal|\n|      CVL|            civilian|\n|      DEV|         development|\n|      EDU|           education|\n|      ELI|              elites|\n|      ENV|       environmental|\n|      HLH|              health|\n|      HRI|        human rights|\n+---------+--------------------+\n```\n\n### Accessing HTML content\n\nThe main difference between an average and an expert data scientist is the level of curiosity and creativity employed in extracting the value latent in the data. \nYou could build a simple model on top of GDELT, or you could notice and leverage all these URLs mentioned, scrape that content, and use these extended results to discover new insights that exceed the original questions.\n\nWe delegate this logic to the excellent Scala library [Goose](https://github.com/GravityLabs/goose/wiki) \n\nI decided to embed my own `Goose` library in that project for the following reasons\n\n1. to make it `Scala 2.11` compatible (currently `2.9`)\n2. to start tuning it towards the extraction of news content and not generic HTML\n3. to start cleaning its messy code base (sorry guys, love this library but code is a mess!)\n4. to scale it for spark (distributed news scanner)\n\n##### The Goose library\n\nInteracting with the Goose library is fairly easy\n\n```scala\nimport com.gravity.goose.{Configuration, Goose}\n\nval conf: Configuration = new Configuration\nconf.setEnableImageFetching(false)\nconf.setBrowserUserAgent(\"Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8\")\nconf.setConnectionTimeout(1000)\nconf.setSocketTimeout(1000)\n\nval url = \"http://www.bbc.co.uk/news/entertainment-arts-35278872\"\nval goose: Goose = new Goose(conf)\nval article = goose.extractContent(url)\n```\n\n##### Spark Pipeline\n\nScraping news articles from URLs in above datasets is done via a Spark [ML pipeline](https://spark.apache.org/docs/2.2.0/ml-pipeline.html)\n\n```scala\nimport com.aamend.spark.gdelt.ContentFetcher\n\nval contentFetcher = new ContentFetcher()\n  .setInputCol(\"sourceUrl\")\n  .setOutputTitleCol(\"title\")\n  .setOutputContentCol(\"content\")\n  .setOutputKeywordsCol(\"keywords\")\n  .setOutputPublishDateCol(\"publishDate\")\n  .setOutputDescriptionCol(\"description\")\n  .setUserAgent(\"Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8\")\n  .setConnectionTimeout(1000)\n  .setSocketTimeout(1000)\n\nval contentDF = contentFetcher.transform(gdeltEventDS)\ncontentDF.show()\n```\n\nThe resulting dataframe is as follows\n\n```\n+--------------------+--------------------+--------------------+--------------------+--------------------+------------+\n|           sourceUrl|         description|             content|            keywords|               title| publishDate|\n+--------------------+--------------------+--------------------+--------------------+--------------------+------------+\n|https://www.thegu...|Parliament passes...|Mariano Rajoy, on...|[MARIANO RAJOY, S...|Mariano Rajoy ous...|  2018-06-01|\n+--------------------+--------------------+--------------------+--------------------+--------------------+------------+\n```\n\n##### Fetching Images\n\nWith a proper installation of [imagemagick](http://www.imagemagick.org/script/index.php), this library can even detect the most representative picture of a given article. \nDownloading GDELT image header for each article opens up lots of data science opportunities (face recognition, fake news / propaganda detection).\nThis, however, requires an installation of `imagemagick` on all executors across your Spark cluster.\n\n```scala\nimport com.aamend.spark.gdelt.ContentFetcher\n\nval contentFetcher = new ContentFetcher()\n  .setInputCol(\"sourceUrl\")\n  .setOutputImageUrlCol(\"imageUrl\")\n  .setOutputImageBase64Col(\"imageBase64\")\n  .setImagemagickConvert(\"/usr/local/bin/convert\")  \n  .setImagemagickIdentify(\"/usr/local/bin/identify\")\n  \nval contentDF = contentFetcher.transform(gdeltEventDS)\nprintln(contentDF.rdd.first.getAs[String](\"imageBase64\"))\n```\n\nThe main image is represented as base64 in data URI\n\n```\ndata:image/jpeg;width=300;height=180;base64,/9j/4AAQSkZJRgABAQEASABIAA...\n```\n\nYou can validate it via any online viewer such as [https://codebeautify.org/base64-to-image-converter](https://codebeautify.org/base64-to-image-converter)\n\n![article](/images/article.jpeg)\n\n*source: https://www.theguardian.com/world/2018/jun/01/mariano-rajoy-ousted-as-spain-prime-minister*\n\n##### Parameters\n\n+ `setInputCol(s: String)`\n\nMandatory, the input column containing URLs to fetch\n\n+ `setOutputTitleCol(s: String)`\n\nOptional, the output column to store article title - title will not be fetched if not specified\n\n+ `setOutputContentCol(s: String)`\n\nOptional, the output column to store article content - content will not be fetched if not specified\n\n+ `setOutputKeywordsCol(s: String)`\n\nOptional, the output column to store article metadata keyword - keywords will not be fetched if not specified\n\n+ `setOutputPublishDateCol(s: String)` \n\nOptional, the output column to store article metadata publishDate - date will not be fetched if not specified\n\n+ `setOutputDescriptionCol(s: String)`\n\nOptional, the output column to store article metadata description - description will not be fetched if not specified\n\n+ `setOutputImageUrlCol(s: String)`\n\nOptional, the output column to store article main image URL - image fetching will be disabled if not specified\n\n+ `setOutputImageBase64Col(s: String)`\n\nOptional, the output column to store article main image data URI - image fetching will be disabled if not specified\n\n+ `setImagemagickConvert(s: String)`\n\nOptional, the path to imagemagick `convert` executable on every spark executors, not used if image fetching is disabled, default: `/usr/local/bin/convert`\n\n+ `setImagemagickIdentify(s: String)`\n\nOptional, the path to imagemagick `identify` executable on every spark executors, not used if image fetching is disabled, default: `/usr/local/bin/identify`\n\n+ `setUserAgent(s: String)` \n\nOptional, the user agent to use in Goose HTTP requests, default: `Mozilla/5.0`\n\n+ `setConnectionTimeout(i: Int)`\n\nOptional, the connection timeout, in milliseconds, default: `1000`\n\n+ `setSocketTimeout(i: Int)`\n\nOptional, the socket timeout, in milliseconds, default: `1000`\n\n#### Fueling data science use cases\n\nThe main reason I decided to embed a Goose extraction as a ML pipeline is to integrate news content with existing Spark ML functionality. \nIn below example, we extract content of web articles, tokenize words and train a [Word2Vec](https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec) model\n\n```scala\nval contentFetcher = new ContentFetcher()\n  .setInputCol(\"sourceUrl\")\n  .setOutputContentCol(\"content\")\n\nval tokenizer = new Tokenizer()\n  .setInputCol(\"content\")\n  .setOutputCol(\"words\")\n\nval word2Vec = new Word2Vec()\n  .setInputCol(\"words\")\n  .setOutputCol(\"features\")\n\nval pipeline = new Pipeline()\n  .setStages(Array(contentFetcher, tokenizer, word2Vec))\n  \nval model = pipeline.fit(df)\nval words = model.transform(df)\n```\n\n## Version\n\n+ 1.0 - Basic parser and reference data\n+ 2.0 - Adding Goose library packaged for Spark / Scala 2.11\n+ 3.0 - Spark 3.0 and Scala 2.12\n\n## Authors\n\nAntoine Amend - [[antoine.amend@gmail.com]](antoine.amend@gmail.com)\n\n## License\n\nApache License, version 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faamend%2Fspark-gdelt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faamend%2Fspark-gdelt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faamend%2Fspark-gdelt/lists"}