Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/adammoses-github/npmjs-googlenews-rss-scraper

This package scrapes a Google News RSS feed for data, returning news stories based on a specified topic or query.
https://github.com/adammoses-github/npmjs-googlenews-rss-scraper

google googlenews news npmjs rss

Last synced: 3 months ago
JSON representation

This package scrapes a Google News RSS feed for data, returning news stories based on a specified topic or query.

Awesome Lists containing this project

README 

        
## About



This package scrapes a Google News RSS feed for data.



The information returned includes news stories based on a specified topic or query.



Topics available are headlines (the default), US National news, world news, entertainment, business, sports, sci/tech, and technology. 



Queries can be anything in the style of a comma-seperated list string, i.e. 'poker' or 'space,spaceships,stars,astronomy'. 



For each news story some information will be returned:

* Title

* Source; i.e. 'Washington Times' or 'New York Times'

* Category

* Publishing Date

* URL; both the full and cleaned URLs

* Description; both the full and cleaned descriptions



## Installation



```

npm install googlenews-rss-scraper

```



## Usage



There is only one method to this package: ```getGoogleNewsRSSScraperData(params, callback)```



This method requries a params object, which has two fields: newsType and newsTypeTerms. These parameters are specified in more detail in the parameters section.



This method requires the use of a callback, which can be specified or anonymous. The callback has one parameters in the form of ```function(data)```. This data object is described in more detail in the return



If there was an error, like failed HTML request or bad parsing, **error** will be true and **data** will be **null**. Otherwise, **error** will be false and **data** will be a javascript object. 



## Paramters Format



The method ```getGoogleNewsRSSScraperData(params, callback)``` requires use of a parameters variable. 



This variable is a JSON object containing two members: **newsType** and **newsTypeTerms**.



**newsType** specifies the type of news to gather, which can either be a pre-defined topic that Google already organizes news by, or it can be a specific query that you request. To get news based on a topic **newsType** should be set to 'TOPIC' and to get news based on a query **newsType** should be set to 'QUERY'.



**newsTypeTerms** specifies the complimentary topic type or query search terms following the selection set in **newsType**. Examples values for a **newsTypeTerms** for a topic are 'HEADLINES' or 'SPORTS'.  Example values for a **newsTypeTerms** for a query are 'nintendo,mario,gaming' or 'spiders,moths,insects'.



### Specifying A Topic Type



The list below specifies the currently available topic types Google News has available. 



When setting the **newsTypeTerms** for a topic the value must be an exact match for a value below, otherwise an error may be returned.



NOTE: This list is subject to change at anytime. 



NOTE: These Google News topic types were developed/tested in the United States, so use in other countries may vary, especially in the case of the 'NATIONAL' news topic type. 



* HEADLINES

* NATIONAL

* WORLD

* ENTERTAINMENT

* BUSINESS

* SPORTS

* SCI/TECH

* TECHNOLOGY



### Specifying A Query



Queries are specified as a single string containing comma-seperated values. The news returned will be based on Google News searches of those values, and are considered to be working on under an OR operation instead of an AND. Individual values can contain spaces but should not contain any other special characters; only letters, numbers, and spaces should be used between comma-seperated values.



In the event of specifying a query that is very niche or not currently news-worthy, it is possible to gather no news.



### Parameter Examples



```javascript

{  newsType: 'TOPIC', newsTypeTerms: 'SPORTS' }



{  newsType: 'TOPIC', newsTypeTerms: 'NATIONAL' }



{  newsType: 'TOPIC', newsTypeTerms: 'SCI/TECH' }



{  newsType: 'QUERY', newsTypeTerms: 'LIONS,TIGERS,CHEETAHS' }



{  newsType: 'QUERY', newsTypeTerms: 'MIAMI DOLPHINS' }



{  newsType: 'QUERY', newsTypeTerms: 'PLANETS,MOONS,STARS' }

```



NOTE: In this example all parameters are specified in upper-case but in fact these values are case insensitive.



### Empty / Default Parameter



Specifying an empty set of parameters, i.e. ```{ }```, will use the default call which is ```{  newsType: 'TOPIC', newsTypeTerms: 'HEADLINES' }```.



## Return Data Format



The method ```getGoogleNewsRSSScraperData(params, callback)``` uses a callback function in the form of ```function(data)```. This function can be specified or anonymous.



This section will describe the format of the returning JSON data object. This object has the following members.



* **error** - boolean, true if an error occured otherwise false

* **errorMessage** - string, if error is true than explains the error otherwise is null

* **type** - string, mirrors the input paramater of **newsType**, either 'TOPIC' or 'QUERY'

** **terms** - string, mirros the input parameter of **newsTypeTerms**, either provided topic type or query values

** **url** - string, the generated Google News RSS URL that was used to fetch the data

** **newsArray** - array, each element is a different news story fetched from the Google News RSS



NOTE: In the event of an error some or all of these values may be set to null.



### News Array Elements



The **newsArray** member contained at the highest level of the returning JSON data object contains the indiviudal news stories fetched from the Google News RSS. This value can be null, likely if an error occured, or even empty if no stories were able to be found for a given query.



The elements of the news array have the following memebers.



* **title** - string, the title of the news story

* **source** - string, the source of the news story, i.e. 'New York Times' or 'New York Post' or 'ABC News', etc.

* **category** - string, mirrors the topic type in the case of news gathered by topic, is null when a query was used

* **pubDate** - string, UTC datetime value for when the story was published

* **fullURL** - string, the complete URL as listed in the RSS feed

* **cleanURL** - string, just the direct link to the new story with the Google News parts stripped out

* **fullDescription** - string, the complete description of the news story as listed in the RSS feed, likely containing many HTML tags

* **cleanDescription** - string, shorter and HTML tag-less of the full description



## Usage Example



```javascript

// Usage example of googlenews-rss-scraper package

// See https://www.npmjs.com/package/googlenews-rss-scraper



var iGoogleNewsRSSScraper = require('googlenews-rss-scraper');



console.log('Usage example of the googlenews-rss-scraper package.')



iGoogleNewsRSSScraper.getGoogleNewsRSSScraperData( { }, function(data) {

        if (!data.error) {

            console.log(JSON.stringify(data, null, 2));      

        }

        else {

            console.log('Some error occured.');

        }

    });

```    



## Return Data Example



```javascript

{

  "error": false,

  "errorMessage": null,

  "type": "TOPIC",

  "terms": "HEADLINES",

  "url": "https://news.google.com/news?cf=all&hl=en&pz=1&ned=us&topic=h&output=rss",

  "newsArray": [

    {

      "title": "The Daily 202: Trump's health care ultimatum is straight out of 'The Art of the Deal.' It just might work.",

      "source": "Washington Post",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 14:10:25 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNFnnExsVOvmXVvoVA-WaXejQpIxrA&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779434739124&ei=4kzVWJChD9jE3QHZ_5-ICg&url=https://www.washingtonpost.com/news/powerpost/paloma/daily-202/2017/03/24/daily-202-trump-s-health-care-ultimatum-is-straight-out-of-the-art-of-the-deal-it-just-might-work/58d458c9e9b69b72b255102d/",

      "cleanURL": "https://www.washingtonpost.com/news/powerpost/paloma/daily-202/2017/03/24/daily-202-trump-s-health-care-ultimatum-is-straight-out-of-the-art-of-the-deal-it-just-might-work/58d458c9e9b69b72b255102d/",

      "fullDescription": "\"\"
Washington Post
\"\"


The Daily 202: Trump's health care ultimatum is straight out of 'The Art of the Deal.' It just might work.
Washington Post
With Breanne Deppisch. THE BIG IDEA: If you read Donald Trump's “The Art of the Deal,” substituting “conservatives" for "contractors," the president's ultimatum to House Republicans on health care is not at all surprising. “You have to be very rough ...
Foreboding signs for Obamacare repeal as Friday vote loomsPolitico
Obamacare Repeal Rolls Toward a Vote: 'It's Time to Act'New York Times
House votes to begin debate on healthcare bill; six Republicans defectThe Hill
Los Angeles Times -Reuters -BBC News -The Boston Globe
all 3,801 news articles »

",

      "cleanDescription": "With Breanne Deppisch. THE BIG IDEA: If you read Donald Trump's “The Art of the Deal,” substituting “conservatives" for "contractors," the president's ultimatum to House Republicans on health care is not at all surprising. “You have to be very rough ..."

    },

    {

      "title": "Toll of London Attack Is Global for an Assailant Born in Britain",

      "source": "New York Times",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 11:41:24 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNGQNr2zBFTygddkHbe4yref5kAz6w&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779433107670&ei=4kzVWJChD9jE3QHZ_5-ICg&url=https://www.nytimes.com/2017/03/23/world/europe/london-attack-uk.html",

      "cleanURL": "https://www.nytimes.com/2017/03/23/world/europe/london-attack-uk.html",

      "fullDescription": "\"\"
New York Times
\"\"


Toll of London Attack Is Global for an Assailant Born in Britain
New York Times
A vigil was held on Thursday in Trafalgar Square in London for the victims of Wednesday's attack. Credit Andrew Testa for The New York Times. LONDON — The roster of the dead and wounded spanned the globe: a veteran Scotland Yard constable, ...
London attack: The victimsBBC News
British police make 'significant' arrests in widening probes after London rampageWashington Post
London attack inquiry focuses on radicalization of perpetratorCNN
Reuters -Fox News -Newsweek -New York Daily News
all 6,677 news articles »

",

      "cleanDescription": "A vigil was held on Thursday in Trafalgar Square in London for the victims of Wednesday's attack. Credit Andrew Testa for The New York Times. LONDON — The roster of the dead and wounded spanned the globe: a veteran Scotland Yard constable, ..."

    },

    {

      "title": "Trump administration grants approval for Keystone XL pipeline",

      "source": "Washington Post",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 15:17:25 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNGnCoQKZuPIFJIg64_ZF9suVaDXqA&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779432958564&ei=4kzVWJChD9jE3QHZ_5-ICg&url=https://www.washingtonpost.com/news/energy-environment/wp/2017/03/24/trump-administration-grants-approval-for-keystone-xl-pipeline/",

      "cleanURL": "https://www.washingtonpost.com/news/energy-environment/wp/2017/03/24/trump-administration-grants-approval-for-keystone-xl-pipeline/",

      "fullDescription": "\"\"
Washington Post
\"\"


Trump administration grants approval for Keystone XL pipeline
Washington Post
President Trump announced Friday morning the granting of a permit for construction of the controversial Keystone XL pipeline, calling it “the first of many infrastructure projects” that he would approve in order to put more Americans to work. Trump ...
US, in Reversal, Issues Permit for Keystone Oil PipelineNew York Times
Trump greenlights Keystone XL pipeline, but other obstacles loomReuters
Keystone XL Pipeline Approved by President Trump in Reversal From ObamaNBCNews.com
The Hill -MarketWatch -USA TODAY -Houston Chronicle
all 382 news articles »

",

      "cleanDescription": "President Trump announced Friday morning the granting of a permit for construction of the controversial Keystone XL pipeline, calling it “the first of many infrastructure projects” that he would approve in order to put more Americans to work. Trump ..."

    },

    {

      "title": "The Latest: Ryan heads to White House to discuss health care",

      "source": "SFGate",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 15:34:24 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNHaKUqvtkuVTDSk0nr1ICf7mAS9Yg&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779430416040&ei=4kzVWJChD9jE3QHZ_5-ICg&url=http://www.sfgate.com/news/medical/article/The-Latest-Mulvaney-warns-against-vote-collapsing-11025037.php",

      "cleanURL": "http://www.sfgate.com/news/medical/article/The-Latest-Mulvaney-warns-against-vote-collapsing-11025037.php",

      "fullDescription": "\"\"
TIME
\"\"


The Latest: Ryan heads to White House to discuss health care
SFGate
WASHINGTON (AP) — The Latest on an effort in Congress to pass a health care bill (all times local):. 12:30 p.m.. House Speaker Paul Ryan is at the White House to brief President Donald Trump on the state of play on the health care bill. The visit ...
Pence postpones travel to fight for Obamacare repeal in WashingtonPolitico
Objections raised over White House photo of health care talksFOX31 Denver
Pence cancels trip to work on healthcare as House vote loomsReuters
The Commercial Appeal -Fox 59 -THV11.com -Washington Post
all 115 news articles »

",

      "cleanDescription": "WASHINGTON (AP) — The Latest on an effort in Congress to pass a health care bill (all times local):. 12:30 p.m.. House Speaker Paul Ryan is at the White House to brief President Donald Trump on the state of play on the health care bill. The visit ..."

    },

    {

      "title": "Maryland school no sanctuary for alleged rape victim",

      "source": "Bucks County Courier Times",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 04:15:00 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNHZMe_cCzS5mF3Q5YVLvJPslLXoDw&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779425560459&ei=4kzVWJChD9jE3QHZ_5-ICg&url=http://www.buckscountycouriertimes.com/opinion/op-ed/maryland-school-no-sanctuary-for-alleged-rape-victim/article_6d435622-5027-5f3d-9c9b-da9d279552d9.html",

      "cleanURL": "http://www.buckscountycouriertimes.com/opinion/op-ed/maryland-school-no-sanctuary-for-alleged-rape-victim/article_6d435622-5027-5f3d-9c9b-da9d279552d9.html",

      "fullDescription": "
\"\"


Maryland school no sanctuary for alleged rape victim
Bucks County Courier Times
A rough translation of Maryland's state motto is "Strong Deeds, Gentle Words." In the case of a 14-year-old girl who was recently raped and sodomized in a restroom at Rockville High School by two males students, both immigrants, one facing a ...

and more »

",

      "cleanDescription": "A rough translation of Maryland's state motto is "Strong Deeds, Gentle Words." In the case of a 14-year-old girl who was recently raped and sodomized in a restroom at Rockville High School by two males students, both immigrants, one facing a ..."

    },

    {

      "title": "Black lawmakers ask FBI to help find missing DC kids",

      "source": "Washington Examiner",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 16:23:00 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNF8EflzoMNmmfanNGcxvuTXG54EyQ&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779433522306&ei=4kzVWJChD9jE3QHZ_5-ICg&url=http://www.washingtonexaminer.com/black-lawmakers-ask-fbi-to-help-find-missing-dc-kids/article/2618363",

      "cleanURL": "http://www.washingtonexaminer.com/black-lawmakers-ask-fbi-to-help-find-missing-dc-kids/article/2618363",

      "fullDescription": "\"\"
Washington Examiner
\"\"


Black lawmakers ask FBI to help find missing DC kids
Washington Examiner
The letter, dated Tuesday, was sent by Congressional Black Caucus chairman Cedric Richmond, D-La., and Del. Eleanor Holmes Norton, who represents the District in Congress. According to the letter, obtained by the Associated Press, D.C., logged 501 ...

and more »

",

      "cleanDescription": "The letter, dated Tuesday, was sent by Congressional Black Caucus chairman Cedric Richmond, D-La., and Del. Eleanor Holmes Norton, who represents the District in Congress. According to the letter, obtained by the Associated Press, D.C., logged 501 ..."

    },

    {

      "title": "Schumer's filibuster promise means GOP won, Gorsuch headed to Supreme Court",

      "source": "Fox News",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 13:45:01 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNFFGxidVTXxs1fb30QvH9fGQm2_Ug&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779422409774&ei=4kzVWJChD9jE3QHZ_5-ICg&url=http://www.foxnews.com/opinion/2017/03/24/schumers-filibuster-promise-means-gop-won-gorsuch-headed-to-supreme-court.html",

      "cleanURL": "http://www.foxnews.com/opinion/2017/03/24/schumers-filibuster-promise-means-gop-won-gorsuch-headed-to-supreme-court.html",

      "fullDescription": "\"\"
Fox News
\"\"


Schumer's filibuster promise means GOP won, Gorsuch headed to Supreme Court
Fox News
Contrary to media reports Thursday, Senate Minority Leader Chuck Schumer's promise to invoke a filibuster signals the success, not the failure, of Judge Neil Gorsuch's Supreme Court nomination. If Democratic Senators had made any progress in attacking ...

and more »

",

      "cleanDescription": "Contrary to media reports Thursday, Senate Minority Leader Chuck Schumer's promise to invoke a filibuster signals the success, not the failure, of Judge Neil Gorsuch's Supreme Court nomination. If Democratic Senators had made any progress in attacking ..."

    },

    {

      "title": "Putin meets French far-right candidate Marine Le Pen at Kremlin",

      "source": "CNN",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 14:39:51 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNEIt-SGZjguRKNUR-10qTWMHDz-Uw&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779434333018&ei=4kzVWJChD9jE3QHZ_5-ICg&url=http://www.cnn.com/2017/03/24/europe/putin-le-pen-kremlin/",

      "cleanURL": "http://www.cnn.com/2017/03/24/europe/putin-le-pen-kremlin/",

      "fullDescription": "\"\"
CNN
\"\"


Putin meets French far-right candidate Marine Le Pen at Kremlin
CNN
(CNN) Russian President Vladimir Putin stressed the "great importance" of ties between his country and France as he met French far-right presidential candidate Marine Le Pen at the Kremlin on Friday, Russian state-run news agency Tass reported.
Ahead Of Its Election, France Is Deeply Fragmented, Its People Often Angry and ScaredHuffington Post
Putin, Le Pen hold surprise meeting in MoscowFox News
This Is What Happened When Marine Le Pen Went To MoscowBuzzFeed News
Foreign Policy (blog) -CNBC -U.S. News & World Report -National Catholic Reporter
all 162 news articles »

",

      "cleanDescription": "(CNN) Russian President Vladimir Putin stressed the "great importance" of ties between his country and France as he met French far-right presidential candidate Marine Le Pen at the Kremlin on Friday, Russian state-run news agency Tass reported."

    },

    {

      "title": "Hosni Mubarak Is Freed, to Dismay of Many in Egypt",

      "source": "New York Times",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 14:53:46 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNEMXt3c2R2_5EbOfFor83Xq4p0wbw&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779435346544&ei=4kzVWJChD9jE3QHZ_5-ICg&url=https://www.nytimes.com/2017/03/24/world/africa/hosni-mubarak-egypt.html",

      "cleanURL": "https://www.nytimes.com/2017/03/24/world/africa/hosni-mubarak-egypt.html",

      "fullDescription": "\"\"
New York Times
\"\"


Hosni Mubarak Is Freed, to Dismay of Many in Egypt
New York Times
Former President Hosni Mubarak of Egypt was returned on March 2 to Maadi Military Hospital in southern Cairo after a court hearing. He was freed from the hospital detention on Friday. Credit Mohamed Hossam/European Pressphoto Agency. CAIRO — Six ...
Key events in the life of the ousted Egyptian presidentFox News
Egypt's Mubarak free, acquitted after years-long detentionWashington Post
Ex-Egyptian President Hosni Mubarak freed after 6 years in detentionCNN
Reuters -CBS News -PRI -NPR
all 95 news articles »

",

      "cleanDescription": "Former President Hosni Mubarak of Egypt was returned on March 2 to Maadi Military Hospital in southern Cairo after a court hearing. He was freed from the hospital detention on Friday. Credit Mohamed Hossam/European Pressphoto Agency. CAIRO — Six ..."

    },

    {

      "title": "Toronto school board bars US trips amid travel ban uncertainty",

      "source": "The Hill",

      "category": "Top Stories",

      "pubDate": "Fri, 24 Mar 2017 15:47:51 GMT",

      "fullURL": "http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNF1HaKbepCfv6YgHSeosil3MNvO1Q&clid=c3a7d30bb8a4878e06b80cf16b898331&cid=52779434475223&ei=4kzVWJChD9jE3QHZ_5-ICg&url=http://thehill.com/policy/international/325589-toronto-school-board-barring-student-staff-trips-to-us-amid-travel-ban",

      "cleanURL": "http://thehill.com/policy/international/325589-toronto-school-board-barring-student-staff-trips-to-us-amid-travel-ban",

      "fullDescription": "\"\"
The Hill
\"\"


Toronto school board bars US trips amid travel ban uncertainty
The Hill
Toronto schools will no longer allow student or staff trips to the U.S. due to uncertainty over whether everyone will be allowed to enter in the wake of President Trump's travel ban. In a statement released Thursday, John Mallow, direct of education ...
Worried about Trump's travel ban, Canada's largest school district calls off US tripsWashington Post
Trump Travel Ban: Toronto School Board Bars New Trips to USNBCNews.com
Toronto Cancels US Class Trips Due To Trump Travel Ban ConcernsHuffington Post
CNN -Fox News -TIME -Slate Magazine (blog)
all 50 news articles »

",

      "cleanDescription": "Toronto schools will no longer allow student or staff trips to the U.S. due to uncertainty over whether everyone will be allowed to enter in the wake of President Trump's travel ban. In a statement released Thursday, John Mallow, direct of education ..."

    }

  ]

}

```