{"id":23353944,"url":"https://github.com/luminati-io/google-news-scraper","last_synced_at":"2025-04-07T19:20:50.427Z","repository":{"id":267760329,"uuid":"901825475","full_name":"luminati-io/Google-News-Scraper","owner":"luminati-io","description":"Extract news data with the #1 Google News API, including top stories, publisher details, article summaries, and trending topics. Start your free trial now!","archived":false,"fork":false,"pushed_at":"2024-12-12T08:37:39.000Z","size":454,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-22T07:02:01.169Z","etag":null,"topics":["api","datasets","google-news","google-news-api","google-news-scraper","python","scraper-api","scraper-python","web-scraper"],"latest_commit_sha":null,"homepage":"https://brightdata.com/products/serp-api/google-search/news","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luminati-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-11T11:40:15.000Z","updated_at":"2024-12-12T08:37:43.000Z","dependencies_parsed_at":"2024-12-12T09:32:38.785Z","dependency_job_id":"2cf67a0b-a6e1-413f-b600-7a4bf8192d8d","html_url":"https://github.com/luminati-io/Google-News-Scraper","commit_stats":null,"previous_names":["luminati-io/google-news-scraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FGoogle-News-Scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FGoogle-News-Scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FGoogle-News-Scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luminati-io%2FGoogle-News-Scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luminati-io","download_url":"https://codeload.github.com/luminati-io/Google-News-Scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247713264,"owners_count":20983683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","datasets","google-news","google-news-api","google-news-scraper","python","scraper-api","scraper-python","web-scraper"],"created_at":"2024-12-21T09:17:13.651Z","updated_at":"2025-04-07T19:20:50.389Z","avatar_url":"https://github.com/luminati-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google News Scraper\n\n[![Promo](https://github.com/luminati-io/Google-News-Scraper/blob/main/Proxies%20and%20scrapers%20GitHub%20bonus%20banner.png)](https://brightdata.com/products/serp-api/google-search/news?promo=github15) \n\nThis repository provides two methods to collect news data from Google News.\n- **Free Method:** Perfect for small projects and learning\n- **Google News API:** Ideal for large-scale, reliable, real-time data extraction\n\n## Table of Contents\n\n- [Method 1: Free Google News Scraper](#method-1-free-google-news-scraper)\n  - [Prerequisites](#prerequisites)\n  - [Installation](#installation)\n  - [Usage](#usage)\n  - [Output](#output)\n- [Common Scraping Challenges](#common-scraping-challenges)\n- [Method 2: Bright Data Google News API](#method-2-bright-data-google-news-api)\n  - [Key Benefits](#key-benefits)\n  - [Getting Started with the Google News API](#getting-started-with-the-google-news-api)\n  - [Key Input Parameters](#key-input-parameters)\n  - [Sample Result](#sample-result)\n  - [Ready-to-Use Python Code](#ready-to-use-python-code)\n  - [Understanding the API Implementation](#understanding-the-api-implementation)\n  - [Customizing Your Data Collection](#customizing-your-data-collection)\n\n## Method 1: Free Google News Scraper\n\u003cimg width=\"700\" alt=\"image\" src=\"https://github.com/user-attachments/assets/a7d34ffe-17c6-4c59-acbf-aaf84ed1b13e\"\u003e\n\nThis free tool lets you collect news articles based on any topic you're interested in. You'll get everything from headlines to publication dates, all neatly organized.\n\n### Prerequisites\n- Python 3.9+\n- Two key packages:\n  - [aiohttp](https://pypi.org/project/aiohttp/) (for making requests)\n  - [beautifulsoup4](https://pypi.org/project/beautifulsoup4/) (for parsing HTML)\n\n### Installation\n1. Clone the repository:\n\n    ```bash\n    git clone https://github.com/luminati-io/Google-News-Scraper.git\n    ```\n3. Navigate to the project directory:\n\n    ```bash\n    cd Google-News-Scraper\n    ```\n4. Install required dependencies:\n\n    ```bash\n    pip install -r requirements.txt\n    ```\n### Usage\n1. Navigate to the `free_scraper` directory and open `main.py`\n2. Define your search terms in the file:\n\n    ```bash\n    search_terms = [\n        \"artificial intelligence\",\n        \"climate change\",\n        \"space exploration\",\n        # Add more search terms as needed\n    ]\n    ```\n3. Run the scraper:\n\n    ```bash\n    python main.py\n    ```\n### Output\nThe scraper generates JSON files:\n- Individual JSON files for each search term\n- A `combined_results.json` file containing data from all search terms\n\nEach article in the JSON output contains:\n```json\n{\n    \"title\": \"OpenAI launches full o1 model with image uploads and analysis, debuts ChatGPT Pro - VentureBeat\",\n    \"link\": \"https://news.google.com/rss/articles/CBMipgFBVV95cUxQTTVmS1I4aW1QanZXTnBfa2tBR3d0Y2JzNjJJNldBZTd1TVVfRmpxaUM3bGJld3RycXhPbU8wM1loT0JGd2JDRzFmU1pLU3FSbkRRZ0FPY29INmdhU1RsWXFqXzdLTjNCbU5ES3pIQXZLbTVmMWVhc0FqVlljeWNPOHZMeFlXV2F5Q21ac0lSZVhIOHlnS05sdkR5ZjhJTU9HazJ6MWJR?oc=5\",\n    \"publication_date\": \"Thu, 05 Dec 2024 18:00:00 GMT\",\n    \"source\": \"VentureBeat\",\n    \"source_url\": \"https://venturebeat.com\",\n    \"guid\": \"CBMipgFBVV95cUxQTTVmS1I4aW1QanZXTnBfa2tBR3d0Y2JzNjJJNldBZTd1TVVfRmpxaUM3bGJld3RycXhPbU8wM1loT0JGd2JDRzFmU1pLU3FSbkRRZ0FPY29INmdhU1RsWXFqXzdLTjNCbU5ES3pIQXZLbTVmMWVhc0FqVlljeWNPOHZMeFlXV2F5Q21ac0lSZVhIOHlnS05sdkR5ZjhJTU9HazJ6MWJR\",\n}\n```\n\n👉 You can find a complete example output in our [free_scraper/data/](https://github.com/luminati-io/Google-News-Scraper/tree/main/free_scraper/data) directory.\n\n## Common Scraping Challenges\nScraping data from Google News can be quite challenging. Here are some common issues you may encounter:\n1. **CAPTCHA and Anti-Bot Mechanisms:** Google often employs CAPTCHAs or rate-limiting mechanisms to prevent bots from accessing its content.\n2. **Scalability:** Scraping large volumes of data or performing high-frequency scraping can overwhelm free scrapers.\n3. **Global and Localized News Access:** Customizing scrapers for different regions and languages often requires significant effort and manual adjustments.\n\n## Method 2: Bright Data Google News API\nWant something more robust? Let's talk about [Bright Data's Google News API](https://brightdata.com/products/serp-api/google-search/news). Here's why it's worth considering:\n\n### Key Benefits\n- **Zero Infrastructure Headaches:** Forget about proxies and CAPTCHAs\n- **Built to Scale:** Handles heavy traffic with exceptional performance\n- **Global Reach:** Get news from any country, any language\n- **Privacy First:** GDPR \u0026 CCPA compliant\n- **Pay for Success:** Only charged for successful requests\n- **Try Before You Buy:** 20 free API calls to test things out\n\n## Getting Started with the Google News API\n\u003e For a detailed guide on setting up the Google News API, check our [Step-by-Step Setup Guide](https://github.com/luminati-io/Google-News-Scraper/blob/main/google_news_api_setup.md).\n### Key Input Parameters\n| **Parameter**| **Required?** | **Description**                                            | **Example**               |\n|---------------|--------------|------------------------------------------------------------|---------------------------|\n| `url`         | Yes          | Base Google News URL                                   | `news.google.com`|\n| `keyword`     | Yes          | Your search topic                        | `\"ChatGPT\"`             |\n| `country`     | No           | Where to get news from                                | `\"US\"`                    |\n| `language`    | No           | What language you want                                | `\"en\"`                    |\n\n### Sample Result\nHere’s what the API returns:\n```json\n{\n    \"url\": \"https://www.tomsguide.com/news/live/12-days-of-openai-live-blog-chatgpt-sora\",\n    \"title\": \"12 Days of OpenAI Day 2 LIVE: o1 full is here and every new ChatGPT AI announcement as it happens\",\n    \"publisher\": \"Tom's Guide\",\n    \"date\": \"2024-12-06T20:54:01.000Z\",\n    \"category\": null,\n    \"keyword\": \"chatgpt\",\n    \"country\": \"US\",\n    \"image\": \"https://news.google.com/api/attachments/CC8iK0NnNW9SbTFVTWtkNGFGSjJSVGhGVFJDb0FSaXNBaWdCTWdhQmtJcWpOQWM=-w200-h112-p-df-rw\",\n    \"timestamp\": \"2024-12-08T10:06:05.122Z\",\n    \"input\": {\n        \"url\": \"https://news.google.com/\",\n        \"keyword\": \"chatgpt\",\n        \"country\": \"US\",\n        \"language\": \"en\",\n    },\n}\n```\n👉 You can find a complete example output in our [news_scraper_output.json](https://github.com/luminati-io/Google-News-Scraper/blob/main/google-news-api-scraper/data/news_scraper_output.json) file.\n\n### Ready-to-Use Python Code\nHere's a script to get you started:\n```python\nimport requests\nimport json\nimport time\n\n\nclass BrightDataNews:\n    def __init__(self, api_token):\n        self.api_token = api_token\n        self.headers = {\n            \"Authorization\": f\"Bearer {api_token}\",\n            \"Content-Type\": \"application/json\",\n        }\n        self.dataset_id = \"gd_lnsxoxzi1omrwnka5r\"\n\n    def collect_news(self, search_queries):\n        \"\"\"\n        Collect Google News articles using BrightData API\n        \"\"\"\n        # 1. Trigger data collection\n        print(\"Starting news collection...\")\n        trigger_response = self._trigger_collection(search_queries)\n        snapshot_id = trigger_response.get(\"snapshot_id\")\n        print(f\"Snapshot ID: {snapshot_id}\")\n\n        # 2. Wait for data to be ready\n        print(\"Waiting for data...\")\n        while True:\n            status = self._check_status(snapshot_id)\n            print(f\"Status: {status}\")\n\n            if status == \"ready\":\n                # Check if data is actually available\n                data = self._get_data(snapshot_id)\n                if data and len(data) \u003e 0:\n                    break\n            time.sleep(10)  # Wait 10 seconds before next check\n        # 3. Get and save the data\n        print(\"Saving data...\")\n        filename = f\"news_scraper_output.json\"\n        with open(filename, \"w\", encoding=\"utf-8\") as f:\n            json.dump(data, f, indent=2, ensure_ascii=False)\n        print(f\"✓ Data saved to {filename}\")\n        print(f\"✓ Collected {len(data)} news articles\")\n        return data\n\n    def _trigger_collection(self, search_queries):\n        \"\"\"Trigger news data collection\"\"\"\n        response = requests.post(\n            \"https://api.brightdata.com/datasets/v3/trigger\",\n            headers=self.headers,\n            params={\"dataset_id\": self.dataset_id, \"include_errors\": \"true\"},\n            json=search_queries,\n        )\n        return response.json()\n\n    def _check_status(self, snapshot_id):\n        \"\"\"Check collection status\"\"\"\n        response = requests.get(\n            f\"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}\",\n            headers=self.headers,\n        )\n        return response.json().get(\"status\")\n\n    def _get_data(self, snapshot_id):\n        \"\"\"Get collected data\"\"\"\n        response = requests.get(\n            f\"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}\",\n            headers=self.headers,\n            params={\"format\": \"json\"},\n        )\n        return response.json()\n```\nHere's how to use it:\n```python\n# Initialize the client\nnews_client = BrightDataNews(\"\u003cYOUR_API_TOKEN\u003e\")\n\n# Define what you want to collect\nqueries = [\n    {\n        \"url\": \"https://news.google.com/\",\n        \"keyword\": \"artificial intelligence startups\",\n        \"country\": \"US\",\n        \"language\": \"en\",\n    },\n    {\n        \"url\": \"https://news.google.com/\",\n        \"keyword\": \"tech industry layoffs\",\n        \"country\": \"US\",\n        \"language\": \"en\",\n    },\n]\n\n# Start collection\ntry:\n    news_data = news_client.collect_news(queries)\n    print(f\"Successfully collected {len(news_data)} articles\")\nexcept Exception as e:\n    print(f\"Collection failed: {str(e)}\")\n```\n### Understanding the API Implementation\n1. **Setting Up Your API Token**\n    - First things first: you'll need an API token\n    - If you haven't got one yet, check out our [setup guide](https://github.com/luminati-io/Google-News-Scraper/blob/main/google_news_api_setup.md)\n2. **Starting the Collection**\n    - Pass your search parameters to the API\n    - You'll get back a `snapshot_id`\n3. **Monitoring Progress**\n    - The process takes a few minutes\n    - Our code checks the status automatically:\n      - \"running\" = Still collecting your data\n      - \"ready\" = Time to collect your results!\n4. **Getting Your Data**\n    - Once the status shows \"ready\", we fetch and save your results\n    - Data comes in clean JSON format\n    - Each article includes all the fields we discussed earlier\n\n## Customizing Your Data Collection\nYou can use the following parameters to fine-tune your results:\n| **Parameter**       | **Type**   | **Description**                                            | **Example**                  |\n|---------------------|------------|------------------------------------------------------------|------------------------------|\n| `limit`             | `integer`  | Max results per input                                   | `limit=10`                   |\n| `include_errors`    | `boolean`  | Get error reports for troubleshooting                     | `include_errors=true`        |\n| `notify`            | `url`      | Webhook notification URL to be notified upon completion  | `notify=https://notify-me.com/` |\n| `format`            | `enum`     | Output format (e.g., JSON, NDJSON, JSONL, CSV)         | `format=json`                |\n\n💡 **Pro Tip:** You can also select whether to deliver the data to an [external storage](https://docs.brightdata.com/scraping-automation/web-data-apis/web-scraper-api/overview#via-deliver-to-external-storage) or to deliver it to a [webhook](https://docs.brightdata.com/scraping-automation/web-data-apis/web-scraper-api/overview#via-webhook).\n\n----\n\nNeed more details? Check the [official API docs](https://docs.brightdata.com/scraping-automation/web-data-apis/web-scraper-api/overview).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluminati-io%2Fgoogle-news-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluminati-io%2Fgoogle-news-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluminati-io%2Fgoogle-news-scraper/lists"}