{"id":27110323,"url":"https://github.com/njraladdin/newspapers-com-scraper","last_synced_at":"2025-04-06T23:50:43.303Z","repository":{"id":262912187,"uuid":"864077377","full_name":"njraladdin/newspapers-com-scraper","owner":"njraladdin","description":"A Node.js scraper for extracting article data from Newspapers.com based on keywords, dates, and locations.","archived":false,"fork":false,"pushed_at":"2024-12-26T17:30:00.000Z","size":164,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T14:39:16.602Z","etag":null,"topics":["archive","data","newspapers","scraper","scraper-api","scraping"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/newspapers-com-scraper","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njraladdin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-27T12:58:56.000Z","updated_at":"2025-02-19T09:42:29.000Z","dependencies_parsed_at":"2024-11-15T01:21:10.679Z","dependency_job_id":"a67ad374-e789-42f3-becd-a3e71f7f66f8","html_url":"https://github.com/njraladdin/newspapers-com-scraper","commit_stats":null,"previous_names":["njraladdin/newspapers-com-scraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njraladdin%2Fnewspapers-com-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njraladdin%2Fnewspapers-com-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njraladdin%2Fnewspapers-com-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njraladdin%2Fnewspapers-com-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njraladdin","download_url":"https://codeload.github.com/njraladdin/newspapers-com-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247569117,"owners_count":20959758,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive","data","newspapers","scraper","scraper-api","scraping"],"created_at":"2025-04-06T23:50:42.784Z","updated_at":"2025-04-06T23:50:43.271Z","avatar_url":"https://github.com/njraladdin.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Newspaper.com Scraper\n\n[![npm version](https://badge.fury.io/js/newspapers-com-scraper.svg)](https://badge.fury.io/js/newspapers-com-scraper)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![npm total downloads](https://img.shields.io/npm/dt/newspapers-com-scraper.svg)](https://www.npmjs.com/package/newspapers-com-scraper)\n\nA Node.js scraper for extracting article data from Newspapers.com based on keywords, dates, and locations.\n\n```javascript\nconst scraper = new NewspaperScraper();\n\nawait scraper.retrieve({\n    keyword: \"elon musk twitter\",\n    limit: 500,\n    dateRange: [2020, 2024],\n    location: \"us\"\n});\n```\n\n## What it Does\n\nSearches Newspapers.com and extracts:\n\n- Newspaper title\n- Page number and URL\n- Publication date\n- Location\n- Number of keyword matches on each page\n\n[Sample Output](https://docs.google.com/spreadsheets/d/1uq366pyEfolITFZ9X507ogsQjssx_pL1bp1pGPIPtt4/edit?gid=0#gid=0)\n\n## Requirements\n\n- Node.js 14+\n- Google Chrome browser\n- GEONODE.com account (optional, for proxy support)\n\n## Installation\n\n```bash\nnpm install newspapers-com-scraper\n```\n\n## Basic Usage\n\n```javascript\nconst NewspaperScraper = require('newspapers-com-scraper');\n\nasync function main() {\n    const scraper = new NewspaperScraper();\n\n    // Listen for articles\n    scraper.on('article', (article) =\u003e {\n        console.log(`Found: ${article.title} (${article.date})`);\n    });\n\n    await scraper.retrieve({\n        keyword: \"elon musk twitter\",  // Required: search term\n        limit: 500,                    // Optional: limit total results\n        dateRange: [2020, 2024],       // Optional: date range\n        location: \"us\"                 // Optional: location code\n    });\n}\n```\n\n## Events\n\nThe scraper emits three types of events:\n\n```javascript\n// 1. Article found\nscraper.on('article', (article) =\u003e {\n    console.log(article);\n    // {\n    //     title: \"The Daily News\",\n    //     pageNumber: 4,\n    //     date: \"2023-05-15\",\n    //     location: \"New York, NY\",\n    //     keywordMatches: 3,\n    //     url: \"https://www.newspapers.com/image/12345678/\"\n    // }\n});\n\n// 2. Progress update\nscraper.on('progress', (progress) =\u003e {\n    console.log(progress);\n    // {\n    //     current: 5,              // Current page\n    //     total: 20,              // Total pages\n    //     percentage: 25.0,       // Progress percentage\n    //     stats: {\n    //         timeElapsed: 45.2,  // Total seconds\n    //         avgPageTime: 9.04   // Avg seconds per page\n    //     }\n    // }\n});\n\n// 3. Scraping complete\nscraper.on('complete', (stats) =\u003e {\n    console.log(stats);\n    // {\n    //     timeElapsed: 180.5,     // Total seconds\n    //     pageTimes: [8.2, 9.1]   // Time per page\n    // }\n});\n```\n\n## Advanced Configuration\n\nFull configuration example:\n\n```javascript\nconst scraper = new NewspaperScraper({\n    // Scraping settings\n    concurrentPages: 2,        // Pages to scrape in parallel\n    resultsPerPage: 50,        // Results per page (max 50)\n    maxConcurrentRequests: 10, // Max parallel requests\n    \n    // Browser settings\n    browser: {\n        headless: false,       // Show browser\n        userAgent: 'Mozilla/5.0...',\n        executablePath: '/path/to/chrome' // Optional\n    },\n    \n    // Proxy settings (optional)\n    proxy: {\n        enabled: false,\n        host: 'proxy.host',\n        port: 9008,\n        username: 'user',\n        password: 'pass'\n    },\n    \n    // Logging\n    logger: {\n        level: 'info',        // 'error' | 'warn' | 'info' | 'debug' | 'silent'\n        custom: null          // Custom logger\n    }\n});\n\nawait scraper.retrieve({\n    keyword: \"elon musk twitter\",\n    limit: 500,\n    dateRange: [2020, 2024],\n    location: \"us\"\n});\n\n// If using proxy, set up .env:\n// PROXY_HOST=your_geonode_proxy_host\n// PROXY_USER=your_geonode_username\n// PROXY_PASS=your_geonode_password\n```\nSee `examples/main.js` for a complete working example.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjraladdin%2Fnewspapers-com-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjraladdin%2Fnewspapers-com-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjraladdin%2Fnewspapers-com-scraper/lists"}