{"id":19373379,"url":"https://github.com/xop/news-scraper-core","last_synced_at":"2026-07-15T12:39:50.206Z","repository":{"id":65457219,"uuid":"69747924","full_name":"XOP/news-scraper-core","owner":"XOP","description":"NewScraper Core","archived":false,"fork":false,"pushed_at":"2016-12-29T13:52:44.000Z","size":32,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-07T07:48:54.413Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/XOP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-01T15:56:48.000Z","updated_at":"2017-10-11T13:03:51.000Z","dependencies_parsed_at":"2023-01-24T12:15:16.558Z","dependency_job_id":null,"html_url":"https://github.com/XOP/news-scraper-core","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOP%2Fnews-scraper-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOP%2Fnews-scraper-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOP%2Fnews-scraper-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XOP%2Fnews-scraper-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/XOP","download_url":"https://codeload.github.com/XOP/news-scraper-core/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240500584,"owners_count":19811559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T08:28:01.089Z","updated_at":"2025-10-19T13:06:06.666Z","avatar_url":"https://github.com/XOP.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NewScraper Core Module\n\n[![npm version](https://badge.fury.io/js/news-scraper-core.svg)](https://badge.fury.io/js/news-scraper-core) [![npm dependencies](https://david-dm.org/stewiekillsloiss/news-scraper-core.svg)](https://david-dm.org/stewiekillsloiss/news-scraper-core)\n\n\n\n\u003e The core module for the NewScraper\n\u003e [https://github.com/XOP/news-scraper](https://github.com/XOP/news-scraper)\n\n\n\n## Goal\n\n**NewScraper Core Module** (NewScraper) is a NodeJS module, that receives specific directives as props and returns scraped pages data.\n\nBoth directives' and output' format is `JSON`.\n\nNewScraper is designed to be used as a **middleware** for a server / hybrid / CLI application.\n\n\n\n## API\n\n### Config\n\n`limit`      \nNumber, default: `undefined` (bypass)  \nDefines the default common limit; will overwrite directive's [Input -\u003e limit](#input)\n\n`output`  \nObject:\n```\n{ \n    path,\n    current\n}\n```\n\n`output.path`  \nString, default: \"./\"  \nPath to the scraped data directory\n\n`output.current`\nString, default: \"data.json\"  \nPath to the current data json file (used to filter previously shown news)\n\n`updateStrategy`  \nString, default: \"\"  \nDefines logic of the post-processing the scraped data:  \n`\"scratch\"` - ignores previous runs, creates new json file every new scraping round  \n`\"compare\"` - compares scraping results to the previous result, stores in `output.current` file (data.json by default)  \n`\"\"` - bypass, no scraping results saved\n\n`scraperOptions`  \nObject, default: {}  \nParameters to pass to the currently used scraper.  \nVersion 1.x - [Nightmare](http://www.nightmarejs.org/), find all options [here](https://github.com/segmentio/nightmare#api).\n\n\n### Input\n\nInput is the collection of directives in a `JSON` format.\n\n\u003e It is recommended for the application to store directives in a most readable format (e.g. `YAML`) and convert it on the fly to the `JSON`.\n\nExample:\n\n```\n[\n    {\n        \"title\": \"Smashing magazine\",\n        \"url\": \"http://www.smashingmagazine.com/\",\n        \"elem\": \"article.post\",\n        \"link\": \"h2 \u003e a\",\n        \"author\": \"h2 + ul li.a a\",\n        \"time\": \"h2 + ul li.rd\",\n        \"image\": \"figure \u003e a \u003e img\",\n        \"limit\": 6\n    },\n    {...},\n    {...}\n]\n```\n\n`title`  \nString  \nName of the resource, **required**\n\n`url`  \nString  \nUrl of the resource, **required**\n\n`elem`  \nString  \nCSS selector of the news item container element, **required**\n\n`link`  \nString  \nCSS selector of the link (\u003ca href=\"\"\u003e...\u003c/a\u003e) _inside_ of the `elem`\nIf the `elem` itself _is_ a link, this is not required\n\n`author`  \nString  \nCSS selector of the author element _inside_ of the `elem`\n\n`time`  \nString  \nCSS selector of the time element _inside_ of the `elem`\n\n`image`  \nString  \nCSS selector of the image element _inside_ of the `elem`\nThis one can be `img` tag or any other - NewScraper will search for `data-src` and `background-image` CSS properties to find proper image data\n\n`limit`  \nNumber  \nHow many `elem`-s from the `url` will be scraped, maximum  \nSee also: [Config -\u003e limit](#config)\n\n\n### Output\n\nOutput includes all Input data  \n`pages -\u003e [] -\u003e {...}`\n\n**Plus** the parsed scraping result, ready for the favourite templating engine  \n`pages -\u003e [] -\u003e {data -\u003e [] -\u003e {...}}`\n\n**Plus** the unmodified markup from the specified pages  \n`pages -\u003e [] -\u003e {data -\u003e [] -\u003e {raw}}`\n\n\nIt also contains some **meta-data**, such as path to the current data file and the exact moment of the scraping start.\n\nExample:\n\n```\n{\n    \"meta\": {\n        \"file\": \"/Users/[...]/data/1474811135645.json\",\n        \"date\": 1474811135645\n    },\n    \"pages\": [\n        {\n            \"url\": \"https://www.smashingmagazine.com\",\n            \"elem\": \"article.post\",\n            \"link\": \"h2 \u003e a\",\n            \"author\": \"h2 + ul li.a a\",\n            \"time\": \"h2 + ul li.rd\",\n            \"image\": \"figure \u003e a \u003e img\",\n            \"limit\": 6,\n            \"data\": [\n                {\n                \"href\": \"https://www.smashingmagazine.com/2016/09/interview-with-matan-stauber/\",\n                \"text\": \"\\n\\t\\t\\tAn Interview With Matan Stauber\\n\\t\\t\\tStretching The Limits Of What’s Possible\\n\\t\\t\",\n                \"title\": \"Read 'Stretching The Limits Of What’s Possible'\",\n                \"raw\": \"\u003carticle class=\\\"post-266432 post type-post status-publish format-standard has-post-thumbnail hentry category-general tag-interviews\\\" vocab=\\\"http://schema.org/\\\" typeof=\\\"TechArticle\\\"\u003e [ ... a lot of markup ... ] \u003c/article\u003e\",\n                \"author\": \"Cosima Mielke\",\n                \"time\": \"September 23rd, 2016\",\n                \"imageSrc\": \"https://www.smashingmagazine.com/wp-content/uploads/2016/09/histography-website-small-opt.png\"\n                },\n                {... x5}\n            ]\n        },\n        {...},\n        {...}\n    ]\n```\n\n\n## Events\n\n:construction: coming up!\n\n\n## [MIT License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxop%2Fnews-scraper-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxop%2Fnews-scraper-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxop%2Fnews-scraper-core/lists"}