Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zgreen/scrape-to-feed
Serve feeds from scraped HTML.
https://github.com/zgreen/scrape-to-feed
feed rss scraper
Last synced: about 23 hours ago
JSON representation
Serve feeds from scraped HTML.
- Host: GitHub
- URL: https://github.com/zgreen/scrape-to-feed
- Owner: zgreen
- Created: 2018-01-30T02:03:31.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2022-12-07T12:41:36.000Z (almost 2 years ago)
- Last Synced: 2024-04-29T13:21:53.820Z (7 months ago)
- Topics: feed, rss, scraper
- Language: JavaScript
- Size: 33.2 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# scrape-to-feed
Serve feeds from scraped HTML.
## Install
`npm install scrape-to-feed`
## Usage
1. At the root of your project, create a `feeds` directory. This directory will contain exported modules for defining your feeds, one feed per file.
2. Add a feed. Here is an example, `./feeds/nyt-example-feed.js`:
```js
module.exports = {
url: "https://www.nytimes.com/",
handler: $ => [
{ title: "NYT example feed" },
...$(".story-heading")
.toArray()
.map(function(story) {
const $story = $(story);
return {
item: {
title: $story.text(),
link: $story.find("a").attr("href")
}
};
})
]
};
```
3. Create and serve your feeds:
```js
#!/usr/bin/env node
require("scrape-to-feed")();
```
4. Your feed will be available at: `localhost:3000/nyt-example-feed`
## Arguments
By default, all feeds will be re-scaped and rebuilt every hour. You can pass a custom interval, like so:
```js
require("scrape-to-feed")(1000 * 60 * 60 * 24); // Update feeds once per day.
```