https://github.com/harrystevens/scraperama
Scrape files from the internet.
https://github.com/harrystevens/scraperama
Last synced: about 1 year ago
JSON representation
Scrape files from the internet.
- Host: GitHub
- URL: https://github.com/harrystevens/scraperama
- Owner: HarryStevens
- License: mit
- Created: 2020-11-05T18:48:44.000Z (almost 6 years ago)
- Default Branch: main
- Last Pushed: 2025-06-24T19:58:00.000Z (about 1 year ago)
- Last Synced: 2025-06-24T20:15:25.800Z (about 1 year ago)
- Language: JavaScript
- Homepage:
- Size: 90.8 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# scraperama
Scrape files from the internet.
## Installation
```bash
npm i scraperama -S
```
## Usage
```js
const scraperama = require("scraperama");
scraperama.csv("file.csv", json => {
console.log(json);
});
scraperama.html("file.html", $ => {
console.log($("body").html());
});
scraperama.json("file.json", json => {
console.log(json);
});
scraperama.text("file.txt", text => {
console.log(text);
});
```
To return a YYYY-MM-DD datestamp (useful for file naming):
```js
scraperama.datestamp(); // current date
scraperama.datestamp(new Date(1999, 0, 1)); // "1999-01-01"
scraperama.datestamp("foo"); // throws a type error
```
To download a file from the Internet:
```js
scraperama.download(
"https://path/to/file.zip", // URL
"path/to/file.zip", // local file path
(pct) => { process.stdout.write(`\r${pct.toFixed(1)}%`); }, // log percentage downloaded
(err) => {
if (err) console.error(err);
console.log("Done!");
} // callback function
);
```
To get an object's file size:
```js
scraperama.filesize(object);
```
To throttle a function:
```js
const logThrottled = scraperama.throttle(console.log, 500);
Array.from({ length: 10 }).forEach((_, i) => logThrottled(i));
```
To untar a local file:
```js
scraperama.untar(
"path/to/file.tar", // input tar file
"path/to/dir", // output directory
(err) => {
if (err) console.error(err);
console.log("Done!");
} // callback function
);
```
To unzip a local file:
```js
scraperama.unzip(
"path/to/file.tar", // input tar file
"path/to/dir", // output directory
(err) => {
if (err) console.error(err);
console.log("Done!");
} // callback function
);
```