https://github.com/kyleross/node-scraper-api
Interface to call https://www.scraperapi.com easily from Node.
https://github.com/kyleross/node-scraper-api
Last synced: over 1 year ago
JSON representation
Interface to call https://www.scraperapi.com easily from Node.
- Host: GitHub
- URL: https://github.com/kyleross/node-scraper-api
- Owner: KyleRoss
- License: mit
- Created: 2019-08-08T17:11:32.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2022-06-19T19:22:32.000Z (about 4 years ago)
- Last Synced: 2024-10-30T00:43:52.624Z (over 1 year ago)
- Language: JavaScript
- Size: 15.6 KB
- Stars: 1
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# scraper-api
[](https://www.npmjs.com/package/scraper-api) [](https://www.npmjs.com/package/scraper-api) [](https://david-dm.org/KyleRoss/node-scraper-api) [](https://travis-ci.org/KyleRoss/node-scraper-api) [](https://coveralls.io/github/KyleRoss/node-scraper-api) [](https://github.com/KyleRoss/node-scraper-api/blob/master/LICENSE) [](https://beerpay.io/KyleRoss/node-scraper-api)
Interface to call [ScraperAPI.com](https://www.scraperapi.com) easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.
## Install
```bash
$ npm install --save scraper-api
```
## Usage
```js
const scraperAPI = require('scraper-api')({
// ... options
});
scraperAPI.get('http://httpbin.org/ip')
.then(result => {
// result => ' ...'
})
.catch(error => {
console.error(error);
//=> 'Internal server error'
});
```
## scraper-api
* [scraper-api](#module_scraper-api)
* [scraperAPI(options)](#exp_module_scraper-api--scraperAPI) ⇒ [ScraperAPI](#ScraperAPI) ⏏
* [.ScraperAPI](#module_scraper-api--scraperAPI.ScraperAPI)
### scraperAPI(options) ⇒ [ScraperAPI](#ScraperAPI) ⏏
Creates new instance of ScraperAPI with the provided options.
**Kind**: Exported function
**Returns**: [ScraperAPI](#ScraperAPI) - New instance of ScraperAPI.
| Param | Type | Default | Description |
| --- | --- | --- | --- |
| options | Object | | Optional configuration options to pass into ScraperAPI. |
| [options.apiKey] | String | process.env.SCRAPER_API_KEY | API key for Scraper API. Defaults to pulling API Key from environment variable. |
| [options.renderJs] | Boolean | false | Render JavaScript on the page before scraping the HTML for the page. |
| [options.keepHeaders] | Boolean | false | Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in `options.gotOptions.headers`. |
| [options.geoCode] | String | 'us' | Geo code in which to use proxies for when scraping. See [documentation](https://www.scraperapi.com/documentation#geographic-location) for more information. |
| [options.premium] | Boolean | false | Whether to use premium proxies. **Caution:** This will cost 10-25 times more than standard proxies. |
| [options.sessionId] | Number | | A numeric session id to use to maintain the same proxy. See [ScraperAPI.session()](#) for more information. |
| [options.gotOptions] | Object | {} | Additional options to pass into [got](https://www.npmjs.com/package/got) for requests to ScraperAPI. |
**Example**
```js
const scraperAPI = require('scraper-api')({
// options...
});
```
#### scraperAPI.ScraperAPI
Access to the uninstantiated ScraperAPI class.
**Kind**: static property of [scraperAPI](#exp_module_scraper-api--scraperAPI)
**Example**
```js
const ScraperAPI = require('scraper-api').ScraperAPI;
const scraperAPI = new ScraperAPI({
// options...
});
```
## ScraperAPI
**Kind**: global class
* [ScraperAPI](#ScraperAPI)
* [new ScraperAPI([options])](#new_ScraperAPI_new)
* [scraperAPI.session(id, [options])](#ScraperAPI+session) ⇒ [ScraperAPI](#ScraperAPI)
* [scraperAPI.get(url, [options])](#ScraperAPI+get) ⇒ Promise.<String>
* [scraperAPI.post(url, data, [options])](#ScraperAPI+post) ⇒ Promise.<Object>
### new ScraperAPI([options])
Constructor for the ScraperAPI class.
| Param | Type | Default | Description |
| --- | --- | --- | --- |
| [options] | Object | {} | Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information. |
### scraperAPI.session(id, [options]) ⇒ [ScraperAPI](#ScraperAPI)
Creates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through
the same proxy. See [documentation](https://www.scraperapi.com/documentation#sessions) for more information.
**Kind**: instance method of [ScraperAPI](#ScraperAPI)
**Returns**: [ScraperAPI](#ScraperAPI) - A new instance of ScraperAPI with `sessionId` set to the provided id.
| Param | Type | Default | Description |
| --- | --- | --- | --- |
| id | Number | | Session ID in which to use for the given session. Must only contain numbers. |
| [options] | Object | {} | Options to override for all subsequent requests to Scraper API. Same as the global options. |
**Example**
```js
const session = scraperAPI.session(1234);
let result = await scraperAPI.get('https://google.com');
// result -> ' ...'
```
### scraperAPI.get(url, [options]) ⇒ Promise.<String>
Calls Scraper API with a GET request to the provided url.
**Kind**: instance method of [ScraperAPI](#ScraperAPI)
**Returns**: Promise.<String> - Promise that resolves with HTML source from requested URL.
| Param | Type | Default | Description |
| --- | --- | --- | --- |
| url | String | | The URL in which to scrape. |
| [options] | Object | {} | Options to override for this specific request. |
**Example**
```js
let result = await scraperAPI.get('https://google.com');
// result -> ' ...'
```
### scraperAPI.post(url, data, [options]) ⇒ Promise.<Object>
Calls Scraper API with a POST request to the provided url with the provided data.
**Kind**: instance method of [ScraperAPI](#ScraperAPI)
**Returns**: Promise.<Object> - Promise that resolves with object response from Scraper API. See [documentation](https://www.scraperapi.com/documentation#post-requests).
| Param | Type | Default | Description |
| --- | --- | --- | --- |
| url | String | | The URL in which to scrape. |
| data | Object \| [form-data](https://github.com/form-data/form-data) | | Data in which to post to the provided URL. Must be either a plain object or instance of [form-data](https://github.com/form-data/form-data). |
| [options] | Object | {} | Options to override for this specific request. May be any of the global options and any additional options below. |
| [options.form] | Boolean | false | Set `true` if provided `data` is form data and should be sent as such. By default, data will be sent as JSON. |
**Example**
```js
let result = await scraperAPI.post('https://example.com/endpoint', {
hello: 'world',
some: 'data'
});
```
---
## More Information
### keepHeaders Option
If you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in `options.gotOptions.headers` and enabling this option.
```js
let result = await scraperAPI.get('https://google.com', {
keepHeaders: true,
gotOptions: {
headers: {
'My-Custom-Header': 'some value'
}
}
});
```
---
## Tests
Tests are written and provided as part of the module. You may run the tests by calling:
```bash
$ npm run test
```
## License
MIT License. See [License](https://github.com/KyleRoss/node-scraper-api/blob/master/LICENSE) in the repository.