https://github.com/scrapingant/scrapingant-client-js
ScrapingAnt API client for JavaScript / Node.js.
https://github.com/scrapingant/scrapingant-client-js
crawler scraper scraping scrapingant webscraping
Last synced: 5 months ago
JSON representation
ScrapingAnt API client for JavaScript / Node.js.
- Host: GitHub
- URL: https://github.com/scrapingant/scrapingant-client-js
- Owner: ScrapingAnt
- License: apache-2.0
- Created: 2021-02-07T18:07:58.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2022-05-06T20:33:12.000Z (about 3 years ago)
- Last Synced: 2024-12-05T22:08:47.702Z (6 months ago)
- Topics: crawler, scraper, scraping, scrapingant, webscraping
- Language: JavaScript
- Homepage: https://www.npmjs.com/package/@scrapingant/scrapingant-client
- Size: 32.2 KB
- Stars: 12
- Watchers: 3
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ScrapingAnt API client for JavaScript
`@scrapingant/scrapingant-client` is the official library to access [ScrapingAnt API](https://docs.scrapingant.com) from your
JavaScript applications. It runs both in Node.js and browser and provides useful features like
automatic retries and parameters encoding to improve the ScrapingAnt usage experience.
- [Quick Start](#quick-start)
- [API key](#api-key)
- [Retries with exponential backoff](#retries-with-exponential-backoff)
- [API Reference](#api-reference)
- [Examples](#examples)
## Quick Start
```js
const ScrapingAntClient = require('@scrapingant/scrapingant-client');
const client = new ScrapingAntClient({ apiKey: '' });
// Scrape the example.com site.
client.scrape('https://example.com')
.then(res => console.log(res))
.catch(err => console.error(err.message));
```
## API key
In order to get API key you'll need to register at [ScrapingAnt Service](https://app.scrapingant.com)
## Retries with exponential backoff
Network communication sometimes fails, that's a given. The client will automatically retry requests that
failed due to a network error, an internal error of the ScrapingAnt API (HTTP 500+).
By default, it will retry up to 8 times. First retry will be attempted after ~500ms, second after ~1000ms
and so on. You can configure those parameters using the `maxRetries` and `minDelayBetweenRetriesMillis`
options of the `ScrapingAntClient` constructor.
## API Reference
All public classes, methods and their parameters can be inspected in this API reference.
### [](#ScrapingAntClient) ScrapingAntClient
ScrapingAntClient is the official library to access [ScrapingAnt API](https://docs.scrapingant.com) from your
JavaScript applications. It runs both in Node.js and browser.
* [ScrapingAntClient](#ScrapingAntClient)
* [`new ScrapingAntClient(options)`](#new_ScrapingAntClient_new)
* [`.scrape(url, [params])`](#ScrapingAntClient+scrape) ⇒ [ScrapingAnt API response](https://docs.scrapingant.com/request-response-format#response-structure)
* * *
#### [](#ScrapingAntClient) `new ScrapingAntClient(options)`
| Param | Type | Default |
|----------------------------------------|---------------------|------------------|
| [options] | object | |
| [options.maxRetries] | number | 8 |
| [options.minDelayBetweenRetriesMillis] | number | 500 |
| [options.timeoutSecs] | number | 60 |
| [options.apiKey] | string | |
* * *
#### [](#ScrapingAntClient+scrape) `scrapingAntClient.scrape(url, [parameters])` ⇒ [ScrapingAnt API response](https://docs.scrapingant.com/request-response-format#response-structure)
https://docs.scrapingant.com/request-response-format#available-parameters
| Param | Type |
|--------------------------------|----------------------|
| url | string |
| [parameters] | object |
| [parameters.browser] | boolean |
| [parameters.cookies] | string |
| [parameters.headers] | object |
| [parameters.js_snippet] | string |
| [parameters.proxy_type] | string |
| [parameters.proxy_country] | string |
| [parameters.wait_for_selector] | string |
| [parameters.return_text] | boolean |
**IMPORTANT NOTE:** parameters.js_snippet will be encoded to Base64 automatically by the ScrapingAnt JS client library.
* * *
### [](#ScrapingAntApiError) ScrapingAntApiError
An `ScrapingAntApiError` is thrown for successful HTTP requests that reach the API,
but the API responds with an error response. Typically, those are internal errors,
which are automatically retried, or validation errors, which are thrown immediately,
because a correction by the user is needed.
**Properties**
| Name | Type | Description |
|------------|---------------------|------------------------------------|
| message | string | Error message returned by the API. |
| statusCode | number | HTTP status code of the error. |
| httpMethod | string | HTTP method of the API call. |
* * *
## Examples
### Using residential proxy
```js
const ScrapingAntClient = require('@scrapingant/scrapingant-client');
const client = new ScrapingAntClient({ apiKey: '' });
// Get the residential IP info using httpbin.org
client.scrape('https://httpbin.org/ip', { proxy_type: 'residential' })
.then(res => console.log(res))
.catch(err => console.error(err.message));
```
### Sending custom cookies
```js
const ScrapingAntClient = require('@scrapingant/scrapingant-client');
const client = new ScrapingAntClient({ apiKey: '' });
// Scrape the httpbin.org site and get all the cookies sent before
client.scrape('https://httpbin.org/cookies', { cookies: 'cookieName1=cookieVal1;cookieName2=cookieVal2' })
.then(res => console.log(res))
.catch(err => console.error(err.message));
```
### Adding custom headers
```js
const ScrapingAntClient = require('@scrapingant/scrapingant-client');
const client = new ScrapingAntClient({ apiKey: '' });
// Scrape the httpbin.org site and get all the headers that would be sent before
client.scrape('https://httpbin.org/headers', { headers: { scraping: "is cool!" } })
.then(res => console.log(res))
.catch(err => console.error(err.message));
```
### Executing custom JS snippet
```js
const ScrapingAntClient = require('@scrapingant/scrapingant-client');
const client = new ScrapingAntClient({ apiKey: '' });
// Scrape the httpbin.org site and replace all the content with "Hello, world"
const customJsSnippet = "var str = 'Hello, world!';\n" +
"var htmlElement = document.getElementsByTagName('html')[0];\n" +
"htmlElement.innerHTML = str;"
client.scrape('https://httpbin.org/cookies', { js_snippet: customJsSnippet })
.then(res => console.log(res))
.catch(err => console.error(err.message));
```