{"id":15705963,"url":"https://github.com/kyleross/node-scraper-api","last_synced_at":"2025-03-30T15:45:50.062Z","repository":{"id":41055486,"uuid":"201303547","full_name":"KyleRoss/node-scraper-api","owner":"KyleRoss","description":"Interface to call https://www.scraperapi.com easily from Node.","archived":false,"fork":false,"pushed_at":"2022-06-19T19:22:32.000Z","size":16,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-30T00:43:52.624Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KyleRoss.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-08T17:11:32.000Z","updated_at":"2019-12-03T09:03:04.000Z","dependencies_parsed_at":"2022-09-20T22:46:18.468Z","dependency_job_id":null,"html_url":"https://github.com/KyleRoss/node-scraper-api","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleRoss%2Fnode-scraper-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleRoss%2Fnode-scraper-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleRoss%2Fnode-scraper-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleRoss%2Fnode-scraper-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KyleRoss","download_url":"https://codeload.github.com/KyleRoss/node-scraper-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246342824,"owners_count":20761939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T20:20:52.033Z","updated_at":"2025-03-30T15:45:50.042Z","avatar_url":"https://github.com/KyleRoss.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scraper-api\n[![npm](https://img.shields.io/npm/v/scraper-api.svg?style=for-the-badge)](https://www.npmjs.com/package/scraper-api) [![npm](https://img.shields.io/npm/dt/scraper-api.svg?style=for-the-badge)](https://www.npmjs.com/package/scraper-api) [![David](https://img.shields.io/david/KyleRoss/node-scraper-api.svg?style=for-the-badge)](https://david-dm.org/KyleRoss/node-scraper-api) [![Travis](https://img.shields.io/travis/KyleRoss/node-scraper-api/master.svg?style=for-the-badge)](https://travis-ci.org/KyleRoss/node-scraper-api) [![Coveralls](https://img.shields.io/coveralls/github/KyleRoss/node-scraper-api.svg?style=for-the-badge)](https://coveralls.io/github/KyleRoss/node-scraper-api) [![license](https://img.shields.io/github/license/KyleRoss/node-scraper-api.svg?style=for-the-badge)](https://github.com/KyleRoss/node-scraper-api/blob/master/LICENSE) [![Beerpay](https://img.shields.io/beerpay/KyleRoss/node-scraper-api.svg?style=for-the-badge)](https://beerpay.io/KyleRoss/node-scraper-api)\n\nInterface to call [ScraperAPI.com](https://www.scraperapi.com) easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.\n\n## Install\n\n```bash\n$ npm install --save scraper-api\n```\n\n## Usage\n```js\nconst scraperAPI = require('scraper-api')({\n    // ... options\n});\n\nscraperAPI.get('http://httpbin.org/ip')\n    .then(result =\u003e {\n        // result =\u003e '\u003c!doctype html\u003e ...'\n    })\n    .catch(error =\u003e {\n        console.error(error);\n        //=\u003e 'Internal server error'\n    });\n```\n\n# API Documentation\n\u003ca name=\"module_scraper-api\"\u003e\u003c/a\u003e\n\n## scraper-api\n\n* [scraper-api](#module_scraper-api)\n    * [scraperAPI(options)](#exp_module_scraper-api--scraperAPI) ⇒ [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI) ⏏\n        * [.ScraperAPI](#module_scraper-api--scraperAPI.ScraperAPI)\n\n\u003ca name=\"exp_module_scraper-api--scraperAPI\"\u003e\u003c/a\u003e\n\n### scraperAPI(options) ⇒ [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI) ⏏\nCreates new instance of ScraperAPI with the provided options.\n\n**Kind**: Exported function  \n**Returns**: [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI) - New instance of ScraperAPI.  \n\n| Param | Type | Default | Description |\n| --- | --- | --- | --- |\n| options | \u003ccode\u003eObject\u003c/code\u003e |  | Optional configuration options to pass into ScraperAPI. |\n| [options.apiKey] | \u003ccode\u003eString\u003c/code\u003e | \u003ccode\u003eprocess.env.SCRAPER_API_KEY\u003c/code\u003e | API key for Scraper API. Defaults to pulling API Key from environment variable. |\n| [options.renderJs] | \u003ccode\u003eBoolean\u003c/code\u003e | \u003ccode\u003efalse\u003c/code\u003e | Render JavaScript on the page before scraping the HTML for the page. |\n| [options.keepHeaders] | \u003ccode\u003eBoolean\u003c/code\u003e | \u003ccode\u003efalse\u003c/code\u003e | Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in `options.gotOptions.headers`. |\n| [options.geoCode] | \u003ccode\u003eString\u003c/code\u003e | \u003ccode\u003e\u0026#x27;us\u0026#x27;\u003c/code\u003e | Geo code in which to use proxies for when scraping. See [documentation](https://www.scraperapi.com/documentation#geographic-location) for more information. |\n| [options.premium] | \u003ccode\u003eBoolean\u003c/code\u003e | \u003ccode\u003efalse\u003c/code\u003e | Whether to use premium proxies. **Caution:** This will cost 10-25 times more than standard proxies. |\n| [options.sessionId] | \u003ccode\u003eNumber\u003c/code\u003e |  | A numeric session id to use to maintain the same proxy. See [ScraperAPI.session()](#) for more information. |\n| [options.gotOptions] | \u003ccode\u003eObject\u003c/code\u003e | \u003ccode\u003e{}\u003c/code\u003e | Additional options to pass into [got](https://www.npmjs.com/package/got) for requests to ScraperAPI. |\n\n**Example**  \n```js\nconst scraperAPI = require('scraper-api')({\n    // options...\n});\n```\n\u003ca name=\"module_scraper-api--scraperAPI.ScraperAPI\"\u003e\u003c/a\u003e\n\n#### scraperAPI.ScraperAPI\nAccess to the uninstantiated ScraperAPI class.\n\n**Kind**: static property of [\u003ccode\u003escraperAPI\u003c/code\u003e](#exp_module_scraper-api--scraperAPI)  \n**Example**  \n```js\nconst ScraperAPI = require('scraper-api').ScraperAPI;\nconst scraperAPI = new ScraperAPI({\n    // options...\n});\n```\n\u003ca name=\"ScraperAPI\"\u003e\u003c/a\u003e\n\n## ScraperAPI\n**Kind**: global class  \n\n* [ScraperAPI](#ScraperAPI)\n    * [new ScraperAPI([options])](#new_ScraperAPI_new)\n    * [scraperAPI.session(id, [options])](#ScraperAPI+session) ⇒ [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI)\n    * [scraperAPI.get(url, [options])](#ScraperAPI+get) ⇒ \u003ccode\u003ePromise.\u0026lt;String\u0026gt;\u003c/code\u003e\n    * [scraperAPI.post(url, data, [options])](#ScraperAPI+post) ⇒ \u003ccode\u003ePromise.\u0026lt;Object\u0026gt;\u003c/code\u003e\n\n\u003ca name=\"new_ScraperAPI_new\"\u003e\u003c/a\u003e\n\n### new ScraperAPI([options])\nConstructor for the ScraperAPI class.\n\n\n| Param | Type | Default | Description |\n| --- | --- | --- | --- |\n| [options] | \u003ccode\u003eObject\u003c/code\u003e | \u003ccode\u003e{}\u003c/code\u003e | Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information. |\n\n\u003ca name=\"ScraperAPI+session\"\u003e\u003c/a\u003e\n\n### scraperAPI.session(id, [options]) ⇒ [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI)\nCreates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through \nthe same proxy. See [documentation](https://www.scraperapi.com/documentation#sessions) for more information.\n\n**Kind**: instance method of [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI)  \n**Returns**: [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI) - A new instance of ScraperAPI with `sessionId` set to the provided id.  \n\n| Param | Type | Default | Description |\n| --- | --- | --- | --- |\n| id | \u003ccode\u003eNumber\u003c/code\u003e |  | Session ID in which to use for the given session. Must only contain numbers. |\n| [options] | \u003ccode\u003eObject\u003c/code\u003e | \u003ccode\u003e{}\u003c/code\u003e | Options to override for all subsequent requests to Scraper API. Same as the global options. |\n\n**Example**  \n```js\nconst session = scraperAPI.session(1234);\n\nlet result = await scraperAPI.get('https://google.com');\n// result -\u003e '\u003c!doctype html\u003e ...'\n```\n\u003ca name=\"ScraperAPI+get\"\u003e\u003c/a\u003e\n\n### scraperAPI.get(url, [options]) ⇒ \u003ccode\u003ePromise.\u0026lt;String\u0026gt;\u003c/code\u003e\nCalls Scraper API with a GET request to the provided url.\n\n**Kind**: instance method of [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI)  \n**Returns**: \u003ccode\u003ePromise.\u0026lt;String\u0026gt;\u003c/code\u003e - Promise that resolves with HTML source from requested URL.  \n\n| Param | Type | Default | Description |\n| --- | --- | --- | --- |\n| url | \u003ccode\u003eString\u003c/code\u003e |  | The URL in which to scrape. |\n| [options] | \u003ccode\u003eObject\u003c/code\u003e | \u003ccode\u003e{}\u003c/code\u003e | Options to override for this specific request. |\n\n**Example**  \n```js\nlet result = await scraperAPI.get('https://google.com');\n// result -\u003e '\u003c!doctype html\u003e ...'\n```\n\u003ca name=\"ScraperAPI+post\"\u003e\u003c/a\u003e\n\n### scraperAPI.post(url, data, [options]) ⇒ \u003ccode\u003ePromise.\u0026lt;Object\u0026gt;\u003c/code\u003e\nCalls Scraper API with a POST request to the provided url with the provided data.\n\n**Kind**: instance method of [\u003ccode\u003eScraperAPI\u003c/code\u003e](#ScraperAPI)  \n**Returns**: \u003ccode\u003ePromise.\u0026lt;Object\u0026gt;\u003c/code\u003e - Promise that resolves with object response from Scraper API. See [documentation](https://www.scraperapi.com/documentation#post-requests).  \n\n| Param | Type | Default | Description |\n| --- | --- | --- | --- |\n| url | \u003ccode\u003eString\u003c/code\u003e |  | The URL in which to scrape. |\n| data | \u003ccode\u003eObject\u003c/code\u003e \\| [\u003ccode\u003eform-data\u003c/code\u003e](https://github.com/form-data/form-data) |  | Data in which to post to the provided URL. Must be either a plain object or instance of [form-data](https://github.com/form-data/form-data). |\n| [options] | \u003ccode\u003eObject\u003c/code\u003e | \u003ccode\u003e{}\u003c/code\u003e | Options to override for this specific request. May be any of the global options and any additional options below. |\n| [options.form] | \u003ccode\u003eBoolean\u003c/code\u003e | \u003ccode\u003efalse\u003c/code\u003e | Set `true` if provided `data` is form data and should be sent as such. By default, data will be sent as JSON. |\n\n**Example**  \n```js\nlet result = await scraperAPI.post('https://example.com/endpoint', {\n   hello: 'world',\n   some: 'data'\n});\n```\n---\n\n## More Information\n\n### keepHeaders Option\nIf you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in `options.gotOptions.headers` and enabling this option.\n\n```js\nlet result = await scraperAPI.get('https://google.com', {\n    keepHeaders: true,\n    gotOptions: {\n        headers: {\n            'My-Custom-Header': 'some value'\n        }\n    }\n});\n```\n\n---\n\n## Tests\nTests are written and provided as part of the module. You may run the tests by calling:\n\n```bash\n$ npm run test\n```\n\n## License\nMIT License. See [License](https://github.com/KyleRoss/node-scraper-api/blob/master/LICENSE) in the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyleross%2Fnode-scraper-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyleross%2Fnode-scraper-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyleross%2Fnode-scraper-api/lists"}