{"id":15013522,"url":"https://github.com/veliovgroup/spiderable-middleware","last_synced_at":"2025-04-12T04:44:55.051Z","repository":{"id":5672743,"uuid":"53707501","full_name":"veliovgroup/spiderable-middleware","owner":"veliovgroup","description":"Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ideal for PWAs, SPAs, and modern JS-driven apps, websites, and webpages","archived":false,"fork":false,"pushed_at":"2025-04-10T10:43:14.000Z","size":631,"stargazers_count":40,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-12T04:44:42.147Z","etag":null,"topics":["crawler","meteor","meteor-package","middleware","nodejs","npm","npm-package","seo","seo-optimization","spiderable"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/spiderable-middleware","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/veliovgroup.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-12T01:39:02.000Z","updated_at":"2025-04-10T10:43:17.000Z","dependencies_parsed_at":"2024-01-31T23:43:43.367Z","dependency_job_id":"c1dda981-c83a-47aa-b318-e5dc6ab270f7","html_url":"https://github.com/veliovgroup/spiderable-middleware","commit_stats":{"total_commits":165,"total_committers":2,"mean_commits":82.5,"dds":"0.042424242424242475","last_synced_commit":"ed4a665cb672a6a62ea974306c8f61c67b267c5e"},"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veliovgroup%2Fspiderable-middleware","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veliovgroup%2Fspiderable-middleware/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veliovgroup%2Fspiderable-middleware/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/veliovgroup%2Fspiderable-middleware/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/veliovgroup","download_url":"https://codeload.github.com/veliovgroup/spiderable-middleware/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248519473,"owners_count":21117757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","meteor","meteor-package","middleware","nodejs","npm","npm-package","seo","seo-optimization","spiderable"],"created_at":"2024-09-24T19:44:23.375Z","updated_at":"2025-04-12T04:44:55.004Z","avatar_url":"https://github.com/veliovgroup.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spiderable middleware\n\nSearch engines and social networks — such as Google, Facebook, Twitter, Yahoo, Bing, and many others — are constantly crawling your website to index its content. However, if your site is built using a modern JavaScript framework (*for example, React, Preact, Vue, Svelte, Angular, Backbone, Ember, Meteor, Blaze*), it serve only a basic HTML skeleton with script tags instead of fully rendered page content. This limitation negatively impacts website's SEO score, performance, and user experience when links shared by users around the web. The mission of `spiderable-middleware` and [`ostr.io`](https://ostr.io) is to enhance SEO and performance for modern JavaScript websites.\n\n## Why Pre-render?\n\n- 🕸 Execute JavaScript, — get rendered HTML page and its content;\n- 🏃‍♂️ Boost response rate and decrease response time with caching;\n- 🚀 Optimized HTML markup for the best SEO score;\n- 🖥 Support PWAs and SPAs;\n- 📱 Support mobile-like crawlers;\n- 💅 Support [`styled-components`](https://styled-components.com);\n- ⚡️ Support [AMP (Accelerated Mobile Pages)](https://www.ampproject.org);\n- 🤓 Works with `Content-Security-Policy` and other complex front-end security rules;\n- 📦 This package shipped with [types](https://github.com/veliovgroup/spiderable-middleware/blob/master/types/index.d.ts) and [TS examples](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/);\n- ❤️ Search engines and social network crawlers love straightforward and pre-rendered pages;\n- 📱 Consistent link previews in messaging apps, like iMessage, Messages, Facebook, Slack, Telegram, WhatsApp, Viber, VK, Twitter, and other apps;\n- 💻 Image, title, and description previews for posted links at social networks, like Facebook, Twitter, Instagram, and other social networks.\n\n## ToC\n\n- [Installation](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#installation)\n- [Basic usage](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#usage)\n- [Meteor.js usage](https://github.com/veliovgroup/spiderable-middleware/blob/master/docs/meteor.md)\n- [Return genuine status code](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#return-genuine-status-code)\n- [Speed-up rendering](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#speed-up-rendering)\n- [Detect request from Prerendering engine during runtime](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#detect-request-from-pre-rendering-engine-during-runtime)\n- [JavaScript redirects](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#javascript-redirects)\n- [AMP Support](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#amp-support)\n- [Rendering Endpoints](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#rendering-endpoints)\n- [API](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#api)\n  - [Constructor `new Spiderable()`](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#constructor)\n  - [Middleware](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#handle)\n  - [TS Types](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#types)\n- [Debugging](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#debugging)\n- [Running Tests](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#running-tests)\n\n## About Package\n\nThis package acts as middleware and intercepts requests to a Node.js application from web crawlers. All requests proxy passed to the Prerendering Service, which returns static, rendered HTML.\n\n__This is SERVER only package. For NPM make sure it's imported only in Node.js.__\n\nWe made this package with developers in mind. It's well written in a very simple way, hackable, and easily tunable to meet all projects needs, it can be used to turn dynamic pages into rendered, cached, and lightweight static pages, just set `botsUA` to `['.*']`. This is the best option to offload servers unless a website gets updated more often than once in 4 hours.\n\n- __Note__: *This package proxies real HTTP headers and response code, to reduce overwhelming requests, try to avoid HTTP-redirect headers, like* `Location` *. Read how to [return genuine status code](https://github.com/veliovgroup/spiderable-middleware#return-genuine-status-code) and [handle JS-redirects](https://github.com/veliovgroup/spiderable-middleware#javascript-redirects)*\n- __Note__: This is __server only package__. This package should be imported/initialized only within server code base\n\nThis middleware was tested and works like a charm with:\n\n- [express](https://www.npmjs.com/package/express): [`example.js`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/express.middleware.js), [`example.ts`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/express.middleware.ts)\n- [connect](https://www.npmjs.com/package/connect): [`example.js`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/connect.middleware.js), [`example.ts`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/connect.middleware.ts)\n- [vanilla http(s) server](https://nodejs.org/api/http.html): [`example.js`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/http.middleware.js), [`example.ts`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/http.middleware.ts)\n- [Nginx](https://nginx.org/en/docs/): [example](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/nginx)\n- [Apache](https://httpd.apache.org/): [example](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/apache.htaccess)\n- See [all examples](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples)\n\nAll other frameworks which follow Node's middleware convention - will work too.\n\n\u003e This package was originally developed for [ostr.io](https://ostr.io) service. But it's not limited to, and can proxy-pass requests to any other rendering-endpoint.\n\n## Installation\n\nInstall [`spiderable-middleware` package from NPM](https://www.npmjs.com/package/spiderable-middleware):\n\n```sh\n# using npmjs\nnpm install spiderable-middleware --save\n\n# using yarn\nyarn add spiderable-middleware\n```\n\n## Usage\n\nGet ready in a few lines of code\n\n### Basic usage\n\nSee [usage examples in `.js` and `.ts`](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples) for quick copy-paste experience.\n\nStart with adding `fragment` meta-tag to HTML template, head, or page:\n\n```html\n\u003chtml\u003e\n  \u003chead\u003e\n    \u003cmeta name=\"fragment\" content=\"!\"\u003e\n    \u003c!-- ... --\u003e\n  \u003c/head\u003e\n  \u003cbody\u003e\n    \u003c!-- ... --\u003e\n  \u003c/body\u003e\n\u003c/html\u003e\n```\n\nImport or require `spiderable-middleware` package:\n\n```js\n// ES6 import\nimport Spiderable from 'spiderable-middleware';\n\n// or CommonJS require\nconst Spiderable = require('spiderable-middleware');\n```\n\nRegister middleware handle:\n\n```js\nimport express from 'express';\nimport Spiderable from 'spiderable-middleware';\n\nconst spiderable = new Spiderable({\n  rootURL: 'http://example.com',\n  auth: 'test:test',\n});\n\nconst app = express();\n// ensure this is the most top registered handle\n// to reduce response time and server load\napp.use(spiderable.handle).get('/', (req, res) =\u003e {\n  res.send('Hello World');\n});\n\napp.listen(3000);\n```\n\nWe provide various options for `serviceURL` as \"[Rendering Endpoints](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md)\", each endpoint has its own features to fit every project needs.\n\n## Return genuine status code\n\nTo pass expected status code of a response from front-end JavaScript framework to browser/crawlers use specially formatted HTML-comment. This comment __can be placed in any part of HTML-page__. `head` or `body` tag is the best place for it.\n\n### Format\n\n__html__:\n\n```html\n\u003c!-- response:status-code=404 --\u003e\n```\n\n__jade__:\n\n```jade\n// response:status-code=404\n```\n\nThis package support __any__ standard and custom status codes:\n\n- `201` - `\u003c!-- response:status-code=201 --\u003e`\n- `401` - `\u003c!-- response:status-code=401 --\u003e`\n- `403` - `\u003c!-- response:status-code=403 --\u003e`\n- `500` - `\u003c!-- response:status-code=500 --\u003e`\n- `514` - `\u003c!-- response:status-code=514 --\u003e` (*non-standard*)\n\n## Speed-up rendering\n\nTo speed-up rendering, JS-runtime __should__ tell to the Spiderable engine when the page is ready. Set `window.IS_RENDERED` to `false`, and once the page is ready set this variable to `true`. Example:\n\n```html\n\u003chtml\u003e\n  \u003chead\u003e\n    \u003cmeta name=\"fragment\" content=\"!\"\u003e\n    \u003cscript\u003e\n      window.IS_RENDERED = false;\n    \u003c/script\u003e\n  \u003c/head\u003e\n  \u003cbody\u003e\n    \u003c!-- ... --\u003e\n    \u003cscript type=\"text/javascript\"\u003e\n      //Somewhere deep in app-code:\n      window.IS_RENDERED = true;\n    \u003c/script\u003e\n  \u003c/body\u003e\n\u003c/html\u003e\n```\n\n## Detect request from Pre-rendering engine during runtime\n\nPre-rendering engine will set `window.IS_PRERENDERING` global variable to `true`. Detecting requests from pre-rendering engine are as easy as:\n\n```js\nif (window.IS_PRERENDERING) {\n  // This request is coming from Pre-rendering engine\n}\n```\n\n__Note__: `window.IS_PRERENDERING` can be `undefined` on initial page load, and may change during runtime. That's why we recommend to pre-define a setter for `IS_PRERENDERING`:\n\n```js\nlet isPrerendering = false;\nObject.defineProperty(window, 'IS_PRERENDERING', {\n  set(val) {\n    isPrerendering = val;\n    if (isPrerendering === true) {\n      // This request is coming from Pre-rendering engine\n    }\n  },\n  get() {\n    return isPrerendering;\n  }\n});\n```\n\n## JavaScript redirects\n\nRedirect browser/crawler inside application when needed while a page is loading (*imitate navigation*), use any of classic JS-redirects can be used, including framework's navigation, or `History.pushState()`\n\n```js\nwindow.location.href = 'http://example.com/another/page';\nwindow.location.replace('http://example.com/another/page');\n\nRouter.go('/another/page'); // framework's navigation !pseudo code\n```\n\n__Note__: *Only 4 redirects are allowed during one request after 4 redirects session will be terminated.*\n\n## API\n\nCreate new instance and pass middleware to server's routes chain;\n\n### *Constructor*\n\n```ts\nnew Spiderable(opts?: SpiderableOptions);\n```\n\n- `opts` {*SpiderableOptions?*} - [Optional] Configuration options\n- `opts.serviceURL` {*string*} - Valid URL to Spiderable endpoint (local or foreign). Default: `https://render.ostr.io`. Can be set via environment variables: `SPIDERABLE_SERVICE_URL` or `PRERENDER_SERVICE_URL`\n- `opts.rootURL` {*string*} - Valid root URL of a website. Can be set via an environment variable: `ROOT_URL`\n- `opts.auth` {*string*} - Auth string in next format: `user:pass`. Can be set via an environment variables: `SPIDERABLE_SERVICE_AUTH` or `PRERENDER_SERVICE_AUTH`. Default `null`\n- `opts.sanitizeUrls` {*boolean*} - Sanitize URLs in order to \"fix\" badly composed URLs. Default `false`\n- `opts.botsUA` {*string[]*} - An array of strings (case insensitive) with additional User-Agent names of crawlers that needs to get intercepted. See default [bot's names](https://github.com/veliovgroup/spiderable-middleware/blob/master/lib/index.js#L198). Set to `['.*']` to match all browsers and robots, to serve static pages to all users/visitors\n- `opts.ignoredHeaders` {*string[]*} - An array of strings (case insensitive) with HTTP header names to exclude from response. See default [list of ignored headers](https://github.com/veliovgroup/spiderable-middleware/blob/master/lib/index.js#L206). Set to `['.*']` to ignore all headers\n- `opts.ignore` {*string[]*} - An array of strings (case __sensitive__) with ignored routes. Note: it's based on first match, so route `/users` will cause ignoring of `/part/users/part`, `/users/_id` and `/list/of/users`, but not `/user/_id` or `/list/of/blocked-users`. Default `null`\n- `opts.only` {*(String|RegExp)[]*} - An array of strings (case __sensitive__) or regular expressions (*could be mixed*). Define __exclusive__ route rules for pre-rendering. Could be used with `opts.onlyRE` rules. __Note:__ To define \"safe\" rules as {*RegExp*} it should start with `^` and end with `$` symbols, examples: `[/^\\/articles\\/?$/, /^\\/article\\/[A-z0-9]{16}\\/?$/]`\n- `opts.onlyRE` {*RegExp*} - Regular Expression with __exclusive__ route rules for pre-rendering. Could be used with `opts.only` rules\n- `opts.timeout` {*number*} - Number, proxy-request timeout to rendering endpoint in milliseconds. Default: `180000`\n- `opts.requestOptions` {*RequestOptions*} - Options for request module (like: `timeout`, `lookup`, `insecureHTTPParser`), for all available options see [`http` API docs](https://nodejs.org/docs/latest-v14.x/api/http.html#http_http_request_url_options_callback)\n- `opts.debug` {*boolean*} - [Optional] Enable debug and extra logging, default: `false`\n\n__Note:__ *Setting* `.onlyRE` *and/or* `.only` *rules are highly recommended. Otherwise, all routes, including randomly generated by bots will be subject of Pre-rendering and may cause unexpectedly higher usage.*\n\n```js\n// CommonJS\n// const Spiderable = require('spiderable-middleware');\n\n// ES6 import\n// import Spiderable from 'spiderable-middleware';\n\nconst spiderable = new Spiderable({\n  rootURL: 'http://example.com',\n  auth: 'test:test'\n});\n\n// More complex setup (recommended):\nconst spiderable = new Spiderable({\n  rootURL: 'http://example.com',\n  serviceURL: 'https://render.ostr.io',\n  auth: 'test:test',\n  only: [\n    /\\/?/, // Root of the website\n    /^\\/posts\\/?$/, // \"/posts\" path with(out) trailing slash\n    /^\\/post\\/[A-z0-9]{16}\\/?$/ // \"/post/:id\" path with(out) trailing slash\n  ],\n  // [Optionally] force ignore for secret paths:\n  ignore: [\n    '/account/', // Ignore all routes under \"/account/*\" path\n    '/billing/' // Ignore all routes under \"/billing/*\" path\n  ]\n});\n```\n\n#### Configuration via env.vars\n\nSame configuration can get achieved via setting up environment variables:\n\n```sh\nROOT_URL='http://example.com'\nSPIDERABLE_SERVICE_URL='https://render.ostr.io'\nSPIDERABLE_SERVICE_AUTH='APIUser:APIPass'\n```\n\nalternatively, when migrating from other pre-rendering service — keep using existing variables, we support the next ones for compatibility:\n\n```sh\nROOT_URL='http://example.com'\nPRERENDER_SERVICE_URL='https://render.ostr.io'\nPRERENDER_SERVICE_AUTH='APIUser:APIPass'\n```\n\n### handle\n\nMiddleware handle\n\n```ts\nconst spiderable = new Spiderable();\nspiderable.handle(req: IncomingMessage, res: ServerResponse, next: NextFunction): void;\n\n// Alias that returns {boolean} \n// true — if prerendering takes over the request\nspiderable.handler(req: IncomingMessage, res: ServerResponse, next: NextFunction): boolean;\n```\n\nExample using `connect` and `express` package:\n\n```js\nimport { createServer } from 'node:http';\nimport Spiderable from 'spiderable-middleware';\n\nconst app = express();\n// const app = connect();\nconst spiderable = new Spiderable();\n\napp.use(spiderable.handle).use((_req, res) =\u003e {\n  res.end('Hello from Connect!\\n');\n});\n\ncreateServer(app).listen(3000;\n```\n\nExample using node.js `http` server:\n\n```js\nimport { createServer } from 'node:http';\nimport Spiderable from 'spiderable-middleware';\n\n// HTTP(s) Server\nhttp.createServer((req, res) =\u003e {\n  spiderable.handle(req, res, () =\u003e {\n    // Callback, triggered if this request\n    // is not a subject of spiderable pre-rendering\n    res.writeHead(200, {'Content-Type': 'text/plain; charset=UTF-8'});\n    res.end('Hello vanilla NodeJS!');\n    // Or do something else ...\n  });\n}).listen(3000);\n```\n\n### Types\n\nImport types right from NPM package\n\n```ts\nimport Spiderable from 'spiderable-middleware';\nimport type { SpiderableOptions, NextFunction } from 'spiderable-middleware';\n\nconst options: SpiderableOptions = {\n  rootURL: 'http://example.com',\n  auth: 'test:test',\n  debug: false,\n  /* ..and other options.. */\n};\nexpectType\u003cSpiderableOptions\u003e(options);\n\nconst spiderable = new Spiderable(options);\nexpectType\u003cSpiderable\u003e(spiderable);\n\nconst next: NextFunction = (_err?: unknown): void =\u003e {};\nexpectType\u003cvoid\u003e(spiderable.handle(req, res, next));\n```\n\n## AMP Support\n\nTo properly serve pages for [Accelerated Mobile Pages](https://www.ampproject.org) (AMP) we support following URI schemes:\n\n```sh\n# Regular URIs:\nhttps://example.com/index.html\nhttps://example.com/articles/article-title.html\nhttps://example.com/articles/article-uniq-id/article-slug\n\n# AMP optimized URIs (prefix):\nhttps://example.com/amp/index.html\nhttps://example.com/amp/articles/article-title.html\nhttps://example.com/amp/articles/article-uniq-id/article-slug\n\n# AMP optimized URIs (extension):\nhttps://example.com/amp/index.amp.html\nhttps://example.com/amp/articles/article-title.amp.html\n```\n\nAll URLs with `.amp.` extension and `/amp/` prefix will be optimized for AMP.\n\n## Rendering Endpoints\n\n- __render (*default*)__ - `https://render.ostr.io` - This endpoint has \"optimal\" settings, and should fit 98% cases. This endpoint respects cache headers of Crawler and origin server\n- __render-bypass (*devel/debug*)__ - `https://render-bypass.ostr.io` - This endpoint will bypass caching mechanisms. Use it when experiencing an issue, or during development, to make sure responses are not cached. It's safe to use this endpoint in production, but it may result in higher usage and response time\n- __render-cache (*under attack*)__ - `https://render-cache.ostr.io` - This endpoint has the most aggressive caching mechanism. Use it to achieve the shortest response time, and when outdated pages (*for 6-12 hours*) are acceptable\n\nTo change default endpoint, grab [integration examples code](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples) and replace `render.ostr.io`, with endpoint from the list above. For NPM integration change value of [`serviceURL`](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#constructor) option.\n\n__Note:__ Described differences in caching behavior related to intermediate proxy caching, `Cache-Control` header will be always set to the value defined in \"Cache TTL\". Cached results at the \"Pre-rendering Engine\" end can be [purged at any time](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cache-purge.md).\n\n## Debugging\n\nPass `{ debug: true }` or set `DEBUG=true` environment variable to enable debugging mode.\n\nTo make sure a server can reach a rendering endpoint run `cURL` command or send request via Node.js to (*replace example.com with your domain name*):\n\n```shell\ncurl -v \"https://test:test@render-bypass.ostr.io/?url=http://example.com\"\n```\n\nIn this example we're using `render-bypass.ostr.io` endpoint to avoid any possible cached results, [read more about rendering endpoints](https://github.com/veliovgroup/spiderable-middleware#rendering-endpoints). As API credentials we're using `test:test`, this part of URL can be replaced with `auth` option from Node.js example. The API credentials and instructions can be found at the very bottom of [Pre-rendering Panel](https://ostr.io/service/prerender) of a particular host, — click on \u003ckbd\u003eIntegration Guide\u003c/kbd\u003e\n\n```sh\n# cURL example:\ncurl -v \"https://test:test@render-bypass.ostr.io/?url=http://example.com\"\n```\n\n```js\n// Node.js example:\nconst https = require('https');\n\nhttps.get('https://test:test@render-bypass.ostr.io/?url=http://example.com', (resp) =\u003e {\n  let data = '';\n\n  resp.on('data', (chunk) =\u003e {\n    data += chunk.toString('utf8');\n  });\n\n  resp.on('end', () =\u003e {\n    console.log(data);\n  });\n}).on('error', (error) =\u003e {\n  console.error(error);\n});\n```\n\n## Running Tests\n\n1. Clone this package\n2. In Terminal (*Console*) go to directory where package was cloned\n3. Then run:\n\n### Node.js/Mocha\n\n```sh\n# Install development NPM dependencies:\nnpm install --save-dev\n# Install NPM dependencies:\nnpm install --save\n\n# Link package to itself\nnpm link\nnpm link spiderable-middleware\n\n# Run tests:\nROOT_URL=http://127.0.0.1:3003 npm test\n\n# Run same tests with extra-logging\nDEBUG=true ROOT_URL=http://127.0.0.1:3003 npm test\n# http://127.0.0.1:3003 can be changed to any local address, PORT is required!\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveliovgroup%2Fspiderable-middleware","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fveliovgroup%2Fspiderable-middleware","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveliovgroup%2Fspiderable-middleware/lists"}