{"id":13623342,"url":"https://github.com/JS-DevTools/rehype-url-inspector","last_synced_at":"2025-04-15T14:32:45.783Z","repository":{"id":57122441,"uuid":"203897175","full_name":"JS-DevTools/rehype-url-inspector","owner":"JS-DevTools","description":"A rehype plugin to inspect, validate, or rewrite URLs anywhere in an HTML document","archived":false,"fork":false,"pushed_at":"2020-07-18T19:57:40.000Z","size":611,"stargazers_count":22,"open_issues_count":3,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T11:51:10.234Z","etag":null,"topics":["broken-links","html","html5","javascript","nodejs","rehype","rehype-plugin","unified","url","url-rewrite","url-validation","urls","validate-url"],"latest_commit_sha":null,"homepage":"https://jstools.dev/rehype-url-inspector","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JS-DevTools.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-23T00:56:54.000Z","updated_at":"2025-01-16T06:02:25.000Z","dependencies_parsed_at":"2022-08-24T14:59:29.043Z","dependency_job_id":null,"html_url":"https://github.com/JS-DevTools/rehype-url-inspector","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":"JS-DevTools/template-node-typescript","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JS-DevTools%2Frehype-url-inspector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JS-DevTools%2Frehype-url-inspector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JS-DevTools%2Frehype-url-inspector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JS-DevTools%2Frehype-url-inspector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JS-DevTools","download_url":"https://codeload.github.com/JS-DevTools/rehype-url-inspector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249061204,"owners_count":21206467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broken-links","html","html5","javascript","nodejs","rehype","rehype-plugin","unified","url","url-rewrite","url-validation","urls","validate-url"],"created_at":"2024-08-01T21:01:30.639Z","updated_at":"2025-04-15T14:32:45.447Z","avatar_url":"https://github.com/JS-DevTools.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","Plugins"],"sub_categories":[],"readme":"Rehype URL Inspector\n==============================\n### A [rehype](https://github.com/rehypejs/rehype) plugin to inspect, validate, or rewrite URLs anywhere in an HTML document\n\n[![Cross-Platform Compatibility](https://jstools.dev/img/badges/os-badges.svg)](https://github.com/JS-DevTools/rehype-url-inspector/actions)\n[![Build Status](https://github.com/JS-DevTools/rehype-url-inspector/workflows/CI-CD/badge.svg)](https://github.com/JS-DevTools/rehype-url-inspector/actions)\n\n[![Coverage Status](https://coveralls.io/repos/github/JS-DevTools/rehype-url-inspector/badge.svg?branch=master)](https://coveralls.io/github/JS-DevTools/rehype-url-inspector)\n[![Dependencies](https://david-dm.org/JS-DevTools/rehype-url-inspector.svg)](https://david-dm.org/JS-DevTools/rehype-url-inspector)\n\n[![npm](https://img.shields.io/npm/v/@jsdevtools/rehype-url-inspector.svg)](https://www.npmjs.com/package/@jsdevtools/rehype-url-inspector)\n[![License](https://img.shields.io/npm/l/@jsdevtools/rehype-url-inspector.svg)](LICENSE)\n[![Buy us a tree](https://img.shields.io/badge/Treeware-%F0%9F%8C%B3-lightgreen)](https://plant.treeware.earth/JS-DevTools/rehype-url-inspector)\n\n\n\nFeatures\n--------------------------\n- Inspect every URL on an HTML page and do whatever you want to, such as:\n  - Normalize URLs\n  - Check for broken links\n  - Replace URLs with different URLs\n  - Add attributes (like `target=\"blank\"`) to certain links\n\n- Finds **all types of URLs** by default, such as:\n  - `\u003ca href=\"http://example.com\"\u003e`\n  - `\u003cimg src=\"img/logo.png\"\u003e`\n  - `\u003clink rel=\"stylesheet\" href=\"/css/main.css\"\u003e`\n  - `\u003clink rel=\"manifest\" href=\"/site.manifest\"\u003e`\n  - `\u003cmeta rel=\"canonical\" content=\"https://example.com/some/page/\"\u003e`\n  - `\u003cmeta property=\"og:image\" content=\"img/logo.png\"\u003e`\n  - `\u003cscript src=\"//example.com/script.js\"\u003e`\n  - `\u003cscript type=\"application/ld+json\"\u003e{\"url\": \"www.example.com\"}\u003c/script\u003e`\n  - `\u003cstyle\u003ebody { background: url(\"/img/background.png\"); }\u003c/style\u003e`\n\n- You can remove the built-in URL rules\n- You can add your own **custom URL rules**\n- You can abort the URL search at any time\n\n\nExample\n--------------------------\n\n**example.html**\u003cbr\u003e\nThis HTML file contains many different types of URLs:\n\n```html\n\u003chtml\u003e\n  \u003chead\u003e\n    \u003clink rel=\"canonical\" href=\"http://example.com/some/page/\"\u003e\n    \u003clink rel=\"manifest\" href=\"/site.webmanifest\"\u003e\n    \u003clink rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/img/favicon.png\"\u003e\n    \u003clink rel=\"stylesheet\" type=\"text/css\" href=\"/css/main.css?v=5\"\u003e\n\n    \u003cmeta name=\"twitter:url\" content=\"http://example.com/some/page/\"\u003e\n    \u003cmeta name=\"twitter:image\" content=\"http://example.com/img/logo.png\"\u003e\n\n    \u003cscript type=\"application/ld+json\"\u003e\n      {\n        \"@context\": \"http://schema.org\",\n        \"headline\": \"Hello, World!\",\n        \"url\": \"http://example.com/some/page/\",\n        \"image\": \"http://example.com/img/logo.png\"\n      }\n    \u003c/script\u003e\n\n    \u003cstyle\u003e\n      body {\n        background: #ffffff url(\"img/background.png\") center center no-repeat;\n      }\n    \u003c/style\u003e\n  \u003c/head\u003e\n  \u003cbody\u003e\n    \u003ch1\u003e\n      \u003ca href=\"/\"\u003e\n        \u003cimg src=\"/img/logo.png\"\u003e Hello World\n      \u003c/a\u003e\n    \u003c/h1\u003e\n    \u003cp\u003e\n      \u003ca href=\"//external.com\" target=\"_blank\"\u003eLorem ipsum\u003c/a\u003e dolor sit amet,\n      non dignissim dolor. Sed diam tellus, \u003ca href=\"some-page.html\"\u003emalesuada, dictum nulla\u003c/a\u003e.\n    \u003c/p\u003e\n\n    \u003cscript src=\"//external.com/script.js\"\u003e\u003c/script\u003e\n  \u003c/body\u003e\n\u003c/html\u003e\n```\n\n**example.js**\u003cbr\u003e\nThis script reads the `example.html` file above and finds all the URLs in it. The script uses [unified](https://unifiedjs.com/), [rehype-parse](https://github.com/rehypejs/rehype/tree/master/packages/rehype-parse), [rehype-stringify](https://github.com/rehypejs/rehype/tree/master/packages/rehype-stringify), and [to-vfile](https://github.com/vfile/to-vfile).\n\n\n```javascript\nconst unified = require(\"unified\");\nconst parse = require(\"rehype-parse\");\nconst inspectUrls = require(\"@jsdevtools/rehype-url-inspector\");\nconst stringify = require(\"rehype-stringify\");\nconst toVFile = require(\"to-vfile\");\n\nasync function example() {\n  // Create a Rehype processor with the inspectUrls plugin\n  const processor = unified()\n    .use(parse)\n    .use(inspectUrls, {\n      inspectEach({ url }) {\n        // Log each URL\n        console.log(url);\n      }\n    })\n    .use(stringify);\n\n  // Read the example HTML file\n  let file = await toVFile.read(\"example.html\");\n\n  // Crawl the HTML file and find all the URLs\n  await processor.process(file);\n}\n\nexample();\n```\n\nRunning this script produces the following output:\n\n```\nhttp://example.com/some/page/\n/site.webmanifest\n/img/favicon.png\n/css/main.css?v=5\nhttp://example.com/some/page/\nhttp://example.com/img/logo.png\nhttp://schema.org\nhttp://example.com/some/page/\nhttp://example.com/img/logo.png\nimg/background.png\n/\n/img/logo.png\n//external.com\nsome-page.html\n//external.com/script.js\n```\n\n\n\nInstallation\n--------------------------\nYou can install Rehype URL Inspector via [npm](https://docs.npmjs.com/about-npm/).\n\n```bash\nnpm install @jsdevtools/rehype-url-inspector\n```\n\nYou'll probably want to install [unified](https://unifiedjs.com/), [rehype-parse](https://github.com/rehypejs/rehype/tree/master/packages/rehype-parse), [rehype-stringify](https://github.com/rehypejs/rehype/tree/master/packages/rehype-stringify), and [to-vfile](https://github.com/vfile/to-vfile) as well.\n\n```bash\nnpm install unified rehype-parse rehype-stringify to-vfile\n```\n\n\n\nUsage\n--------------------------\nUsing the URL Inspector plugin requires an understanding of how to use Unified and Rehype. [Here is an excelleng guide](https://unifiedjs.com/using-unified.html) to learn the basics.\n\nThe URL Inspector plugin works just like any other Rehype plugin. Pass it to [the `.use()` method](https://github.com/unifiedjs/unified#processoruseplugin-options) with an [options object](#options).\n\n```javascript\nconst unified = require(\"unified\");\nconst inspectUrls = require(\"@jsdevtools/rehype-url-inspector\");\n\n// Use the Rehype URL Inspector plugin with custom options\nunified().use(inspectUrls, {\n  inspect(urls) { ... },      // This function is called once with ALL of the URLs\n  inspectEach(url) { ... },   // This function is called for each URL as it's found\n  selectors: [\n    \"a[href]\",                // Only search for links, not other types of URLs\n    \"div[data-image]\"         // CSS selectors for custom URL attributes\n  ]\n});\n```\n\n\n\nOptions\n--------------------------\nRehype URL Inspector supports the following options:\n\n|Option                |Type                |Default                |Description\n|:---------------------|:-------------------|:----------------------|:-----------------------------------------\n|`selectors`           |array of strings, objects, and/or functions |[built-in selectors](src/selectors.ts) |Selectors indicate where to look for URLs in the document. Each selector can be a CSS attribute selector string, like `a[href]` or `img[src]`, or a function that accepts a [HAST node](https://github.com/syntax-tree/hast) and returns its URL(s). See [`extractors.ts`](src/extractors.ts) for examples.\n|`keepDefaultSelectors`|boolean             |`false`                |Whether to keep the default selectors in addition to any custom ones.\n|`inspect`             |function            |no-op                  |A function that is called _once_ and receives an array containing all the URLs in the document\n|`inspectEach`         |function            |no-op                  |A function that is called for _each_ URL in the document as it's found. Return `false` to abort the search and skip the rest of the document.\n\n\n\nURL Objects\n--------------------------\nThe `inspectEach()` function receives a [`UrlMatch` object](src/types.ts).  The `inspect()` function receves an array of `UrlMatch` objects.  Each object has the following properties:\n\n|Property               |Type                 |Description\n|:----------------------|:--------------------|:------------------------------------\n|`url`                  |string               |The URL that was found\n|`propertyName`         |string or undefined  |The name of the [HAST node property](https://github.com/syntax-tree/hast#properties) where the URL was found, such as `\"src\"` or `\"href\"`. If the URL was found in the text content of the node, then `propertyName` is `undefined`.\n|`node`                 |object               |The [HAST Element node](https://github.com/syntax-tree/hast#element) where the URL was found. **You can make changes to this node**, such as re-writing the URL, adding additional attributes, etc.\n|`root`                 |object               |The [HAST Root node](https://github.com/syntax-tree/hast#root). This gives you access to the whole document if you need it.\n|`file`                 |object               |The [File object](https://github.com/vfile/vfile) that gives you information about the HTML file itself, such as the path and file name.\n\n\n\nContributing\n--------------------------\nContributions, enhancements, and bug-fixes are welcome!  [Open an issue](https://github.com/JS-DevTools/rehype-url-inspector/issues) on GitHub and [submit a pull request](https://github.com/JS-DevTools/rehype-url-inspector/pulls).\n\n#### Building\nTo build the project locally on your computer:\n\n1. __Clone this repo__\u003cbr\u003e\n`git clone https://github.com/JS-DevTools/rehype-url-inspector.git`\n\n2. __Install dependencies__\u003cbr\u003e\n`npm install`\n\n3. __Build the code__\u003cbr\u003e\n`npm run build`\n\n4. __Run the tests__\u003cbr\u003e\n`npm test`\n\n\n\nLicense\n--------------------------\nRehype URL Inspector is 100% free and open-source, under the [MIT license](LICENSE). Use it however you want.\n\nThis package is [Treeware](http://treeware.earth). If you use it in production, then we ask that you [**buy the world a tree**](https://plant.treeware.earth/JS-DevTools/rehype-url-inspector) to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.\n\n\n\nBig Thanks To\n--------------------------\nThanks to these awesome companies for their support of Open Source developers ❤\n\n[![Travis CI](https://jstools.dev/img/badges/travis-ci.svg)](https://travis-ci.com)\n[![SauceLabs](https://jstools.dev/img/badges/sauce-labs.svg)](https://saucelabs.com)\n[![Coveralls](https://jstools.dev/img/badges/coveralls.svg)](https://coveralls.io)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJS-DevTools%2Frehype-url-inspector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJS-DevTools%2Frehype-url-inspector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJS-DevTools%2Frehype-url-inspector/lists"}