Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/billdietrich/linkcheckerhtml

A VSCode extension that check links in HTML, XML, and RSS files to ensure they are valid.
https://github.com/billdietrich/linkcheckerhtml

diagnostics html html-files https-equivalents link-checker markdown onion php rss-link-checker semantic-web tor-network vscode-extension xml

Last synced: about 2 months ago
JSON representation

A VSCode extension that check links in HTML, XML, and RSS files to ensure they are valid.

Host: GitHub
URL: https://github.com/billdietrich/linkcheckerhtml
Owner: BillDietrich
License: mit
Created: 2018-11-20T20:58:27.000Z (about 6 years ago)
Default Branch: master
Last Pushed: 2022-12-10T11:05:22.000Z (about 2 years ago)
Last Synced: 2023-03-04T11:18:45.764Z (almost 2 years ago)
Topics: diagnostics, html, html-files, https-equivalents, link-checker, markdown, onion, php, rss-link-checker, semantic-web, tor-network, vscode-extension, xml
Language: TypeScript
Homepage:
Size: 14.5 MB
Stars: 6
Watchers: 1
Forks: 1
Open Issues: 7
Metadata Files:
- Readme: README.md
- License: LICENSE.md

Awesome Lists containing this project

README

# HTML / XML / RSS link checker
VSCode extension that checks for broken links in an HTML, XML, RSS, PHP, or Markdown file.

## Functionality
Checks for broken links in anchor-href, link-href, img-src, and script-src tags in currently-open HTML or PHP file. It checks HTTP/HTTPS links by trying to access them on the internet, and checks relative links (../folder/file.html) by checking if the file exists on the local file system.

Checks both clearnet and onion (Tor) links.

Also checks for badly-formatted mailto links, and duplicate local anchors (anchor-name, anchor-id).

Also checks for working HTTPS equivalents of HTTP links.

Also checks for broken links in currently-open XML or RSS or Markdown file.

Optionally checks for invalid characters and common mistakes (missing tag content, empty attribute value, more).

Also checks for errors in a small subset of semantic HTML tags (in HTML and PHP files): checks that each page has header, main, footer; checks that each heading is inside a section, article, or aside; checks that each section/article/aside has exactly one heading in it; checks that heading values are nested properly.

## Use
Open an editor window on an HTML, XML, RSS, PHP, or Markdown file, and then press `Alt+H`.

Broken links are reported via the standard error/warning/information diagnostic icons in lower-left of UI.

Click on the diagnostic icons and numbers to open the diagnostics pane.

Click on a diagnostic line, see that link highlighted in the source file, press `Alt+T` to open that URL in your browser.

If it's an HTTP link, press `Alt+M` to try to open the HTTPS equivalent of that URL in your browser.

Press `Alt+L` to clear all diagnostic messages generated by this extension.

![Using the extension](UsingTheExtension.jpg "Using the extension")

Tip: After you do `Alt+H` and get diagnostics, work on the problems from bottom (last diagnostic) to top (first diagnostic). That way the line numbers in the diagnostics don't change as you delete or add lines in the source.

To see/change settings for this extension, open Settings (`Ctrl+,`) / Extensions / "HTML / XML / RSS link checker".

To change the key-combinations for this extension, open File / Preferences / Keyboard Shortcuts and search for Alt+H or Alt+T or Alt+M or Alt+L.

### Onion (Tor) links

Onion URLs look like ```https://1234567890123456.onion/something``` (16 chars before '.onion') or ```https://12345678901234567890123456789012345678901234567890123456.onion/something``` (56 chars before '.onion'). They are used to access dark-web sites through Tor Browser (usually).

#### Checking validity of Onion (Tor) links

To use `Alt+H` to check onion links, you must have a Tor/socks proxy listening on 127.0.0.1:9050. On Linux:
```shell
sudo systemctl status tor # should show an active Tor service
# if it's not active, try:
sudo systemctl start tor

sudo ss -lptu | grep :9050 # should show an active Tor listener
```
For more information see https://github.com/talmobi/tor-request#requirements If you don't have a Tor/socks proxy listening, each onion link will give an error "Can't check onion URLs: no Tor/socks service listening on 127.0.0.1:9050".

While checking links, the Tor Browser can be running or not, it doesn't matter. Only the proxy is used.

#### Opening Onion (Tor) links in Tor Browser

[THIS FEATURE SEEMS TO BE BROKEN]

To use `Alt+T` or `Alt+M` to open onion links in the Tor Browser, you must have Tor Browser installed and running already. You have to launch it yourself; this extension won't launch it.

Also, on Linux, you must install "xdotool":
```shell
sudo apt install xdotool
xdotool --version
```

Then for any bad onion link reported in the diagnostics, do ```Alt+T``` on it. If it's an "http://" onion link (illegal, I think), also you can do ```Alt+M``` on it. You should see focus switch to the Tor Browser, and the URL will be typed in the address bar, then accessed.

The connection to Tor Browser is not 100% reliable. The extension is using xdotools to send key-presses to the Tor Browser, and it's fairly timing-dependent and one-way. If your system is busy, or Tor Browser is busy, or something else goes wrong, you may see the wrong things happen in Tor Browser (chars missing from the URL, or some dialogs popping open).

### Semantic HTML

The body of the HTML page is expected to be structured like:
```

STUFF

HEADING

CONTENT

HEADING

\

,\\,\\,\\").

* **reportRedirect**: Report links that get redirected (default is "as Warning").

* **reportSemanticErrors**: Report errors in semantic HTML tags such as main, section, article, aside, h1, etc (default is "as Information").

* **timeout**: Timeout (seconds) for accessing a link (range is 5 to 30; default is 15).

* **torOpenURLCmd1**: command (1) to open an URL in Tor Browser ('URL' will be appended; default is "xdotool search --onlyvisible --name 'Tor Browser' windowactivate --sync key --clearmodifiers --window 0 ctrl+t type --delay 100 "

* **torOpenURLCmd2**: command (2) to open an URL in Tor Browser (default is "xdotool search --onlyvisible --name 'Tor Browser' windowactivate --sync key --clearmodifiers --window 0 Return").

* **userAgent**: User-Agent value used in Get requests (default is "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0").

### Limitations

* HTML and PHP: Tag name and href/src/id attribute must be on the same line.

* XML and RSS: Entire tag (for link, guid, and url tags) must be on the same line.

* Doesn't know about comments; will find and check tags inside comments.

* Checks "#name" links to targets in current file, but not in other local or remote files.

* Doesn't check EVERY detail of the email address spec in mailto links. Just a cursory check.

* XML: There are no standard tag and attribute names, so some links may not be checked.

Note that checking for broken links is more of an art than a science. Some sites don't actually return 404, but send you to a landing page. For example, Azure.com works this way. You can go to https://Azure.com/foo/bar and it will happily redirect you to a sub-page of https://azure.microsoft.com/, with no 404 status returned. So take a status of "OK" with a grain of salt - you may not be arriving at the page you intend.

Also, browsers seem to be more tolerant than the library used by this extension. This extension will report a lot of certificate-errors and such that browsers mostly ignore.

And checking is getting harder, with more URLs redirecting through GDPR-consent or cookie pages and such, or redirecting to same URL with a tracking parameter added, causing false positives.

#### Quirks

* If there are multiple identical tags with identical link-targets on same line (for example two Anchor tags with identical href targets), clicking on diagnostic for any of them takes you to first one in the source line.

* Doesn't check ANY of the email address format after "?", as in "mailto:[email protected]?subject=xyz".

* "://" is prepended to items in dontCheckURLsThatStartWith before matching; e.g. if you specify "localhost" the code searches for "://localhost" in URLs.

* The checking in XML and RSS files is permissive, allowing known stuff from RSS, and likely stuff that could be in XML. Any attribute of the form *url="something" or *href="something" will be checked, as well as the standard RSS tags: link, guid, url.

* Onion: an URL is considered "onion" if it starts with "https://" and contains ".onion" ANYWHERE in it.

* Onion: if an URL starts with "http://", it will be treated as non-Onion, and the "https://" form of it will be checked as non-Onion too.

* Semantic HTML: assumes that a section/article/aside will have a heading in it before it has any sub-section/article/aside.

* Semantic HTML: section/article/aside without heading will be flagged (correctly) but may screw up the tracking of headings from that point on. Fix first such error and then scan again.

* PHP: links are found if they look like links in HTML. For example, the PHP code is expected to look something like:

```echo 'test1
';```

and not like:

```echo 'test1
';```

But the following would be okay:

```echo '' + theTextOfTheLink + '
';```

* Markdown: in reference-style links, the URL will be checked, but there will be no check that both halves of a reference-style link exist. A reference-style link looks like:
```
[hobbit-hole] [1]

[1]: "Hobbit hole"
```

---

## Install
### From the Marketplace
Open Visual Studio Code and press `F1`; a field will appear at the top of the window. Type `ext install linkcheckerhtml`, hit enter, and reload the window to enable.

### From VSIX file
Either:
* In CLI, do
```shell
code --install-extension linkcheckerhtml-n.n.n.vsix
```
or
* In VSCode GUI, in the Extensions view "..." drop-down, select the "Install from VSIX" command.

### From source code
* Do a ```git clone``` to copy the source code to "linkcheckerhtml" in your home directory.
* In CLI, ```cd linkcheckerhtml``` and then ```./CopyToHomeToRunInNormal.sh```

---

## Releases
### 0.2.0
* Copied from "Microsoft / linkcheckermd" and then greatly modified.
* Extension works, but probably has memory leaks, not much testing.

### 0.3.0
* Changed to use node-fetch module (https://github.com/bitinn/node-fetch) instead of broken-link. But has bad hangs.
* Changed to use got module (https://github.com/sindresorhus/got) instead of node-fetch. But has bad hangs.
* Changed to use axios module (https://github.com/axios/axios) instead of node-fetch. Works, but hangs on some URLs.
* Changed to throttle so that it checks max of 4 links in parallel.
* Changed settings to use pull-downs: error, warn, info, don't report

### 0.4.0
* Finally nailed that hang bug.
* Added setting for timeout.
* Fixed timeout and redirect settings.

### 0.5.0
* Added `Alt+T` to open an URL in a browser.
* First release with a VSIX file.

### 0.6.0
* Got rid of: "href" or "src" has to be first attribute in the tag.
* Require at least one "." in mailto address's domain.
* Try to dispose memory properly to avoid leaks.
* Handle local files with "?args" on the end.

### 0.7.0
* Added localRoot setting.
* Fixed mailto that ends with "?".
* Added userAgent setting, and it definitely makes some sites happier.

### 1.0.0
* Increased default timeout to 12.
* Check local anchors (#name) in current file.
* Support anchor-id (HTML5) as well as anchor-name.

### 1.1.0
* Added settings about checking local anchors (#name) and ID attributes in current file.

### 1.2.0
* Moved repeated add-diagnostic code into a function.

### 1.3.0
* Added setting and code to check if HTTPS equivalent exists for HTTP address.
* Added `Alt+M` to open current HTTP URL as an HTTPS URL in browser.

### 1.4.0
* Briefly tested IPv6 addresses to see that at least they don't cause anything to blow up.
* Set default user-agent string to latest Firefox.
* Added dontCheckURLsThatStartWith setting and code.
* Increased default timeout to 15.

### 1.5.0
* Added "Using the extension" image.
* Better message when 0 files left to do.
* Added addExtensionToLocalURLsWithNone setting and code.

### 1.6.0
* Added `Alt+L` to clear all diagnostics belonging to this extension.
* Changed my email address.

### 1.7.0
* Fixed README.

### 2.0.0
* Added support for XML and RSS files.

### 2.1.0
* Changed to Axios 0.19.0.
* On redirected link, give new URL.

### 2.2.0
* Updated package dependencies because of security warnings.
* Don't report link that redirects to same link (but rare, usually something is different).
* Don't report link that redirects to same link with a tracking parameter added (but rare, usually something else is different too).
* Fix status when file contains zero links.

### 3.0.0
* Added support for checking onion links. Simple pass or fail, consider redirect as pass, no way to control timeout or user-agent.
* Onto new versions of VSCode and npm and node.
* Updated default user-agent string to Firefox 76.

### 3.1.0
* Made `Alt+T` or `Alt+M` on onion link open it in Tor Browser, using xdotool.

### 3.2.0
* Flag onion links where domain name is illegal length.
* Moved xdotool command line strings into settings. (Wayland will use ydotool ?)
* Treat onion links that start with "http:" as clearnet links.

### 3.3.0
* Somehow using xdotool to send onion links to Tor Browser has stopped working.

### 4.0.0
* Added checking for bad characters and possible mistakes.
* Onto new versions of VSCode and npm and node.
* Updated default user-agent string to Firefox 83.
* Various code cleanup.
* Made regex's case-insensitive.

### 5.0.0
* Added checking for semantic HTML errors.
* Updated default user-agent string to Firefox 84.

### 5.1.0
* Added support of PHP files as if they were HTML files.
* Updated default user-agent string to Firefox 86.

### 6.0.0
* Added support of Markdown files.

### 6.1.0
* Added support for local Markdown links to heading IDs as in ```[link1](#heading1)```.
* Added support for Markdown headings automatically becoming local IDs, with spaces converted to dashes.
* Updated default user-agent string to Firefox 89.

### 6.2.0
* Tweaked support for Markdown headings automatically becoming local IDs: take a heading, remove any leading spaces, change it to lowercase, remove everything not letter digit hyphen space, then change spaces to hyphens.

### 6.3.0
* In Markdown, added requirement for ```[identifier``` at start of link.
* Updated npm and modules.

---

## Development
### To-Do list
* In Markdown, prevent collisions when generating implicit header IDs ? "# H" twice should generate IDs "h" and "h-1" ?
* Add tasks to open and close all HTML files in directory, so linter reports any errors.
* Maybe new axios has broken timeout ?
* Somehow using xdotool to open onion link in Tor Browser has gotten broken.
* Test onion links a lot more, maybe indicate redirects, any way to control timeout, set user-agent.
* Better way to open onion link in Tor Browser ?
* Way to open onion link in Tor Browser on Windows ?
* Add setting "do/don't check onion links".
* Snap version of VSCode uses `Alt+H` for Help menu.
* Create automated tests.
* Extension really is supposed to remove each diagnostic line after the corresponding source line is edited.
* Bundle extension to make it smaller/faster ?
https://code.visualstudio.com/api/working-with-extensions/bundling-extension
https://webpack.js.org/guides/getting-started/
* Can't really test IPv6 because my system and ISP have it turned off.
* Allow single-quotes on attributes ? I thought HTMLHint didn't allow them, so I didn't support them.
* Don't check a link if it has rel="nofollow" ? Probably should leave it as-is: check it.
* Any way to do retries inside axios ? Apparently not.
* Memory leaks ? Doesn't seem to be any tool to check an extension for leaking. Maybe not possible, since extensions are running inside a huge framework of Electron or Node or something.
* Display a "busy" cursor ? Can't. Window.withProgress could put up a dialog, but then user would have to close the dialog manually every time, don't want that. Doesn't seem to be a way to close that dialog programmatically.
* Click on diagnostic, do `Alt+T` or `Alt+M` to browser, come back to VSCode, cursor is in filter field of diagnostics pane instead of in source file. More convenient if in source file. But seems to be no way to do it.
* Multi-line tag (tag name and href/src attribute on different lines) silently ignored. Would be a lot of work to deal with, given the simple way the code does parsing.

### Development Environment
I'm no expert on this stuff, maybe I'm doing some things stupidly.

Now using:
* Fedora 34 KDE with X.
* VSCode deb 1.58.2 (which says Node.js: 14.16.0)
* ```node --version``` # v14.17.0
* ```npm --version``` # 6.14.13
* axios
* path
* fs
* tor-request

I did:
* ```sudo apt install npm```
* ```sudo npm install -g vsce```
In project directory:
* ```npm install```
* ```npm audit```
* ```npm audit fix```

* ```sudo npm -g install --save axios```

[GitHub repo for this extension](https://github.com/BillDietrich/linkcheckerhtml)

[Visual Studio Marketplace page for this extension](https://marketplace.visualstudio.com/items?itemName=BillDietrich.linkcheckerhtml)

[My web site](https://www.billdietrich.me/)

---

## Privacy Policy
This extension doesn't collect, store or transmit your identity or personal information in any way. All it does is read the current editor window, do existence-tests on local files, open links to internet sites, and send internet links to your browser.