Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/UmbrellaDocs/linkspector

Uncover broken links in your content.
https://github.com/UmbrellaDocs/linkspector

hacktoberfest link-check link-checker markdown-link-check markdown-links

Last synced: 2 months ago
JSON representation

Uncover broken links in your content.

Host: GitHub
URL: https://github.com/UmbrellaDocs/linkspector
Owner: UmbrellaDocs
License: apache-2.0
Created: 2023-08-23T01:11:20.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-10-04T01:11:55.000Z (4 months ago)
Last Synced: 2024-11-07T06:43:50.036Z (2 months ago)
Topics: hacktoberfest, link-check, link-checker, markdown-link-check, markdown-links
Language: JavaScript
Homepage:
Size: 380 KB
Stars: 66
Watchers: 6
Forks: 11
Open Issues: 16
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md

Awesome Lists containing this project

project-awesome - UmbrellaDocs/linkspector - Uncover broken links in your content. (JavaScript)
awesome-docs - Linkspector

README

[![GitHub Marketplace](https://img.shields.io/badge/GitHub%20Marketplace-action%20linkspector-brightgreen?style=for-the-badge)](https://github.com/marketplace/actions/run-linkspector-with-reviewdog)
[![NPM](https://img.shields.io/npm/v/@umbrelladocs/linkspector?style=for-the-badge)](https://www.npmjs.com/package/@umbrelladocs/linkspector)

Uncover broken links in your content.

Linkspector

Linkspector is a CLI app that checks for dead hyperlinks in files.
It supports multiple markup languages such as Markdown, AsciiDoc (limited - hyperlinks only), and ReStructured Text (coming soon).

With Linkspector, you can easily check all hyperlinks in your files, ensuring that they are not broken and that your readers can access all the relevant content.
The app allows you to quickly and easily identify any broken links, so you can fix them before publishing your content.

Linkspector is a powerful tool for anyone who creates content using markup languages.

## How this is different from existing tools?

1. **Enhanced Link Checking with Puppeteer**: It uses [Puppeteer](https://pptr.dev/) to check links in Chrome's headless mode, reducing the number of false positives.
2. **Addresses limitations and adds user-requested features**: It is built to adress the shortcomings in [GitHub Action - Markdown link check](https://github.com/gaurav-nelson/github-action-markdown-link-check) and adds many user requested features.
3. **Single repository for seamless collaboration**: All the code it needs to run is in a single repository, making it easier for community to collaborate.
4. **Focused for CI/CD use**: Linkspector is purposefully tailored to run into your CI/CD pipelines. This ensures that link checking becomes an integral part of your development workflow.

## Installation

Before you can use Linkspector, you need to install it. You can do this using the following command:

```bash
npm install -g @umbrelladocs/linkspector
```

This command installs Linkspector globally, allowing you to use it from anywhere in your terminal. If you don't want to install using `npm` you can download the binary from GitHub releases.

### GitHub action

For more details, see [action-linkspector](https://github.com/UmbrellaDocs/action-linkspector)

## Checking Hyperlinks

To check hyperlinks in your markup language files, follow these steps:

1. Open your terminal.

1. Navigate to the directory containing the files you want to check.

1. (**Optional**) Create a [configuration](#configuration) file called `.linkspector.yml`. By default, Linkspector looks for a configuration file named `.linkspector.yml` in the current directory. If you have a custom configuration file or want to specify its path, you can use the `-c` or `--config` option.

1. Use the `linkspector check` command to initiate the hyperlink check. For example:

```bash
linkspector check
```

- To specify a custom configuration file path:

```bash
linkspector check -c /path/to/custom-config.yml
```

- To output the results in JSON format:

```bash
linkspector check -j
```

The JSON output follows [rdjson](https://github.com/reviewdog/reviewdog/tree/master/proto/rdf#rdjson) format.

1. Linkspector starts checking the hyperlinks in your files based on the configuration provided in the configuration file or using the default configuration. It then displays the results in your terminal.

1. After the check is complete, Linkspector provides a summary of the results. If any dead links are found, they are listed in the terminal, along with their status codes and error messages.

1. If no dead links are found, Linkspector displays a success message, indicating that all links are working.

## Configuration

Linkspector uses a configuration file named `.linkspector.yml` to customize its behavior. If this file is not found in the current directory when the program is run, Linkspector displays a message saying "Configuration file not found. Using default configuration." and uses a default configuration.

### Default Configuration

The default configuration is as follows:

```yaml
dirs:
- .
useGitIgnore: true
```

If you are defining a custom configuration, you must include the `dirs` or `files` section in the configuration file.

Following are the available configuration options:

| Option | Description | Required |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | --------------------------------- |
| [`files`](#files-to-check) | The list of Markdown files to check for broken links. | Yes, if `dirs` is not specified. |
| [`dirs`](#directories-to-search) | The list of directories to search for Markdown files. | Yes, if `files` is not specified. |
| [`excludedFiles`](#excluded-files) | The list of Markdown files to exclude from the link checking process. | No |
| [`excludedDirs`](#excluded-directories) | The list of directories to exclude from the link checking process. | No |
| [`baseUrl`](#base-url) | The base URL to use when checking relative links in Markdown files. | No |
| [`ignorePatterns`](#ignore-patterns) | The list of regular expressions that match URLs to be ignored during link checking. | No |
| [`replacementPatterns`](#replacement-patterns) | The list of regular expressions and replacement strings to modify URLs during link checking. | No |
| [`aliveStatusCodes`](#alive-status-codes) | The list of HTTP status codes that are considered as "alive" links. | No |
| [`useGitIgnore`](#use-gitignore) | Indicates whether to use the rules defined in the `.gitignore` file to exclude files and directories. | No |
| [`modifiedFilesOnly`](#check-modified-files-only) | Indicates whether to check only the files that have been modified in the last git commit. | No |

### Files to Check

The `files` section specifies the Markdown files that Linkspector should check for broken links. You can add the file paths you want to include in this list. For example:

```yaml
files:
- README.md
- file2.md
- file3.md
```

### Directories to Search

The `dirs` section lists the directories where Linkspector should search for Markdown files. You can specify directories relative to the current working directory. For example:

```yaml
dirs:
- ./
- folder2
```

### Excluded Files

The `excludedFiles` section allows you to specify Markdown files that should be excluded from the link checking process. Add the paths of the files you want to exclude. For example:

```yaml
excludedFiles:
- ./check.md
- excluded-file2.md
```

### Excluded Directories

The `excludedDirs` section lets you specify directories that should be excluded from the link checking process. Provide the paths of the directories you want to exclude. For example:

```yaml
excludedDirs:
- ./lib
- excluded-folder2
```

### Base URL

The `baseUrl` option sets the base URL that will be used when checking relative links in Markdown files. In this example:

```yaml
baseUrl: https://example.com
```

The base URL is set to `https://example.com`.

### Ignore Patterns

The `ignorePatterns` section allows you to define regular expressions that match URLs to be ignored during the link checking process. For example:

```yaml
ignorePatterns:
- pattern: '^https://example.com/skip/.*$'
- pattern: "^(ftp)://[^\\s/$?#]*\\.[^\\s]*$"
```

In this example, URLs matching the specified patterns will be skipped during link checking.

### Replacement Patterns

The `replacementPatterns` section lets you define regular expressions and replacement strings to modify URLs during link checking. For example:

```yaml
replacementPatterns:
- pattern: "(https?://example.com)/(\\w+)/(\\d+)"
replacement: '$1/id/$3'
- pattern: "\\[([^\\]]+)\\]\$(https?://example.com)/file\$"
replacement: '$1'
```

These patterns and replacements will be applied to URLs found in the Markdown files.

### Alive Status Codes

The `aliveStatusCodes` section allows you to specify a list of HTTP status codes that are considered as "alive" links. In this example:

```yaml
aliveStatusCodes:
- 200
- 201
- 204
```

Links returning any of these status codes will be considered valid.

### Use .gitignore

The `useGitIgnore` option, when set to `true`, indicates that Linkspector should use the rules defined in the `.gitignore` file to exclude files and directories. For example:

```yaml
useGitIgnore: true
```

When enabled, the app will respect the `.gitignore` rules during link checking.

### Check Modified Files Only

The `modifiedFilesOnly` option, when set to `true`, indicates that Linkspector should only check the files that have been modified in the last git commit. For example:

```yaml
modifiedFilesOnly: true
```

When enabled, Linkspector will use `git` to find the list of modified files and only check those files. Please note that this option requires `git` to be installed and available on your system path. If `git` is not installed or not found in the system path, Linkspector will throw an error.

Also, if no modified files are found in the list of files to check, Linkspector will skip link checking and exit with a message indicating that modified files are not specified in the configuration.

### Sample configuration

```yml
files:
- README.md
- file2.md
- file3.md
dirs:
- ./
- folder2
excludedFiles:
- ./check.md
- excluded-file2.md
excludedDirs:
- ./lib
- excluded-folder2
baseUrl: https://example.com
ignorePatterns:
- pattern: '^https://example.com/skip/.*$'
- pattern: "^(ftp)://[^\\s/$?#]*\\.[^\\s]*$"
replacementPatterns:
- pattern: "(https?://example.com)/(\\w+)/(\\d+)"
replacement: '$1/id/$3'
- pattern: "\\[([^\\]]+)\\]\$(https?://example.com)/file\$"
replacement: '$1'
aliveStatusCodes:
- 200
- 201
- 204
useGitIgnore: true
```

## Sample output

If there are failed links, linkspector shows the output as comma-seprated values and exit with error.
`File, HTTP status code, Line number, Error message`

```
REDISTRIBUTED.md, https://unlicense.org/, null, 186, net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH at https://unlicense.org/]
💥 Error: Some hyperlinks in the specified files are invalid.
```

If there are no errors, linkspector shows the following message:

```
✨ Success: All hyperlinks in the specified files are valid.
```

## Using Linkspector with Docker

To use Linkspector with Docker, follow these steps:

1. Clone the Linkspector repository to your local machine and switch to the cloned directory:
```bash
git clone [email protected]:UmbrellaDocs/linkspector.git
cd linkspector
```
1. Build the docker image locally, while being at the root (`.`) of this project:

```bash
docker build --no-cache --pull --build-arg LINKSPECTOR_PACKAGE= -t umbrelladocs/linkspector .
```

1. To perform a check using the default configuration, while being at the root (`$PWD`) of the project to be checked:

```bash
docker run --rm -it -v $PWD:/app \
--name linkspector umbrelladocs/linkspector \
bash -c 'linkspector check'
```

To specify a custom configuration file path:

```bash
docker run --rm -it -v $PWD:/app -v $PWD/custom-config.yml:/path/to/custom-config.yml \
--name linkspector umbrelladocs/linkspector \
bash -c 'linkspector check -c /path/to/custom-config.yml'
```

## What's planned

- [x] Spinner for local runs.
- [x] Create a GitHub action. See [action-linkspector](https://github.com/UmbrellaDocs/action-linkspector)
- [x] Modified files only check.
- [!] Asciidoc support. (Limited to hyperlinks only)
- [ ] ReStructured Text support.
- [ ] Disable binary files downlaod.
- [x] JSON output for `failed-only` ~~or `all`~~ links.
- ~~[ ] CSV output for `all` links.~~ (dropped for now)
- ~~[ ] Experimaental mode to gather all links and check them in batches to study performance gains.~~ (dropped for now)
- ~~[ ] Proxy support to connect puppeteer to a remote service.~~ (dropped for now)
- ~~[ ] Puppeteer config support.~~ (dropped for now)

## Contributing

If you would like to contribute to Linkspector, please read the [contributing guidelines](/CONTRIBUTING.md).