https://github.com/excoriate/inspector-gadget-cli
A CLI tool for inspecting and analyzing web links.
https://github.com/excoriate/inspector-gadget-cli
cli link-checker rust seo-tools web-crawler
Last synced: 6 months ago
JSON representation
A CLI tool for inspecting and analyzing web links.
- Host: GitHub
- URL: https://github.com/excoriate/inspector-gadget-cli
- Owner: Excoriate
- Created: 2024-08-24T09:04:56.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-08-25T07:57:16.000Z (about 1 year ago)
- Last Synced: 2025-02-05T12:18:34.290Z (8 months ago)
- Topics: cli, link-checker, rust, seo-tools, web-crawler
- Language: Rust
- Size: 120 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Inspector Gadget CLI
A powerful CLI tool for inspecting and analyzing web links. Handy when it comes to feeding your LLM with the right context.
[](https://github.com/Excoriate/inspector-gadget-cli/actions/workflows/rust.yml)
## Installation
### Option 1: Install pre-built binary (Recommended)
You can install the Inspector Gadget CLI using our installation script:
```bash
curl -sSf https://raw.githubusercontent.com/Excoriate/inspector-gadget-cli/main/scripts/install.sh | sh
```
This script will download the appropriate binary for your platform, extract it, and install it to `/usr/local/bin` with the correct permissions. You can then run the `inspector` command directly without needing to use `./inspector-gadget`.
### Option 2: Install from Source
To install Inspector Gadget CLI from source, you can use the provided installation script:
```bash
bash scripts/install_from_source.sh
```
This will clone the repository, build the project, and install the binary to `/usr/local/bin`.
## Usage
```bash
inspector [OPTIONS]
```
Options:
| Option | Description |
|--------|-------------|
| `--output-format ` | Choose between json, yaml, txt, or clipboard (default: json) |
| `--output-file ` | Specify the output file name (default: inspect-result-.) |
| `--log-level ` | Adjust the verbosity of logs (e.g., info, debug, error) (default: info) |
| `--help` | Displays help information |
| `--version` | Shows version information |
| `--show-links` | Show links in the terminal |
| `--detailed` | Show detailed information including ignored links |
| `--config ` | Sets a custom config file |
| `--ignore-domains ` | Comma-separated list of domains to ignore |
| `--ignore-regex ` | Comma-separated list of regex patterns to ignore URLs |
| `--forbidden-domains ` | Comma-separated list of forbidden domains |
| `--ignored-childs ` | Comma-separated list of child paths to ignore |
| `--timeout ` | Timeout in seconds for each HTTP request |
Example:
```bash
inspector https://docs.dagger.io --show-links --output-format=txt --output-file=dagger-doc-links
```
## Configuration
The inspector tool uses a YAML configuration file named `.inspector-config.yml` in the user's home directory. This file allows you to customize various aspects of the link inspection process.
Here's a description of the configuration options:
| Field | Type | Description |
|-------|------|-------------|
| `url` | String | The base URL to start the inspection from (required) |
| `ignore` | Object | Contains settings for ignoring certain URLs |
| `ignore.domains` | Array of Strings | List of domain suffixes to ignore |
| `ignore.regex` | Array of Strings | List of regex patterns to ignore URLs |
| `forbidden_domains` | Array of Strings | List of domain suffixes that are forbidden to scan |
| `ignored_childs` | Array of Strings | List of URL path prefixes to ignore |
| `timeout` | Integer | Timeout in seconds for each HTTP request |
| `default_output` | String | Default output format if not specified in CLI arguments |
### Example Configuration
```yaml
url: https://docs.dagger.io
ignore:
domains:
- "example.com"
- "test.org"
regex:
- "^https?://localhost"
- "^https?://127\\.0\\.0\\.1"
forbidden_domains:
- "forbidden.com"
- "restricted.org"
ignored_childs:
- "/api/"
- "/internal/"
timeout: 30
default_output: "json"
```
You can use a custom configuration file by specifying its path:
```bash
inspector https://example.com --config /path/to/custom-config.yml
```
## Contributing
Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request.
## License
Inspector CLI is licensed under the [MIT License](LICENSE).
## Examples
### Terragrunt Documentation
To fetch all the Terragrunt documentation, you can use the following commands:
```bash
# Locally
just run https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/ --show-links --output-format=txt --output-file=terragrunt-docs-links
# or using the inspector cli
inspector https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/ --show-links --output-format=txt -o terragrunt-docs-links
```
Alternatively, you can use a configuration file:
```bash
# Locally
just run https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/ --config docs/examples/terragrunt-docs/terragrunt-inspector-config.yml
# or using the inspector cli
inspector https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/ --config docs/examples/terragrunt-docs/terragrunt-inspector-config.yml
```
The `terragrunt-inspector-config.yml` file contains the following configuration:
```yaml
url: https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/
ignore:
domains:
- "github.com"
- "twitter.com"
- "linkedin.com"
regex:
- "^https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/#.*$"
forbidden_domains:
- "example.com"
ignored_childs:
- "/community/"
- "/plugins/"
timeout: 30
default_output: "json"
```
### Terraform Documentation
To fetch all the Terraform documentation, you can use the following commands:
```bash
# Locally
just run https://terraform-docs.io/user-guide/introduction/ --show-links --output-format=txt --output-file=terraform-docs-links
# or using the inspector cli
inspector https://terraform-docs.io/user-guide/introduction/ --show-links --output-format=txt -o terraform-docs-links
```
Alternatively, you can use a configuration file:
```bash
# Locally
just run https://terraform-docs.io/user-guide/introduction/ --config docs/examples/terraform-docs/terraform-docs-inspector-config.yml
# or using the inspector cli
inspector https://terraform-docs.io/user-guide/introduction/ --config docs/examples/terraform-docs/terraform-docs-inspector-config.yml
```
The `terraform-docs-inspector-config.yml` file contains the following configuration:
```yaml
url: https://terraform-docs.io/user-guide/introduction/
ignore:
domains:
- "github.com"
- "twitter.com"
- "linkedin.com"
regex:
- "^https://terraform-docs.io/user-guide/.*#.*$"
forbidden_domains:
- "example.com"
ignored_childs:
- "/community/"
- "/plugins/"
timeout: 30
default_output: "json"