https://github.com/alex-page/get-site-urls

🔗 Get all of the URL's from a website.
https://github.com/alex-page/get-site-urls

crawler sitemap-generator urls

Last synced: about 2 months ago
JSON representation

🔗 Get all of the URL's from a website.

• Host: GitHub
• URL: https://github.com/alex-page/get-site-urls
• Owner: alex-page
• License: mit
• Created: 2018-05-21T10:22:09.000Z (over 6 years ago)
• Default Branch: main
• Last Pushed: 2023-07-29T18:30:17.000Z (over 1 year ago)
• Last Synced: 2024-10-12T23:15:21.185Z (2 months ago)
• Topics: crawler, sitemap-generator, urls
• Language: JavaScript
• Homepage:
• Size: 414 KB
• Stars: 27
• Watchers: 3
• Forks: 8
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-x - get-site-urls

README

        
# 🔗 get-site-urls

> Crawl a URL to generate a sitemap and find 404 errors with one command

## Usage

### Command line interface

```shell

npx get-site-urls polaris.shopify.com

```

#### CLI options

```shell

$ npx get-site-urls --help

Get all of the URL's from a website.

Usage

	$ get-site-urls 

Options

	--output=, -o   File saved to the system

	--max-depth=,   Maximum depth of routes to search

	--alias=,       Replace  with  for sitemap generation

Examples

	$ get-site-urls polaris.shopify.com --output=sitemap.xml

	✅ Created sitemap.xml with 137 urls

```

### NodeJS

```js

import getSiteUrls from "get-site-urls";

const urls = await getSiteUrls("alexpage.dev");

```

#### NodeJS options

`maxDepth` - Maximum depth of routes to search

```js

import getSiteUrls from "get-site-urls";

const maxDepth = 2;

const urls = await getSiteUrls("alexpage.dev", maxDepth);

```

## Release History

- v3.0.0 - stdout replaced with output file

- v2.0.4 - Remove exports, relative bin entry, index.js !# to run in node env

- v2.0.3 - Rename main to exports in package.json

- v2.0.2 - Add name to bin entry in package.json

- v2.0.1 - Add missing bin entry in package.json

- v2.0.0 - Convert get-site-urls to a cli tool

- v1.1.7 - Update dependencies and GitHub actions to yml syntax

- v1.1.6 - Use `xo` and `ava`, update dependencies

- v1.1.5 - Add tests for urls with ending slash, update documentation

- v1.1.4 - Use files instead of `.npmignore`

- v1.1.3 - Replace travis with GitHub actions

- v1.1.2 - Update package.json link

- v1.1.1 - Fix issue with CI

- v1.1.0 - Fixing bugs with urls, adding tests and basic auth support

- v1.0.0 - 💥 First commit