https://github.com/zachleat/spider-pig

Get a list of local URL links from a root URL.
https://github.com/zachleat/spider-pig

Last synced: 3 days ago
JSON representation

Get a list of local URL links from a root URL.

• Host: GitHub
• URL: https://github.com/zachleat/spider-pig
• Owner: zachleat
• License: mit
• Created: 2018-01-17T13:33:45.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2024-02-22T22:51:19.000Z (9 months ago)
• Last Synced: 2024-10-31T16:56:59.472Z (10 days ago)
• Language: JavaScript
• Homepage:
• Size: 65.4 KB
• Stars: 67
• Watchers: 3
• Forks: 6
• Open Issues: 13
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# spider-pig

Get a list of local URL links from a root URL. Works with JavaScript generated content. Can also act as a live-DOM CSS search across multiple files (find all the templates that are using the CSS selector I want to change).

```

Test

Test

Duplicate Test

URL to self

Email link

External

```

Normalizes all of the matching URLs to be full absolute URLs (including host and protocol and path, etc).

## Installation

Available on [npm](https://www.npmjs.com/package/@zachleat/spider-pig).

```

npm install @zachleat/spider-pig

```

## Usage

```

$ spiderpig http://zachleat.localhost/web/

http://zachleat.localhost/web/

http://zachleat.localhost/web/about/

http://zachleat.localhost/web/best-of/

…

```

### Filter URLs

```

$ spiderpig http://zachleat.localhost/web/ --filter="/about/"

http://zachleat.localhost/web/about/

```

### Show URLs that contain a CSS Selector

Useful to see where CSS changes might regress on a project. This takes a bit of time. You can also mix in `--filter` here too.

```

$ spiderpig http://zachleat.localhost/web/ --selector=".header"

Found 180 urls.

Looking for urls with CSS selector ".header":

http://zachleat.localhost/web/ ✅  Yes

http://zachleat.localhost/web/about/ ✅  Yes

http://zachleat.localhost/web/best-of/ ✅  Yes

http://zachleat.localhost/web/projects/ ✅  Yes

```

#### Add a limit

Use `--selectorlimit` to set an upper bound on the number of URLs that get spidered.

```

$ spiderpig http://zachleat.localhost/web/ --selector=".header" --selectorlimit=3

Found 180 urls

Looking for urls with CSS selector ".header" (limit 3):

http://zachleat.localhost/web/ ✅  Yes

http://zachleat.localhost/web/about/ ✅  Yes

http://zachleat.localhost/web/best-of/ ✅  Yes

…

```

### Debug mode

```

$ DEBUG=SpiderPig spiderpig http://zachleat.localhost/web/

```

### API

#### Get URLs

```

const SpiderPig = require("@zachleat/spider-pig");

(async function() {

	let sp = new SpiderPig();

	await sp.start();

	let urls = await sp.fetchLocalUrls("http://localhost/myproject/");

	// Optional, filter (case sensitive)

	urls = sp.filterUrls(urls, "views");

})();

```

#### Search for a selector

```

const SpiderPig = require("@zachleat/spider-pig");

(async function() {

	let sp = new SpiderPig();

	let selector = ".test-css-selector:nth-child(2)";

	await sp.start();

	let urls = await sp.fetchLocalUrls("http://localhost/myproject/");

	for(let url of urls) {

		if( await sp.hasSelector(url, selector) ) {

			// has it

		} else {

			// doesn’t have it

		}

	}

})();

```

##

![Homer Simpson holding the Spider Pig](docs/spider-pig.jpg)