https://github.com/ernesto-jimenez/emit_urls

Crawl a website and print all crawled URLs to STDOUT as they are discovered
https://github.com/ernesto-jimenez/emit_urls

Last synced: 24 days ago
JSON representation

Crawl a website and print all crawled URLs to STDOUT as they are discovered

• Host: GitHub
• URL: https://github.com/ernesto-jimenez/emit_urls
• Owner: ernesto-jimenez
• Created: 2013-10-30T17:14:02.000Z (about 11 years ago)
• Default Branch: master
• Last Pushed: 2013-10-30T17:30:36.000Z (about 11 years ago)
• Last Synced: 2024-04-09T16:26:27.205Z (7 months ago)
• Language: Go
• Size: 117 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# emit_urls commandline tool

This is an small tool that will crawl a website and print in STDOUT all crawled URLs.

# Possible uses

* See all the URLs crawlable in the local development server:

```emit_urls http://localhost:3000```

* Run some other command for all each crawlable url with GNU/Parallel:

```emit_urls http://localhost:3000 | parallel "some_command {}"```

* Find out about your links producing 404:

```emit_urls --logfile=/dev/stdout http://0.0.0.0:8000 | grep 404```

# Installation

```

go get github.com/ernesto-jimenez/emit_urls

go install github.com/ernesto-jimenez/emit_urls

```