https://github.com/qubyte/sitemap

A simple webcrawler
https://github.com/qubyte/sitemap

Last synced: about 1 month ago
JSON representation

A simple webcrawler

• Host: GitHub
• URL: https://github.com/qubyte/sitemap
• Owner: qubyte
• License: mit
• Created: 2016-09-09T12:41:14.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2017-01-02T11:36:42.000Z (over 8 years ago)
• Last Synced: 2025-02-14T09:32:22.316Z (3 months ago)
• Language: Go
• Size: 18.6 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
[![Build Status](https://travis-ci.org/qubyte/sitemap.svg)](https://travis-ci.org/qubyte/sitemap)

# sitemap

sitemap is a library that can crawl a given URL and produce a map of sites

within that domain and subdomains. It will honour `rel="nofollow"` attributes

on links.

sitemap will attempt to gather information on links, scripts, images, videos,

and audio.