https://github.com/nuraj250/sitemap-crawler
A simple web-based sitemap crawler built with Node.js, Express, and SimpleCrawler. This tool allows users to input a URL and crawl the website, generating lists of successful and failed URLs. The results can be downloaded for further analysis.
https://github.com/nuraj250/sitemap-crawler
express link-checker nodejs seo-tools simplecrawler sitemap-generator web-crawler web-scraping website-analysis
Last synced: about 2 months ago
JSON representation
A simple web-based sitemap crawler built with Node.js, Express, and SimpleCrawler. This tool allows users to input a URL and crawl the website, generating lists of successful and failed URLs. The results can be downloaded for further analysis.
- Host: GitHub
- URL: https://github.com/nuraj250/sitemap-crawler
- Owner: Nuraj250
- Created: 2024-03-19T10:17:04.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-16T16:11:25.000Z (over 1 year ago)
- Last Synced: 2025-05-20T07:32:26.325Z (about 1 year ago)
- Topics: express, link-checker, nodejs, seo-tools, simplecrawler, sitemap-generator, web-crawler, web-scraping, website-analysis
- Language: JavaScript
- Homepage:
- Size: 20.5 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Sitemap Crawler
## Overview
Sitemap Crawler is a simple web-based tool built with **Node.js, Express, and SimpleCrawler**. It allows users to input a URL, crawl the website, and generate lists of successful and failed URLs. The results can be downloaded for further analysis.
## Features
- Web-based UI for entering a URL and starting a crawl.
- Uses **SimpleCrawler** to fetch and analyze pages.
- Lists **successful and failed URLs** separately.
- Allows downloading results for further analysis.
- Built with **Node.js, Express, and Bootstrap** for easy use and customization.
## Installation & Setup
### Prerequisites
Ensure you have **Node.js** installed on your system.
### Clone the Repository
```bash
git clone https://github.com/Nuraj250/sitemap-crawler.git
cd sitemap-crawler
```
### Install Dependencies
```bash
npm install
```
### Run the Server
```bash
node app.js
```
By default, the server runs on **http://localhost:3000**.
## Usage
1. Open `index.html` in your browser.
2. Enter a URL in the input field.
3. Click **"Start Crawl"** to begin.
4. The progress will be displayed in a modal.
5. Once completed, view the results and download the successful or failed URLs.
## Technologies Used
- **Node.js** - Backend framework
- **Express.js** - Server setup
- **SimpleCrawler** - Web crawling library
- **Bootstrap** - UI framework
- **JavaScript** - Frontend scripting
## Contribution
Feel free to fork this repository and submit pull requests with improvements or additional features.
## License
This project is licensed under the **MIT License**.