https://github.com/ssujitx/sitemap-postlink-scraper
A Python script for scraping website sitemap links and saving them to text files.
https://github.com/ssujitx/sitemap-postlink-scraper
Last synced: about 2 months ago
JSON representation
A Python script for scraping website sitemap links and saving them to text files.
- Host: GitHub
- URL: https://github.com/ssujitx/sitemap-postlink-scraper
- Owner: SSujitX
- Created: 2023-09-17T15:11:44.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-07-16T16:00:41.000Z (about 2 years ago)
- Last Synced: 2025-10-23T17:50:00.121Z (10 months ago)
- Language: Python
- Homepage:
- Size: 12.7 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Website Sitemap Scraper
The Website Sitemap Scraper is a Python script that allows you to fetch and extract sitemap links from a website. This tool is useful for collecting information about a website's structure and content.
## Features
- Fetches sitemap links from a specified website.
- Saves the sitemap links to a text file for future reference.
## Prerequisites
Before you begin, ensure you have met the following requirements:
- Python 3.7 or higher installed on your system.
- The following Python libraries installed:
- `httpx`: Used for making asynchronous HTTP requests.
- `selectolax`: Used for parsing HTML/XML content.
You can install the required libraries using pip:
```bash
pip install -r requirements.txt
```
## Usage
1. Clone this repository to your local machine:
```bash
git clone https://github.com/your-username/Sitemap-Postlink-Scraper.git
```
2. Navigate to the project directory:
```bash
cd Sitemap-Postlink-Scraper
```
3. Run the script:
```bash
python sitemap_post_scraper.py
```
4. Follow the on-screen instructions to provide the URL of the website you want to scrape.
5. If a sitemap is found on the website, the script will fetch and save the sitemap links to a text file named _sitemap_links.txt.