https://github.com/cuongnb14/web2md
Convert any website into markdown data for LLM
https://github.com/cuongnb14/web2md
Last synced: 8 months ago
JSON representation
Convert any website into markdown data for LLM
- Host: GitHub
- URL: https://github.com/cuongnb14/web2md
- Owner: cuongnb14
- Created: 2025-02-21T15:26:11.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-02-21T15:45:27.000Z (8 months ago)
- Last Synced: 2025-02-21T16:30:25.098Z (8 months ago)
- Language: Python
- Homepage:
- Size: 4.88 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Website to Markdown
## Overview
This script is a simple web crawler that extracts content from a given website and converts it into Markdown format. It supports multi-threaded crawling to improve speed and allows customization of parameters such as maximum workers.
## Features
- Extracts the main content from web pages and converts it to Markdown.
- Crawls internal links.
- Uses multi-threading for faster execution.
- Saves the extracted content to a Markdown file.
## Installation
Ensure you have Python installed, then install the required dependencies:
```bash
pip install -r requirements.txt
```
## Usage
Run the script with the following command:
```bash
python main.py --output_file --max_workers
```
Or with docker
```bash
docker run --rm -v ./:/tmp/out cuongnb14/web2md:1.0.1 --output_file /tmp/out.md
```
### Arguments:
- ``: The starting URL for crawling.
- `--output_file`: The output file to store the extracted content (default: `crawled_content.md`).
- `--max_workers`: The number of concurrent workers (default: `5`).
- `--rendering`: Use playwright to render page
- `--only-main`: Only use playwright for first page
### Example:
```bash
python main.py "https://example.com/docs" --output_file output.md --max_workers 10
```