{"id":26093156,"url":"https://github.com/yamasammy/web2llm","last_synced_at":"2025-10-29T01:36:10.780Z","repository":{"id":280691351,"uuid":"942851423","full_name":"yamasammy/Web2LLM","owner":"yamasammy","description":"An advanced Python tool for extracting data from websites, cleaning the content, and converting it to high-quality Markdown for optimal use by LLM systems.","archived":false,"fork":false,"pushed_at":"2025-03-04T19:56:48.000Z","size":41,"stargazers_count":15,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-04T17:50:50.344Z","etag":null,"topics":["ai","api","llm","markdown","scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yamasammy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-04T19:25:01.000Z","updated_at":"2025-07-03T09:14:58.000Z","dependencies_parsed_at":"2025-03-04T20:22:56.541Z","dependency_job_id":"1c4d41f8-52ac-4a55-b281-f49f1d0aeec0","html_url":"https://github.com/yamasammy/Web2LLM","commit_stats":null,"previous_names":["yamasammy/web2llm"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/yamasammy/Web2LLM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yamasammy%2FWeb2LLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yamasammy%2FWeb2LLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yamasammy%2FWeb2LLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yamasammy%2FWeb2LLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yamasammy","download_url":"https://codeload.github.com/yamasammy/Web2LLM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yamasammy%2FWeb2LLM/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281544212,"owners_count":26519552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","api","llm","markdown","scraping"],"created_at":"2025-03-09T11:55:42.151Z","updated_at":"2025-10-29T01:36:10.623Z","avatar_url":"https://github.com/yamasammy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web2LLM\n\nAn advanced Python tool for extracting data from websites, cleaning the content, and converting it to high-quality Markdown for optimal use by LLM systems.\n\n## Features\n\n- **LLM-Optimized Content Extraction**: Intelligently extracts and cleans web content specifically formatted for Large Language Models and AI-powered IDEs like Cursor\n- **AI-Ready Documentation Generation**: Creates markdown files that can be used to feed AI tools with the latest framework documentation, API references, or technical guides\n- **Context Window Optimization**: Removes non-essential elements (headers, footers, navbars) to maximize the useful information within LLM context windows\n- **Knowledge Base Enhancement**: Generates clean, structured markdown perfect for building custom knowledge bases to augment AI capabilities\n- **Framework Documentation Updates**: Easily capture the latest documentation for programming frameworks to keep your AI tools up-to-date\n- **Intelligent Content Processing**:\n  - Removal of distracting UI elements that confuse AI parsers\n  - Complete elimination of CSS and JavaScript that waste token space\n  - Smart detection of navigation elements through semantic analysis\n- **Multiple Output Formats** optimized for different AI consumption patterns\n- **REST API** for seamless integration into AI workflows\n- **Automatic File Management** with intelligent naming for organized knowledge repositories\n\n## Installation\n\n```bash\npip install -r requirements.txt\n```\n\n## Usage\n\n### Command Line\n\n```bash\n# Scrape a URL and display the result\npython run.py scrape https://example.com\n\n# Scrape a URL and save as Markdown\npython run.py scrape https://example.com --save\n\n# Specify an output filename\npython run.py scrape https://example.com --save --output my-file.md\n```\n\n### Start the API\n\n```bash\npython -m app.main\n```\n\n### Use as a Library\n\n```python\nfrom app.scraper import scrape_url\nfrom app.converter import html_to_markdown\n\n# Scrape a URL\nresult = scrape_url(\"https://example.com\")\nhtml_content = result[\"html\"]\n\n# Convert to markdown\nmarkdown_content = html_to_markdown(html_content)\n\n# Save to a file\nwith open(\"output.md\", \"w\") as f:\n    f.write(markdown_content)\n```\n\n## API Endpoints\n\n- `POST /scrape`: Scrape a URL and return the content in Markdown\n- `POST /scrape/save`: Scrape a URL and save the content as a Markdown file\n\n## Major Improvements\n\n### 1. AI-Optimized Content Extraction\n\n- **Token Efficiency**: Removes headers, footers, and navigation elements to maximize useful content within LLM context windows\n- **Advanced AI-Confusing Element Detection**:\n  - Identifies and removes elements by standard CSS selectors\n  - Uses link density analysis to detect navigation menus\n  - Employs semantic content analysis to identify non-essential sections\n  - Recognizes positional patterns typical of UI elements\n  - Detects sidebar elements through structural analysis\n- **Smart Content Preservation**:\n  - Retains information-rich sections (\u003e1000 characters)\n  - Applies adaptive cleaning based on content type\n  - Uses configurable thresholds for different website categories\n\n### 2. LLM Context Window Optimization\n\n- **Complete removal of token-wasting elements** like scripts, styles, and decorative markup\n- **Elimination of interactive JavaScript attributes** irrelevant to AI processing\n- **Removal of styling information** that consumes valuable context space\n- **Filtering of code snippets** not relevant to the main content\n- **Cleaning of metadata sections** that don't contribute to understanding\n\n### 3. AI-Ready Markdown Generation\n\n- **Multi-layered conversion strategy**:\n  - Primary conversion optimized for AI readability\n  - Structured extraction fallback for complex layouts\n  - Plain text preservation when structure is less important\n- **Enhanced semantic structure** for better AI comprehension\n- **Special handling** for data-rich elements like tables, quotes, and code blocks\n- **Optimized whitespace** for improved token efficiency\n\n### 4. LLM Integration Reliability\n\n- **Fallback mechanisms** to ensure content is always retrievable\n- **Format consistency** for predictable AI processing\n- **Encoding normalization** for cross-platform compatibility\n- **Intelligent file organization** for systematic knowledge management\n\n## Adjustable Parameters\n\nTo adapt the tool to specific sites, you can modify:\n\n1. **Detection thresholds** in `detect_nav_by_content()`:\n   - Number of links (currently 8)\n   - Percentage of short links (currently 85%)\n   - Text length considered significant (currently 50 characters per link)\n\n2. **CSS selectors** in `remove_headers_footers()`:\n   - Add specific selectors for certain sites\n   - Modify the `header_selectors`, `footer_selectors`, etc. lists\n\n3. **Content thresholds** in `clean_html()`:\n   - Modify the 500 character threshold for additional extraction\n   - Adjust the 70% threshold for applying advanced detection\n\n## AI Integration Use Cases\n\n### Enhancing AI-Powered IDEs like Cursor\n\n- **Framework Documentation Updates**: Keep your AI coding assistant up-to-date with the latest framework documentation by scraping official docs\n- **API Reference Integration**: Create clean markdown files from API documentation for more accurate code suggestions\n- **Tutorial Conversion**: Transform web tutorials into markdown for better context when asking for implementation help\n- **Error Solution Repository**: Build a collection of cleaned Stack Overflow or GitHub issue solutions for common errors\n\n### Augmenting LLM Knowledge\n\n- **Technical Documentation**: Feed your LLM with the latest technical documentation that may not be in its training data\n- **Research Papers**: Convert academic papers and research findings into clean markdown for better AI comprehension\n- **Product Documentation**: Create markdown versions of product documentation for more accurate product-specific assistance\n- **Custom Knowledge Base**: Build specialized knowledge repositories for domain-specific AI applications\n\n### Practical Examples\n\n```python\n# Update your AI IDE with the latest React documentation\npython run.py scrape https://reactjs.org/docs/getting-started.html --save --output react_latest.md\n\n# Create a knowledge base from multiple pages\nfrom app.scraper import scrape_url\nfrom app.converter import html_to_markdown\n\nurls = [\n    \"https://docs.python.org/3/library/asyncio.html\",\n    \"https://docs.python.org/3/library/concurrent.futures.html\"\n]\n\nfor url in urls:\n    result = scrape_url(url)\n    markdown = html_to_markdown(result[\"html\"])\n    filename = f\"python_async_{url.split('/')[-1].replace('.html', '.md')}\"\n    with open(filename, \"w\") as f:\n        f.write(markdown)\n```\n\n## Result Examples\n\nWith these improvements, Web2LLM produces:\n\n- **AI-Optimized Content**: Clean, structured markdown without distracting elements\n- **Token-Efficient Format**: No wasted tokens on JavaScript, CSS, or UI elements\n- **Context Window Maximization**: Only the most informative content is preserved\n- **Semantic Structure**: Properly formatted headings, lists, and code blocks for better AI comprehension\n- **Consistent Formatting**: Predictable structure for reliable AI processing\n\n### Before \u0026 After Example\n\n**Before processing (raw HTML):**\n```html\n\u003chtml\u003e\n\u003chead\u003e\n  \u003ctitle\u003eAPI Documentation\u003c/title\u003e\n  \u003cstyle\u003e/* 250KB of CSS */\u003c/style\u003e\n  \u003cscript\u003e/* 500KB of JavaScript */\u003c/script\u003e\n\u003c/head\u003e\n\u003cbody\u003e\n  \u003cheader\u003e\n    \u003cnav\u003e\u003c!-- Complex navigation menu --\u003e\u003c/nav\u003e\n    \u003cdiv class=\"search\"\u003e\u003c!-- Search form --\u003e\u003c/div\u003e\n  \u003c/header\u003e\n  \u003caside\u003e\u003c!-- Sidebar with links --\u003e\u003c/aside\u003e\n  \u003cmain\u003e\n    \u003ch1\u003eAPI Reference\u003c/h1\u003e\n    \u003cp\u003eThis documentation describes the REST API...\u003c/p\u003e\n    \u003c!-- Actual valuable content --\u003e\n  \u003c/main\u003e\n  \u003cfooter\u003e\u003c!-- Copyright, links, etc. --\u003e\u003c/footer\u003e\n\u003c/body\u003e\n\u003c/html\u003e\n```\n\n**After processing (markdown for LLM consumption):**\n```markdown\n# API Reference\n\nThis documentation describes the REST API...\n\n## Endpoints\n\n### GET /users\n\nReturns a list of users.\n\n**Parameters:**\n- `limit`: Maximum number of results (default: 20)\n- `offset`: Pagination offset (default: 0)\n\n**Response:**\n```json\n{\n  \"users\": [\n    {\n      \"id\": 1,\n      \"name\": \"Example User\"\n    }\n  ],\n  \"total\": 100\n}\n```\n```\n\n## Maintenance and Troubleshooting\n\nIf you encounter problems with certain sites:\n\n1. **Check the HTML structure** of the site to identify particular elements\n2. **Add specific CSS selectors** to the appropriate lists\n3. **Adjust detection thresholds** to be more or less aggressive\n4. **Use the raw HTML saving option** to analyze the original content\n\n## Configuration\n\nSee the `.env.example` file for available configuration options. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyamasammy%2Fweb2llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyamasammy%2Fweb2llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyamasammy%2Fweb2llm/lists"}