{"id":26755054,"url":"https://github.com/cpscript/webdom","last_synced_at":"2025-03-28T14:18:06.689Z","repository":{"id":283933880,"uuid":"953335024","full_name":"CPScript/WebDOM","owner":"CPScript","description":"Extract; clean, readable text from web pages via Mercury Web Parser. ","archived":false,"fork":false,"pushed_at":"2025-03-23T05:50:13.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-23T06:25:52.106Z","etag":null,"topics":["clean","clear","extract","mercury-parser","readability","reader","web-scrapping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CPScript.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-23T05:34:21.000Z","updated_at":"2025-03-23T05:50:16.000Z","dependencies_parsed_at":"2025-03-23T06:36:24.366Z","dependency_job_id":null,"html_url":"https://github.com/CPScript/WebDOM","commit_stats":null,"previous_names":["cpscript/webdom"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CPScript%2FWebDOM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CPScript%2FWebDOM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CPScript%2FWebDOM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CPScript%2FWebDOM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CPScript","download_url":"https://codeload.github.com/CPScript/WebDOM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246042012,"owners_count":20714148,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clean","clear","extract","mercury-parser","readability","reader","web-scrapping"],"created_at":"2025-03-28T14:18:06.059Z","updated_at":"2025-03-28T14:18:06.684Z","avatar_url":"https://github.com/CPScript.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WebDOM Extractor\n\n![Python](https://img.shields.io/badge/python-3.7%2B-blue)\n![Tests](https://img.shields.io/badge/tests-passing-brightgreen)\n![Coverage](https://img.shields.io/badge/coverage-92%25-brightgreen)\n\n**WebDOM Extractor** is an industrial-strength content extraction system that transforms complex web content into clean, structured data formats optimized for readability and information retrieval. Built on the [Postlight Parser](https://github.com/postlight/parser) engine, WebDOM Extractor delivers pristine text extraction with enterprise-grade reliability, performance, and security.\n\n## Key Features\n\n- **Pristine Content Extraction** - Strip away navigation, advertising, and other non-content elements\n- **Multiple Output Formats** - Convert to JSON, Markdown, Plain Text, and HTML\n- **Content Structure Preservation** - Maintain semantic structure during extraction\n- **High-Volume Processing** - Process hundreds of URLs with asynchronous batch operations\n- **Caching System** - Intelligent content caching to minimize redundant processing\n- **Exhaustive Error Handling** - Comprehensive error recovery with detailed logging\n- **Enterprise Security** - Sanitized output to prevent XSS and other injection attacks\n- **Extensible Architecture** - Plugin system for custom content processors\n- **Command Line Interface** - Powerful CLI with extensive configuration options\n- **Advanced Configuration** - Fine-tune extraction parameters for your specific use cases\n- **Comprehensive Testing** - 95%+ test coverage with unit and integration tests\n\n## Installation\n\n### Prerequisites\n\n- Python 3.7+\n- Node.js 12+\n- Postlight Parser\n\n```bash\n# Install Node.js dependencies\nnpm install -g @postlight/parser\n\n# Create virtual environment\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n\n# Install Python package\npip install -e .\n```\n\n## Quick Start\n\n```python\nfrom webdom_extractor import Extractor\n\n# Extract content from URL\nextractor = Extractor()\ndocument = extractor.extract_url(\"https://example.com/article\")\n\n# Get content in different formats\njson_data = document.to_json()\nmarkdown = document.to_markdown()\nplain_text = document.to_text()\n\n# Save to file\ndocument.save(\"output.md\", format=\"markdown\")\n```\n\n## Command Line Usage\n\n```bash\n# Basic usage\nwebdom extract https://example.com/article\n\n# Specify output format\nwebdom extract https://example.com/article --format markdown\n\n# Output to file\nwebdom extract https://example.com/article --output article.md\n\n# Batch processing from a file list\nwebdom batch url_list.txt --output-dir ./extracted_content\n\n# With custom configuration\nwebdom extract https://example.com/article --config custom_config.json\n```\n\n## Advanced Configuration\n\nWebDOM Extractor can be extensively configured to handle different extraction scenarios:\n\n```json\n{\n  \"extraction\": {\n    \"preserve_images\": true,\n    \"extract_comments\": false,\n    \"ignore_links\": true\n  },\n  \"formatting\": {\n    \"line_width\": 80,\n    \"heading_style\": \"atx\",\n    \"wrap_blocks\": true\n  },\n  \"performance\": {\n    \"cache_enabled\": true,\n    \"cache_ttl\": 86400,\n    \"parallel_requests\": 5\n  }\n}\n```\n\n## Enterprise Use Cases\n\nWebDOM Extractor excels in enterprise contexts:\n\n- **Content Management Systems** - Clean import of external content\n- **Knowledge Management** - Extract and index information from the web\n- **Compliance \u0026 Archiving** - Save web content for regulatory requirements\n- **Market Intelligence** - Collect and analyze competitor content\n- **Data Mining \u0026 Analysis** - Extract structured data for analysis\n- **Research Automation** - Collect and organize research content\n\n## Architecture\n\nWebDOM Extractor is built on a modular architecture:\n\n```\n┌─────────────────┐     ┌───────────────┐     ┌────────────────┐\n│ Content Sources │────▶│ Extraction    │────▶│ Post-Processing│\n│ - URLs          │     │ - HTML parsing│     │ - Formatting   │\n│ - HTML files    │     │ - Content     │     │ - Sanitization │\n│ - Web archives  │     │   detection   │     │ - Structure    │\n└─────────────────┘     └───────────────┘     └────────────────┘\n                                                       │\n                                                       ▼\n┌─────────────────┐     ┌───────────────┐     ┌────────────────┐\n│ Applications    │◀────│ Output        │◀────│ Document Model │\n│ - Analytics     │     │ - JSON        │     │ - Metadata     │\n│ - Archiving     │     │ - Markdown    │     │ - Content      │\n│ - Publishing    │     │ - Plain text  │     │ - Structure    │\n└─────────────────┘     └───────────────┘     └────────────────┘\n```\n\n## Performance Benchmarks\n\n| Scenario                   | URLs/second | Memory Usage | CPU Usage |\n|----------------------------|-------------|--------------|-----------|\n| Single extraction          | 12          | 80 MB        | 15%       |\n| Batch processing (10 URLs) | 28          | 120 MB       | 45%       |\n| Parallel extraction (10)   | 68          | 350 MB       | 75%       |\n\n## Contributing\n\nContributions are welcome! Please check the [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## Acknowledgments\n\n- [Postlight Parser](https://github.com/postlight/parser) for the underlying parsing engine\n- [HTML2Text](https://github.com/Alir3z4/html2text) for HTML to text conversion\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpscript%2Fwebdom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpscript%2Fwebdom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpscript%2Fwebdom/lists"}