{"id":25300113,"url":"https://github.com/deftio/simple-py-crawlbot","last_synced_at":"2025-04-06T23:40:35.500Z","repository":{"id":234347093,"uuid":"788713762","full_name":"deftio/simple-py-crawlbot","owner":"deftio","description":"Simple python cli website crawler with output summary","archived":false,"fork":false,"pushed_at":"2025-03-25T20:20:54.000Z","size":1439,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T21:27:24.030Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deftio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-18T23:59:12.000Z","updated_at":"2025-03-25T20:20:58.000Z","dependencies_parsed_at":"2024-04-19T00:38:21.001Z","dependency_job_id":"f11579d4-7af8-4940-8d38-69c6d85e8525","html_url":"https://github.com/deftio/simple-py-crawlbot","commit_stats":null,"previous_names":["deftio/simple-py-crawlbot"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deftio%2Fsimple-py-crawlbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deftio%2Fsimple-py-crawlbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deftio%2Fsimple-py-crawlbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deftio%2Fsimple-py-crawlbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deftio","download_url":"https://codeload.github.com/deftio/simple-py-crawlbot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247569130,"owners_count":20959758,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-13T05:38:24.561Z","updated_at":"2025-04-06T23:40:35.495Z","avatar_url":"https://github.com/deftio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple Python Website Crawler\n\n## Overview\n\nThis is a collection of Python tools for web crawling, content extraction, and document processing. The main tools include:\n\nGUI/API version\n- **SPyCrawl**: A web-based interface for crawling websites and processing documents and api\n\nSeparate Command Line Tools\n- **crawler.py**: A command-line web crawler using BeautifulSoup and Selenium\n- **clean_and_strip.py**: Tools for cleaning HTML and converting between formats\n- **merge_docs_into_pdf.py**: Tools for merging multiple document types into PDFs (or separately)\n\nEach tool captures HTML content, saves it locally, and provides various output formats. These tools are intended for testing and validating site content and structure.\n\n📚 **[Comprehensive documentation available in the docs folder](/docs/user-guide.md)**\n\n### Note:\nThese tools are meant to be simple - if the site you wish to crawl has complex javascript it may not work. Also please be responsible in crawling sites as it can take much site bandwidth and respect appropriate copyrights and other information.  \n\nThe crawler will not attempt to handle logins or firewalls.\n\n## Features\n\n- **Headless Browser Crawling**: Utilizes a headless Chrome browser to navigate and download web pages.\n- **Duplicate Handling**: Normalizes URLs to avoid processing and storing duplicate content.\n- **Local Storage**: Saves each page as a static HTML file.\n- **JSON Summary**: Generates a summary of all processed pages, including titles, file paths, and URLs.\n- **Web Interface**: Browser-based GUI for the crawler (spycrawl.py).\n- **Content Extraction**: Clean and extract structured text from HTML files (clean_and_strip.py).\n- **Format Conversion**: Convert between YAML and JSON formats (yaml_to_json.py).\n- **PDF Generation**: Merge multiple document types into a single PDF (merge_docs_into_pdf.py).\n\nFor detailed documentation on all features, see the [documentation folder](/docs/).\n\n## Prerequisites \u0026 Installation\n\nBefore you run the web crawler, you must install the following:\n\n- Python 3.9 or newer (technically 3.6 should work but some of the deps may soon no longer be supported)\n- Selenium\n- BeautifulSoup4\n- ChromeDriver (make sure it matches your Chrome version and is placed in your PATH)\n\nA separate script : clean_and_strip.py can be used to extract text while preserving the document hierarchy.\n\nYou can install the required Python packages using pip:\n\n```bash\npip install -r requirements.txt\n```\n\nIf you have issues or conflicts with packages try creating a virtual environment with conda or simlar.  Virtual environments isolate the packages for this project from your current python packages and set up.\n\n[How to install Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html#regular-installation)\n\nThen at the command line run this command such as :\n```bash\nconda create -n simple-py-crawlbot python=3.10\nconda activate simple-py-crawlbot\n```\nuse this to deactivate the virtual env:\n\n```bash\nconda deactivate\n```\n\n## Usage\n\n### SPyCrawl Web Interface\nThe easiest way to use the crawler is through the SPyCrawl web interface:\n\n```bash\npython spycrawl.py\n```\n\nThen open your browser to http://127.0.0.1:8803\n\nThis interface provides a user-friendly way to:\n- Crawl websites\n- Clean HTML content\n- Convert between YAML and JSON formats\n- Generate PDFs from multiple document types\n\n### Simple Crawler CLI (stand alone command line crawler)\nThe crawler takes several cli (command line interface) arguments:\n\n--url: Required. Specifies the starting URL for the crawler.\nExample: --url \"https://my-website-to-crawl/\"\n\n--output-dir: Specifies the directory where the HTML files will be stored. Default is \"output\".\nExample: --output_dir custom_directory\n\n--summary: Specifies the filename for the JSON summary. Default is \"summary.json\".\nExample: --summary custom_summary.json\n\n--progress: Enables progress display during the crawling process. This is a flag; include it to activate.\nExample: --progress\n\n--clean: Removes non-informational content (such as scripts and styles) from the HTML files. This is a flag; include it to activate.\nExample: --clean\n\n--max_links: Limits the number of links to crawl. If omitted, the crawler processes the entire site.\nExample: --max_links 100\n\nEach CLI argument can be used in combination to fine-tune the behavior of the crawler based on the needs of the user. You can customize the input parameters to control various aspects like the extent of crawling, output customization, and content processing.\n\n### Example \nHere is an example using the cli arguments\n```bash\npython crawler.py --url https://my-website-to-crawl.com --output output_pages --summary_file detailed_summary.json --progress --clean --max_links 50\n```\n\n--url https://my-website-to-crawl.com: This sets the starting URL for the crawler to the specified website.\n\n--output_dir output_pages: This directs the script to save all crawled HTML files in a directory named output_pages.\n\n--summary detailed_summary.json: Specifies that the JSON summary of the crawl should be saved with the filename detailed_summary.json.\n\n--progress: Includes a progress flag that enables real-time output showing the progress of the crawl, indicating how many links have been saved and what the current URL being processed is.\n\n--clean: Activates the cleaning function, which will strip non-essential content such as scripts and CSS from the HTML files to focus only on the informational content.\n\n--max_links 50: Limits the crawler to processing a maximum of 50 links from the website, which is useful for keeping the scope of the crawl manageable or for testing purposes.  (if left out it will attempt to crawl the whole site)\n \n## Clean and Strip (clean_and_strip.py cli utility)\nThe script clean_and_script.py is a cli program which takes a directory of html files (such as output from the crawler.py script) and outputs a yaml file for each intput html file which has just the text (no css or styles or html attributes) from the source directory.  Files are stored in yaml format but are human readable.\n\nUsage is as follows:\n```bash\nclean_and_strip -input_dir input_directory_of_crawled_files  -output_dir output_directory_of_extracted_text\n```\n\n## yaml_to_json.py\nAlso included is yaml_to_json.py which can take a directory of yaml files and convert to json.  Can be used with clean_and_strip.py above\n\n```bash \nyaml_to_json.py -input_dir input_directory_of_crawled_files  -output_dir output_directory_of_extracted_text\n```\n\n## Document Merging Tool (Create PDFs)\n\nThe `merge_docs_into_pdf.py` script allows you to combine multiple documents of different formats into a single PDF file. This tool supports the following file formats:\n- HTML (.html)\n- Markdown (.md) \n- Text (.txt)\n- YAML (.yaml)\n- PDF (.pdf)\n\n### Prerequisites\n\nSkip the package installation if you have already used pip to install requirments for the crawler.py\n\nInstall the required Python packages:\n\n```bash\npip install weasyprint PyPDF2 markdown2 reportlab pyyaml\n```\nor \n```bash\npip install -r requirments.txt\n```\n\n### Usage\n\nRun the script using the following command:\n\n```bash\npython merge_docs_into_pdf.py -d \u003cdirectory\u003e -o \u003coutput.pdf\u003e [--no-merge]\n```\n\nArguments:\n- `-d, --directory`: Directory containing the files to merge\n- `-o, --output`: Name of the output PDF file or directory (when using --no-merge)\n- `--no-merge`: Optional flag to create separate PDFs instead of merging into one\n\n### Examples\n\nMerge all files into a single PDF:\n```bash\npython merge_docs_into_pdf.py -d docs/ -o combined_documentation.pdf\n```\n\nCreate separate PDFs for each file:\n```bash\npython merge_docs_into_pdf.py -d docs/ -o separate_pdfs/ --no-merge\n```\n\nThis will:\n1. Scan the specified directory for supported files\n2. Convert each file to PDF format\n3. Either:\n   - Merge all PDFs into a single output file (default behavior)\n   - Create separate PDFs in the output directory (when using --no-merge)\n4. Clean up temporary files automatically\n\n### Notes\n- Files are processed in alphabetical order\n- Existing PDF files in the directory will be included\n- Unsupported file types are automatically skipped\n- The script preserves the formatting of the original documents\n- When using --no-merge, the output filename becomes the directory name where individual PDFs are saved\n\n## Testing SPyCrawl API\n\nThe repository includes a comprehensive test suite specifically for the `spycrawl.py` API endpoints. These tests are separate from other CLI tools in the repository.\n\n### Test Files\n\n- `test_spycrawl_api.py` - Contains all the test cases for SPyCrawl API endpoints\n- `run_spycrawl_tests.py` - Runner script with command-line options\n- `test_spycrawl_README.md` - Detailed documentation for the test suite\n\n### Setup and Running Tests\n\n1. Ensure all dependencies are installed:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n2. Run the test suite:\n   ```bash\n   python run_spycrawl_tests.py\n   ```\n\n3. Control verbosity level (optional):\n   ```bash\n   # Quiet mode (only show errors)\n   python run_spycrawl_tests.py -v 0\n\n   # Normal mode (show test names and status)\n   python run_spycrawl_tests.py -v 1\n\n   # Verbose mode (show detailed output)\n   python run_spycrawl_tests.py -v 2\n   ```\n\n### What's Tested\n\nThe test suite verifies all SPyCrawl API endpoints, including:\n- Basic routing (/ and /pages)\n- Web crawling functionality (/api/crawl)\n- Crawl session management (list, get, stop, clear)\n- HTML cleaning (/api/clean) \n- YAML to JSON conversion (/api/convert)\n- PDF generation (/api/pdf)\n- File operations (list files, download file)\n\nThe tests will automatically start and stop the SPyCrawl server in a separate process and create temporary test directories to avoid affecting your existing data.\n\nFor more details about the tests, see `test_spycrawl_README.md`.\n\n## Docker Support\n\nSPyCrawl can be run in a Docker container, which eliminates the need to install dependencies locally and ensures consistent behavior across different environments.\n\n### Prerequisites\n\n- Docker and Docker Compose installed on your system\n  - [Install Docker](https://docs.docker.com/get-docker/)\n  - [Install Docker Compose](https://docs.docker.com/compose/install/)\n\n### Running SPyCrawl with Docker\n\nThere are two ways to run SPyCrawl with Docker:\n\n#### Option 1: Using Docker Compose (Recommended)\n\n1. Build and start the container:\n   ```bash\n   docker-compose up\n   ```\n\n2. Access the web interface at http://localhost:8803\n\n3. To run in the background:\n   ```bash\n   docker-compose up -d\n   ```\n\n4. To stop the container:\n   ```bash\n   docker-compose down\n   ```\n\n#### Option 2: Using Docker directly\n\n1. Build the Docker image:\n   ```bash\n   docker build -t spycrawl .\n   ```\n\n2. Run the container:\n   ```bash\n   docker run -p 8803:8803 -v $(pwd)/output:/app/output -v $(pwd)/logs:/app/logs spycrawl\n   ```\n\n3. Access the web interface at http://localhost:8803\n\n### Persistent Data\n\nThe Docker configuration mounts two volume directories:\n- `./output`: Where crawled websites and generated files are stored\n- `./logs`: Where log files are stored\n\nThis ensures that your data persists even when containers are stopped or removed.\n\n### Customizing Docker Settings\n\nYou can modify the `docker-compose.yml` file to change:\n- Port mappings\n- Volume mounts\n- Environment variables\n\nFor example, to change the port:\n```yaml\nports:\n  - \"8080:8803\"  # Maps host port 8080 to container port 8803\n```\n\n## LICENSE\nBSD-2\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeftio%2Fsimple-py-crawlbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeftio%2Fsimple-py-crawlbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeftio%2Fsimple-py-crawlbot/lists"}