https://github.com/siddharth-nandagopal/web-scraper

a simple web scraper in Python
https://github.com/siddharth-nandagopal/web-scraper

asdf beautifulsoup bs4 poetry python ray requests webscraper

Last synced: 7 days ago
JSON representation

a simple web scraper in Python

• Host: GitHub
• URL: https://github.com/siddharth-nandagopal/web-scraper
• Owner: siddharth-nandagopal
• Created: 2024-10-06T20:53:09.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2024-10-06T21:41:55.000Z (4 months ago)
• Last Synced: 2024-11-14T01:23:32.711Z (2 months ago)
• Topics: asdf, beautifulsoup, bs4, poetry, python, ray, requests, webscraper
• Language: Python
• Homepage:
• Size: 33.2 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# web-scraper

a simple web scraper in Python

## steps to run

1. install dependencies

    ```

    asdf install

    poetry install

    ```

2. execute web-scraper

    ```

    python3 main.py

    ```

    sample output:

    ```

    begin web-scraper...

    number of links scraped from base https://docs.ray.io/en/latest/index.html till depth 2: 288

    ```

    (OR)

    ```

    time python3 main.py

    ```

    sample output:

    ```

    begin web-scraper...

    number of links scraped from base https://docs.ray.io/en/latest/index.html till depth 2: 288

    python3 main.py  11.48s user 0.24s system 25% cpu 46.391 total

    ```