https://github.com/jonathanalvares9009/web-crawler

https://github.com/jonathanalvares9009/web-crawler

Last synced: 26 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/jonathanalvares9009/web-crawler
• Owner: jonathanalvares9009
• Created: 2023-08-23T10:05:37.000Z (about 1 year ago)
• Default Branch: master
• Last Pushed: 2023-10-22T04:35:31.000Z (about 1 year ago)
• Last Synced: 2024-10-04T13:19:10.380Z (about 1 month ago)
• Language: JavaScript
• Size: 78.1 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Web Crawler

## Description

This is a web crawler that crawls the website and displays it.

## Installation for web-crawler

1. Clone the repository

2. Install the requirements (`npm i`)

3. Run `npm start ` to crawl through a URL

4. Run `npm start --yc-companies` to crawl through YC companies

## Installation for web-scrapper

1. In a new terminal cd into the folder

2. Run `virtualenv venv` to create a virtual env

3. Run `source venv/bin/activate` to activate the virtual env

4. Run `pip install -r requirements` to install the requirements

Now to scrape a website, run `python main.py` and follow the instructions provided in the terminal.

Now to find YC companies run `yc_scrapper.py` and this will create a file called yc_companies.txt in the output folder which is a dependency for the `--yc-companies` parameter in the web-crawler which recursively crawls through the YC websites.