https://github.com/ondiekelijah/simple-web-crawler

A Simple Web scraper. Using a URL and an HTML tag provided by a user scraps the page and returns the total number of elements fetched and then display the results from the scrap.
https://github.com/ondiekelijah/simple-web-crawler

beautifulsoup4 flask urllib3

Last synced: 3 months ago
JSON representation

A Simple Web scraper. Using a URL and an HTML tag provided by a user scraps the page and returns the total number of elements fetched and then display the results from the scrap.

• Host: GitHub
• URL: https://github.com/ondiekelijah/simple-web-crawler
• Owner: ondiekelijah
• License: mit
• Created: 2021-04-13T13:07:03.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2022-02-12T14:37:08.000Z (almost 3 years ago)
• Last Synced: 2023-03-10T18:33:09.834Z (almost 2 years ago)
• Topics: beautifulsoup4, flask, urllib3
• Language: CSS
• Homepage: https://scrap-the-web.herokuapp.com/
• Size: 1.98 MB
• Stars: 7
• Watchers: 1
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
[![CI to Docker Hub](https://github.com/Dev-Elie/Simple-Web-Crawler/actions/workflows/github_registry.yml/badge.svg)](https://github.com/Dev-Elie/Simple-Web-Crawler/actions/workflows/github_registry.yml)

## Requirements ,Packages used and Installation

Download and install Python. Make sure you install Python v3.6.+

 

## Installation

          

Navigate to your directory of choice the proceed as follows;


          

### 1 .Clone the git repo and create a virtual environment 

          

Depending on your operating system,make a virtual environment to avoid messing with your machine's primary dependencies

          

> **Windows**

          

```

git clone https://github.com/Dev-Elie/Simple-Web-Crawler.git

cd Simple-Web-Crawler

py -3 -m venv venv

```

          

> **macOS/Linux**

          

```

git clone https://github.com/Dev-Elie/Simple-Web-Crawler.git

cd Simple-Web-Crawler

python3 -m venv venv

```

### 2 .Activate the virtual environment (venv)

          

> **Windows** 

```venv\Scripts\activate```

          

> **macOS/Linux**

```. venv/bin/activate```

or

```source venv/bin/activate```

### 3 .Install the requirements

Applies for windows/macOS/Linux

```pip install -r requirements.txt```

### 4. Run the application 

> **For linux and macOS**

Make the run file executable by running the code

```chmod 777 run```

Then start the application by executing the run file

```./run```

> **On windows**

```

set FLASK_APP=main

flask run

```

Then on your browser open `localhost:5000` or `http://127.0.0.1:5000/` 



You should receive a response — a page similar to the one seen in the screenshot.

![scrappy home](https://github.com/Dev-Elie/Simple-Web-Crawler/blob/main/static/images/scrappy-home.png)

Let's connect on Twitter