https://github.com/illimag/webspider0

https://github.com/illimag/webspider0

Last synced: 20 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/illimag/webspider0
• Owner: Illimag
• Created: 2022-05-23T16:44:35.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2022-05-23T17:15:36.000Z (over 2 years ago)
• Last Synced: 2024-05-02T04:49:51.006Z (6 months ago)
• Language: Python
• Size: 4.17 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Web Spider

To run this spider, use Ubuntu command line, install following with either env or install individually, run webspider behind VPN. Run test first to see if installed correctly, then run webspider.

# run test

    python3 test.py

# Run Webspider

    python3 webspider.py

# install

Ubuntu 20.02

Python 3

Pip

VirtualEnv

## Install Pip3

    sudo apt-get install python3-pip

## Install VirtualEnv

    python3 -m pip install --user virtualenv

## Install ensurepip

    sudo apt-get install python3-venv

## Create Virtual Env

    python3 -m venv env

## Activate virtual Env

    source env/bin/activate

## Install bs4

    python3 -m pip install bs4

## Install requests

    python3 -m pip install requests

## deactivate Virtual Env

    env/bin/deactivate

## Freeze and create requirements

    pip freeze > requirements.txt