https://github.com/ahmadhuss/scrapy_universal
A universal, plug-and-play Scrapy boilerplate for scalable web scraping projects.
https://github.com/ahmadhuss/scrapy_universal
python scrapy scrapy-crawler scrapy-spider
Last synced: 15 days ago
JSON representation
A universal, plug-and-play Scrapy boilerplate for scalable web scraping projects.
- Host: GitHub
- URL: https://github.com/ahmadhuss/scrapy_universal
- Owner: ahmadhuss
- License: mit
- Created: 2025-04-01T19:50:21.000Z (about 2 months ago)
- Default Branch: master
- Last Pushed: 2025-04-01T20:29:06.000Z (about 2 months ago)
- Last Synced: 2025-05-03T03:45:42.280Z (20 days ago)
- Topics: python, scrapy, scrapy-crawler, scrapy-spider
- Language: Python
- Homepage:
- Size: 3.91 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Scraping
This repository contains various Python scripts to check web scraping techniques for certain sites.
**How we can use on our local machine?**
Make sure your OS has already installed `Python 3.x.x` , that comes with `pip`, `pip` is a package manager for Python and is included by default with the Python installer. It helps to install and uninstall Python packages (such as Django!).
**Setting up a virtual environment:**
After cloning this repo, enter the following command on your terminal:
python -m venv venv
It will create the virtual environment for the following project, where we will install all our project dependencies. There is a `venv` directory automatically created in your project that Git will not track.
After that we have to activate the virtual environment and install the project dependencies there.
**For Windows:**
Enter the following command at the root of your cloned repo.
venv\Scripts\activate
**For Unix:**
Enter the following command at the root of your cloned repo.
. venv\bin\activate
**Install dependencies:**
pip install -r requirements.txt
### License
MIT