Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pozharliev/adatapro-internship-2021
Site, which shows information about the current scraping status. Built with Django and Scrapy.
https://github.com/pozharliev/adatapro-internship-2021
bulma django scrapy
Last synced: 18 days ago
JSON representation
Site, which shows information about the current scraping status. Built with Django and Scrapy.
- Host: GitHub
- URL: https://github.com/pozharliev/adatapro-internship-2021
- Owner: pozharliev
- Created: 2021-07-16T15:03:00.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2021-07-25T20:59:46.000Z (over 3 years ago)
- Last Synced: 2024-07-02T19:41:55.059Z (4 months ago)
- Topics: bulma, django, scrapy
- Language: HTML
- Homepage:
- Size: 9.31 MB
- Stars: 3
- Watchers: 6
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Team Red
Site, which shows information about the current scraping status.
💻 About
Our project consists of a site that aims to help its users by providing them with up-to-date information on the availability of the PlayStation 5. The information is retrieved automatically through crawling. Once a day, registered users are sent an email with information about an available PS5, and users can set up the system to receive information only from the places they are interested in.
🎥 Interface
Our Documentation
⌨️ How it works
Our work can be split into 4 different parts:
- The Site (Our Front-end)
- Home - gives basic information about the site.
- Information - gives more information about the site and how to use the functions of the site.
- Registration and login - to create an account and log in.
- Account - to control user settings and functions.
- The Django Application (Our Back-end)
- Our Back-end is entirely written on
Django.
- It is centered around views, which are in different 📁 folders.
- It can be started by the 📁
manage.py script.
- You can find in the in 📁 API/ folder.
- It is hosted on
localhost on port 8000, although you can change it.
- 📊 The Database
- We are using
MariaDB because of its worldwide popularity and its simplicity.
- We have a few of tables, but the main ones are the auth_user, accountview_profilepreferences and historyview_historyscraping.
- The auth_user stores all ®️ registered accounts.
- The accountview_profilepreferences stores all the profile preferences related to which sites you want to see.
- The historyview_historyscraping stores all the information from the scrapers.
- All of the tables are generated from Django models.
- The Scraper
⛏️ Installation
Step 1: Clone the repository
git clone https://github.com/ABPozharliev19/adatapro-internship-2021.git
cd adatapro-internship-2021/
Step 2: Install dependencies
You can do this by either using pipenv:
pip3 install pipenv # Install pipenv with pip
pipenv shell # Activate the environment
pipenv install # Install all dependencies in the Pipfile.lock file
Or by using pip:
pip3 install -r requirements.txt # Install all dependencies in the requirements.
Step 3: Rename env.example to .env
Step 4: Change the information in the .env file
SECRET_KEY = 'Your django secret key'
DB_NAME = 'Your database name'
DB_USER = 'Your database user'
DB_PASSWORD = 'Your database password'
DB_HOST = 'Your host'
DB_PORT = 3306 # The port you want to use
EMAIL_HOST_USER = 'Your email username' # Should be gmail
EMAIL_HOST_PASSWORD = 'Your email password'
Step 5: Make the migrations and migrate the models
python manage.py makemigrations
python manage.py migrate
Step 6: Run the server
python manage.py runserver
Or specify the port for the server
python manage.py runserver PORT
Step 7: Go to localhost:PORT and use the site!
🧒 Our Team
Atanas Pozharliev - ABPozharliev19 - Back-End Developer
Kostadin Taligadzhiev - KNTaligadzhiev19 - Front-End Developer
Igor Simeonov - GrandMaster-42 - Scraping
Velin Kurtev - Velin1234 - Scraping
Diyan Nedelchev - dido322 - Scraping
Kaloyan Totev - KaloyanTotev339 - Grafana & Docker Setup