https://github.com/sumitm01/integrated-skill-tracker-using-websraping
A skill development tracking application which uses web scraping to collect data from various coding websites and displays stats for each user in one place making it easy to learn coding.
https://github.com/sumitm01/integrated-skill-tracker-using-websraping
celery-redis django-rest-framework html-css-javascript python3 scraping-web selenium-webdriver
Last synced: 2 months ago
JSON representation
A skill development tracking application which uses web scraping to collect data from various coding websites and displays stats for each user in one place making it easy to learn coding.
- Host: GitHub
- URL: https://github.com/sumitm01/integrated-skill-tracker-using-websraping
- Owner: SumitM01
- License: mit
- Created: 2023-06-01T10:07:36.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-11-07T04:07:46.000Z (over 2 years ago)
- Last Synced: 2025-05-15T18:47:31.900Z (about 1 year ago)
- Topics: celery-redis, django-rest-framework, html-css-javascript, python3, scraping-web, selenium-webdriver
- Language: Python
- Homepage:
- Size: 91.4 MB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Integrated Skill Tracker using Web Scraping
This project is an Integrated Skill Tracker web application that scrapes job listing websites to track the skills that are currently in demand in the job market. The application uses Django as the web framework, Selenium for web scraping, and Celery for task scheduling. This application serves as a central hub for various skill practicing websites, including HackerRank, Codechef, etc. Users can access these websites through a single platform, making it easier to track progress and improve skills...
## Prerequisites
To run this app on your local machine, you will need to have the following installed:
- Python 3.6+
- Django 3.2+
- Selenium 3.141+
- Celery 5.0+
## Installation
Clone the repository:
```bash
git clone https://github.com/SumitM01/Integrated-skill-tracker-using-webscraping.git
```
Install the required dependencies using pip
```pip
pip install -r requirements.txt
```
Set up your MongoDB and create a `.env` file with your MongoDB connection string:
```Python
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'database_name',
'USER': 'database_user',
'PASSWORD': 'database_password',
'HOST': 'localhost',
'PORT': '5432',
}
}
```
Run the database migrations
```Python
python manage.py migrate
```
Start the Celery worker
```Python
celery -A Integrated_Coding worker -l info
```
Start the Django development server
```Python
python manage.py runserver
```
Access the web application at `http://localhost:8000/`
## Usage/Examples
- Create an account and add your desired coding platform profiles to track.
- Save the details in the user page and you are done.
- The scraping process is run asynchronously using Celery, so the user can continue using the application while the scraping is ongoing.
- Once the scraping is complete, the user can view the Stats for each platform, Leaderboard position, and ongoing contests on the homepage and contests page.
- We've also implemented practice problems with links to external sites to give an idea of where to start DSA.
## Screenshots
## License
This project is licensed under the MIT License. see the
[MIT](https://choosealicense.com/licenses/mit/) file for details.
## Acknowledgements
This project was built as a part of a university assignment. Special thanks to our professor for giving us the opportunity to work on this project and learn new technologies.