{"id":18565002,"url":"https://github.com/sumitm01/integrated-skill-tracker-using-websraping","last_synced_at":"2026-05-02T02:34:35.836Z","repository":{"id":183703262,"uuid":"648147483","full_name":"SumitM01/Integrated-skill-tracker-using-websraping","owner":"SumitM01","description":"A skill development tracking application which uses web scraping to collect data from various coding websites and displays stats for each user in one place making it easy to learn coding. ","archived":false,"fork":false,"pushed_at":"2023-11-07T04:07:46.000Z","size":95879,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-15T18:47:31.900Z","etag":null,"topics":["celery-redis","django-rest-framework","html-css-javascript","python3","scraping-web","selenium-webdriver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SumitM01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-01T10:07:36.000Z","updated_at":"2024-06-28T08:17:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"c35d711e-62c1-4ff5-9486-be733488f2fc","html_url":"https://github.com/SumitM01/Integrated-skill-tracker-using-websraping","commit_stats":null,"previous_names":["sumitm01/integrated-skill-tracker-using-websraping"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SumitM01/Integrated-skill-tracker-using-websraping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumitM01%2FIntegrated-skill-tracker-using-websraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumitM01%2FIntegrated-skill-tracker-using-websraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumitM01%2FIntegrated-skill-tracker-using-websraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumitM01%2FIntegrated-skill-tracker-using-websraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SumitM01","download_url":"https://codeload.github.com/SumitM01/Integrated-skill-tracker-using-websraping/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SumitM01%2FIntegrated-skill-tracker-using-websraping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32521092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-02T01:12:54.858Z","status":"online","status_checked_at":"2026-05-02T02:00:05.923Z","response_time":132,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery-redis","django-rest-framework","html-css-javascript","python3","scraping-web","selenium-webdriver"],"created_at":"2024-11-06T22:17:12.616Z","updated_at":"2026-05-02T02:34:35.821Z","avatar_url":"https://github.com/SumitM01.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Integrated Skill Tracker using Web Scraping\nThis project is an Integrated Skill Tracker web application that scrapes job listing websites to track the skills that are currently in demand in the job market. The application uses Django as the web framework, Selenium for web scraping, and Celery for task scheduling. This application serves as a central hub for various skill practicing websites, including HackerRank, Codechef, etc. Users can access these websites through a single platform, making it easier to track progress and improve skills...\n\n## Prerequisites\n\nTo run this app on your local machine, you will need to have the following installed:\n\n- Python 3.6+\n- Django 3.2+\n- Selenium 3.141+\n- Celery 5.0+\n\n## Installation\n\nClone the repository:\n\n```bash\n  git clone https://github.com/SumitM01/Integrated-skill-tracker-using-webscraping.git\n```\n\nInstall the required dependencies using pip\n\n```pip\n  pip install -r requirements.txt\n```\nSet up your MongoDB and create a `.env` file with your MongoDB connection string:\n\n```Python\nDATABASES = {\n    'default': {\n        'ENGINE': 'django.db.backends.postgresql',\n        'NAME': 'database_name',\n        'USER': 'database_user',\n        'PASSWORD': 'database_password',\n        'HOST': 'localhost',\n        'PORT': '5432',\n    }\n}\n```\nRun the database migrations\n```Python\npython manage.py migrate\n\n```\nStart the Celery worker\n```Python\ncelery -A Integrated_Coding worker -l info\n\n```\nStart the Django development server\n```Python\npython manage.py runserver\n\n```\nAccess the web application at `http://localhost:8000/`\n\n## Usage/Examples\n\n- Create an account and add your desired coding platform profiles to track.\n- Save the details in the user page and you are done.\n- The scraping process is run asynchronously using Celery, so the user can continue using the application while the scraping is ongoing.\n- Once the scraping is complete, the user can view the Stats for each platform, Leaderboard position, and ongoing contests on the homepage and contests page.\n- We've also implemented practice problems with links to external sites to give an idea of where to start DSA.\n\n## Screenshots\n\n![profile image](https://github.com/SumitM01/Integrated-skill-tracker-using-websraping/assets/65524561/3367f29c-a5c6-4563-a113-3877473a7a4b)\n![dashborad image](https://github.com/SumitM01/Integrated-skill-tracker-using-websraping/assets/65524561/4fbea5d2-e55b-474d-9123-353c93cd927f)\n![contest image](https://github.com/SumitM01/Integrated-skill-tracker-using-websraping/assets/65524561/69fafb3f-b14b-4f9b-9175-b2044aaad019)\n![leaderboard image](https://github.com/SumitM01/Integrated-skill-tracker-using-websraping/assets/65524561/260c811b-a592-4ee1-8da8-bda4c07b9eee)\n![report Image](https://github.com/SumitM01/Integrated-skill-tracker-using-websraping/assets/65524561/abb6c14f-a7a6-433a-9f7f-97018229b152)\n![problem image](https://github.com/SumitM01/Integrated-skill-tracker-using-websraping/assets/65524561/d86115ef-eee7-4d4f-aa6d-cdbd029f64a8)\n\n## License\n\nThis project is licensed under the MIT License. see the\n[MIT](https://choosealicense.com/licenses/mit/) file for details.\n\n## Acknowledgements\n\nThis project was built as a part of a university assignment. Special thanks to our professor for giving us the opportunity to work on this project and learn new technologies.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumitm01%2Fintegrated-skill-tracker-using-websraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsumitm01%2Fintegrated-skill-tracker-using-websraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumitm01%2Fintegrated-skill-tracker-using-websraping/lists"}