{"id":23257018,"url":"https://github.com/solvro/script-promochator-usos-scraper","last_synced_at":"2025-08-20T14:33:19.040Z","repository":{"id":254627000,"uuid":"847067411","full_name":"Solvro/script-promochator-usos-scraper","owner":"Solvro","description":"Some data scrapers from USOS via the official API and other methods made in Python for the promoCHATor project in the ML section.","archived":false,"fork":false,"pushed_at":"2024-10-06T21:40:53.000Z","size":37,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T17:11:10.515Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Solvro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-24T18:48:56.000Z","updated_at":"2024-11-21T08:48:06.000Z","dependencies_parsed_at":"2024-08-26T23:51:29.913Z","dependency_job_id":"c1839f58-e5f2-451e-927a-6670f41246c4","html_url":"https://github.com/Solvro/script-promochator-usos-scraper","commit_stats":null,"previous_names":["solvro/script-promochator-usos-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Solvro/script-promochator-usos-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Solvro%2Fscript-promochator-usos-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Solvro%2Fscript-promochator-usos-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Solvro%2Fscript-promochator-usos-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Solvro%2Fscript-promochator-usos-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Solvro","download_url":"https://codeload.github.com/Solvro/script-promochator-usos-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Solvro%2Fscript-promochator-usos-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271332592,"owners_count":24741287,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-19T12:26:39.290Z","updated_at":"2025-08-20T14:33:18.695Z","avatar_url":"https://github.com/Solvro.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./assets/solvro.png\"\u003e\n\u003c/p\u003e\n\n# Solvro ML - USOS scraper for PromoCHATor project\n\u003cp align=\"justify\"\u003e\nThis repository contains scripts scraping data from USOS for PromoCHATor project in Solvro ML section. These scripts allow to collect data about university teachers, their scientific achievements, abstracts of students' scientific papers. The project will be developed as necessary.\n\u003c/p\u003e\n\n## Table of contents\n\n1. **[Description](#description)**\n2. **[Technologies](#technologies)**\n3. **[Development](#development)**\n   1. [Quick start](#quick-start)\n      - [Setup project locally](#setup-project-locally)\n      - [Run usos-teachers-scraper](#run-usos-teachers-scraper)\n      - [Run usos-abstracts-scraper](#run-usos-abstracts-scraper)\n   2. [Github workflow](#github-workflow)\n4. **[Current team](#current-team)**\n\n## Description\n\u003cp align=\"justify\"\u003e \nThis repository contains scripts scraping data from USOS. These scripts allow to collect data about university teachers, their scientific achievements, abstracts of students' scientific papers.\n\u003c/p\u003e\n\n## Technologies\nProject uses following languages and technologies\n* Python 3.9.13\n\n## Development\n### Quick start\n#### Setup project locally\n\n1. Clone the repository:\n\n   ```\n   git clone https://github.com/Solvro/script-promochator-usos-scraper.git\n   ```\n\n2. Change directory:\n\n   ```\n   cd script-promochator-usos-scraper\n   ```\n\n3. Create new virtual environment:\n\n   ```\n   python -m venv \u003cyour_env_name\u003e\n   ```\n\n4. Activate environment:\n\n   ```\n   ./venv/Scripts/activate\n   ```\n\n5. Install the required modules:\n\n   ```\n   python -m pip install -r requirements.txt\n   ```\n\n#### Run usos-teachers-scraper\n\n1. Sign up for an API key:\n\n   ```\n   https://apps.usos.pwr.edu.pl/developers/\n   ```\n\n2. Change directory:\n\n   ```\n   cd usos-teachers-scraper\n   ```\n\n3. Create a `config.json` file. Then come up with a secret key. Finally, paste your Consumer Key and Consumer Secret:\n\n   ```\n   {\n    \"secret_key\": \"\u003cyour-secret-key\u003e\",\n    \"consumer_key\": \"\u003cgenerated-consumer-key\u003e\",\n    \"consumer_secret\": \"\u003cgenerated-consumer-secret\u003e\"\n   }\n   ```\n\n4. Run the script:\n\n   ```\n   python ./usos-teachers-scraper.py\n   ```\n\n5. Visit the USOS authorization page:\n   \n   ```\n   http://127.0.0.1:5000/start_oauth\n   ```\n\n6. Fetch teachers data by visiting page:\n   ```\n   http://127.0.0.1:5000/fetch_staff\n   ```\n\n#### Run usos-abstracts-scraper\n\n1. Change directory:\n\n   ```\n   cd usos-abstracts-scraper\n   ```\n\n2. Run the script:\n\n   ```\n   python ./usos-abstracts-scraper.py\n   ```\n\n3. Input initial thesis id and final thesis id.\n\n### Github workflow\n\nWhen you had assigned yourself to new task, you should stick to these steps\n1. `git checkout main` Check out main branch\n2. `git pull origin main` Pull current changes from main branch\n3. `git fetch` Be up to date with remote branches\n4. `git checkout -b type/task` Create new task branch\n5. `git add .` Add all changes we have made\n6. `git commit -m \"My changes description\"` Commit changes with proper description\n7. `git push origin type/task` Pushing our changes to remote branch\n8. On Github we are going to make Pull Request (PR) from our remote branch\n \n\u003e [!WARNING]\n\u003e Do not push changes directly to main branch\n\nFor further information read Solvro handbook\n\n**Github Solvro Handbook 🔥** - https://docs.google.com/document/d/1Sb5lYqYLnYuecS1Essn3YwietsbuLPCTsTuW0EMpG5o/edit?usp\n\n## Current team\nThis is our current team\n- [@LukiLenkiewicz](https://github.com/LukiLenkiewicz) - Tech Lead\n- [@Micz26](https://github.com/Micz26) - ML Engineer\n- [@farqlia](https://github.com/farqlia) - ML Engineer\n- [@AgataGro](https://github.com/AgataGro) - ML Engineer\n- [@dekompot](https://github.com/dekompot) - ML Engineer\n- [@b4rt4s](https://github.com/b4rt4s) - ML Engineer\n- [@Woleek](https://github.com/Woleek) - ML Engineer\n- [@WiktoriaFrost](https://github.com/WiktoriaFrost) - ML Engineer\n- [@Barionetta](https://github.com/Barionetta) - Project Manager\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolvro%2Fscript-promochator-usos-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsolvro%2Fscript-promochator-usos-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolvro%2Fscript-promochator-usos-scraper/lists"}