{"id":17798874,"url":"https://github.com/jazzpi/mts-scraper","last_synced_at":"2025-04-02T03:41:52.068Z","repository":{"id":86827088,"uuid":"255980008","full_name":"jazzpi/mts-scraper","owner":"jazzpi","description":"Selenium-based scraper for the MTS","archived":false,"fork":false,"pushed_at":"2020-04-20T15:23:14.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-07T18:24:21.391Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://moseskonto.tu-berlin.de","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jazzpi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-15T16:48:06.000Z","updated_at":"2021-09-01T12:10:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"2c56a937-d6eb-4f21-a83e-d9a4794cba4e","html_url":"https://github.com/jazzpi/mts-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jazzpi%2Fmts-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jazzpi%2Fmts-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jazzpi%2Fmts-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jazzpi%2Fmts-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jazzpi","download_url":"https://codeload.github.com/jazzpi/mts-scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246752608,"owners_count":20827987,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-27T12:01:59.324Z","updated_at":"2025-04-02T03:41:52.050Z","avatar_url":"https://github.com/jazzpi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MTS Scraper\n\nThis is a scraper for the *Modultransfersystem* of TUB's *MOSES* system.\n\nSince the *MTS* uses [PrimeFaces](https://www.primefaces.org), which is really\nannoying to reverse engineer, this scraper uses Selenium to do the requests.\n\n## Installation\n\nTODO\n\n## Running\n\n``` sh\npython -m mts_scraper -h\n```\n\n## Output\n\nThe scraper saves the results in a SQLite database. This database has the\nfollowing tables:\n\n### Degree Programs\n\n- ID (from MTS)\n- Title\n- Degree (MSc/BSc/etc.)\n\n### Study Areas\n\n- ID (only for unique identification)\n- Title\n- Parent ID (NULL if there is no parent study area)\n- Degree Program ID\n\n### Modules\n\n- ID (from MTS)\n- Version\n- Title\n- ECTS\n- Exam type\n- Details/Parts Fetched?\n- Faculty\n- Department\n- Learning Outcomes\n- Content\n\n### Modules \u003c\u003e Study Areas\n\n- Study area ID\n- Module ID\n- Module version\n\n### Module Parts\n\n- ID (only for unique identification)\n- Title\n- Language\n- Type (VL/UE/SEM etc.)\n- Turnus\n- SWS\n- Veranstaltungsnummer (if available)\n- Module ID\n- Module version\n\n## Continuing a scraping session\n\nIf the database specified by `-d` exists, only modules (and module parts) that\nare not already in it are fetched.\n\nModules are identified by ID and version (i.e., two modules with the same title\nbut different IDs count as two different modules, as do two modules with the\nsame ID but different versions).\n\nThe scraper executes these steps:\n\n1. Check if the degree program exists. If no, start from scratch. If yes, we can\n   be sure the study areas and their respective module lists have been fetched\n   already.\n2. Compile a list of all modules in any study area in the degree program (from\n   the modules table) _that have not been fetched already_.\n3. Sequentially fetch each unfetched module.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjazzpi%2Fmts-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjazzpi%2Fmts-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjazzpi%2Fmts-scraper/lists"}