{"id":35118625,"url":"https://github.com/markbeep/vvzapi","last_synced_at":"2026-02-28T01:25:00.537Z","repository":{"id":319989759,"uuid":"1075505752","full_name":"markbeep/vvzapi","owner":"markbeep","description":"REST API and search for VVZ","archived":false,"fork":false,"pushed_at":"2026-01-31T18:54:08.000Z","size":45656,"stargazers_count":5,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-01T06:45:46.332Z","etag":null,"topics":["api","catalog","course","eth","ethz","vvz"],"latest_commit_sha":null,"homepage":"https://vvzapi.ch","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/markbeep.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-13T15:38:44.000Z","updated_at":"2026-01-31T18:53:54.000Z","dependencies_parsed_at":"2025-12-05T13:01:02.292Z","dependency_job_id":null,"html_url":"https://github.com/markbeep/vvzapi","commit_stats":null,"previous_names":["markbeep/vvzapi"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/markbeep/vvzapi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markbeep%2Fvvzapi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markbeep%2Fvvzapi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markbeep%2Fvvzapi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markbeep%2Fvvzapi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/markbeep","download_url":"https://codeload.github.com/markbeep/vvzapi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markbeep%2Fvvzapi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29196719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T12:38:28.597Z","status":"ssl_error","status_checked_at":"2026-02-07T12:38:23.888Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","catalog","course","eth","ethz","vvz"],"created_at":"2025-12-27T22:54:57.324Z","updated_at":"2026-02-21T23:09:40.333Z","avatar_url":"https://github.com/markbeep.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VVZ REST API\n\n![banner](./assets/banner.png)\n\n#### Community-made simple search and REST API for [ETHZ's Vorlesungsverzeichnis](https://www.vvz.ethz.ch/Vorlesungsverzeichnis)\n\n## Quick Start\n\nHead to https://vvzapi.ch and start playing around with the search or API!\n\n## Terms of Use\n\n\u003e [!NOTE]  \n\u003e This project is not affiliated with ETH Zurich in any way. It is a personal project that is not endorsed by ETH Zurich. The data is scraped from the public [VVZ](https://www.vvz.ethz.ch/Vorlesungsverzeichnis) website and is not used for any commercial purposes. The API is provided as-is and without any warranties. Use it at your own risk.\n\u003e\n\u003e This repository is licensed under the GPLv3 license. See [LICENSE](./LICENSE) for more information. Any forks must also be licensed under the GPLv3 license and made open-source if they are distributed.\n\u003e\n\u003e If you use the API or the downloadable database, I ask that you please credit the project and link back to the repository.\n\u003e I'm happy to see any projects that use the API and would like to link them here.\n\n## Schema\n\nThe schema is inspired by the [VVZ Manual](https://www.bi.id.ethz.ch/soapvvz-2023-1/manual/SoapVVZ.pdf#page=18) (starts page 18).\n\nAttributes have been translated to english, dropped (in cases where the value was internal and not visible on VVZ), or additional attributes have been added that were not present in the documentation.\n\nThe word choices might be confusing if you're not used to them. **Importantly, the term \"unit\" (or \"learning unit\" as the 1:1 translated from the German \"Lerneinheit\") is used for what is commonly understood as a course (Discrete Mathematics, Big Data, etc.).** \"Unit\" is the more general term, as VVZ also lists non-courses like thesis and other projects.\n\nA unit can have multiple times and places it can take place at. These individual slots are called **\"courses\"** in the API (a somewhat loose translation of the German \"Lehrveranstaltung\").\n\n## Search Design\n\nThe search is inspired by [Scryfall](https://scryfall.com/).\n\n## Semester Status\n\n\u003e [!NOTE]  \n\u003e For some reason all semesters 2009-2019 (both S and W) are simply not available. Accessing any of them throws a 403 Forbidden. I wonder if this\n\u003e is just some short-term problem or if they'll never come back. Some of the data (for all courses) is available in the\n\u003e [Complete Catalogue](https://www.vvz.ethz.ch/Vorlesungsverzeichnis/gesamtverzeichnis.view?lang=en), but I currently do not have any plans\n\u003e of parsing data from PDFs.\n\n## Versioning\n\nThis project uses semantic versioning. Breaking changes will result in a bump of the major version. There should not be any breaking changes to the endpoints of any endpoints that are the same or lower version than the major version. If the current version is `2.x.x`, the endpoints under `/v1` and `/v2` will not be _intentionally_ updated in a way that would break or completely change their usage. But `/v3` would then still be in prerelease and might change anytime.\n\n---\n\n## Contribution\n\nThe idea behind the VVZ API is to more easily enable the creation of various cool tools requiring course/VVZ data. If you have an idea for something that should absolutely be in the API, but is missing, open up an issue and let's start discussing it!\n\nI'm grateful for any form of contribution, may it be adding documentation, implementing new features, opening issues for errors or something else. Head to the `Local Development` section below to learn more about how to get the API running locally.\n\n---\n\n## Local Development\n\nDepending on what you intend to test locally, you can opt to download a dump of the database (head to the API docs to find the endpoint) to develop locally with the most up-to-date data.\n\nAdditionally, for ease of development, there is both a devcontainer (`.devcontainer/devcontainer.json`) and a mise (`mise.toml`) config that can help with setting up the local development environment. The devcontainer setup will initialze the essentials in a docker/podman container without cluttering your system with dependencies, while [mise](https://mise.jdx.dev/) installs packages into your shell environment and might be easier.\n\n### Alembic Migrations\n\nLocally, a SQLite database is used. Running the migrations automatically creates the database.\n\n#### Run migrations\n\n```sh\nuv run alembic upgrade heads\n```\n\n#### Create revision\n\nRequired if any model was modified.\n\n```sh\nuv run alembic revision --autogenerate -m \"message\"\n```\n\n### Scraper\n\n#### Create scraper\n\n```sh\nuv run scrapy genspider \u003cscraper name\u003e \u003cscraper name\u003e.py\n```\n\n#### Run scraper\n\n```sh\nuv run -m scraper.main\n```\n\nOr for just one of the spiders:\n\n```sh\nuv run scrapy crawl units\nuv run scrapy crawl lecturers\nuv run scrapy crawl ratings\n```\n\n#### Run in shell (for debug)\n\n```sh\nuv run scrapy shell \"\u003curl\u003e\"\n```\n\n#### Debug spider\n\n```sh\n# general syntax\nuv run scrapy parse --spider=units -c \u003ccb func\u003e \"\u003curl\u003e\"\n# scrape all units\nuv run scrapy parse --spider=units -c parse_start_url \"https://www.vvz.ethz.ch/Vorlesungsverzeichnis/sucheLehrangebot.view?lang=de\u0026semkez=2003S\u0026seite=0\"\n# scrape a single unit\nuv run scrapy parse --spider=units -c parse_unit \"https://www.vvz.ethz.ch/Vorlesungsverzeichnis/lerneinheit.view?semkez=2025W\u0026ansicht=ALLE\u0026lerneinheitId=192945\u0026lang=en\"\n# scrape all lecturers\nuv run scrapy parse --spider=lecturers -c parse_start_url \"https://www.vvz.ethz.ch/Vorlesungsverzeichnis/sucheDozierende.view?lang=de\u0026semkez=2026S\u0026seite=0\"\n```\n\n#### Cleanup html cache directory\n\nThere might be outdated or unused files in the html cache directories. Using the cleanup script everything that is not needed can be removed. Additionally it can also be used to purposely delete at most `amount` valid cached files from one or more `semester`s that are older than `age-seconds`.\n\n```sh\nuv run scraper/util/cleanup_scrapy.py [--dry-run] [--amount \u003cint\u003e] [--age-seconds \u003cint\u003e] [-d \u003csemester\u003e]*\n```\n\n#### Scrape locally\n\nThe scraper can also be started locally by running the docker image directly, if desired.\n\n```sh\ndocker run \\\n    -e SEMESTER=W \\\n    -e START_YEAR=2024 \\\n    -e END_YEAR=2024 \\\n    -v $PWD/data:/app/.scrapy \\\n    markbeep/vvzapi-scraper:nightly\n```\n\nIn the data directory there'll be a `httpcache` directory containing all crawled HTML files and a `scrapercache` directory containing scraper specific files and potentially a file called `error_pages.jsonl` with errors.\n\n---\n\n### API Server\n\n```sh\nuv run fastapi dev api/main.py\n```\n\n#### Tailwindcss\n\nTailwind is used in combination with [DaisyUI](https://daisyui.com). Download the source files using the following commands:\n\n```sh\ncurl -sLo api/static/daisyui.mjs https://github.com/saadeghi/daisyui/releases/latest/download/daisyui.mjs\ncurl -sLo api/static/daisyui-theme.mjs https://github.com/saadeghi/daisyui/releases/latest/download/daisyui-theme.mjs\n```\n\nThen run tailwindcss:\n\n```sh\ntailwindcss -i api/static/tw.css -o api/static/globals.css --watch\n```\n\n### Type Check\n\n```sh\nuv run basedpyright\n```\n\n---\n\n## Benchmarking\n\nWebsite can be benchmarked with lighthouse or k6 locally.\n\n### Lighthouse\n\nCan be used to test SEO and generally if the website form is correct.\n\n```sh\njust lighthouse /unit/193444\n```\n\n### K6\n\nCan be used to benchmark the website.\n\n```sh\nk6 run --vus 50 --duration 5m k6.js\n```\n\n### JaegerUI\n\nOpenTelemetry can be used for more performance details and what slows down certain things.\n\nJaegerUI can be started up with the following command (runs a docker container):\n\n```sh\njust jaeger\n```\n\nIn your local `.env` file you want to add the following line:\n\n```sh\nJAEGER_ENDPOINT=http://localhost:4317\n```\n\nThe jaeger UI can be viewed under http://localhost:16686\n\nNOTE: This works well in combination with K6 to stress test the endpoints.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkbeep%2Fvvzapi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkbeep%2Fvvzapi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkbeep%2Fvvzapi/lists"}