{"id":16406830,"url":"https://github.com/terror/mcgill.courses","last_synced_at":"2025-07-25T06:40:55.458Z","repository":{"id":150108977,"uuid":"611099086","full_name":"terror/mcgill.courses","owner":"terror","description":"A course search and review platform for McGill University","archived":false,"fork":false,"pushed_at":"2025-02-02T00:23:03.000Z","size":35916,"stargazers_count":41,"open_issues_count":62,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-14T22:03:18.271Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://mcgill.courses","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/terror.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-08T05:18:41.000Z","updated_at":"2025-03-07T00:53:49.000Z","dependencies_parsed_at":"2024-10-11T06:11:14.836Z","dependency_job_id":"5d7c07db-adb2-4e77-b9ac-eb775adf2056","html_url":"https://github.com/terror/mcgill.courses","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terror%2Fmcgill.courses","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terror%2Fmcgill.courses/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terror%2Fmcgill.courses/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/terror%2Fmcgill.courses/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/terror","download_url":"https://codeload.github.com/terror/mcgill.courses/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243725010,"owners_count":20337659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T06:11:05.134Z","updated_at":"2025-07-25T06:40:55.443Z","avatar_url":"https://github.com/terror.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## mcgill.courses \u003ca\u003e\u003cimg src=\"https://github.com/user-attachments/assets/3d835599-381c-4d82-8e71-4d80be190909\" height=\"40\" valign=\"top\" /\u003e\u003c/a\u003e\n\nA course search and review platform for McGill university.\n\n![](https://github.com/user-attachments/assets/8807c010-6d85-4b3a-9199-4cc101417519)\n\n## Development\n\nYou'll need [docker](https://www.docker.com/),\n[cargo](https://doc.rust-lang.org/cargo/) and [pnpm](https://pnpm.io/) installed\non your machine to spawn the various components the project needs to run\nlocally.\n\nFirst, join the discord server to get access to the development environment\nvariables:\n\n[https://discord.gg/fSVSqfPHSV](https://discord.gg/fSVSqfPHSV)\n\nIn `.env` within the root directory you'll have to set\n\n```\nMS_CLIENT_ID=\nMS_CLIENT_SECRET=\nMS_REDIRECT_URI=http://localhost:8000/api/auth/authorized\n```\n\n...and then in `client/.env` you'll have to set the server url\n\n```\nVITE_API_URL=http://localhost:8000\n```\n\nSecond, mount a local [mongodb](https://www.mongodb.com/) instance with docker\nand initiate the replica set:\n\n```bash\ndocker compose up --no-recreate -d\nsleep 5\ndocker exec mongodb mongosh --quiet --eval 'rs.initiate()' \u003e /dev/null 2\u003e\u00261 || true\n```\n\nSpawn the server with a data source (in this case the `/seed` directory) and\ninitialize the database (note that seeding may take some time on slower\nmachines):\n\n```bash\ncargo run -- --source=seed serve --initialize --db-name=mcgill-courses\n```\n\nFinally, spawn the react frontend:\n\n```bash\npnpm install\npnpm run dev\n```\n\n_n.b._ If you have [just](https://github.com/casey/just) installed, we provide a\n`dev` recipe for doing all of the above in addition to running a watch on the\nserver:\n\n```bash\njust dev\n```\n\nSee the\n[justfile](https://github.com/terror/mcgill.courses/blob/master/justfile) for\nmore recipes.\n\n### Gathering seed data\n\nThe server command-line interface provides a load subcommand for scraping all\ncourses from various McGill course information websites and building a JSON data\nsource, for example:\n\n```\nRUST_LOG=info cargo run -- --source=seed \\\n  load \\\n  --batch-size=200 \\\n  --scrape-vsb \\\n  --user-agent \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36\"\n```\n\nThe current defaults include scraping all current 10,000+ courses offered by\nMcGill, current schedule information from the official\n[visual schedule builder](https://vsb.mcgill.ca), and courses offered in\nprevious terms going back as far the 2009-2010 term.\n\nFor full usage information, see the output below:\n\n```present just run load --help\nUsage: server load [OPTIONS] --user-agent \u003cUSER_AGENT\u003e\n\nOptions:\n      --user-agent \u003cUSER_AGENT\u003e      A user agent\n      --course-delay \u003cCOURSE_DELAY\u003e  Time delay between course requests in milliseconds [default: 0]\n      --retries \u003cRETRIES\u003e            Number of retries [default: 10]\n      --batch-size \u003cBATCH_SIZE\u003e      Number of pages to scrape per concurrent batch [default: 20]\n      --mcgill-terms \u003cMCGILL_TERMS\u003e  The mcgill terms to scrape [default: 2009-2010 2010-2011 2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 2018-2019 2019-2020 2020-2021 2021-2022 2022-2023 2023-2024 2024-2025]\n      --vsb-terms \u003cVSB_TERMS\u003e        The schedule builder terms to scrape [default: 202405 202409 202501]\n      --scrape-vsb                   Scrape visual schedule builder information\n  -h, --help                         Print help\n```\n\nAlternatively, if you have [just](https://github.com/casey/just) installed, you\ncan run:\n\n```\njust load\n```\n\n## Tools\n\nWe have a few tools that we use throughout the project, below documents some of\nthem. You can find them all under the\n[`/tools`](https://github.com/terror/mcgill.courses/tree/master/tools) directory\nfrom the project root.\n\nFor python-based tools, we highly recommend you install\n[uv](https://docs.astral.sh/uv/) on your system. On macOS or linux, you can do\nit as follows:\n\n```bash\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\nFollow the\n[documentation](https://docs.astral.sh/uv/getting-started/installation/) for\nother systems.\n\n### `changelog-generator`\n\nOur changelog page\n([https://mcgill.courses/changelog](https://mcgill.courses/changelog)) is\nautomated by this tool.\n\nWe feed PR titles and descriptions to a large language model (in this case\nhard-coded to GPT-3.5) to generate a user-friendly summary using\n[this prompt](https://github.com/terror/mcgill.courses/blob/master/tools/changelog-generator/prompt.txt).\n\nThe tool assumes you have an [OpenAI](https://openai.com/) API key set in the\nenvironment, and you can use it from the project root like:\n\n```bash\ncargo run --manifest-path tools/changelog-generator/Cargo.toml \\\n  -- \\\n  --output client/src/assets/changelog.json\n```\n\nThis will run the changelog generator on all\n[up-to-date merged PRs](https://github.com/terror/mcgill.courses/pulls?q=is:pr+is:closed)\nfrom our GitHub repository, populating\n[`changelog.json`](https://github.com/terror/mcgill.courses/blob/master/client/src/assets/changelog.json)\nwith the results.\n\nThere are a few other options the tool supports:\n\n```present cargo run --manifest-path tools/changelog-generator/Cargo.toml -- --help\nUsage: changelog-generator [OPTIONS]\n\nOptions:\n      --output \u003cOUTPUT\u003e               [default: ../../client/src/assets/changelog.json]\n      --regenerate [\u003cREGENERATE\u003e...]\n      --regenerate-all\n      --repo \u003cREPO\u003e                   [default: mcgill.courses]\n      --user \u003cUSER\u003e                   [default: terror]\n  -h, --help                          Print help\n```\n\nFor instance, you can regenerate single entries by specifying their pull request\nnumber.\n\n### `course-average-fetcher`\n\nThis tool is used to populate a\n[JSON file](https://github.com/terror/mcgill.courses/blob/master/client/src/assets/courseAveragesData.json)\nwith course average information we display on\n[course pages](https://mcgill.courses/course/econ208).\n\nWe read and parse a\n[crowdsourced google sheet](https://docs.google.com/spreadsheets/d/1NGUBQuF8FI6ebna86S1RHpc27srxpMbaSyjipIkr-gk/edit?gid=233834959#gid=233834959)\nwith historical course averages provided generously by the\n[McGill enhanced](https://demetrios-koziris.github.io/McGillEnhanced/) team.\n\n### `requirement-parser`\n\nWe parse prerequisites and corequisites using a fine-tuned large language model\nwith custom examples, all the code lives in\n[`/tools/requirement-parser`](https://github.com/terror/mcgill.courses/tree/master/tools/req-parser).\n\nIf you need to run the requirement parser on a file, simply:\n\n```bash\ncd tools/requirement-parser\nuv sync\nuv run main.py \u003cfile\u003e\n```\n\n_n.b._ This will require an [OpenAI](https://openai.com/) API key and the name\nof the fine-tuned model to be set in the environment.\n\nFor more information about how this works, check out our\n[research project](https://github.com/SamZhang02/llmbda).\n\n### `search-index-aggregator`\n\nThis tool selectively includes only the\n[JSON fields](https://github.com/terror/mcgill.courses/blob/master/client/src/assets/searchData.json)\n(from database\n[seed files](https://github.com/terror/mcgill.courses/tree/master/seed))\nrequired by the search component, significantly reducing payload size and\nimproving resource efficiency.\n\n## Deployment\n\nWe continuously deploy our site with [Render](https://render.com/) using a\n[docker image](https://github.com/terror/mcgill.courses/blob/master/Dockerfile),\nand have a [MongoDB](https://en.wikipedia.org/wiki/MongoDB?useskin=vector)\ninstance hosted on [Atlas](https://www.mongodb.com/atlas/database).\n\nWe also use\n[S3](https://aws.amazon.com/pm/serv-s3/?trk=936e5692-d2c9-4e52-a837-088366a7ac3f\u0026sc_channel=ps)\nto host a bucket for referring to a hash when deciding whether or not to seed\ncourses in our production environment, and Microsoft's\n[identity platform](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-auth-code-flow)\nfor handling our OAuth 2.0\n[authentication flow](https://github.com/terror/mcgill.courses/blob/master/src/auth.rs).\n\n## Prior Art\n\nThere are a few notable projects worth mentioning that are similar in nature to\n[mcgill.courses](https://mcgill.courses), and have either led to inspiration or\nnew ideas with regard to its functionality and design, namely:\n\n- [uwflow.com](https://uwflow.com/) - A course search and review platform for\n  the University of Waterloo\n- [cloudberry.fyi](https://www.cloudberry.fyi/) - A post-modern schedule builder\n  for McGill students\n- [mcgill.wtf](https://github.com/terror/mcgill.wtf) - A fast full-text search\n  engine for McGill courses\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterror%2Fmcgill.courses","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fterror%2Fmcgill.courses","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fterror%2Fmcgill.courses/lists"}