{"id":28211031,"url":"https://github.com/darinm223/ucla-web-scraper","last_synced_at":"2025-06-11T03:32:07.012Z","repository":{"id":78918410,"uuid":"21379112","full_name":"DarinM223/ucla-web-scraper","owner":"DarinM223","description":"A web scraper in Node.js that scrapes the UCLA schedule of classes","archived":false,"fork":false,"pushed_at":"2015-01-18T08:42:19.000Z","size":400,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-17T17:11:19.202Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DarinM223.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-01T07:39:49.000Z","updated_at":"2024-03-19T17:36:30.000Z","dependencies_parsed_at":"2023-02-23T23:40:48.776Z","dependency_job_id":null,"html_url":"https://github.com/DarinM223/ucla-web-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DarinM223%2Fucla-web-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DarinM223%2Fucla-web-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DarinM223%2Fucla-web-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DarinM223%2Fucla-web-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DarinM223","download_url":"https://codeload.github.com/DarinM223/ucla-web-scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DarinM223%2Fucla-web-scraper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259191651,"owners_count":22819391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-17T17:10:57.594Z","updated_at":"2025-06-11T03:32:06.999Z","avatar_url":"https://github.com/DarinM223.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"UCLA schedule of classes web scraper\n====================================\n\n![Main image](http://i.imgur.com/oCrLA6e.png)\n\nProof-of-concept web scraper that uses Node.js and the request and cheerio libraries to scrape\ndata from the UCLA schedule of classes and store the data in\nmongodb. The data is not intended for commercial use.\n\nInstallation and usage\n----------------------\n\n### Requirements: mongodb\n### Tested on: Ubuntu 12.04\n\nFirst clone the repository with\n\n    git clone https://github.com/DarinM223/ucla-web-scraper.git\n\nThen in the root directory of the project, enter:\n\n    npm install\n\nTo install the dependencies. To run the server, enter:\n\n    node main.js\n\nRight now after it finishes adding all of the courses it will hang\nand you will have to end the process by pressing Ctrl-C.\n\nThe scraper should take around 3-4 minutes to run on an i5\nprocessor on UCLA's network connection :). Because it uses the cluster library to efficiently use\nall of the cores, processors with more cores will be faster. Right now it only stores data from the Fall 2014 quarter.\nTo view some of the data, enter:\n\n    mongo\n    use ucla\n    db[\"14F\"].find()\n\nTo view how much data was saved, enter inside the mongo console:\n\n    db[\"14F\"].find().count()\n\nIt should return around 2700-2900 elements\n\n![Time proof](http://i.imgur.com/JPpGk4S.png?1)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarinm223%2Fucla-web-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdarinm223%2Fucla-web-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarinm223%2Fucla-web-scraper/lists"}