https://github.com/darinm223/ucla-web-scraper
A web scraper in Node.js that scrapes the UCLA schedule of classes
https://github.com/darinm223/ucla-web-scraper
Last synced: about 1 year ago
JSON representation
A web scraper in Node.js that scrapes the UCLA schedule of classes
- Host: GitHub
- URL: https://github.com/darinm223/ucla-web-scraper
- Owner: DarinM223
- Created: 2014-07-01T07:39:49.000Z (about 12 years ago)
- Default Branch: master
- Last Pushed: 2015-01-18T08:42:19.000Z (over 11 years ago)
- Last Synced: 2025-05-17T17:11:19.202Z (about 1 year ago)
- Language: JavaScript
- Size: 391 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
UCLA schedule of classes web scraper
====================================
Proof-of-concept web scraper that uses Node.js and the request and cheerio libraries to scrape
data from the UCLA schedule of classes and store the data in
mongodb. The data is not intended for commercial use.
Installation and usage
----------------------
### Requirements: mongodb
### Tested on: Ubuntu 12.04
First clone the repository with
git clone https://github.com/DarinM223/ucla-web-scraper.git
Then in the root directory of the project, enter:
npm install
To install the dependencies. To run the server, enter:
node main.js
Right now after it finishes adding all of the courses it will hang
and you will have to end the process by pressing Ctrl-C.
The scraper should take around 3-4 minutes to run on an i5
processor on UCLA's network connection :). Because it uses the cluster library to efficiently use
all of the cores, processors with more cores will be faster. Right now it only stores data from the Fall 2014 quarter.
To view some of the data, enter:
mongo
use ucla
db["14F"].find()
To view how much data was saved, enter inside the mongo console:
db["14F"].find().count()
It should return around 2700-2900 elements
