https://github.com/cloudyyoung/ucalgary-spider
A crawler to grab information from academic calendar for all the courses and people directory for all the staffs of the University of Calgary.
https://github.com/cloudyyoung/ucalgary-spider
scrapy
Last synced: 10 months ago
JSON representation
A crawler to grab information from academic calendar for all the courses and people directory for all the staffs of the University of Calgary.
- Host: GitHub
- URL: https://github.com/cloudyyoung/ucalgary-spider
- Owner: cloudyyoung
- Created: 2020-07-15T19:43:29.000Z (almost 6 years ago)
- Default Branch: main
- Last Pushed: 2025-03-06T09:17:11.000Z (about 1 year ago)
- Last Synced: 2025-04-10T20:31:38.778Z (about 1 year ago)
- Topics: scrapy
- Language: Python
- Homepage:
- Size: 15.3 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Nero - UCalgary Spider
## Development
Pip is needed to install the packages in `requirements.txt`
```
pip install -r requirements.txt
```
## Run
Run a crawler
`scrapy crawl `
List of crawlers are under directory `nero/spiders`
## Clean raw data
Run clean script
`python3 grigio/main.py`
## Endpoints
Coursedog Curriculum API: https://coursedogcurriculum.docs.apiary.io/#reference/programs/get-all-programs
### Departments
https://app.coursedog.com/api/v1/ucalgary_peoplesoft/departments
### Courses
https://app.coursedog.com/api/v1/cm/ucalgary_peoplesoft/courses/
https://app.coursedog.com/api/v1/cm/ucalgary_peoplesoft/courses/1604171-2018-07-01?includeDependents=true&formatDependents=true
### Programs
https://app.coursedog.com/api/v1/cm/ucalgary_peoplesoft/programs/
https://app.coursedog.com/api/v1/cm/ucalgary_peoplesoft/courseSets/
# Terms
https://app.coursedog.com/api/v1/ucalgary_peoplesoft/general/terms
# Requisites
https://app.coursedog.com/api/v1/ucalgary_peoplesoft/requisite-sets
AOS - Area of Study
Requisites example (simple):
- https://app.coursedog.com/api/v1/cm/ucalgary_peoplesoft/courses/1072711-2020-07-01?includeDependents=true&formatDependents=true
- https://app.coursedog.com/api/v1/cm/ucalgary_peoplesoft/courses/1072161-2020-07-01?includeDependents=true&formatDependents=true