https://github.com/jdm79/web-scraper-checker
I built a web scraper to text me when my MA course was listed on the university website
https://github.com/jdm79/web-scraper-checker
httparty nikkou nokogiri ruby twilio twilio-ruby webscraper whenever
Last synced: 8 months ago
JSON representation
I built a web scraper to text me when my MA course was listed on the university website
- Host: GitHub
- URL: https://github.com/jdm79/web-scraper-checker
- Owner: jdm79
- Created: 2019-08-14T10:33:11.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2023-03-17T01:32:59.000Z (over 3 years ago)
- Last Synced: 2025-07-05T20:48:52.427Z (12 months ago)
- Topics: httparty, nikkou, nokogiri, ruby, twilio, twilio-ruby, webscraper, whenever
- Language: Ruby
- Homepage:
- Size: 27.3 KB
- Stars: 2
- Watchers: 0
- Forks: 1
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# web-scraper-checker
web scraper program using Ruby to check when my MA course is finally listed on their website so I can see when everything happens etc. twilio code allows me to sms myself with updates.
this is mainly to get back into coding, to practise webscraping for my Interactive Journalism MA at City University, London working on data journalism.
# Stack
* 'httparty' gem to scrape the html from the web
* 'nokogiri' gem to parse the html into whatever I want to parse it into
* 'nikkou' gem to perform regex check for key words in the parsed html
* 'whenever' gem to create a chron job which automatically pings the website every hour to see if anything has changed
* 'twilio' gem which allows free sms (low usage) service
# How to use (this is more for future me than anything else)
* clone this repo
* bundle install the gems
* personalise (twilio account, website and tag to scrape, dotenv config etc)
* run the program: $ ruby main.rb
# How to use the cron job
[Click here](https://github.com/javan/whenever)