https://github.com/agentender/openpayrollcrawler

A simple webcrawler targeted at https://openpayrolls.com
https://github.com/agentender/openpayrollcrawler

Last synced: about 1 month ago
JSON representation

A simple webcrawler targeted at https://openpayrolls.com

• Host: GitHub
• URL: https://github.com/agentender/openpayrollcrawler
• Owner: AgentEnder
• Created: 2019-10-29T13:12:34.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2019-10-29T13:14:59.000Z (about 5 years ago)
• Last Synced: 2024-10-11T17:08:40.959Z (3 months ago)
• Language: Python
• Size: 2.93 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

        
# OpenPayroll Crawler

This is a simple web crawler meant toextract payroll data from OpenPayroll.

## Dependencies ##

* Scrapy `pip install scrapy`

## How to run: ##

Running the crawler is simple. 

* Open crawler.py

* Edit line 44 with whatever location you are wanting to crawl (check http://openpayrolls.com for valid locations)

	* Optionally change the output file name here as well.

* Run crawler.py as main

## *WARNING*: Running this for large locations will likely get you IP banned from OpenPayroll ##

### A VPN is strongly recommended ###