https://github.com/jshrake/lacity-council-crawl
A crawler and scraper for LA city council voting data
https://github.com/jshrake/lacity-council-crawl
Last synced: 10 months ago
JSON representation
A crawler and scraper for LA city council voting data
- Host: GitHub
- URL: https://github.com/jshrake/lacity-council-crawl
- Owner: jshrake
- License: other
- Created: 2015-07-14T16:30:16.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2015-07-18T16:58:36.000Z (almost 11 years ago)
- Last Synced: 2025-03-31T01:14:42.530Z (over 1 year ago)
- Language: Clojure
- Size: 133 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# lacity-council-crawl
A crawler and scraper for [LA city council voting data](http://citycouncil.lacity.org/cvvs/search/search.cfm) built with Clojure
## Usage
This service crawls votes [10 seconds at a time](http://lacity.org/robots.txt). As of this commit, there are roughly ~80,000 votes in the system.
Once this service has crawled all existing votes, it will attempt to retrieve newly added votes indefintely.
### Dev
Stores the data in an SQLite database under `db`:
```bash
mkdir db
lein run
```
### Production
Create a production configuration file at `resources/production_config.edn` that specifies the database information. You can model it after [resources/dev_config.edn](resources/dev_config.edn).
```bash
lein with-profile production run
```
## Deploy
As this is a service that should run indefinitely, it may be desriable to deploy it to the great big cloud in the sky. I've chosen to deploy to a free instance of Amazon's [Elastic Beanstalk](https://aws.amazon.com/elasticbeanstalk/):
0. Create a production configuration file
1. Install the Amazon EB CLI `brew install awsebcli`
2. `eb init`
3. `eb create`
4. `eb deploy`
## License
MIT