https://github.com/palewire/git-scraper-example
A example of a git scraper that download, lints, commits and archives a data set
https://github.com/palewire/git-scraper-example
git-scraper python scraper
Last synced: 8 months ago
JSON representation
A example of a git scraper that download, lints, commits and archives a data set
- Host: GitHub
- URL: https://github.com/palewire/git-scraper-example
- Owner: palewire
- License: mit
- Created: 2021-06-30T02:07:22.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2024-06-02T15:03:49.000Z (almost 2 years ago)
- Last Synced: 2024-06-02T16:54:57.157Z (almost 2 years ago)
- Topics: git-scraper, python, scraper
- Language: Python
- Homepage:
- Size: 726 KB
- Stars: 0
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# git-scraper-example
A example of a git scraper that downloads, commits and archives a data set.
The source for this demonstration is the list of [liquor stores in the state of Iowa](https://data.iowa.gov/Regulation/Iowa-Liquor-Stores/ykb6-ywnd). The scraping routine is automated by the [scrape.yaml workflow](https://github.com/palewire/git-scraper-example/actions/workflows/scrape.yaml).
## Instructions
Clone the repository
```zsh
git clone git@github.com:palewire/git-scraper-example.git
```
Move into the directory
```zsh
cd git-scraper-example
```
Install the Python tools
```zsh
pipenv install
```
Run the downloader
```zsh
pipenv run python src/download.py
```