https://github.com/zehengl/scrapy-darksky-api
A scrapy app to crawl weather data from Dark Sky Api
https://github.com/zehengl/scrapy-darksky-api
darksky python scrapy
Last synced: 7 months ago
JSON representation
A scrapy app to crawl weather data from Dark Sky Api
- Host: GitHub
- URL: https://github.com/zehengl/scrapy-darksky-api
- Owner: zehengl
- License: mit
- Created: 2019-10-08T06:37:56.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-04-08T14:55:07.000Z (about 1 year ago)
- Last Synced: 2024-04-08T18:15:53.589Z (about 1 year ago)
- Topics: darksky, python, scrapy
- Language: Python
- Homepage: https://yyc-darksky-api.streamlit.app/
- Size: 6.79 MB
- Stars: 6
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# scrapy-darksky-api
A scrapy app to crawl weather data from Dark Sky Api
> Note: Dark Sky has joined Apple and the API will not be available after end of March, 2023. See [source](https://blog.darksky.net/).
## Environment
- Python 3.9
## Install
1. create virtualenv
2. activate virtualenv
3. update pip
4. install deps
Use `pip install -r requirements-dev.txt` for development.
It will install `pylint` and `black` to enable linting and auto-formatting.
## Usage
1. set environment variables for darksky secret key, latitude, longitude, year, and hour for crawling
2. run scrapy to crawl the forecast data and save in json
This scrapy app would crawl the temperatures in all days within {year} at {hour} o'clock.
Keep in mind that [Dark Sky Api][1] only allows 1000 daily requests for free tier.
## Example
( _51.0447° N_, _114.0719° W_ ) is Calgary's coordinates.
### Windows
```powershell
python -m venv .venv
.\.venv\Scripts\activate
python -m pip install -U pip
pip install -r requirements-dev.txt
$Env:darksky_secret_key="xxx"
$Env:darksky_latitude="51.05011"
$Env:darksky_longitude="-114.08529"
$Env:darksky_year="2013"
$Env:darksky_hour="6"
scrapy crawl forecast -O data\forecast_$Env:darksky_year`_$Env:darksky_hour.json
```
### Linux
```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
pip install -r requirements-dev.txt
export darksky_secret_key="xxx"
export darksky_latitude="51.05011"
export darksky_longitude="-114.08529"
export darksky_year="2013"
export darksky_hour="6"
scrapy crawl forecast -O data/forecast_$darksky_year\_$darksky_hour.json
```
## Demo
See the demo app at [https://yyc-darksky-api.streamlit.app/][2] for some visualization done on Calgary's weather data
## Credits
- [Icon][3] by [The Pictographers][4]
[1]: https://darksky.net/dev
[2]: https://yyc-darksky-api.streamlit.app/
[3]: https://www.iconfinder.com/icons/667368/celcius_clouds_farenheit_sunshine_temerature_thermometer_weather_icon
[4]: https://www.iconfinder.com/bluewolfski