https://github.com/linde/retrosheet-spark
This is a simple project to create data frames in spark for the data contained in retrosheets baseball archive. It is distributed as a python notebook.
https://github.com/linde/retrosheet-spark
baseball spark
Last synced: about 2 months ago
JSON representation
This is a simple project to create data frames in spark for the data contained in retrosheets baseball archive. It is distributed as a python notebook.
- Host: GitHub
- URL: https://github.com/linde/retrosheet-spark
- Owner: linde
- Created: 2016-04-10T08:16:47.000Z (about 10 years ago)
- Default Branch: master
- Last Pushed: 2021-12-13T09:58:53.000Z (over 4 years ago)
- Last Synced: 2025-09-08T13:58:52.140Z (10 months ago)
- Topics: baseball, spark
- Language: Jupyter Notebook
- Homepage:
- Size: 2.24 MB
- Stars: 0
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
# Retrosheet Spark
This is a simple project to create data frames in spark for the data
contained in retrosheets baseball archive. It is distributed as a
python notebook.
This project owes a lot to the helpful comment here:
http://stackoverflow.com/questions/31227363/creating-spark-data-structure-from-multiline-record
## Getting the retrosheets data
```bash
$ mkdir retrosheet-data
$ cd retrosheet-data
$ for yyyy in `seq 1910 10 2010`; do echo getting $yyyy; wget http://www.retrosheet.org/events/${yyyy}seve.zip; done
$ for yyyy in `seq 1910 10 2010`; do mkdir ${yyyy}seve; done
$ for yyyy in `seq 1910 10 2010`; do unzip -d ${yyyy}seve ${yyyy}seve.zip; done
```
## Some notes on getting up and going...
This verion targets the latest python3 in brew at the moment (Python
3.6.5) and spark v2.3.0 running pyspark within jupyter.. You can do
something akin to the following to get them installed.
```bash
$ brew install apache-spark
$ brew install python
$ pip3 install virtualenv
$ virtualenv .py
$ . .py/bin/activate
$ pip install jupyter
```
if you want to do the plotting example in [etrosheet-spark-queries.ipynb], then also install the following:
```bash
$ pip install plotly
```
## The finally, to run things
```bash
PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark --executor-memory 2GB
```
Then run python notebook and execute each buffer, then query and enjoy!