https://github.com/datitran/pyspark-app-cf
https://github.com/datitran/pyspark-app-cf
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/datitran/pyspark-app-cf
- Owner: datitran
- License: mit
- Created: 2017-01-25T15:20:05.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-02-06T14:23:26.000Z (over 9 years ago)
- Last Synced: 2024-10-29T08:04:39.440Z (over 1 year ago)
- Language: Jupyter Notebook
- Size: 125 KB
- Stars: 2
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# PySpark-App-CF
A simple example which uses the [PySpark buildpack](https://github.com/andreasf/pyspark-buildpack) to deploy an [Apache Spark](http://spark.apache.org/) application, particularly using its Python API, on Cloud Foundry.
### Getting Started
- Use `cf push` to deploy the application
### Testing
A local `apache-spark` instance and `nosetests` need to be installed before running the tests. For more information see [here](https://github.com/datitran/spark-tdd-example).
- Run tests with: `nosetests -vs tests/`
### CI/CD
[Concourse](https://concourse.ci/) is used as our CI tool due to its seamless integration with Cloud Foundry. The fastest way to use Concourse is with [Vagrant](https://www.vagrantup.com/):
- Install Vagrant and run `vagrant init concourse/lite && vagrant up`
- Connect to the CI: `fly -t pyspark-app-cf login -c http://192.168.100.4:8080`
- Fill in the credential details in `credentials.yml.example` and rename the file to `credentials.yml`
- Register the pipeline: `fly -t pyspark-app-cf set-pipeline -p pyspark-app-ci -c pipeline.yml -l credentials.yml`
- Unpause the pipeline: `fly -t pyspark-app-cf unpause-pipeline -p pyspark-app-ci`
## Dependencies
- [Apache Spark 2.1.0](http://spark.apache.org/)
- OpenJDK 1.8.0_91
- [Anaconda](https://www.continuum.io/downloads) Python 3.5.0
- Python conda environment (install with `conda env create --file environment.yml`)
## Copyright
See [LICENSE](LICENSE) for details.
Copyright (c) 2017 [Dat Tran](http://www.dat-tran.com/), [Andreas Fleig](https://github.com/andreasf).