https://github.com/frankyang0529/spark-with-spot-instance

This is python script which use spark_c2.py in Spark to start spot-instance.
https://github.com/frankyang0529/spark-with-spot-instance

Last synced: about 1 month ago
JSON representation

This is python script which use spark_c2.py in Spark to start spot-instance.

• Host: GitHub
• URL: https://github.com/frankyang0529/spark-with-spot-instance
• Owner: FrankYang0529
• Created: 2014-11-25T16:53:40.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2014-11-26T03:58:28.000Z (almost 10 years ago)
• Last Synced: 2024-10-03T20:12:20.778Z (about 1 month ago)
• Language: Python
• Size: 125 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Spark-with-Spot-instance

This is the python script which use spark_ec2.py in Spark to automatize.

## Document

First you have to movie spot-script.py to /spark and then run with python2.7.

### AWS_ACCESS_KEY_ID & AWS_SECRET_KEY_ID

You can get AWS_ACCESS_KEY_ID and AWS_SCRET_KEY_ID from [Security Credentials](https://console.aws.amazon.com/iam/home?#home)

### Key pair name & Key file direction

You can get key pair from amazon ec2.

Example:

key file direction: /Users/frank/test-ec2.pem

key pair name: test-ec2

### Slave number

Type an integer which mean how many slave you want.

### zone

Type a zone where you want to launch your spot instance.

Example:

zone: us-east-1b

### Spot price

Type an float which mean the max price you want to launch instance.

### Cluster name

Type a cluster name which you want.

Example:

cluster name: test

### Program name

Type the program name which you want to run on spot instance.

Example:

program name: premovie.py

### Program file direction

Type the program file direction which you put program file on local.

Example:

program file direction: /Users/frank/premovie.py