https://github.com/gerhalt/mining-camp

Easy automated configuration and deployment of Minecraft servers on AWS spot instances, featuring automatic backups and restoration using S3.
https://github.com/gerhalt/mining-camp

ansible aws ec2 ec2-spot minecraft minecraft-server python route53 s3 server spot-instances terraform

Last synced: about 14 hours ago
JSON representation

Easy automated configuration and deployment of Minecraft servers on AWS spot instances, featuring automatic backups and restoration using S3.

• Host: GitHub
• URL: https://github.com/gerhalt/mining-camp
• Owner: gerhalt
• Created: 2017-08-15T18:44:23.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2021-09-23T23:21:48.000Z (about 3 years ago)
• Last Synced: 2023-10-26T14:49:33.071Z (about 1 year ago)
• Topics: ansible, aws, ec2, ec2-spot, minecraft, minecraft-server, python, route53, s3, server, spot-instances, terraform
• Language: Python
• Homepage:
• Size: 87.9 KB
• Stars: 46
• Watchers: 4
• Forks: 6
• Open Issues: 4
• Metadata Files:
  • Readme: README.adoc

Awesome Lists containing this project

README

        
Mining Camp

===========

:toc:

:toc-placement: preamble

:toclevels: 3

Easy automated configuration and deployment of Minecraft servers on AWS spot

instances, with features such as instance shutdown monitoring and automatic

backups and restorations using S3.

image::https://i.imgur.com/jvJzU6v.png[Mining Camp]

== Introduction

Amazon sells spare EC2 instances that aren't currently reserved, called "spot

instances", for a fraction of the cost of a reserving them. If you can mitigate

their volatility risk, they're an excellent deal and powerful enough to host

demanding game servers.

_Advantages:_

* More control over everything. SSH into the box, troubleshoot, adjust anything

you like on the fly.

* Better bang for your buck. More powerful systems than most hosting services

offer.

* Pay for exactly what you use. Spot instances are billed by the second, so if

your server isn't going to be used for a few days, or even overnight, just turn

it off.

* Linux-based. This might be a disadvantage, depending on your comfort level.

Windows instances are not only more of a pain, they're also far more expensive.

* Works with any AWS region, so you can minimize latency for your player base.

_Disadvantages:_

* Spot instances can be terminated at any time. This is mitigated by both the

emergency shutdown monitoring script, but in practice reasonable bids for spot

instances result in excellent uptime.

* Requires initial work to setup. This isn't a push-button solution, but once

you're done with the setup day-to-day interactions are extremely easy.

=== Sample Cost Breakdowns

I've provided a couple samples of pricing based on different instance types

below. When deciding what to use, cross reference your requirements against

the https://aws.amazon.com/ec2/spot/pricing/[spot instance pricing] and the

https://aws.amazon.com/ec2/instance-types/[EC2 instance types]. Be aware that

spot instance prices can differ not only between AWS regions, but between

availability zones as well.

All calculations below are based on an instance in `us-east-1`, and assume a

31-day month.

NOTE: AWS charges for absolutely everything. It's critical that you monitor

your bill and be aware of where you're incurring charges.

==== i3.large

An _i3.large_ is a quite powerful instance with ~16GB of RAM and a 500GB NVMe

drive, meaning an EBS (beyond the 8GB root volume) isn't required. Price

estimates for the various moving pieces:

* Current spot instance price for an _i3.large_ is $0.0371 per hour at present.

* An 8GB EBS is used as the root volume for the instance, at $0.10 per

    GB-month, or roughly $0.00013 an hour. This is deleted on instance shutdown,

    so no charges are incurred when the server is off.

* Elastic IPs are free when assigned to a running instance, and $0.005 per hour

    when unassigned.

* AWS data transfer is pretty expensive, at $0.09 per GB. I use about 1.4GB per

    day, but my server has a lot of time when it's completely idle. For the

    sake of round numbers, 2GB of bandwidth a day winds up being $0.0075 an

    hour. This cost is very subjective.

* S3 is actually quite cheap, especially with lifecycle management rules

    cleaning up old backups. 20GB of combined server archives and rotating

    backups is $0.023 per GB, $0.46 a month, or $0.0006 an hour.

From these numbers, when my server is running, I'm paying roughly *$0.045* an

hour or *$33.73* a month to run it full-time. When my server is off, but I want

to preserved my backups and elastic IP address, I'm paying *$4.17* a month. Just

preserving backups is only *$0.46*, which is very reasonable.

==== t2.medium

Options with less RAM more be suited to smaller loads. One example is a

_t2.medium_, which has only 4GB of RAM and no SSD, which means it requires an

EBS volume. The upside is that it only costs $0.0134 an hour. If we use a 30GB

ephemeral EBS volume that's deleted on instance termination, that adds roughly

$0.0004 an hour.

Using this less powerful, less expensive instance brings considerable savings:

*$0.0255* an hour, or *$18.97* when run for an entire month.

== Setup

Start by checking out this repository, and navigating to your checkout-out

directory. Sample bash commands provided are from the root directory unless

otherwise noted.

You'll need to download and install the following requirements to get started

(versions I've tested with are in parentheses):

* `pip` (20.1)

* `terraform` (0.12.24)

You'll also need:

* An appropriate version of the JDK/JVM to run the server locally at least once,

  allowing it to generate the necessary configuration files.

* A fully configured AWS developer account; you'll be connecting to it to set

  up the required infrastructure.

=== Configuration

==== Credentials

You'll need your AWS credentials available for most of these operations, under

the `minecraft` profile. `~/.aws/credentials` will look like:

```

[minecraft]

aws_access_key_id = 

aws_secret_access_key = 

```

To tell Ansible to use this profile, you can do one of the following:

* `export AWS_PROFILE=minecraft` in the shell you'll be running the Ansible

    commands in.

* Prefix each command with `AWS_PROFILE=minecraft`, which sets the environment

    for that particular invocation.

* Change the `minecraft` section title to `default` (or duplicate it). This

    will cause those credentials to be tried by default.

==== SSH Key Pair

You'll need a key pair for accessing your instance. Generate a public-private

key pair. As an example, you can do this with `ssh-keygen`:

```

ssh-keygen -t rsa -b 4096 -C "AWS"

```

In the EC2 console, select _Import Key Pair_ on the

_NETWORK & SECURITY -> Key Pairs_ page. Upload your public key, and name it

"aws-public". The launch configuration Terraform creates includes this key,

allowing SSH access to Ansible (and for troubleshooting!)

==== Instance Access

You'll need to _choose one of the following methods_ to access your instance:

* Use the public IP automatically assigned to your spot instance. This

  requires no setup on your part, but the address will be different each time

  the instance is started. The IP of your instance can be found under

  _EC2 → Instances_ in the "IPv4 Public IP" column.

* Use an elastic IP. This associates a known IP with your

  instance each time it's started. Setup for this must be done by hand.

  Elastic IPs also incur a cost while they're not in use, as well as

  additional charges if you make too many assignments per month.

  **(DEPRECATED)**

* Use Route 53 to host an owned domain or subdomain, pointed at your server.

  Requires a domain and minor manual setup with your registrar.

===== via Elastic IP (DEPRECATED)

You'll need to create an elastic IP for association with your instance,

providing a convenient public-facing IP. In the AWS console, do the following:

1. Enter the EC2 service.

2. Click on _Elastic IPs_, under the _NETWORK & SECURITY_ menu on the left-hand

side of the screen.

3. Click _Allocate new address_.

4. Leave the scope as "VPC", and click close.

5. You should see your new elastic IP in the list. Save the _Allocation ID_ for

later use during the setup.

Once a server has been spun up, this elastic IP will be attached to it.

An allocated elastic IP is included in the price of a running instance, but

you will be billed for any unassigned EIPs by the hour. For this reason, if

you plan to stop your Minecraft server for long periods of time, be sure to

delete your EIPs and create new ones when you're ready to begin hosting again.

===== via Domain

Proceed through the rest of the setup, specifying a server hostname. Once

you've applied your Terraform config, return to do this section and do the

following:

1. Enter the AWS console, and navigate to the _Route 53_ service.

2. Select the hosted zone for the hostname you chose.

3. You should see an `NS` record with four hosts.

4. Add a corresponding `NS` record to your domain for each host. I leave this

  as an exercise for the reader.

5. Now, when your server is booted, it will automatically add an `A` record

  with a short TTL pointed at your server's public IP.

==== Virtual Environment & Python Requirements

Using pip, install the necessary Python 2.7 requirements. I recommend using

https://virtualenv.pypa.io/en/stable/[virtualenv] and

https://pypi.python.org/pypi/virtualenvwrapper/[virtualenvwrapper]. Running the

following installs Ansible, the AWS command-line interface, and libraries

required for interacting with AWS programmatically.

```

$ mkvirtualenv minecraft

(minecraft) $ pip install -r requirements.txt

```

==== Minecraft Server Archive

You'll need to create a Minecraft server archive to be pulled onto your

instance each time the box is spun up. In this example, I'll be creating an

archive for my Feed the Beast server named `daftcyborg`.

```

$ # Create a base directory named after your server name

$ mkdir daftcyborg

$ cd daftcyborg

$ # Get your base server pack. In my case, I've already downloaded the FTB server

$ ls

FTBRevelationServer_1.0.0.zip

$ unzip FTBRevelationServer_1.0.0.zip

$ # Install the server requirements, if the pack requires it

$ sh ./FTBInstall.sh

$ # Create and populate the bootstrap script - mining camp will use this to

$ # launch your server. Copy from _utitilies/_, and update with memory limits

$ # and the `.jar` you're using.

$ cp /utilities/server-start.sh .

$ # Launch the server. You'll need to do this twice, once to create the

$ # eula.txt and once to generate the base

$ sh ./server-start.sh

Missing eula.txt. Startup will fail and eula.txt will be created

Make sure to read eula.txt before playing!

To continue press 

```

Open `eula.txt`, and agree (or don't) to the terms and conditions.

Launch the server again, and wait for it to complete. This will generate the

world base, and any settings and properties files necessary. Quit the server,

and do the following as desired:

* Remove the `world` directory, which is the world directory name used by

default and which will (assuming you update the `server.properties` file) be

named differently when your server is run.

* Edit `server.properties` as desired. It is important that the _server-port_

be left as _25565_, otherwise you'll need to adjust the Terraform

configuration. Fields I recommend changing are _level-name_, _level-seed_, and

_motd_.

* Add yourself and any other players desired to `ops.json`.

* Update `server-icon.png` to a custom icon.

Copy server.properties to `ansible/files/server.properties`, which Ansible will

install every time over the top of the properties file in the archive, allowing

easy configuration changes.

Now, clean up your leftover base archive, since you don't need it anymore:

```

$ rm FTBRevelationServer_1.0.0.zip

```

Navigate up a level, and create a gzipped tarball:

```

$ cd ..

$ tar -cvzf daftcyborg-server-12-20-2017.tgz daftcyborg/

```

Lastly, push the archive to S3:

```

$ # The parameterized command is 'aws s3 cp  s3:////'

$ # My version looks like:

$ aws s3 cp daftcyborg-server-12-20-2017.tgz s3://josh-minecraft/daftcyborg/

```

Lastly, save the full name, including file extension, of the archive you

generated; it will be required when you run the setup wizard.

==== Settings

The recommended way to configure the system is to run the setup wizard

from the root repo directory, like so:

```

$ ./utilities/setup.py

```

This guides you through each required setting, offering default values if

available. It then takes your input and renders out `terraform/variables.tf`

and `ansible/group_vars/all` from corresponding `*.j2` templates. If you like,

you can populate those templates by hand as well.

* It's important you choose the right _aws_availability_zone_, since spot

  prices can vary substantially from zone to zone.

* Maximum spot price determines the maximum price you're willing to pay per

  hour. Setting this wisely will prevent you from being surprised by a large

  bill at the end of the month.

==== Terraform

Terraform allows you to easily setup EC2 and S3 to match your needs. To apply

the terraform configuration, run:

```

terraform apply terraform/

```

Once this has successfully completed, your AWS configuration is done. Unless

you change your configuration, you won't need to run this again.

==== Ansible

The first time you do the configurations, you'll need to bundle your local

Ansible configuration and push it to your S3 bucket, under the same directory

as your server is stored. A provided playbook takes care of this for you:

```

cd ansible/

ansible bundle.yml

```

NOTE: You'll need to repeat this step each time you adjust any server settings.

== Instance Interactions

=== Launch

Jump to the `ansible` directory, and run the `start.yml` playbook to configure

the instance and launch the minecraft server:

```

cd ansible

ansible-playbook start.yml

```

This merely increases the size of the auto-scaling group from 0 → 1; the server

won't be available for a few more minutes as it provisions.

=== Shutdown

Shutting down your server is very similar, with a few extra command options

that allow Ansible to SSH into the host and save the server state before

killing the box itself.

```

cd ansible

ansible-playbook -i ec2.py --private-key=~/.ssh/aws -u ubuntu -c ssh stop.yml

```

When this playbook finishes, your instance will be gone, but the state of the

server will have been preserved and pushed to S3, ready for the next time you

launch it.

NOTE: If the auto-inventory script is taking too long, you can update

`ansible/ec2.ini`'s `regions` entry with the particular AWS region you're using.

NOTE: If using an older version of Ansible, the Paramiko library used by

default may run into errors when gathering facts from the remote host. If this

happens, add `-c ssh` to the `ansible-playbook` command above.

== Troubleshooting

Logs from the initial provision are available via the console as well as in

`/var/log/user-data`.

== Tests

Tests are currently available for the Prospector tool. You'll need to install

the requirements in the test directory in order to run them. From the root,

with your virtual environment active:

```

(minecraft) $ pip install -r utilities/tests/requirements.txt

```

Now you can launch the test suite:

```

(minecraft) $ python -m unittest -v utilities.tests.test_prospector

```