https://github.com/mozilla/hashitalk2015
INACTIVE - http://mzl.la/ghe-archive - Notes from the Hashiconf talk by @jdotp and @phrawzty
https://github.com/mozilla/hashitalk2015
inactive unmaintained
Last synced: 3 months ago
JSON representation
INACTIVE - http://mzl.la/ghe-archive - Notes from the Hashiconf talk by @jdotp and @phrawzty
- Host: GitHub
- URL: https://github.com/mozilla/hashitalk2015
- Owner: mozilla
- Archived: true
- Created: 2015-09-28T19:59:23.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2019-03-29T17:03:38.000Z (almost 7 years ago)
- Last Synced: 2025-01-12T10:31:09.298Z (12 months ago)
- Topics: inactive, unmaintained
- Size: 117 KB
- Stars: 9
- Watchers: 9
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Good reads and tools we use
* 12factor.net : A great write up by Heroku on making an app scalable, modular, and easy to upgrade. Basically, how to build apps in the new world.
* https://github.com/phrawzty/hiera-s3: Nice hiera plugin from phrawzty (but maybe just use vault! :) )
* Visible Ops Book: http://www.amazon.com/The-Visible-Ops-Handbook-Implementing/dp/0975568612
* Phoenix Project Book: http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
# Code Examples We Use
* Our Terraform code: https://github.com/mozilla/socorro-infra/blob/master/terraform
* Our Puppet code: https://github.com/mozilla/socorro-infra/blob/master/puppet
* Our rustic deployment script: https://github.com/mozilla/socorro-infra/blob/master/bin/deploy-socorro.sh
* Our Packer code: https://github.com/mozilla/socorro-infra/blob/master/packer
* Our user data code: https://github.com/mozilla/socorro-infra/blob/master/terraform/webapp/socorro_role.sh
# Some of our example public dashboards / metrics
* A cool way to cycle through dashboards: http://dashplay.io/0p98yofx+
* Our current events: https://p.datadoghq.com/sb/282da3b8c-2916c1621c
* Today vs yesterday: https://p.datadoghq.com/sb/282da3b8c-3c0ed4ed8d
* Prod stats: https://p.datadoghq.com/sb/282da3b8c-bfee0403b5
* How to send custom and arbitrary metrics to cloudwatch (you can track # of coffees consumed, or anything else) https://www.loggly.com/blog/send-custom-metrics-to-cloudwatchs-api/
# Packer Hint
* Packer can create an AMI for any region, and even create an AMI for multiple regions. Super useful for deployment!
# S3 Specific Hints
* US Standard is simply (US-East-1 and US-West-1) in failover mode. This is why things in US Standard are 'eventual consistency' for reads after writes.
* If you need immediate, reliable consistency, choose a specific region when creating the bucket.
* US-Standard does not cost extra compared to single-region non-redundant S3 regions.
* Cloudfront costs the same per request as S3.
* You *do* get SSL with S3, but not on http endpoints. ( Here, have this page explain it to you...it's a little weird. http://docs.aws.amazon.com/general/latest/gr/rande.html )
* Otherwise, use SSL in cloudfront. Use the old IAM tools to upload it...
# Other AWS Hints
* Our naming standards for AWS infrastructure(do this ASAP!): https://dl.dropboxusercontent.com/u/2273146/aws-naming-conventions.txt
* When given a choice, download/request your SSL certs in Apache format. This goes right up to AWS easily.
**iam-servercertupload -b ./mysite.crt -k ./mysite.key -c ./intermediatechain.crt -p /cloudfront/ -s mysite-2015-09-25**
* Always use autoscaling groups for everything, even things that never need to scale. You always want 1 of some server up...autoscaling groups make it so if that one dies, a new one automatically replaces it.
* You can have multiple autoscaling groups scale into a single ELB.
* You can have a single autoscaling group add/remove servers from multiple ELBs.
* These concepts are useful if you want to balance out the # of instance store vs. EBS backed EC2 instances...EBS is usually the culprit on AWS @#$@storm days
* Go multi-region early rather than later...it doesn't get easier the longer you wait!
* When load testing or expecting heavy traffic, it's a great idea to call AWS support and have them "prewarm" your ELB. ELBs are basically just EC2 instances running load balancing software, and 'prewarming' them is scaling them up.
* To get great ratings for SSL security, check out Mozilla's SSL cipher tool (credit to Gene!) https://mozilla.github.io/server-side-tls/ssl-config-generator/
* If you have to have multiple AZ's in staging but only have one node up, enable cross zone balancing.
* CNAME everything! Let's say you have a database server: webappdatabase.rdsaddress.2342342342.aws.com... If you cname that to webapp-db.internaldomain.com instead, if you ever have to switch databases or regions you have a super easy way to switch over!
* Health check based routing is awesome, as is geo-ip routing.
* Decide on a tagging strategy early on....Mozilla Tools and Services team uses: environment, role, project. These tags play a major role in monitoring, metrics, and even permissions.
* Always, immediately, first thing...enable Cloudtrail in AWS.
# Alerting Hint
* Setup an IFTTT recipe which will monitor the status RSS feeds for AWS, and SMS/email/alert you and your team when those RSS feeds get updated.
* Be sure you have multiple sources watching your apps!
# Load Testing Hint
* Easy 'pound one address' tool: bees with machineguns (it's awesome to make an endpoint that exercises the entire app/funnel/workflow that you can then pound on with bees...and the logs are so fun to read)
# Ways to Not Spend All of Your Organizations Money
* Setup detailed billing and a bucket to store it im
* Try out Netflix ICE cost analysis software
* When you know you need an instance for at least a year, setup reserved instances
# Team Hint
* A Great Intro to 'Gamedays' (The Way to Make Apps and Teams More Resilient by Dylan Richard): https://www.youtube.com/watch?v=LCZT_Q3z520
# A hint from @rhelmer
ABCD: ALWAYS BE CHECKING DATADOG