{"id":19730579,"url":"https://github.com/begriffs/microservice-template","last_synced_at":"2026-01-27T17:01:20.473Z","repository":{"id":25311654,"uuid":"28738328","full_name":"begriffs/microservice-template","owner":"begriffs","description":"Basic architecture for running and monitoring workers","archived":false,"fork":false,"pushed_at":"2015-05-05T22:16:27.000Z","size":633,"stargazers_count":70,"open_issues_count":10,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-12T19:53:23.751Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/begriffs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-03T07:53:30.000Z","updated_at":"2023-12-11T21:23:53.000Z","dependencies_parsed_at":"2022-07-25T12:32:12.181Z","dependency_job_id":null,"html_url":"https://github.com/begriffs/microservice-template","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begriffs%2Fmicroservice-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begriffs%2Fmicroservice-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begriffs%2Fmicroservice-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begriffs%2Fmicroservice-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/begriffs","download_url":"https://codeload.github.com/begriffs/microservice-template/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224192997,"owners_count":17271238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T00:16:50.746Z","updated_at":"2026-01-27T17:01:15.453Z","avatar_url":"https://github.com/begriffs.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Microservice Template ([see video](http://begriffs.com/posts/2015-02-15-microservice-template.html))\n\nThe scalable monitored template for cluster infrastructure.\n\n* Provision servers with Packer and Chef\n* Deploy and scale with Terraform\n* Service discovery with Consul\n* Data collection with collectd and statsd\n* Analysis and visualization with InfluxDB and Grafana\n\n#### Installation\n\n1. Sign up for Amazon Web Services free tier [here](http://aws.amazon.com/free/).\n    It is **extremely important** to ensure the region in your\n    Amazon Web Console is set to \"N. California\" as the defaults \n    are all configured to use this region in future steps. If you decide\n    to use a different region you will need to specify the region in future\n    steps when noted.\n\n2. Create Amazon IAM credentials for deployment.\n    Assuming you are signed into the AWS console, visit the [user\n    admin](https://console.aws.amazon.com/iam/home#users) page. Create\n    a new user and note their security credentials (ID and secret key).\n    Then attach a user policy of \"Power User\" to the newly created user.\n\n3. [Create EC2 keypair](https://console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName).\n\n4. Clone this repo including its chef recipe submodules.\n    ```bash\n    git clone --recursive https://github.com/begriffs/microservice-template.git\n\n    # if you've already cloned the repo you can do: git submodule update --init\n    ```\n\n5. Install \u003ca href=\"https://www.packer.io/\" target=\"_blank\"\u003ePacker\u003c/a\u003e and\n    \u003ca href=\"https://www.terraform.io/\" target=\"_blank\"\u003eTerraform\u003c/a\u003e. On a\n    Mac you can install them with homebrew:\n    ```bash\n    brew tap homebrew/binary\n    brew install packer\n    brew install terraform\n    ```\n\n6. Create machine images (AMI) using your credentials.\n    When running each of these commands **write down** the AMI ids\n    each one generates. They will be of the form `ami-[hash]`. You\n    will need to remember which command created which AMI.\n    AWS credentials are required and can be provided on the command line or\n    accessed from environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.\n\n    Optionally the region can be specified via command line. If specifying a region the proper\n    source ami must also be provided via command line.\n\n    ####Region to source ami mappings.\n    ```bash\n       us-east-1: ami-b66ed3de\n       us-west-1: ami-4b6f650e\n       us-west-2: ami-b5a7ea85\n       eu-west-1: ami-607bd917\n       ap-southeast-1: ami-ac5c7afe\n       ap-southeast-2: ami-63f79559\n       ap-northeast-1: ami-4985b048\n       sa-east-1: ami-8737829a\n    ```\n\n    If no region is specified on the command line a default region of `us-west-1` is used\n    with a source_ami of `ami-4b6f650e`.\n\n    ```bash\n    packer build -var 'aws_access_key=xxx' -var 'aws_secret_key=xxx' -var 'region=us-east-1' -var 'source_ami=ami-b66ed3de' consul.json \n    packer build -var 'aws_access_key=xxx' -var 'aws_secret_key=xxx' statsd.json\n    packer build -var 'aws_access_key=xxx' -var 'aws_secret_key=xxx' influx.json\n    packer build -var 'aws_access_key=xxx' -var 'aws_secret_key=xxx' grafana.json\n    packer build -var 'aws_access_key=xxx' -var 'aws_secret_key=xxx' rabbitmq.json\n\n    # for haskell workers (optional)\n    packer build -var 'aws_access_key=xxx' -var 'aws_secret_key=xxx' halcyon.json\n    ```\n\n7. Deploy machine images.\n    ```bash\n    cp terraform/terraform.tfvars{.example,}\n    ```\n    Edit `terraform/terraform.tfvars` and fill in the ami instances created by\n    the previous steps, the key name associated with your keypair created on EC2,\n    and your AWS keys. Optionally provide a region matching the region of your amis.\n    A default region of `us-west-1`. If you built a halcyon ami, you will need to\n    specify the number of halcyon workers for the ami to be used.\n\n    Now the fun part. Go into the terraform directory and run `make`.\n\n    At the end it will output the public IP address of the monitoring\n    server for the cluster. You can use it to watch server health and\n    resource usage.\n\n#### Monitoring\n\nThe server exposes web interfaces for several services.\n\nPort  | Service\n----- | -------------------------------------\n80    | [Grafana](http://grafana.org/) charts\n8500  | Consul server status and key/vals\n8080  | InfluxDB admin panel\n8086  | InfluxDB API used by Grafana\n15672 | RabbitMQ management console\n\nInflux has been configured with two databases, `metrics` and\n`grafana`. Cluster data accumulates in the former and Grafana stores\nyour chart settings in the latter. The Influx user `grafana` (password\n`grafpass`) has full access to both tables. RabbitMQ is set up with\nuser `guest` password `guest`.\n\n#### Collecting more stats\n\nThe cluster exposes [StatsD](https://github.com/etsy/statsd/)\nserver(s) at `statsd.node.consul`. Your applications should send\nit lots of events. The statsd protocol is UDP and incurs little\napplication delay. The statsd server relays all info to InfluxDB\nwhich makes it accessible for graphing.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbegriffs%2Fmicroservice-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbegriffs%2Fmicroservice-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbegriffs%2Fmicroservice-template/lists"}