{"id":13879838,"url":"https://github.com/coinbase/geoengineer","last_synced_at":"2025-07-16T16:30:28.704Z","repository":{"id":56873993,"uuid":"63974097","full_name":"coinbase/geoengineer","owner":"coinbase","description":"DEPRECATED — Infrastructure As Code","archived":true,"fork":false,"pushed_at":"2020-01-16T23:50:36.000Z","size":1585,"stargazers_count":401,"open_issues_count":0,"forks_count":57,"subscribers_count":32,"default_branch":"master","last_synced_at":"2024-11-09T11:18:11.695Z","etag":null,"topics":["balance","cloud-resources","codifying-resources","dsl","infrastructure","ruby","workflow"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coinbase.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-07-22T18:15:01.000Z","updated_at":"2024-09-29T01:04:55.000Z","dependencies_parsed_at":"2022-08-20T22:30:29.656Z","dependency_job_id":null,"html_url":"https://github.com/coinbase/geoengineer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fgeoengineer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fgeoengineer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fgeoengineer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coinbase%2Fgeoengineer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coinbase","download_url":"https://codeload.github.com/coinbase/geoengineer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226143895,"owners_count":17580245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["balance","cloud-resources","codifying-resources","dsl","infrastructure","ruby","workflow"],"created_at":"2024-08-06T08:02:35.321Z","updated_at":"2024-11-24T08:31:53.475Z","avatar_url":"https://github.com/coinbase.png","language":"Ruby","readme":"# GeoEngineer\n\n[![CircleCI](https://circleci.com/gh/coinbase/geoengineer.svg?style=shield)](https://circleci.com/gh/coinbase/geoengineer)\n\n\u003ca href=\"https://commons.wikimedia.org/wiki/File:Mantle_of_Responsibility.png\"\u003e\u003cimg src=\"./assets/mantle.png\" align=\"right\" alt=\"Mantle_of_Responsibility\" /\u003e\u003c/a\u003e\n\nGeoEngineer provides a Ruby DSL and command line tool (`geo`) to *codify*  then plan and execute changes to cloud resources.\n\nGeoEngineer's goals/requirements/features are:\n\n0. **DSL based on Terraform**: GeoEngineer uses [Terraform](https://github.com/hashicorp/terraform) to plan and execute changes, so the DSL to describe resources is similar to Terraform's. GeoEngineer's DSL also provides programming and object oriented features like inheritance, abstraction, branching and looping.\n1. **Development Workflow**: GeoEngineer is built to be used within existing development workflows, e.g. branching, creating pull requests, code reviewing and merging. To simplify these workflows, GeoEngineer dynamically generates Terraform state files using cloud APIs and opinionated tagging.\n2. **Extensible Validation**: Every team has their own standards when managing cloud resources e.g. naming patterns, tagging requirements, security rules. GeoEngineer resources can have custom validations added to ensure the resources conform to required standards.\n3. **Describe Existing Resources**: Existing resources can be described with GeoEngineer without having to destroy and recreate them.\n4. **Principle of Least Astonishment**: show the exact plan before execution; do nothing without confirmation; do not allow a plan to be executed with failing validations; by default do not allow deletions; show warnings and hints to make code better.\n5. **One File per Project**: Managing dozens of projects with hundreds of files is difficult and error-prone, especially if a single project's resources are described across many files. Projects are easier to manage when they are each described in one file.\n6. **Dependencies**: resources have dependencies on other resources, projects have dependencies on other projects. Using Ruby's `require` file that describe resources can be included and referenced without having to hard-code any values.\n\n## Getting Started\n\n### Install Terraform\n\nInstructions to install Terraform can be found [here](https://www.terraform.io/downloads.html).\n\n    brew install terraform\n\n### Install Ruby\n\nInstructions to install Ruby can be found [here](https://www.ruby-lang.org/en/documentation/installation/).\n\n    rbenv install `cat .ruby-version`\n\n### Install GeoEngineer\n\nBuild the gem locally and then refer to it with `geo` on the command line.\n\n    bundle install\n    gem build geoengineer.gemspec\n    gem install geoengineer-version.gem\n    geo --help\n\n### Running GeoEngineer\n\nInstall and configure [`assume-role`](https://github.com/coinbase/assume-role).\n\n    assume-role \u003caccount-id\u003e \u003crole\u003e\n    ./geo --help\n\n### First GeoEngineer Project\n\nGeoEngineer can use the folder structure where projects and environments are in the `projects` and `environments` directories respectively, however everything can also be defined in a single file, e.g. `first_project.rb`:\n\n```ruby\n# First define the environment which is available with the variable `env`\n# This is where project invariants are stored, e.g. subnets, vpc ...\nenvironment(\"staging\") {\n  account_id  \"1\"\n  subnet      \"1\"\n  vpc_id      \"1\"\n  allow_destroy true ## Defaults to false.  Set to true to support `geo destroy ...`\n}\n\n# Create the first_project to be in the `staging` environment\nproject = project('org', 'first_project') {\n  environments 'staging'\n}\n\n# Define the security group for the ELB to allow HTTP\nelb_sg = project.resource(\"aws_security_group\", \"allow_http\") {\n  name         \"allow_http\"\n  description  \"Allow All HTTP\"\n  vpc_id       env.vpc_id\n  ingress {\n      from_port    80\n      to_port      80\n      protocol     \"tcp\"\n      cidr_blocks  [\"0.0.0.0/0\"]\n  }\n  tags {\n    Name \"allow_http\"\n  }\n}\n\n# Define the security group for EC2 to allow ingress from the ELB\nec2_sg = project.resource(\"aws_security_group\", \"allow_elb\") {\n  name         \"allow_elb\"\n  description  \"Allow ELB to 80\"\n  vpc_id       env.vpc_id\n  ingress {\n      from_port    8000\n      to_port      8000\n      protocol     \"tcp\"\n      security_groups  [elb_sg]\n  }\n  tags {\n    Name \"allow_elb\"\n  }\n}\n\n# cloud_config to run webserver\nuser_data = %{\n#cloud-config\nruncmd:\n  - docker run -d --name nginx -p 8000:80 nginx\n}\n\n# Create an EC2 instance to run nginx server\ninstance = project.resource(\"aws_instance\", \"web\") {\n  ami           \"ami-1c94e10b\" # COREOS AMI\n  instance_type \"t1.micro\"\n  subnet_id     env.subnet\n  user_data     user_data\n  tags {\n    Name \"ec2_instance\"\n  }\n}\n\n# Create the ELB connected to the instance\nproject.resource(\"aws_elb\", \"main-web-app\") {\n  name             \"main-app-elb\"\n  security_groups  [elb_sg]\n  subnets          [env.subnet]\n  instances        [instance]\n  listener {\n    instance_port     8000\n    instance_protocol \"http\"\n    lb_port           80\n    lb_protocol       \"http\"\n  }\n}\n```\n\nThe GeoEngineer command line tool `geo` can:\n\n1. **Create a plan** with `geo plan -e staging first_project.rb`\n2. **Execute the plan** with `geo apply -e staging first_project.rb`\n3. **Create a graph** with `geo graph -e staging --quiet first_project.rb | dot -Tpng \u003e graph.png \u0026\u0026 open graph.png`\n4. **Status of Codified Resources** with `geo status first_project.rb -e staging`\n5. **Query GPS Resource Graph** with `geo query \"*:*:*:*:*\"`\n\n*There are more examples in the `examples` folder.*\n\n## Customizations\n\nGeoEngineer's DSL can be customized to your needs using validations, GPS and reusable methods on resources.\n\n### Validations\n\nBelow is an example which will add the validation to ensure that all listeners on all ELB's must be HTTPS, for security reasons.\n\n```ruby\nclass GeoEngineer::Resources::AwsElb \u003c GeoEngineer::Resource\n  validate :validate_listeners_must_be_https\n\n  def validate_listeners_must_be_https\n    errors = []\n    all_listener.select{ |i| i.lb_protocol != 'https' }.each do\n      errors \u003c\u003c \"ELB must use https protocol #{for_resource}\"\n    end\n    return errors\n  end\nend\n```\n\n### Geo Planning System (GPS)\n\nGeoEngineer describes resources in the **cloud** domain, not **your** application domain. For example, security group ingress is the \"cloud\" way of defining \"what can call your service\". The friction between these two domains makes communication with others (e.g. developers) difficult.\n\nGPS is an abstraction that helps you describe your cloud in the language of your domain. GPS:\n\n1. **Uses Higher Level Vocabulary** to build configurations.\n1. **Explicit Configurations** means no tricks; What you see is what you get.\n1. **YAML and JSON Schema** to strictly configure using known standards.\n1. **Extensible Configuration** lets GPS express any domain.\n1. **Backwards Compatible**: GPS is built to work with current GeoEngineer resources.\n\nGPS files look like `gps/org/first-project.yml`:\n\n```yml\n\u003cenvironment\u003e:\n  \u003cconfiguration\u003e:\n    \u003cnode_type\u003e:\n      \u003cnode_name\u003e:\n        \u003cattributes\u003e:\n```\n\nThe filename is used to define the project. The `environment` and `configuration` are used to group nodes. Each configuration has multiple nodes, defined under their types. You can define your own node types that can allow multiple `attributes`.\n\nFor example, the file `./gps/org/first-project.yml` describes a node `service` named `api` with configuration `staging` in the `development` environment:\n\n```yml\ndevelopment:\n  staging:\n    service:\n      api:\n        ports: \"80:80\"\n```\n\nIf you have multiple environments and wish something to be applied to all of them evenly, you can use `_default` as a special environment keyword. This will be applied to all known environments, unless they are already defined. For example, if you had a project that was deployed to all environments except one named `internal`, you could use the following example:\n\n```yml\n_default:\n  common:\n    service:\n      api:\n        ports: \"80:80\"\ninternal: {}\n```\n\nThe `service` node type is defined to take a string of ports and build a Load balancer:\n\n```ruby\n# Load Balancer Node\nclass GeoEngineer::GPS::Nodes::Service \u003c GeoEngineer::GPS::Node\n  # explicity define the exposed resources from this node\n  define_resource \"aws_elb\", :elb\n\n  # define the types of attributes using JSON schema\n  def json_schema\n    {\n      \"type\":  \"object\",\n      \"additionalProperties\" =\u003e false,\n      \"properties\":  {\n        \"ports\":  {\n          \"type\":  \"string\",\n          \"default\":  \"80:80\"\n        }\n      }\n    }\n  end\n\n  # called by GPS when creating resources\n  def create_resources(project)\n    create_elb(project) # method created with `define_resource`\n    setup_elb\n  end\n\n  def setup_elb\n    # Set the values of the resource here\n    elb.ports = attributes[\"ports\"]\n  end\nend\n```\n\nTo integrate with a project use:\n\n```ruby\nproject = gps.project(\"org\", \"first-project\", env) do |nodes|\n  # query for api filling in the default env, config, project...\n  nodes.find(\":::service:api\")\nend\n\n# Find the service\n# query syntax is `\u003cproject\u003e:\u003cenvironment\u003e:\u003cconfig\u003e:\u003ctype\u003e:\u003cname\u003e`\nservice = gps.find(\"org/first-project:development:staging:service:api\")\n\n# method to get the GeoEngineer resource ELB\nservice.elb\n\n# method to get the terraform reference to the resource\nservice.elb_ref\n\n# return all service nodes\ngps.where(\"org/first-project:*:*:service:*\").each do |node|\n  node.elb.tags { ... }\nend\n```\n\n### Methods\n\nDefine methods to be used in your own resources, e.g. a custom method to security group to add a rule:\n\n```ruby\nclass GeoEngineer::Resources::AwsSecurityGroup \u003c GeoEngineer::Resource\n  # ...\n  def all_egress_everywhere\n    egress {\n        from_port        0\n        to_port          0\n        protocol         \"-1\"\n        cidr_blocks      [\"0.0.0.0/0\"]\n    }\n  end\n  # ...\nend\n\nproject.resource('aws_security_group', 'all_egress') {\n  all_egress_everywhere # use the method to add egress\n}\n```\n\n## Adding New Resources\n\nThe best way to contribute is to add resources that exist in Terraform but are not yet described in GeoEngineer.\n\nTo define a resource:\n\n0. checkout and fork/branch GeoEngineer\n1. create a file `./lib/geoengineer/resources/\u003cprovider_type\u003e/\u003cresource_type\u003e.rb`\n2. define a class `class GeoEngineer::Resources::\u003cResourceType\u003e \u003c GeoEngineer::Resource`\n3. define `_terraform_id`, and potentially `_geo_id` and `self._fetch_remote_resources` method (more below).\n4. write a test file for the resource that follows the style of other similar resources\n\n### Codified to Remote Resources\n\nA fundamental problem with codifying resources is matching the in code resource to the real remote resource. Terraform does this by maintaining an `id` in a state file which is matched to a remote resources attribute. This attribute is different per resource, e.g. for ELB's it is their `name`, for security groups it is their `group_name` that is generated so cannot be codified.\n\nWithout a state file GeoEngineer uses API's to match resources, this makes generated `id`'s likes security groups difficult. For these generated ids GeoEngineer uses tags e.g. for ELB's the GeoEngineer id is its `name` (just like Terraform) and for security groups it is their `Name` tag.\n\nIn a GeoEngineer resource the `_terraform_id` is the id used by Terraform and the `_geo_id` is GeoEngineer ID. By default a resources `_geo_id` is the same as the `_terraform_id`, so for most resources only the `_terraform_id` is required.\n\nIf `_terraform_id` is generated then the remote resource needed to be fetched via API and matched to the codified resource with `_geo_id`. This is done by implementing the `self._fetch_remote_resources` method to use the API and return a list of resources as an array of hashes each containing keys `_terraform_id` and `_geo_id`, then GeoEngineer will automatically match them.\n\nFor example, in `aws_security_group`'s the resource is matched based on the `Name` tag, implements as:\n\n```ruby\nclass GeoEngineer::Resources::AwsSecurityGroup \u003c GeoEngineer::Resource\n  after :initialize, -\u003e { _terraform_id -\u003e { NullObject.maybe(remote_resource)._terraform_id } }\n  after :initialize, -\u003e { _geo_id -\u003e { NullObject.maybe(tags)[:Name] } }\n\n  def self._fetch_remote_resources(provider)\n    AwsClients.ec2(provider)\n      .describe_security_groups['security_groups']\n      .map(\u0026:to_h).map do |sg|\n        sg[:name] = sg[:group_name]\n        sg[:_terraform_id] = sg[:group_id]\n        sg[:_geo_id] = sg[:tags] ? sg[:tags].select { |x| x[:key] == \"Name\" }.first[:value] : nil\n        sg\n    end\n  end\nend\n```\n\n### Adding a New Provider\n\nAdding resources for a new provider requires creating a new subfolder and resources referencing the provider name in `lib/geoengineer/resources/`. If necessary, utility methods for the new provider client are stored at `lib/geoengineer/utils/`. Once the resources files are defined, no further setup is needed as provider information is pulled in from resource definitions in the project files being planned and applied.\n\n### Validations\n\nTerraform does not validate a lot of attributes before they are sent to the cloud. This means that often plans will fail for reasons that could have been initially validated. When creating a resource think about what validations could be done to ensure a plan is successful.\n\nFor example, a security groups needs a `Name` tag, requires a `name` and `description`, and a more complicated example is that its `cidr_blocks` should be valid:\n\n```ruby\nclass GeoEngineer::Resources::AwsSecurityGroup \u003c GeoEngineer::Resource\n  # ...\n  validate :validate_correct_cidr_blocks\n  validate -\u003e { validate_required_attributes([:name, :description]) }\n  validate -\u003e { validate_has_tag(:Name) }\n\n  def validate_correct_cidr_blocks\n    errors = []\n    (self.all_ingress + self.all_egress).each do |in_eg|\n      next unless in_eg.cidr_blocks\n      in_eg.cidr_blocks.each do |cidr|\n        begin\n          NetAddr::IPv4Net.parse(cidr)\n        rescue NetAddr::ValidationError\n          errors \u003c\u003c \"Bad cidr block \\\"#{cidr}\\\" #{for_resource}\"\n        end\n      end\n    end\n    errors\n  end\n  # ...\nend\n```\n\n### Terraform State\n\nTerraform by default will attempt to sync its resources with the API so that its state file is up to date with the real world. Given that GeoEngineer uses Terraform in a different way this sometimes causes plans to list changes that have already happened.\n\nTo fix this issue a resource can override `to_terraform_state` method, e.g. `aws_db_instance` has issues with `final_snapshot_identifier` updating:\n\n\n```ruby\nclass GeoEngineer::Resources::AwsDbInstance \u003c GeoEngineer::Resource\n  # ...\n  def to_terraform_state\n    tfstate = super\n    tfstate[:primary][:attributes] = {\n      'final_snapshot_identifier' =\u003e final_snapshot_identifier,\n    }\n    tfstate\n  end\n  # ...\nend\n```\n\n## GeoEngineer Reference\n\nThe core models in GeoEngineer are:\n\n```\n +-------------+ 1\n | Environment +-----------+\n +-------------+           |\n       | 1                 |\n       |                   |\n       v *                 v *\n +-----+-------+ 1  * +-------------+ 1  *  +-------------+\n | Project     +-----\u003e+ Resource    +------\u003e+ SubResource |\n +-------------+      +-------------+       +-------------+\n```\n\n\n1. `Environment` contains many resources that may exist outside of a project, like VPCs or routing tables. Also every project defined to be in the environment, for example the `test_www` project is in `staging` but `monorail` is in `staging` and `production` environments.\n2. `Project` contains many resources and services grouped together into a name.\n3. `Resource` and `SubResource` are based off of how terraform models cloud resources. A `Resource` instance can have many `SubResource` instances, but a `SubResource` instance belongs to only one `Resource` instance, e.g. a load balancer resource may have a `health_check` sub-resource to only allow specific incoming ports.\n\nAll these models can have arbitrary attributes assigned to them either by directly assigning on the instance, or through passing a block to the constructor. For example:\n\n```ruby\nresource = Resource.new('type','id') { |res|\n  # CORRECT\n  res.hello = 'hey'\n  puts res.hello # 'hey'\n  hello 'hey again' #\n  puts res.hello # 'hey again'\n\n  # INCORRECT way of assigning variables\n  goodbye = 'nooo' # This assigns a local variable, not an attribute on the resource\n  puts res.goodbye # nil\n}\n\nputs resource.hello # 'hey again'\n\nresource.goodbye = 'see ya'\nputs resource.goodbye # 'see ya'\n```\n\nAdditionally, if the value is expensive to calculate or requires other attributes not yet assigned, an attribute can be assigned a `Proc` or `lambda` which will be calculated lazily:\n\n```ruby\nresource = Resource.new('type','id')\nresource.lazy_attr = -\u003e { puts \"CALCULATING THE VALUE\"; 'value' }\n# ...\nputs resource.lazy_attr\n#$ \"CALCULATING THE VALUE\"\n#$ \"value\"\n```\n\n### Environment\n\nThe top level class in GeoEngineer is the `environment`: it contains all projects, resources and services, and there should only ever be one initialized at a time.\n\nAn environment can mean many things to different people, e.g. an AWS account, an AWS region, or a specific AWS VPC. The only real constraint is that a resource has one instance per environment, e.g. a load balancer that is defined to be in `staging` and `production` environments, will have an instance in each.\n\nThe function `environment` is provided as a factory to build an environment:\n\n```ruby\nenvironment = environment(\"environment_name\") { |e|\n  e.attr_1 = [1,2,3]\n  attr_2 'value'\n}\nenvironment.attr_3 = \"another value\"\n```\n\n### Project\n\nA project is a group of resources typically provisioned to deploy one code base. A project has an `organization` and `name`, to mimic the github `username`/`organiztion` and `repository` structure.\n\nA project is defined like:\n\n```ruby\nproject = project('org', 'project_name') {\n  environments 'staging', 'production'\n}\n```\n\nThis projects organization is `org`, its name `project_name` and will be provisioned in the `staging` and `production` environments. The `org` and `name` must be unique across all other projects.\n\nThe method `project` will automatically add the project to the instantiated environment object **only if** that environment's name is in the list of environments, otherwise it is ignored.\n\n### Resources and SubResources\n\nResources are defined to be similar to the [terraform resource](https://www.terraform.io/docs/configuration/resources.html) configuration. The main difference is to not use `=` as this will create a local ruby variable and not assign the value.\n\nA `Resource` can be created with and `environment` or `project` object (this will add that resource to that object):\n\n```ruby\nenvironment.resource('type', 'identifier') {\n  name \"resource_name\"\n  subresource {\n    attribute \"attribute\"\n  }\n}\n\nproject.resource('type', 'identifier') {\n  # ...\n}\n```\n\nThe `type` of a resource must be a valid terraform type, where AWS types are listed [here](https://www.terraform.io/docs/providers/aws/index.html). Some resources are not supported yet by GeoEngineer.\n\n`identifier` is used by GeoEngineer and terraform to reference this resource must be unique, however it is not stored in the cloud so can be changed without affecting a plan.\n\nA resource also has a ruby block sent to it that contains parameters and sub-resources. These values are defined by terraform so for reference to what values are required please refer to the [terraform docs](https://www.terraform.io/docs/providers/aws/index.html).\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fgeoengineer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoinbase%2Fgeoengineer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoinbase%2Fgeoengineer/lists"}