Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/linkorb/infra

🚀 INFRA: your infrastructure as a GraphQL service
https://github.com/linkorb/infra

ansible devops firewall-rules graphql infra infrastructure monitoring

Last synced: about 14 hours ago
JSON representation

🚀 INFRA: your infrastructure as a GraphQL service

Awesome Lists containing this project

README 

        



## Introduction



Infra allows you to define all objects in your infrastructure in a way that you can load it

into a GraphQL service. For example: Hosts, HostGroups, Users, Dns, Monitoring, BackupRules, Deployments, etc, and all of their relationships.



Accessing your infrastructure as a graph allows you to do a couple of cool things:



* Generate multiple inventory definitions from a single source (ansible, puppet, rundeck, etc)

* Generate configuration files for backup and monitoring services

* Generate complex firewall scripts

* Configure objects in context. I.e. configure dns records as part of your Deployment, or monitoring rules as part of your HostGroup

* Generate dashboard configurations based on infra objects

* Generate comprehensive infrastructure documentation

* Create scripts that operate on your infrastructure data in language agnostic ways

* As your configuration consists of YAML files only, everything is heavily scriptable (for both importing and exporting data from all relevant systems) and managable through version control.



## Resources



You define your infrastructure as a set of Resources in YAML files. The format is heavily inspired by Kubernetes. Here's an example:



```yaml

---

kind: Host

metadata:

  name: my-app-server

  description: This my application server

spec:

  os: Ubuntu 16.04

  publicIp: 192.168.1.1

  privateIp: 10.0.1.1

  hostGroups: app-servers

---

kind: HostGroup

metadata:

  name: app-servers

  description: All my application server

  labels:

    color: green

spec: ~

```



For a more comprehensive example, see the `example/` directory in this repository.



You can define your resources in standalone YAML files, or (as in this example) configure multiple resources in one file using YAML's [multiple document feature](https://en.wikipedia.org/wiki/YAML#Advanced_components)



If you're familiar with Kubernetes, you'll feel right at home.



Each resource specifies it's type using the `kind` key. A list of available resource types is provided below.



Resources contain a `metadata` key that allows you to specify the resource name, description, a set of labels and annotations (more on metadata below).



The `spec` key configures the resource, and it's available keys depend on the `kind` of resource you're creating.



## Installation



    cd /opt

    git clone [email protected]:linkorb/infra.git

    cd infra

    composer install # See https://getcomposer.org/download/ if you don't have composer installed yet



**NOTE:** It is **strongly** recommended to add `bin/` to your environment's `PATH` variable, so you can easily invoke the `infra` command-line tool from anywhere on your system.



If you're using Bash, you can do this by adding the following line to your `~/.bashrc` file:



    PATH=/opt/infra/bin:$PATH



Don't forget to open a new bash session for the changes to take effect (close your terminal, or just run `bash` again).



## Configuration



Infra decides where to load it's infrastructure data based on the `INFRA_CONFIG` environment variable. It should contain a path to directory that contains a set of YAML files describing your infrastructure.



You can define the variable wherever you like. For example in your `~/.bashrc`, or a `.env` file in the root of the infra directory (i.e. `/opt/infra/.env`). A `.env.dist` file is provided.



If `INFRA_CONFIG` is undefined, it will point to the `example/` directory as a default.



## Example project



The `example/` directory contains an example infrastructure configuration (You can view `example/README.md` for it's details). To load it, make sure your `INFRA_CONFIG` variable points to the `example/` directory (absolute path required), or simply leave `INFRA_CONFIG` undefined: the example project is loaded by default.



It is recommended to explore the example project first, before using infra to configure your own infrastructure. The example configuration will contain examples for all relevant resource types, configured in a sensible way to show you all available functionality.



Once you're ready to define your own infrastructure, simply point `INFRA_CONFIG` to a new directory (ideally a git repository), and start creating configuration files there.



## Usage



You can get a list of available commands by simply typing `infra list` or just `infra`.



### Exploring using the `get` command.



The `infra get` command lets you explore your infrastructure configuration. Type `infra get --help` for it's options. All arguments are optional, and the more you specify, the more specific your response will be. Some examples:



* `infra get`: returns a list of available resource types (and their aliases)

* `infra get hosts`: returns all defined objects of that resource type. 

* `infra get hg`: you can use any of the available resource type aliases as shortcuts: hg, hostGroup, HostGroup, hostGroups, HostGroups are all equivalent here.

* `infra get Host app0`: returns a resource definition by type and name.

* `infra get HostGroup db hosts`: returns a calculated property from a resource by type, name and propertyname.

 

### Running GraphQL queries



The `infra query` command let's you perform a GraphQL query against your infrastructure configuration.

It reads the query from `stdin` and outputs the response (as json) to `stdout`.



For example:



    infra query < /opt/infra/example/graphql/example.graphql



Introspection is also enabled, so you can execute the following command to get a full export of all supported types with detailed information about available fields:



    infra query < /opt/infra/example/graphql/introspection.graphql



## Listing hosts (host expansion algorithm)



The `infra host:list ` allows you to list a set of hosts. The list you can provide is quite flexible. It's using the host expansion algorithm which is used in all places in infra where you can specify a set of hosts. Some of the command-line utilities allow you to specify a list of hosts, but also linking to hosts from `spec` sections in your resource definitions follow this algorithm. This command helps you to test this. Some examples:



* `app0`: (i.e., a host name) returns an list containing only the app0 host

* `db`: (i.e., a hostgroup name) returns an list containing all hosts in this hostgroup.

* `app0,db0m`: (i.e., two host name) returns a list containing these specific hosts.

* `app0,db`: (i.e., a host name and a host group name) returns a list containing hosts in the `db` host group, plus the specific `app0` host.

* `db0m,db`: (i.e., a host group name and a host name already in that host group) returns a list containing hosts in the `db` host, not listing the host `db0m` twice.

* `prod-east`: (i.e., a child host group name of `prod`) returns a list in the `prod-east` host group.

* `prod-west`: (i.e., a child host group name of `prod`) returns a list in the `prod-west` host group.

* `prod`: (i.e., a parent host group name) returns a list of all hosts linked to the `prod` host group directly, or through one of it's child host groups (`prod-east` and `prod-west`).



As you can see, you can freely mix and match host names, host group names, and take advantage of hierarchy in host groups.



## Executing commands in bulk over SSH



The `infra host:exec  ` command lets you perform a standard shell command one one or more hosts in bulk.



The first argument is the list of hosts you want to perform the command on (following the *host expansion algorithm* described earlier), and the second argument is the actual command. 



## Generating firewall rules



Infra can generate complex firewall configurations based on very simple rule defintions. For example, it allows you to set up a rule that allows access to a specified port from all hosts in group X to all hosts in group Y over public or private IP addresses. It will dynamically generate the required rules based on your infra graph. You can leverage the full power of the host expansion algorithm (described earlier) to specify source and remote target host lists.



Run `infra firewall:show ` (or short-hand notation `infra f:s `) to let infra generate firewall rules for the specified host.



Running `infra f:s db0m` will output the generated iptables rules to the console like this:



```

#### iptables boilerplate removed for brevity ####



# rule=allow-mysql hosts='db' remoteHosts='app'

-A INPUT -d 10.0.2.1 -s 10.0.1.1 -p tcp --dport 3306 -j ACCEPT -m comment --comment "host='db1m' remoteHost='app1' rule='allow-mysql'"

-A INPUT -d 10.0.2.1 -s 10.0.1.2 -p tcp --dport 3306 -j ACCEPT -m comment --comment "host='db1m' remoteHost='app2' rule='allow-mysql'"



# rule=allow-ssh hosts='prod' remoteHosts=''

-A INPUT -d 10.0.2.1 -s 192.168.99.99 -p tcp --dport 22 -j ACCEPT -m comment --comment "host='db1m' rule='allow-ssh'"



COMMIT

```



Compare these to the definitions in `example/resources/FirewallRule/`;



Note how you'll find rules defined for both the "db" hostgroup (of which db0m is a direct member), and rules for the "prod" hostgroup (of which db0m is only indirectly a member through the "prod" child hostgroup "prod-east").



Additionally, you'll notice that the db0m server has 2 rules generated for the 2 app servers (app1 and app2). In case you'll add 5 more app servers, infra will automatically generate a rule for each of them (7 in total).



## Installing firewall rules



Run `infra firewall:install ` to install the firewall. Infra will ssh into the host(s) and:



1. backup the current iptable rules using `iptables-save`. The backup is created in /tmp, and the exact filename will be displayed on the console.

2. Copy the firewall rules to the server

3. Run `iptables-restore` on the rules file and output the results.



## Scripts (in your language of choice)



If you'd like to build tools that leverage your infrastructure graph in your language of choice (php, node, python, ruby, bash, etc), scripts are your solution.



### Creating your own script



To create your own custom script:



1. Add an executable script in the `scripts/` directory of your infrastructure configuration repository

2. Make sure to `chmod 755` the file, as only executable files are detected by infra

3. Let your script execute GraphQL queries using the `infra query` command (pass in the query over stdin, getting the response json from stdout)

4. Use the returned data to apply any logic you see fit: export to common config formats, call a set of APIs, generate documentation, etc.



### SDKs



An SDK is not required, but may make your life as an infra script developer more convenient. Currently available SDKs:



* https://github.com/linkorb/infra-sdk-php



Let us know if you've created an SDKs for other languages, we'll be glad to link to it from this section.



### Example scripts



Examples are included in the `example/scripts` directory. (examples in more languages welcomed as PRs!)



### Core scripts



A set of "core" scripts are included in the `scripts/` directory of this repository. The core scripts are for commonly used applications that will be of use to many infra users. If you've created a reusable script that you'd like to be included in the core scripts, be sure to send a PR!



## Todo



* [x] Command: Add `infra script` command for listing and executing discovered scripts

* [ ] Command: Add `infra serve` command that spins up a GraphQL server

* [ ] Feature: Allow in-line resources in order to define resources as part of their container (i.e. DnsRecord as part of Deployment etc)

* [ ] Feature: Allow variable expansion in resource configuration (i.e. to access parent container variables)

* [ ] Feature: Set resource index value (incremental, from 1) and output in resource listings. Allow to fetch resource by index. This should simplify exploring using the `get` command.

* [x] Feature: Create minimal PHP SDK for scripts to simplify querying

* [x] Feature: script discovery

* [ ] Feature: infra manifest file (for base variables, directory references, bulk script commands, etc)

* [ ] Feature: support wildcards in host expansion algorithm (i.e. by using fnmatch)

* [x] Refactor: extract sensu commands into scripts

* [ ] Refactor: extract ansible commands into scripts

* [ ] Refactor: Loader in external class

* [ ] Refactor: SSH or other infra-specific methods in external services (so Infra becomes general resource container)

* [ ] Script: Add script to generate HTML documentation from infra

* [ ] Script: Add script to fetch host facts

* [ ] Script: Add grafana dashboard generator 

* [x] Script: Add DNS zone publisher

* [x] Script: Add Rundeck inventory generator

* [ ] Feature: Add option to augment loaded resources with labels (i.e. for host facts)

* [ ] Resource: BackupRule + script for BackupPC, rsync, or similar

* [ ] Resource: CronJob + conf.d/infra file generator

* [ ] Resource: DockerEngine, DockerApp. With docker-compose YAML as inline spec

* [ ] Resource: Secret

* [ ] Resource: DatabaseCluster or MysqlCluster to setup and monitor replication

* [ ] Resource: Database or MysqlDatabase

* [ ] Resource: Ingress or NginxSite or ApacheVhost for ingress configuration

* [ ] Resource: GitRepository

* [ ] Resource: Deployment / Ansistrano / Capistrano / DeployerPHP configurations

* [ ] Resource: File to manage deployment configuration files



## License



MIT. Please refer to the [license file](LICENSE) for details.



## Brought to you by the LinkORB Engineering team






Check out our other projects at [linkorb.com/engineering](http://www.linkorb.com/engineering).



Btw, we're hiring!