Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/get-bridge/muss

For when your docker-compose projects are a mess
https://github.com/get-bridge/muss

internal

Last synced: 3 months ago
JSON representation

For when your docker-compose projects are a mess

Awesome Lists containing this project

README 

          
[![Build Status](https://travis-ci.org/instructure/muss.svg?branch=master)](https://travis-ci.org/instructure/muss)



# muss



Definition:



> noun: mess; scramble; a confused conflict

> verb: to put into disorder; to make messy



Or if you prefer, an acronym:



> Manage User Selected Services



`muss` is a command line application to allow users to choose what services

they want to run and how they want to run them.



`muss` is a wrapper around `docker-compose`.

When you run a `muss` subcommand it will:



- process (and validate) the config files

- generate a `docker-compose.yml`

- prepare any defined bind mounts (volumes)

- load secrets into the environment

- delegate to `docker-compose`



Projects (and their dependent services) can be configured

in a familiar style through a series of yaml files.



# Installation



    brew tap instructure/muss

    brew install muss



# Building, Testing, etc



`muss` is a CLI written in go.



`make install` to install it into your `$GOPATH`.



You can use `make test` to test all of the included go packages

or you can run any `go` commands directly (`go test ./cmd`).



# Usage



`muss help` will describe available subcommands.



Many docker-compose commands are supported directly

so that config files are always up to date and secrets

are loaded into the environment.



The simplest possible usage is to call `muss up`

to start all the services and type Ctrl-C to shut it down.



    $ muss help

    Configure and run project services



    Usage:

      muss [command]



    Available Commands:

      attach      Attach local stdio to a running container

      build       Build or rebuild services

      config      muss configuration

      dc          Call aribtrary docker-compose commands

      down        Stop and remove containers, networks, images, and volumes

      exec        Execute a command in a running container

      help        Help about any command

      logs        View output from services

      ps          List containers

      pull        Pull the latest images for services

      restart     Restart services

      rm          Remove stopped containers

      run         Run a one-off command

      start       Start services

      stop        Stop services

      up          Create and start containers

      version     Show version information

      wrap        Execute arbitrary commands



    Flags:

      -h, --help   help for muss



    Use "muss [command] --help" for more information about a command.



## Advanced Usage



For less common docker-compose commands that muss does not define

you can use the `dc` subcommand, e.g. `muss dc events --json`.



You can also run any arbitrary commands using `muss wrap`.

This allows you to run a command after files have been generated and

the environment has been loaded.



muss has its own `config` subcommand (different from the docker-compose

config command).



`muss config save` will generate the files.  This happens before running other

commands (like `muss up`) but this can be useful if you just want to inspect the

files.



`muss config show` will print out the whole configuration.  The `--format`

parameter takes a go template string to allow you to limit or manipulate the

config (useful for scripting and debugging).



# Configuration



A muss project is configured via a series of yaml files:



1. project config

2. user config

3. module definitions



The project configuration defines:



- default preferences for module configs

- secret commands

- status command

- the locations of additional config files

- docker-compose settings



This file should be committed into source control.



A muss user file allows you to specify preferences and customizations:



- specify a different module order than the project's default

- select specific module configs

- disable modules that you don't intend to use

- a compose override map to be merged in at the end



This file should be ignored by version control.



The project file can include the location of module definition files,

each of which:



- has a name

- defines one or more config options for how to run the service

  (example use cases would be local repos, pre-baked docker images,

  or remote services).



## Project Config



The syntax and options for the main `muss.yaml` file:



```yaml

    ---

    # Path to user customization file.

    user_file: muss.user.yaml



    # If present will set COMPOSE_PROJECT_NAME (unless already set).

    project_name: myproject



    # Alternate target for compose config (default is docker-compose.yml).

    # If present will set COMPOSE_FILE (unless already set).

    # Note that when COMPOSE_FILE is set docker-compose will not automatically

    # use docker-compose.override.yml.

    # The override section of the muss user file will still work, however.

    compose_file: "docker-compose.muss.yml"



    # Define the order of which configuration option to use

    # for any module that has multiple options.

    default_module_order:

      - registry

      - repo

      - internal



    # Module files are yaml files containing module definitions.

    module_files:

      - ./dev/database.yml

      - ./dev/microservice/service.yml



    # Secret commands define aliases that can be used by module definitions.

    secret_commands:

      vault:



        # Arguments will be prepended to the secret arguments.

        exec: ["vault", "kv", "get", "-field"]



        # Env Commands will be run once to setup the environment

        # if any secrets are requested.

        env_commands:

          # When you specify a varname the output of the command

          # will be set in that environment variable.  If that variable

          # is already set in the environment the command will not be run.

          - varname: VAULT_TOKEN

            exec: ["bin/vault-token"]



    # A passphrase is required for local caching of the secrets.

    # Use an env var representing your auth token.

    secret_passphrase: $VAULT_TOKEN



    # A status line will be fixed to the bottom of the screen during "up".

    status:

      # Stdout from this command will appear in the status line.

      exec: ["bin/muss-status"]

      # Each line of status output will be formatted with this spec:

      line_format: "# %s"

      # Specify how often to execute the command to update the status:

      interval: 5s

```



## User Config



Users can customize what they want to run with a user file:



```yaml

    ---

    # Users can set their own order for which module config to use.

    # This list will be used before the default_module_order of the

    # project config.

    module_order:

      - remote

      - repo



    # Modules can also be chosen specifically (to override order lists)

    modules:

      microservice:

        config: remote



      # ...or removed completely.

      stats:

        disabled: true



    # An override section can be defined that will be merged onto the

    # docker-compose config.  By defining it here, muss extensions (like file

    # volumes) can be utilized.

    override:

      services:

        app:

          environment:

            HOW_I_LIKE_IT: nifty

```



## Module Definitions



Module files contain module definitions that will generate chunks of a

docker-compose file (which will all be merged on top of each other).



A module can define multiple configs.

The names of the configs are arbitrary;

These are the values used in the

`module_order` and `default_module_order` lists.

Use common names among your modules so that the

order options apply as consistently as possible.



In the example below:

- "local" implies "run from local directory"

- "registry" implies "an image from a docker registry"

- "edge" and "staging" are different options for external integrations



When multiple configs are defined

the option will be chosen in this order:

- `MUSS_MODULE_ORDER` env var (split on commas)

- a specific user choice

- the first of any `module_order`

- the first of any `default_module_order`



The body of a module config can contain the following:

- "include" is a list of items to merge in before the rest of the map

and can be:

  - a string referring to another config name

  - a map of `file: path` to read in a file (relative to muss.yaml)

- "secrets" is a list of secrets to load

- "services" is a subset of the "services" section of a docker-compose

  configuration... it will be passed through.

- "volumes" is also just a piece of docker-compose syntax that will be passed.



```yaml

    ---

    # Module name.

    name: microservice



    configs:



      # Configs with a leading underscore are private/internal

      # and can be used as common bases for merging.

      _base:



        # The "services" map will be merged into the final docker-compose yaml.

        # We define attributes of multiple services here that will all be merged

        # on top of each other (the way a docker-compose override file works).

        services:

          app:

            environment:

              # The "app" service (defined more fully elsewhere)

              # will only receive this environment variable

              # if this service definition is enabled.

              MICROSERVICE_ENABLED: '1'

          microservice:

            environment:

              APP_ENV: 'development'



      local:

        include:

          # A config can start by including other maps that will be merged in first.

          # This can be the name of another config

          - _base

          # or the contents of another yaml file.

          - file: compose-snippet.yml



        # Any "volumes" will also pass to docker-compose.

        volumes:

          microservice-data: {}

        services:

          microservice:

            # Paths will be relative to the project root.

            build: ../microservice

            volumes:

              - microservice-data:/var/lib/microservice



      registry:

        include:

          - _base

        services:

          microservice:

            image: our-registry/microservice



      _remote:

        services:

          # This service can define how another service will reach it.

          app:

            environment:

              MICROSERVICE_URL:

              MICROSERVICE_KEY:

          # Note that there is no "microservice" here.

          # By pointing to a remote instance

          # we have eliminated the need to run it locally.



      edge:

        include:

          - _base

          - _remote

        # A config can define secrets that will be loaded into the environment.

        secrets:

          MICROSERVICE_URL: {vault: ["MICROSERVICE_URL", "app/edge/common"]}

          MICROSERVICE_KEY: {vault: ["MICROSERVICE_KEY", "app/edge/common"]}



      staging:

        include:

          - _base

          - _remote

        secrets:

          MICROSERVICE_URL: {vault: ["MICROSERVICE_URL", "app/staging/common"]}

          MICROSERVICE_KEY: {vault: ["MICROSERVICE_KEY", "app/staging/common"]}

```



Any "services" defined that do not contain a "build" or an "image" will not be

included in the final docker-compose file.  This allows one service to define

what secrets are available for another service even if that service won't be

used in the end.



Service integration can now be organized around how services will be selected.

All of the docker-compose configuration for a given service can be placed in the

same module file, including not only that service (or multiple services)

but also how it integrates with another service.



For example, if a microservice can be local, remote, or completely optional,

the definition file can point the environment variables for other services

to this one.  That way they update according to which ever option is chosen,

or the service can be completely disabled and the other services won't receive

the environment variables at all.



As another example, you could have a stats container that collects and prints

stats from all other services.  If the stats service file defines not only the

stats service but also the environment variables for all the other services

that will send stats to it, then you can simply disable the stats service

from your user config file and all the other services will no longer be

configured to send any stats.



# Secrets



Module definitions can specify secrets to be loaded by external commands.



The project configuration defines aliases that simplify the usage

and define commands that setup the environment

(for example, logging in to vault and returning the token).



When a module config is chosen that includes secrets:



- the setup commands are executed and the environment is populated

- the individual secret commands are then run and added as environment variables

- muss then delegates to the subcommand (docker-compose, etc).



Secrets are cached and encrypted with the `secret_passphrase`.

So if you use your auth token as your passphrase the secrets will be cached

for as long as your token is valid.  When you get a new token it will force

fetching new secrets.



Secret commands can either specify a `varname` and the STDOUT of the script

will be assigned to that var.

Alternatively the commands can specify: `parse: true`

and the output will be parsed as lines of `NAME=VALUE`.



STDIN and STDERR will pass directly so that users can response to password

prompts and see errors.



To provide a more concrete example:



`muss.yaml`:



    secret_commands:

      vault:

        exec: ["vault", "kv", "get", "-field"]

        env_commands:

          - exec: ["bin/vault-token"]

            parse: true

        # The passphrase will be parsed and env vars will be interpolated.

        passphrase: $VAULT_TOKEN

        # The "cache" value can be "passphrase" (the default)

        # "none" to disable caching, or a duration ("24h", "168h")

        # to expire the cache (if the passphrase hasn't already changed by then).

        cache: "passphrase"

    # You can set a global passphrase that will be used for any secrets

    # that do not define their own.

    secret_passphrase: $VAULT_TOKEN

    module_files:

      - dev/microservice.yml



The exec command can be something global

but will often be a command that is part of your project.



The `bin/vault-token` script would probably do something like this:



    #!/bin/bash



    # Setup any necessary env like (perhaps VAULT_ADDR).

    export VAULT_ADDR=...



    if ! token-is-valid; then

      vault login ...

      # persist the token and expiration so that next time

      # the script runs it won't have to login again.

    fi



    # These lines will be parsed and set in the environment.

    echo VAULT_ADDR="$VAULT_ADDR"

    echo VAULT_TOKEN="$(< ~/.vault-token )"



The actual secrets in the module definition (`dev/microservice.yml`):



    name: microservice

    configs:

      somewhere-far:

        secrets:

          SECRET_KEY: {vault: ["MICROSERVICE_KEY", "path/to/key"]}

        services:

          app:

            environment:

              SECRET_KEY: # no value, it will get it from the environment.



So when the "somewhere-far" option is chosen for the module

it will load those secrets.



First `bin/vault-token` will be executed and `VAULT_ADDR` and `VAULT_TOKEN`

will be set in the environment.



Then `vault kv get -field MICROSERVICE_KEY path/to/key` will be executed

and the value stored in the `SECRET_KEY` environment variable.



Then muss will continue and delegate to `docker-compose` to run your services

and the populated environment variables will be passed along.



# Additional Behavior



A few additional behaviors are defined beyond the normal docker-compose

features.



## Volumes



When bind mounts (host volumes) are specified muss will attempt to ensure

that they already exist.  This helps prevent permission problems that can occur

by letting the docker daemon create a directory under your project directory.



## File volumes



Additionally volumes can be specified to be files instead of directories

which can be useful for mounting config files into the container.

Just use the long syntax for volumes and add `file: true`:



    volumes:

      - target: /home/docker/.somerc

        source: ~/.somerc

        file: true



This way if the specified path isn't already a file muss will create it

so that docker doesn't make it a directory and cause confusing errors later.



The `file: true` will be removed from the resulting docker-compose file.