{"id":13879389,"url":"https://github.com/bobf/orchestration","last_synced_at":"2025-07-16T15:32:21.943Z","repository":{"id":33783350,"uuid":"158604101","full_name":"bobf/orchestration","owner":"bobf","description":"Toolkit for consistent and easy Docker orchestration","archived":false,"fork":false,"pushed_at":"2024-05-31T19:38:57.000Z","size":887,"stargazers_count":9,"open_issues_count":11,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-18T01:55:07.748Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bobf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-21T20:44:59.000Z","updated_at":"2024-05-31T19:39:00.000Z","dependencies_parsed_at":"2023-02-11T22:16:08.491Z","dependency_job_id":"f1a56dfa-ecb5-4480-8c1b-a22361898c78","html_url":"https://github.com/bobf/orchestration","commit_stats":{"total_commits":360,"total_committers":2,"mean_commits":180.0,"dds":"0.011111111111111072","last_synced_commit":"b608ed388bceae818182220c7535adeee1b3b8f0"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobf%2Forchestration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobf%2Forchestration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobf%2Forchestration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bobf%2Forchestration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bobf","download_url":"https://codeload.github.com/bobf/orchestration/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226143895,"owners_count":17580245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-06T08:02:19.360Z","updated_at":"2024-11-24T08:31:20.425Z","avatar_url":"https://github.com/bobf.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Orchestration\n\n_Orchestration_ aims to provide a convenient and consistent process for working with _Rails_ and _Docker_ without obscuring underlying components.\n\nAt its core _Orchestration_ is simply a `Makefile` and a set of `docker-compose.yml` files with sensible, general-purpose default settings. Users are encouraged to tailor the generated build-out to suit their application; once the build-out has been generated it belongs to the application.\n\nA typical _Rails_ application can be tested, built, pushed to _Docker Hub_, and deployed to _Docker Swarm_ with the following commands:\n\n```bash\nmake setup test build push\nmake deploy manager=user@swarm.example.com env_file=/var/configs/myapp.env\n```\n\n_Orchestration_ has been successfully used to build continuous delivery pipelines for numerous production applications with a wide range or requirements.\n\nSee [upgrade guide](UPGRADE.md) if you are upgrading from `0.5.x` to `0.6.x`.\n\n## Example\n\nThe below screenshot demonstrates _Orchestration_ being installed in a brand new _Rails_ application with support for _PostgreSQL_ (via the _PG_ gem) and _RabbitMQ_ (via the _Bunny_ gem):\n![example](doc/images/example.png \"https://hub.docker.com/r/rubyorchestration/sampleapp/tags\")\n\n## Getting Started\n\n[_Docker_](https://www.docker.com/get-started) and [_Docker Compose_](https://docs.docker.com/compose/install/) must be installed on your system.\n\n### Install\n\nAdd _Orchestration_ to your `Gemfile`:\n\n_Ruby 3.x_:\n\n```ruby\ngem 'orchestration', '~\u003e 0.7.9'\n```\n\n_Ruby 2.x_:\n\n```ruby\ngem 'orchestration', '~\u003e 0.6.12'\n```\n\nInstall:\n\n```bash\nbundle install\n```\n\n### Setup\n\nGenerate configuration files and select your deployment server:\n\n#### Generate build-out\n\n```bash\nrake orchestration:install server=unicorn # (or 'puma' [default], etc.)\n```\n\nTo rebuild all build-out at any time, pass `force=yes` to the above install command.\n\nYou will be prompted to enter values for your _Docker_ organisation and repository name. For example, the _organisation_ and _repository_ for https://hub.docker.com/r/rubyorchestration/sampleapp are `rubyorchestration` and `sampleapp` respectively. If you are unsure of these values, they can be modified later by editing `.orchestration.yml` in the root of your project directory.\n\nOverride these values from the command line by passing `project` and `organization` variables:\n\n```bash\nrake orchestration:install project=myapp organization=myorg\n```\n\n#### Configuration files\n\n_Orchestration_ generates the following files where appropriate. Backups are created if a file is replaced.\n\n* `config/database.yml`\n* `config/mongoid.yml`\n* `config/rabbitmq.yml` (see [RabbitMQ Configuration](#rabbitmq-configuration) for more details)\n* `config/redis.yml` (see [Redis Configuration](#redis-configuration) for more details)\n* `config/unicorn.rb`\n* `config/puma.rb`\n\nYou may need to merge your previous configurations with the generated files. You will only need to do this once.\n\nTest and development dependency containers bind to a randomly-generated [at install time] local port to avoid collisions. You may compare e.g. `orchestration/docker-compose.test.yml` with the `test` section of the generated `config/database.yml` to see how things fit together.\n\nWhen setup is complete, add the generated build-out to _Git_:\n\n```bash\ngit add .\ngit commit -m \"Add Orchestration gem\"\n```\n\n## Usage\n\nAll `make` commands provided by _Orchestration_ (with the exception of `test` and `deploy`) recognise the `env` parameter. This is equivalent to setting the `RAILS_ENV` environment variable.\n\ne.g.:\n```\n# Stop all test containers\nmake stop RAILS_ENV=test\n```\n\nThe default value for `env` is `development`.\n\nAs with any `Makefile` targets can be chained together, e.g.:\n```\n# Run tests, build, and push image\nmake setup test build push\n```\n\n### Containers\n\nAll auto-detected services will be added to the relevant `docker-compose.\u003cenvironment\u003e.yml` files at installation time.\n\n#### Start services\n\n```bash\nmake start\n```\n\n#### Stop services\n\n```bash\nmake stop\n```\n\n#### Interface directly with `docker-compose`\n\n```bash\n$(make compose RAILS_ENV=test) logs -f database\n```\n\n### Images\n\nImage tags are generated using the following convention:\n\n```\n# See .orchestration.yml for `organization` and `repository` values.\n\u003corganization\u003e/\u003crepository\u003e:\u003cgit-commit-hash\u003e\n\n# e.g.\nacme/anvil:abcd1234\n```\n\n#### Build an application image\n\nNote that `git archive` is used to generate the build context. Any uncommitted changes will _not_ be included in the image.\n```\nmake build\n```\n\nThe `include` option can also be passed to provide a manifest file. Any files listed in this file will also be built into the _Docker_ image. Files **must** be located within the project directory.\n\n```bash\nmake build include=manifest.txt\n```\n\n```bash\n# manifest.txt\ndoc/api/swagger.json\ndoc/api/soap.xml\ndoc/api/doc.html\n```\n\nSee also [build environment](#build-environment) if you use gems hosted on private _GitHub_/_Bitbucket_ repositories.\n\n#### Push latest image\n\nYou must be logged in to a _Docker_ registry. Use the `docker login` command (see [Docker documentation](https://docs.docker.com/engine/reference/commandline/login/) for further reference).\n\n```\nmake push\n```\n\n### Development\n\nA [`.env` file](https://docs.docker.com/compose/env-file/) is created automatically in your project root. This file is _not_ stored in version control. Set all application environment variables in this file.\n\n#### Launching a development server\n\nUse vanilla _Rails_ commands to load a server or console. If present, a `.env` file will be read and loaded (via the [dotenv](https://github.com/bkeepers/dotenv) gem) to populate the environment.\n\n```bash\nbundle exec rails s # Launch a server\nbundle exec rails c # Launch a console\n```\n\n### Testing\n\nA default `test` target is provided in your application's main `Makefile`. You are encouraged to modify this target to suit your application's requirements.\n\nTo launch all dependency containers, run database migrations, and run tests:\n```bash\nmake setup test\n```\n\nThe default `test` command can (and should) be extended. This command is defined in the root `Makefile` in the project and, by defaults, runs `rspec` and `rubocop`.\n\nTo skip the setup step and just run tests (i.e. once test containers are up and running and ready for use) simply run:\n\n```bash\nmake test\n```\n\nNote that _Orchestration_ will wait for all services to become fully available (i.e. running and providing valid responses) before attempting to run tests. This is specifically intended to facilitate testing in continuous integration environments.\n\n_(See [sidecar containers](#sidecar-containers) if you are running your test/development server inside _Docker_)_.\n\nDependencies will be launched and then tested for readiness. The retry limit and interval time for readiness tests can be controlled by the following environment variables:\n\n```\nORCHESTRATION_RETRY_LIMIT # default: 15\nORCHESTRATION_RETRY_INTERVAL # default: 10 [seconds]\n```\n\n### (Local) Deployment\n\nRun a deployment environment locally to simulate your deployment platform:\n\n```bash\nmake deploy manager=localhost\n```\n\nEnsure you have passwordless _SSH_ access to your own workstation and that you have approved the host authenticity/fingerprint.\n\n#### Deploy to a remote swarm\n\nTo connect via _SSH_ to a remote swarm and deploy, pass the `manager` parameter:\n```bash\nmake deploy manager=user@manager.swarm.example.com\n```\n\nThe file `orchestration/docker-compose.deployment.yml` is created automatically. This file will be used for all deployments, regardless of _Rails_ environment. Other environments should be configured using a separate [`.env` file](#env-file) for each environment. i.e. to deploy a staging environment, create a `staging.env` (for example), set `RAILS_ENV=staging` and run:\n```bash\nmake deploy manager=user@manager.swarm.example.com env_file=staging.env\n```\n\nThis way you can set different publish ports and other application configuration variables for each stage you want to deploy to.\n\n#### Roll back a deployment\n\nRoll back the `app` service of your stack:\n```\nmake rollback manager=user@manager.swarm.example.com\n```\n\nRoll back a specific service:\n```\nmake rollback manager=user@manager.swarm.example.com service=database\n```\n\nThe `project_name` parameter is also supported.\n\n\n#### Use a custom stack name\n\nThe [_Docker_ stack](https://docs.docker.com/engine/reference/commandline/stack/) name defaults to the name of your repository (as defined in `.orchesration.yml`) and the _Rails_ environment, e.g. `anvil_staging`.\n\nTo override this default, pass the `project_name` parameter:\n```\nmake deploy project_name=custom_stack_name\n```\n\nThis variable will also be available as `COMPOSE_PROJECT_NAME` for use within your `docker-compose.yml`. e.g. to explicitly name a network after the project name:\n\n```yaml\nnetworks:\n  myapp:\n    name: \"${COMPOSE_PROJECT_NAME}\"\n```\n\n#### Use a custom `.env` file\n\u003ca name=\"env-file\"\u003e\u003c/a\u003e\n\nSpecify a path to a local `.env` file (see [Docker Compose documentation](https://docs.docker.com/compose/environment-variables/#the-env-file)):\n```\nmake deploy env_file=/path/to/.env\n```\n\nNote that the following two variables _must_ be set in the relevant `.env` file (will look in the current working directory if no path provided):\n\n```\n# Published port for your application service:\nPUBLISH_PORT=3000\n\n# Number of replicas of your application service:\nREPLICAS=5\n```\n\nIt is also recommended to set `SECRET_KEY_BASE`, `DATABASE_URL` etc. in this file.\n\n## Logs\n\nThe output from most underlying components is hidden in an effort to make continuous integration pipelines clear and concise. All output is retained in `log/orchestration.stdout.log` and `log/orchestration.stderr.log`. i.e. to view all output during a build:\n\n```bash\ntail -f log/orchestration*.log\n```\n\nA convenience `Makefile` target `dump` is provided. The following command will output all consumed _stdout_, _stderr_, and _Docker Compose_ container logs for the test environment:\n\n```bash\nmake dump RAILS_ENV=test\n```\n\nAll commands also support the `verbose` flag which will output all logs immediately to the console:\n```bash\nmake build verbose=1\n```\n\n\u003ca name=\"build-environment\"\u003e\u003c/a\u003e\n## Build Environment\n\nThe following environment variables will be passed as `ARG` variables when building images:\n\n```\nBUNDLE_BITBUCKET__ORG\nBUNDLE_GITHUB__COM\n```\n\nSet these variables in your shell if your `Gemfile` references privately-hosted gems on either _Bitbucket_ or _GitHub_.\n\nSee related documentation:\n\n* https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/\n* https://confluence.atlassian.com/bitbucket/app-passwords-828781300.html\n\n## Healthchecks\n\n[Healthchecks](https://docs.docker.com/engine/reference/builder/#healthcheck) are automatically configured for your application. A healthcheck utility is provided in `orchestration/healthcheck.rb`. The following environment variables can be configured (in the `app` service of `orchestration/docker-compose.deployment.yml`):\n\n| Variable | Meaning | Default Value |\n|-|-|-|\n| `WEB_HOST` | Host to reach application (from inside application container) | `localhost` |\n| `WEB_PORT` | Port to reach application (from inside application container) | `8080` |\n| `WEB_HEALTHCHECK_PATH` | Path expected to return a successful response | `/` |\n| `WEB_HEALTHCHECK_READ_TIMEOUT` | Number of seconds to wait for data before failing healthcheck | `10` |\n| `WEB_HEALTHCHECK_OPEN_TIMEOUT` | Number of seconds to wait for connection before failing healthcheck | `10` |\n| `WEB_HEALTHCHECK_SUCCESS_CODES` | Comma-separated list of HTTP status codes that will be deemed a success | `200,201,202,204` |\n\nIf your application does not have a suitable always-available route to use as a healthcheck, the following one-liner may be useful:\n\n```ruby\n# config/routes.rb\nget '/healthcheck', to: proc { [200, { 'Content-Type' =\u003e 'text/html' }, ['']] }\n```\n\nIn this case, `WEB_HEALTHCHECK_PATH` must be set to `/healthcheck`.\n\n## Dockerfile\n\nA `Dockerfile` is automatically generated based on detected dependencies, required build steps, _Ruby_ version, etc.\n\nReal-world applications will inevitably need to make changes to this file. As with all _Orchestration_ build-out, the provided `Dockerfile` should be treated as a starting point and customisations should be applied as needed.\n\n## Entrypoint\n\nAn [entrypoint](https://docs.docker.com/engine/reference/builder/#entrypoint) script for your application is provided at `orchestration/entrypoint.sh` which does the following:\n\n* Runs the `CMD` process as the same system user that launched the container (rather than the default `root` user);\n* Creates various required temporary directories and removes stale `pid` files;\n* Adds a route `host.docker.internal` to the host machine running the container (mimicking the same route provided by _Docker_ itself on _Windows_ and _OS\n  X_).\n\n\u003ca name=\"sidecar-containers\"\u003e\u003c/a\u003e\n## Sidecar Containers\n\nIf you need to start dependency services (databases, etc.) and connect to them from a _Docker_ container (an example use case of this would be running tests in _Jenkins_ using its _Docker_ agent) then the container that runs your tests must join the same _Docker_ network as your dependency services.\n\nTo do this automatically, pass the `sidecar` parameter to the `start` or `test` targets:\n\n```bash\nmake setup test sidecar=1\n```\n\nWhen running in sidecar mode container-to-container networking is used so there is no benefit to binding dependency containers to a specific port on the host machine (only the target port will be used). For this reason a random, ephemeral port (chosen by _Docker_) will be used to allow multiple instances of each dependency to run alongside one another.\n\nThe _Docker Compose_ project name (and derived network name) is also suffixed with a random token to avoid container/network name conflicts.\n\nNote that a temporary file `orchestration/.sidecar` containing the random project name suffix will be created when sidecar mode is used. If this file exists then sidecar mode is always assumed to be _on_. This is to allow (e.g.) stopping services that have been started separately with another command, for example:\n\n```bash\n# Start dependencies and run tests in sidecar mode\nmake setup test sidecar=1\n\n# Stop test dependencies in sidecar mode\nmake stop RAILS_ENV=test\n```\n\n\u003ca name=\"rabbitmq-configuration\"\u003e\u003c/a\u003e\n## RabbitMQ Configuration\n\n_Orchestration_ generates the following `config/rabbitmq.yml`:\n\n```\ndevelopment:\n  url: amqp://127.0.0.1:51070\n  management_url: http://127.0.0.1:5069\n\ntest:\n  url: amqp://127.0.0.1:51068\n  management_url: http://127.0.0.1:5067\n\nproduction:\n  url: \u003c%= ENV['RABBITMQ_URL'] %\u003e\n  management_url: \u003c%= ENV['RABBITMQ_MANAGEMENT_URL'] %\u003e\n```\n\nThe [Bunny](https://github.com/ruby-amqp/bunny) _RabbitMQ_ gem does not recognise `config/rabbitmq.yml` or `RABBITMQ_URL`. If your application uses _RabbitMQ_ then you must manually update your code to reference this file, e.g.:\n\n```ruby\nconnection = Bunny.new(Rails.application.config_for(:rabbit_mq)['url'])\nconnection.start\n```\n\nUsing this approach, the environment variable `RABBITMQ_URL` can be used to configure _Bunny_ in production (similar to `DATABASE_URL` and `MONGO_URL`).\n\nThis is a convention of the _Orchestration_ gem intended to make _RabbitMQ_ configuration consistent with other services.\n\n\u003ca name=\"redis-configuration\"\u003e\u003c/a\u003e\n## Redis Configuration\n\n_Orchestration_ generates the following `config/redis.yml`:\n\n```\ndevelopment:\n  url: redis://127.0.0.1:51071\n\ntest:\n  url: redis://127.0.0.1:51069\n```\n\nThe [Redis](https://github.com/redis/redis-rb) gem does not recognise `config/redis.yml`. If your application uses _Redis_ then you must manually update your code to reference this file, e.g.:\n\n```ruby\n# config/initializers/redis.rb\n\nENV['REDIS_URL'] ||= Rails.application.config_for(:redis)['url']\n```\n_Redis_ will then use `REDIS_URL` for all connections.\n\nThis allows `development` and `test` environments to auto-load the correct config for the relevant containers while also allowing `production` to use either the auto-generated _Redis_ service or an external _Redis_ instance.\n\n\n## Alternate Database Configuration Files\n\nIf you have multiple databases configured in various `config/database.*.yml` files then the `make wait` command will automatically detect database configurations.\n\nIf a service `database-example` is included in the relevant _Docker Compose_ configuration then `config/database.example.yml` will be used to load the connection configuration. Note that the service name _must_ begin with `database-`.\n\n## License\n\n[MIT License](LICENSE)\n\n## Contributing\n\nFeel free to make a pull request. Use `make test` to ensure that all tests, _Rubocop_ checks, and dependency validations pass correctly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobf%2Forchestration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbobf%2Forchestration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbobf%2Forchestration/lists"}