{"id":23078210,"url":"https://github.com/gathercole3/dev-env","last_synced_at":"2025-09-07T01:04:53.762Z","repository":{"id":195725158,"uuid":"87628745","full_name":"gathercole3/dev-env","owner":"gathercole3","description":null,"archived":false,"fork":false,"pushed_at":"2020-10-27T10:49:27.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-01T03:49:46.201Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gathercole3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-04-08T11:22:01.000Z","updated_at":"2020-10-27T10:49:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"19c94315-db9b-4e2c-9dd5-535aeda4aaf0","html_url":"https://github.com/gathercole3/dev-env","commit_stats":null,"previous_names":["gathercole3/dev-env"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/gathercole3/dev-env","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gathercole3%2Fdev-env","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gathercole3%2Fdev-env/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gathercole3%2Fdev-env/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gathercole3%2Fdev-env/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gathercole3","download_url":"https://codeload.github.com/gathercole3/dev-env/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gathercole3%2Fdev-env/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273983110,"owners_count":25202095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-16T10:47:10.332Z","updated_at":"2025-09-07T01:04:53.714Z","avatar_url":"https://github.com/gathercole3.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Common Development Environment\n\nThis repository contains the code for a development environment that can be used across teams and projects. It is designed to allow collections of applications to be loaded from a separate configuration repository, and maximising consistency between them.\n\nIt provides several hooks for applications to take advantage of, including:\n\n* Docker container creation and launching via docker-compose\n* Automatic creation of commodity systems such as Postgres or Elasticsearch (with further hooks to allow for initial provisoning such as running SQL, Alembic DB upgrades or Elasticsearch index creation)\n\n## Getting started\n\n### Prerequisites\n\n* **Docker and Docker Compose**. Exactly what toolset you use depends on your OS and personal preferences, but recommended are:\n  * [Docker For Mac](https://docs.docker.com/docker-for-mac/)\n  * [Docker for Windows 10](https://docs.docker.com/docker-for-windows/) (See [the wiki](https://github.com/LandRegistry/common-dev-env/wiki/Windows-setup) for more information on getting a working Windows environment set up)\n  * [Docker CE for Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/)\n* **Git**\n* **Ruby 2.3+**\n  * The `highline` gem must also be installed\n\n### Git/SSH\n\nYou must ensure the shell you are starting the dev-env from can access all the necessary Git repositories - namely the config repo and the application repos it specifies. If they are to be accessed via SSH, [this wiki page](https://github.com/LandRegistry/common-dev-env/wiki/Git---SSH-setup) has some proven techniques for generating keys and making them available to the shell.\n\n### Controlling the dev-env\n\nTo begin:\n\n1. Start Docker.\n1. Using Git, clone this repository into a directory of your choice.\n1. Open a terminal in that directory, and run `source run.sh up`\n1. If this is the first time you are launching the machine you will be prompted for the url of a configuration repository. Paste it in and press enter.\n\nOther `run.sh` parameters are:\n\n* `halt` - stops all containers\n* `reload` - stops all containers then rebuilds and restarts them (including running any commodity fragments)\n* `destroy` - stops/removes all containers, removes all built images (i.e. leaving any pulled from Docker Hub) and resets all dev-env configuration files.\n* `repair` - sets the Docker-compose configuration to use the fragments from applications in _this_ dev-env instance (in case you are switching between several)\n\n## Usage guide\n\n### Configuration Repository\n\nThis is a Git repository that must contain a single file  -\n`configuration.yml`. The configuration file lists the applications that will be running in the dev-env, and specifies the URL of their Git repository along with which branch/tag should be initially checked out. The name of the application should match the repository name so that things dependent on the directory structure like volume mappings in the app's docker-compose-fragment.yml will work correctly.\n\n[Example](snippets/configuration.yml)\n\nThe local application repositories will be pulled and updated on each `up` or `reload`, _unless_ the current checked out branch does not match the one in the configuration. This allows you to create and work in feature branches while remaining in full control of updates and merges.\n\n### Application support\n\nFor an application repository to leverage the full power of the dev-env...\n\n#### Docker\n\nDocker containers are used to run all apps. So some files are needed to support that.\n\n##### `/fragments/docker-compose-fragment.yml` (Mandatory)\n\nThis is used by the envvironment to construct the application container and then launch it. Standard [Compose file](https://docs.docker.com/compose/compose-file/) structure applies - and all apps must use the same Compose file version - but some recommendations are:\n\n* Container name and service name should match\n* Any ports that need to be accessed from the host machine (as opposed to from other containers) should be mapped\n* A `volumes` entry should map the path of the app folder to wherever the image expects source files to be (if they are to be accessed dynamically, and not copied in at image build time)\n* If the provided log collator is to be used, then a syslog logging driver needs to be present, forwarding to logstash:25826.\n\nAlthough generally an application should only have itself in it's compose fragment, there is no reason why other containers based on other Docker images cannot also be listed in this file, if they are not provided already by the dev-env.\n\nNote that when including directives such as a Dockerfile build location or host volume mapping for the source code, the Compose context root `.` is considered to be the dev-env's /apps/ folder, not the location of the fragment. Ensure relative paths are set accordingly.\n\n[Example](snippets/docker-compose-fragment.yml)\n\n##### `/Dockerfile`\n\nThis is a file that defines the application's Docker image. The Compose fragment should point to this file. Extend an existing image and install/set whatever is needed to ensure that containers created from the image will run. See the [Dockerfile reference](https://docs.docker.com/engine/reference/builder/) for more information.\n\n[Example - Python/Flask](snippets/flask_Dockerfile)\n\n[Example - Java](snippets/java_Dockerfile)\n\n#### Commodities\n\n##### `/configuration.yml` (Mandatory)\n\nThis file lives in the root of the application and specifies which commodities the dev-env should create and launch for the application to use.\n\nThe list of allowable values is:\n\n* postgres\n* db2\n* elasticsearch\n* elasticsearch5\n* nginx\n* rabbitmq\n* redis\n* swagger\n* wiremock\n\nIndividual commodities may require further files in order to set them up correctly, these are detailed below. Note that unless specified, any fragment files will only be run once. This is controlled by a generated `.commodities.yml` file in the root of the this repository, which you can change to allow the files to be read again - useful if you've had to delete and recreate a commodity container.\n\n[Example](snippets/app_configuration.yml)\n\n##### PostgreSQL\n\n**`/fragments/postgres-init-fragment.sql`**\n\nThis file contains any one-off SQL to run in Postgres - at the minimum it will normally be creating a database and an application-specific user.\n\n[Example](snippets/postgres-init-fragment.sql)\n\nIf you want to spatially enable your database see the following example:\n\n[Example - Spatial](snippets/spatial_postgres-init-fragment.sql)\n\nThe default Postgres port 5432 will be available for connections from other containers and on the host.\n\n**`/manage.py`**\n\nThis is a standard Alembic management file - if it exists, then a database migration will be run on every `up` or `reload`.\n\n##### DB2\n\n**`/fragments/db2-init-fragment.sql`**\n\nThis file contains any one-off SQL to run in DB2 - at the minimum it will normally be creating a database.\n\n[Example](snippets/db2-init-fragment.sql)\n\n##### ElasticSearch\n\n**`/fragments/elasticsearch-fragment.sh`**\n**`/fragments/elasticsearch5-fragment.sh`**\n\nThis file is a shell script that contains curl commands to do any setup the app needs in elasticsearch - creating indexes etc. It will be passed a single argument, the host and port, which can be accessed in the script using `$1`.\n\nThe ports 9200/9300 and 9202/9302 are exposed on the host for Elasticsearch versions 2 and 5 respectively.\n\n[Example](snippets/elasticsearch-fragment.sh)\n\n##### Nginx\n\n**`/fragments/nginx-fragment.conf`**\n\nThis file forms part of an NGINX configration file. It will be merged into the server directive of the main configuration file.\n\nImportant - if your app is adding itself as a proxied location{} behind NGINX, NGINX must start AFTER your app, otherwise it will error with a host not found. So your app's docker-compose-fragment.yml must actually specify NGINX as a service and set the depends_on variable with your app's name. Compose will automatically merge this with the dev-env's own NGINX fragment. See the end of the [example Compose fragment](snippets/docker-compose-fragment.yml) for the exact code.\n\n[Example](snippets/nginx-fragment.conf)\n\n##### RabbitMQ\n\nThere are no fragments needed when using this. The Management Console will be available on \u003chttp://localhost:45672\u003e (guest/guest).\n\n##### Redis\n\nThere are no fragments needed when using this. Redis will be available at \u003chttp://localhost:16379\u003e on the host and at \u003chttp://redis:6379\u003e inside the Docker network.\n\nYou can monitor Redis activity using the CLI:\n\n```shell\nbashin redis\nredis-cli monitor\n```\n\n#### Other files\n\n**`/fragments/custom-provision.sh`**\n\nThis file contains anything you want - for example, initial data load. It will be executed on the host, not in the container. It is only ever run once (during the first `run.sh up`), and the file `.custom_provision.yml` is used by the dev-env to keep track of whether they have been run or not. Like `.commodities.yml`, this can be manually modified to trigger another run.\n\n**`/fragments/custom-provision-always.sh`**\n\nThis works the same way as `custom-provision.sh` except it is executed on every `run.sh up`.\n\n**`/fragments/host-fragments.yml`**\n\nThis file contains details of hosts to be forwarded on the host; if it exists then requests to the second address shall be forwarded to the first address.\n\n[Example](snippets/host-fragments.yml)\n\n**`/fragments/docker-compose-\u003cany value\u003e-fragment.yml`**\n\nThis file can be used to override the default settings for a docker container such as environment variables. It will not be loaded by default but can be applied using the add-to-docker-compose alias.\n\n## Logging\n\nAny messages that get forwarded to the logstash\\* container on TCP port 25826 will be output both in the logstash container's own stdout (so `livelogs logstash` can be used to monitor all apps) and in ./logs/log.txt.\n\n\\* Note that it is not really logstash, but we kept the container name that for backwards compatibility purposes.\n\n## Hints and Tips\n\n* Ensure that you give Docker enough CPU and memory to run all your apps.\n* The `run.sh destroy` command should be a last resort, as you will have to rebuild all images from scratch. Try the `fullreset` alias as that will just remove your app containers and recreate them. They are most likely to be the source of any corruption. Remember to alter `.commodities.yml` and `.custom_provision.yml` if you need to, and `run.sh reload`.\n\n### Useful commands\n\nIf you hate typing long commands then the commands below have been added to the dev-env for you:\n\n```bash\nstatus                                           -     view the status of all running containers\nstop \u003cname of container\u003e                         -     stop a container\nstart \u003cname of container\u003e                        -     start a container\nrestart \u003cname of container\u003e                      -     restart a container\nlogs \u003cname of container\u003e                         -     view the logs of a container (from the past)\nlivelogs \u003cname of container\u003e                     -     view the logs of a container (as they happen)\nex \u003cname of container\u003e \u003ccommand to execute\u003e      -     execute a command in a running container\nrun \u003coptions\u003e \u003cname of container\u003e \u003ccommand\u003e      -     creates a new container and runs the command in it.\nremove \u003cname of container\u003e                       -     remove a container\nfullreset \u003cname of container\u003e                    -     Performs stop, remove then rebuild. Useful if a container (like a database) needs to be wiped. Remember to reset .commodities if you do though to ensure init fragments get rerun\nrebuild \u003cname of container\u003e                      -     checks if a container needs rebuilding and rebuilds/recreates/restarts it if so, otherwise does nothing. Useful if a file has changed that the Dockerfile copies into the image.\nbashin \u003cname of container\u003e                       -     bash in to a container\nunit-test \u003cname of container\u003e                    -     run the unit tests for an application (this expects there to a be a Makefile with a unittest command). If you add -r it will output reports to the test-output folder.\nintegration-test \u003cname of container\u003e             -     run the integration tests for an application (this expects there to a be a Makefile with a integrationtest command).\nacceptance-test | acctest                        -     run the acceptance tests run_tests.sh script inside the given container. If using the skeleton, any further parameters will be passed to cucumber.\n          \u003cname of container\u003e \u003ccucumber args\u003e\nacceptance-lint | acclint                     -     run the acceptance tests run_linting.sh script inside the given container.\n          \u003cname of container\u003e\npsql \u003cname of database\u003e                          -     run psql in the postgres container\ndb2                                              -     run db2 command line in the db2 container\nmanage \u003cname of container\u003e \u003ccommand\u003e             -     run manage.py commands in a container\nalembic \u003cname of container\u003e \u003ccommand\u003e            -     run an alembic db command in a container, with the appropriate environment variables preset\nadd-to-docker-compose\n  \u003cname of new compose fragment\u003e                 -     looks in fragments folder of loaded apps to search for a new docker-compose-fragment including the provided parameter eg docker-compose-syt2-fragment then runs docker-compose up\n```\n\nIf you prefer using docker or docker-compose directly then below is list of useful commands (note: if you leave out \\\u003cname of container\\\u003e then all containers will be affected):\n\n```bash\ndocker-compose run --rm \u003cname of container\u003e \u003ccommand\u003e    -    spin up a temporary container and run a command in it\ndocker-compose rm -v -f \u003cname of container\u003e              -    remove a container\ndocker-compose down --rmi all -v --remove-orphans        -    stops and removes all containers, data, and images created by up. Don't use `--rmi all` if you want to keep the images.\ndocker-compose stop|start|restart \u003cname of container\u003e    -    (aliases: stop/start/restart) starts, stops or restarts a container (it must already be built and created)\ndocker exec -it \u003cname of container\u003e bash                 -    (alias: bashin) gets you into a bash terminal inside a running container (useful for then running psql etc)\n```\n\nFor those who get bored typing docker-compose you can use the alias dc instead. For example \"dc ps\" rather than \"docker-compose ps\".\n\n### Adding Breakpoints to Applications Running in Containers\n\nIn order to interact with breakpoints that you add to your applications you need to run the container in the foreground and attach to the container terminal. You do that like so:\n\n```bash\ndocker-compose stop \u003cname of container\u003e\ndocker-compose run --rm --service-ports \u003cname of container\u003e\n```\n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available and the changelog, see the [releases page](https://github.com/LandRegistry/common-dev-env/releases).\n\n## Authors\n\n* Simon Chapman ([GitHub](https://github.com/sdchapman))\n* Ian Harvey ([GitHub](https://github.com/IWHarvey))\n\nSee also the list of [contributors](https://github.com/LandRegistry/common-dev-env/contributors) who participated in this project.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details\n\n## Acknowledgments\n\n* Matthew Pease ([GitHub](https://github.com/Skablam)) - for helping create the original internal version\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgathercole3%2Fdev-env","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgathercole3%2Fdev-env","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgathercole3%2Fdev-env/lists"}