{"id":23008166,"url":"https://github.com/kangaroot/rootstack-devfactory","last_synced_at":"2025-08-14T04:31:59.984Z","repository":{"id":37966085,"uuid":"472442986","full_name":"kangaroot/rootstack-devfactory","owner":"kangaroot","description":"The Devfactory is a solution meant to provide teams a quick container/native/vm runtime enviroment. The goal is as follows:  Use the hashicorp toolset to provide Service discovery/mesh Runtime environment for vm/containers/native/springboot Secure and transparant routing Auditability and logging out of the box, with minimal adjustments needed Monitored environment","archived":false,"fork":false,"pushed_at":"2023-07-05T07:44:23.000Z","size":676,"stargazers_count":31,"open_issues_count":5,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2023-08-02T00:51:36.073Z","etag":null,"topics":["consul","grafana","loki","minio","nomad","promtail","traefik"],"latest_commit_sha":null,"homepage":"","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kangaroot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-21T17:28:52.000Z","updated_at":"2023-06-19T10:22:39.000Z","dependencies_parsed_at":"2023-02-14T05:00:30.144Z","dependency_job_id":null,"html_url":"https://github.com/kangaroot/rootstack-devfactory","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangaroot%2Frootstack-devfactory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangaroot%2Frootstack-devfactory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangaroot%2Frootstack-devfactory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kangaroot%2Frootstack-devfactory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kangaroot","download_url":"https://codeload.github.com/kangaroot/rootstack-devfactory/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229800407,"owners_count":18126027,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","grafana","loki","minio","nomad","promtail","traefik"],"created_at":"2024-12-15T09:00:44.937Z","updated_at":"2024-12-15T09:01:24.562Z","avatar_url":"https://github.com/kangaroot.png","language":"Jinja","funding_links":[],"categories":["Infrastructure setup"],"sub_categories":["Monitoring and Logging"],"readme":"# Rootstacks Devfactory\n\nThe Devfactory is a solution meant to provide teams a quick container/native/vm runtime enviroment.\nThe goal is as follows:\n\n* Use the hashicorp toolset to provide\n  * Service discovery/mesh\n  * Runtime environment for vm/containers/native/springboot\n* Secure and transparant routing\n* Auditability and logging out of the box, with minimal adjustments needed\n* Monitored environment\n\nThe initial phase of this project will be focussed on Azure deployment.\n\n# Table of contents\n\n- [Rootstacks Devfactory](#rootstacks-Devfactory)\n- [Table of contents](#table-of-contents)\n- [Architecture](#architecture)\n  - [Integrations](#integrations)\n- [Native component list](#native-component-list)\n- [Containerized component list](#containerized-component-list)\n- [Hashicorp Nomad](#hashicorp-nomad)\n  - [Networking](#networking)\n  - [Integrations](#integrations-1)\n    - [Consul](#consul)\n    - [docker (and plugins)](#docker-and-plugins)\n- [Hashicorp Consul](#hashicorp-consul)\n  - [Networking](#networking-1)\n  - [Key/value](#keyvalue)\n- [Docker](#docker)\n- [Promtail](#promtail)\n- [Grafana loki](#grafana-loki)\n- [Grafana](#grafana)\n- [Minio](#minio)\n- [Prometheus](#prometheus)\n- [traefik](#traefik)\n- [Installation](#installation)\n  - [dependencies for local development](#dependencies-for-local-development)\n  - [python environment](#python-environment)\n\n\n# Architecture\n\nWhen deploying we need minimum 4 vm's (3 admin, 1 worker), however to be able to upgrade we suggest to start with 5 or 6 systems.\n\n![](./img/OSSSystemdiagram.png)\n\nThe applicative high level overview is following:\n\n![](./img/hashi-in-a-box-high-level.png)\n\nIn the setup the runtime stack runs natively on the Linux system, they provide the base for the nomad runtime and observability around it.\n\n## Integrations\n\nThe main integrations are as such: \n\n![](./img/OSSIntegrations.png)\n\n\n\n# Native component list\n\nThere is a dual stack deployment of base components, the native installation and the Nomad job setup.\nWe will first list and discuss the native components:\n\n* Hashicorp Nomad\n* Hashicorp Consul\n* Docker runtime\n* Promtail \n* Grafana Loki\n\n# Containerized component list\n\n* Grafana\n* Minio\n* Prometheus\n* Traefik\n\n# Hashicorp Nomad\n\n\u003e Dependency: Docker, Consul\n\nNomad is used as runtime for the environment, it has 3 masters and 3 clients. By default we setup 2 namespaces:\n\n* admin: The namespace containing all administrative tools\n* client: The namespace used for applications, client deployments.\n\nThe namespaces are defined in the `nomad_namespaces` variable, set to `[\"admin\", \"client\"]` in the defaults.\n\nNomad is masters are configured to expect 3 nodes for a full bootstrap to be successfull.\nClients are configured on all nodes, and get the meta tag `admin` set to `1` when the node is an admin node.\n\n**telemetry**  is enabled by default, we enable prometheus metrics and set the collection interval to 1s.\n\n## Networking\n\nAll server bind addresses are limited to the `vm_ip` variable set in the `host_vars/host` inventory. \nThe client configuration uses the `network_cidr` variable for binding and defaults to `10.42.0.0/24` as defined in `group_vars/nomad/all.yml` in the inventory.\n\n## Integrations\n\n### Consul\n\nThe consul integration is non-optional and is set to use `local_ip:8500` as consul_address.\nNomad uses the standard integration, meaning that all services defined in jobs will be part of the consul_catalog.\n\n\n\n```hcl\n  create_from_role = \"nomad-server\"\n  token = \"{{ nomad_token.auth.client_token }}\" \n}\n```\n\n### docker (and plugins)\n\nDocker is configured in the `playbooks/roles/nomad/templates/nomad.hcl.j2` template and enables following:\n\n* volumes, with selinuxlabel `z`\n* garbage collection\n* allow privileged containers\n* set extra labels on containers\n\nWe also enable the loki docker plugin in case this is not an arm64 platform.\n\n# Hashicorp Consul\n\n\u003e Dependency: None\n\nConsul is used for service mesh and also internal dns server.\n\nConsul is setup without namespaces, as this is an enterprise option that we did not get for this deployment.\n\nConsul has 1 datacenter and this is configured in the `consul_dc_name` variable configured in `group_vars/consul_node/all.yml`, this dc name is used for multiple purposes outsite consul as well: \n\n* filename generation for consul certificates\n* CA certificate\n\nWe enabled **Consul Connect** in case this will be used in the jobs.\nUI is enabled by default.\n\nConsul uses 3 servers as master nodes and 1 node **must** have the `consul_bootstrap_node` set to true in the `host_vars/host` config file.\n\n## Networking\n\nAll nodes bind to the `vm_ip` variable and enable: `[\"https\", \"http\", \"dns\", \"grpc\"]`.\n\nWe also configure `iptables` to enable a redirect prerouting/output chain from port 53 to 8600 (standard dns port of Consul) in the nat tables.\n\nTo see the exact rules see `playbooks/roles/consul/tasks/iptables.yml`\n\n## Key/value\n\nConsul is also used as key/value store for other services in case they need this. \nTraefik will use the Consul catalog store and an extra k/v store with the name `{{ consul_dc_name }}_traefik`. More on this later.\n\n# Docker\n\n\u003e Dependency: None\n\nWe use the default docker installation and configure the docker/loki log driver when we are running on amd64 (x86_64) architectures.\n\n# Promtail\n\n\u003e Dependency: None, but will forward to Loki\n\nPromtail is used to scrape the logfiles of the system and jobs when deployed.\nWe listen to :9080 and use loki as our push target. \n\nFor the exact scraping config I refer to: `playbook/roles/promtail/templates/promtail.yaml.j2`\n\n\n# Grafana loki\n\n\u003e Dependency: Minio deployed\n\nLoki is used as log collector/indexer/search engine. \nIt receives logs from docker (in case the log-driver is installed see [Docker](#docker)).\n\nLoki has 1 node that does everything, the other 2 nodes are considered queriers only.\n\nWe use an internal boltdb index database that is stored the shared store named `aws` but is the bucket `loki` on minio using the servicename `s3.service.consul:9000`, with the values of `s3_access_key` and `s3_secret_key` as user/pass respectively.\n\n\n# Grafana\n\n\u003e Dependency: Nomad\n\nGrafana is used to display the metrics and logs, it is listening on `grafana.{{ zone}}` address.\n\nWe also add Loki and Prometheus as datasource by default.\nFollowing dashboards are added during setup:\n\nLoki \u0026 promtail: https://grafana.com/grafana/dashboards/10880\nTraefik 2.2: https://grafana.com/grafana/dashboards/12250\nMinio Dashboard: https://grafana.com/grafana/dashboards/13502\n\ndefault username/password: `admin/test`\n\n# Minio\n\n\u003e Dependency: Nomad\n\nMinio is used to provide s3 storage for loki mainly. It can be used for other purposes as well.\n\nMinio listens to `s3.{{ zone }}` on port 9000 with console port 9001. \nMinio uses the `s3_access_key` and `s3_secret_key` defined in `group_vars/all/s3.yml` as credentials during startup.\n\n# Prometheus\n\n\u003e Dependency: Nomad\n\nPrometheus is setup as a job in Nomad, and has the configuration in it's job description.\nYou can find the exact scraping information in `playbooks/templates/prometheus.hcl.j2`\n\n# traefik\n\n\u003e Dependency: Nomad, consul\n\nTraefik is our ingress controller and runs on all master nodes. \nIt's configured to enable: \n\n* Let's encrypt certificates for any domain you allow it (CAA method)\n* Custom certificate as default\n\nIt listens to following ports:\n\n* 80: http\n* 443: https\n* 8081: admin\n\nIt uses the Consul Catalog to enable services as they arrive, and allows for custom configuration through the `{{ consul_dc_name }}_traefik` prefix in the Consul K/V.\n\nIf a default_cert is defined in the `job_fact` it will add a the certs to the default tls store, enabling a self provided certificate.\n\n\n# Installation\n\nFor installing the environment one needs to create an inventory (both `molecule/ubuntu/molecule.yml` and `molecule/rocky/molecule.yml`) show the minimal variables needed for a runtime.\n\n## dependencies for local development\n\n* VirtualBox opensource version\n* Vagrant\n* Python3\n\n## python environment\n\nInstallation python environment is done as such:\n\n```bash\npython3 -m venv venv\nsource venv/bin/activate\npip install -r requirements.txt\n```\n\nTo setup a local environment, one can use molecule:\n\n`molecule converge -s ubuntu` or `molecule converge -s rocky`\n\n\nWhen using the molecule local setups, the password for the user is \"test\"\n\n\n## Azure \n\nTo install on azure you first need to create an inventory. The azure-inv in the root folder is a working example,\nbut given that you need specific information for your azure environment you must make modifications.\n\n### Requirements\n\nOne needs the following resources in Azure available already:\n\n* A resource group (ours is jochen) \n* A dns zone (ours is jochen.kangaroot.net)\n\nMake sure you have permissions to create resources and storage accounts within that resource group.\n\nAfter following the local dev install one needs to run following extra commands to enable the azure collection:\n\n```bash\n$ ansible-galaxy collection install azure.azcollection\n$ pip install -r ~/.ansible/collections/ansible_collections/azure/azcollection/requirements-azure.txt\n```\n\nTo actually deploy the needed resources one needs to run this command:\n\n```bash\n$ ansible -i \u003cpath-to-inventory\u003e playbooks/azure.yml --ask-vault-pass -t create\n```\n\nTo destroy the environment run the previous command with the destroy tag instead of create.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkangaroot%2Frootstack-devfactory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkangaroot%2Frootstack-devfactory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkangaroot%2Frootstack-devfactory/lists"}