{"id":14967006,"url":"https://github.com/golemfactory/concent-deployment","last_synced_at":"2025-10-19T09:31:29.436Z","repository":{"id":86423480,"uuid":"108838807","full_name":"golemfactory/concent-deployment","owner":"golemfactory","description":"Scripts and configuration for Concent deployment","archived":false,"fork":false,"pushed_at":"2022-02-19T08:45:18.000Z","size":712,"stargazers_count":5,"open_issues_count":85,"forks_count":8,"subscribers_count":26,"default_branch":"master","last_synced_at":"2024-10-01T01:56:34.911Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/golemfactory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-30T11:01:11.000Z","updated_at":"2023-04-18T13:18:31.000Z","dependencies_parsed_at":"2023-03-17T16:00:20.910Z","dependency_job_id":null,"html_url":"https://github.com/golemfactory/concent-deployment","commit_stats":{"total_commits":452,"total_committers":12,"mean_commits":"37.666666666666664","dds":"0.24778761061946908","last_synced_commit":"67e6f8d2a86b6fc8c6951dc45ac476dabefc8667"},"previous_names":[],"tags_count":38,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golemfactory%2Fconcent-deployment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golemfactory%2Fconcent-deployment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golemfactory%2Fconcent-deployment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golemfactory%2Fconcent-deployment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/golemfactory","download_url":"https://codeload.github.com/golemfactory/concent-deployment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219869245,"owners_count":16555571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:37:17.159Z","updated_at":"2025-10-19T09:31:24.029Z","avatar_url":"https://github.com/golemfactory.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# concent-deployment\nScripts and configuration for Concent deployment\n\n## GKE cluster configuration\n\n### Storage\n\nThe `nginx-storage` pod assumes that an ext4-formatted persistent disk with name defined by the `nginx_storage_disk` variable in `var.yml` is provisioned and mounts it in read-write mode.\nTo provision such a disk for the development cluster use the following command:\n\n``` bash\ngcloud compute disks create --size 30GB \u003cdisk name\u003e\n```\n\n### Creating the database\n\nBefore creating a new cluster in GKE, a PosgreSQL database and role have to be created for it.\nThis operation requires privileges for creating and deleting arbitrary databases in Cloud SQL.\nFor security reasons the role used to access the database from within the cluster should not have such wide privileges and this step needs to be performed outside of cluster deployment process.\nTo perform this step you need to have the .vault files with encrypted secrets in your local `concent-secrets/` directory.\nBoth cloud and cluster secrets are required.\nAnsible will prompt you for password required to decrypt them.\n\n``` bash\ncd concent-deployment/cloud/\nansible-playbook create-databases.yml                              \\\n    --extra-vars cluster=$cluster                                  \\\n    --vault-id @prompt                                             \\\n    --ask-vault-pass                                               \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\n\n### Creating the cluster\n\nThe playbook needs to be executed from a machine that is authorized to run `gcloud` commands on the Google Cloud project and has appropriate privileges for creating and deleting clusters.\nTo perform this step you need to have a configuration file for the specific clusters environment that cluster will represent (`concent-deployment-values/var-\u003ccluster\u003e.yml`).\nThe configuration files are stored in the [concent-deployment-values](https://github.com/golemfactory/concent-deployment-values/) repository.\nThe `$cluster` and `$concent_version` shell variables are explained below.\n\n\n``` bash\ncd concent-deployment-values/cloud/\nansible-playbook create-cluster-instance.yml \\\n    --extra-vars \"cluster=$cluster concent_version=$cluster_version\"\n```\n\n## Build scenarios\n\nScripts in this repository allow you to build containers and cluster configuration in three different scanarios. Each one has its own requirements:\n\n- **Build on your local machine.**\n    To do this you need to run Linux and install all the packages required to build and deploy containers.\n    You build containers by running Makefiles and deploy with shell scripts.\n    This mode is meant solely for deploying to the test cluster in development.\n- **Build inside the `concent-builder-vm` virtual machine.**\n    In this scenario all you need is Vagrant and VirtualBox.\n    You use Ansible to configure the machine and then run playbooks that take care of executing all build and deployment steps.\n    This mode is meant for development and for testing configuration changes meant for the `concent-builder` server itself.\n- **Build on the remote `concent-builder` server.**\n    In this senario you run the Ansible playbooks on a remote machine.\n    You obviously need access that machine to do this.\n    This is the recommended way to deploy in production.\n\n## Deployment\n\n### Cloning the repositories\n\nIn every scenario you need local copies of `concent-deployment` and `concent-deployment-values` repositories:\n\n``` bash\ngit clone git@github.com:golemfactory/concent-deployment.git\ngit clone git@github.com:golemfactory/concent-deployment-values.git\n\n```\n\nThe above assumes that the `$cluster` shell variable is set to the name of the cluster you're deploying to and that `concent-deployment-values` contains the configuration for that cluster.\n\n### Secrets\n\nPasswords and keys required for deployment are not stored in the repository.\nTo deploy you need to get access to them and put them in the following locations:\n\n- `concent-secrets/$cluster/cluster-secrets.yml.vault`\n- `concent-secrets/cloud/cloud-secrets.yml.vault`\n\n#### SSL certificates\n\nThe nginx instances need certificates and private keys to be able to serve HTTPS traffic.\nPut them in the following locations:\n\n- `concent-secrets/$cluster/nginx-proxy-ssl.crt.vault`\n- `concent-secrets/$cluster/nginx-proxy-ssl.key.vault`\n- `concent-secrets/$cluster/nginx-storage-ssl.crt.vault`\n- `concent-secrets/$cluster/nginx-storage-ssl.key.vault`\n\n#### Generating self-signed certificates\n\nIt's best if your certificates are signed by a Certificate Authority (CA) because then it's possible for the client to verify their authenticity without having to know the public key ahead of time.\nThis is not required though.\nYou can generate and use a self-signed certificate.\n\nHere's an example command that generates a 2048-bit RSA certificate valid for a year:\n\n``` bash\nopenssl req                      \\\n    -x509                        \\\n    -nodes                       \\\n    -sha256                      \\\n    -days   365                  \\\n    -newkey rsa:2048             \\\n    -keyout nginx-proxy-ssl.key  \\\n    -out    nginx-proxy-ssl.crt  \\\n    -config extensions.cnf\n```\n\n`extensions.cnf` file\n```\n[req]\ndefault_bits       = 2048\nprompt             = no\ndefault_md         = sha256\nx509_extensions    = x509_ext\ndistinguished_name = dn\n\n[dn]\nC            = \u003ccountry\u003e\nST           = \u003cstate\u003e\nO            = \u003corganization\u003e\nOU           = \u003corganization_unit\u003e\nCN           = \u003cdomain_name\u003e\nemailAddress = \u003cemail_address\u003e\n\n[x509_ext]\nbasicConstraints     = CA:FALSE\nsubjectAltName       = @alt_names\nsubjectKeyIdentifier = hash\n\n[alt_names]\nDNS.1 = \u003cdomain_name\u003e\n```\nReplace `\u003ccountry\u003e`, `\u003cstate\u003e`, `\u003corganization\u003e`, `\u003corganization_unit\u003e`, `\u003cdomain_name\u003e`, `\u003cemail_address\u003e` with actual values.\n\n### Setting up the `concent-builder-vm` virtual machine.\n\nDo this if you want to use the virtual machine for deployment.\n\n- Install Vagrant, VirtualBox and Ansible using your system package manager.\n- Create and configure the virtual machine:\n\n    ``` bash\n    cd concent-deployment/concent-builder-vm\n    vagrant up\n    ```\n\n    This will run the `configure.yml` playbook for you.\n\n### Creating `concent-deployment-server` machine\n\nThis step creates the `concent-deployment-server` machine meant to be used for deployment to `mainnet`, `testnet`, `staging` clusters, configuring machines, creating disks, etc.\nThe playbook needs to be executed from a machine that is authorized to run `gcloud` commands on the Google Cloud project.\n\n- Run the `create-compute-instance-for-deployment-server.yml` playbook.\n\n    ``` bash\n    cd concent-deployment-values/cloud/\n    ansible-playbook create-compute-instance-for-deployment-server.yml\n    ```\n### Configuring `concent-builder` machine\n\nDo this if you want to use the remote server for building and deploying.\n\n- Run the `configure.yml` playbook.\n\n    ``` bash\n    cd concent-deployment/concent-builder/\n    ansible-playbook configure.yml                                    \\\n        --inventory ../../concent-deployment-values/ansible_inventory \\\n        --user      $user\n    ```\n\n    Where the `$user` shell variable contains the name of your shell account on the remote machine.\n\n### Configuring `concent-deployment-server` machine\n\nThis step configures the `concent-deployment-server` machine.\n\n- Run the `configure-concent-deployment-server.yml` playbook.\n\n    ``` bash\n    cd concent-deployment/cloud/\n    ansible-playbook configure-concent-deployment-server.yml           \\\n        --inventory ../../concent-deployment-values/ansible_inventory  \\\n        --user      $user\n    ```\n\n### Setting up Ethereum client on a separate machine\nThis step installs and configures Geth on a separate machine in Google Compute Engine.\nThis is optional since Geth can be deployed automatically as a part of a Concent cluster but when you have multiple clusters, having one shared instance of the client allows you to use less resources.\nConcent does not use Geth very heavily so one instance of the blockchain client can handle many Concent instances.\nIt also makes deployment faster and more reliable since it's not necessary to wait for Geth synchronization every time it's updated or redeployed.\n\nThis machine comes in two flavors: `ethnode-testnet` and `ethnode-mainnet`.\nIf you're running both test clusters and a production cluster, you need both.\n\nThe playbooks needs to be executed from `concent-deployment-server` machine that is authorized to run `gcloud` commands on the Google Cloud project.\nTo create and configure the machine run the playbooks listed below.\nThe first one creates the cloud instance, reserves IP and provisions a disk.\nThis operation requires permissions to the Google Cloud project\nThe `$ethnode` variable specifies the flavor of the machine to be created: `ethnode-testnet` or `ethnode-mainnet`.\nThe second one connects to the machine, installs everything on it and starts Geth.\n``` bash\ncd concent-deployment-values/cloud/\nansible-playbook create-vm-instances-for-geth.yml                   \\\n    --extra-vars \"ethnode=$ethnode\"\n\ncd concent-deployment/ethnode/\n ansible-playbook configure.yml                                     \\\n    --extra-vars \"ethnode=$ethnode\"                                 \\\n    --inventory  ../../concent-deployment-values/ansible_inventory  \\\n    --user      $user\n```\n\nAll the instructions below assume that you're using local playbooks to run build and deployment commands on a new server.\n\nThe `$cluster` variable determines which server the playbook will be executed on.\nFor `concent-dev` it connects to `concent-builder`.\nFor other values (`concent-staging`, `concent-testnet` or `concent-mainnet`) - to `concent-deployment-server`.\nThis behavior applies to all playbooks, except for `build-test-and-push-containers.yml`.\n\nNote that if you're running the playbooks themselves from within that server too, you need to add `--connection=local` to your `ansible-playbook` calls.\nOtherwise Ansible will run its commands over SSH (rather than directly) using the public IP specified in `ansible_inventory`, which will likely fail because you're not supposed to have your private SSH key on the remote server you're connecting to.\n\n### Building containers and cluster configuration\nBefore following these instructions, please make sure that the Concent version you're building (i.e. `concent_version` in `containers/versions.yml`) is listed in `concent_versions` dictionary in `var-concent-\u003ccluster\u003e.yml` file.\nAt any given time there can be multiple Concent versions deployed to different clusters within the same environment (e.g. `v1.8` and `v1.9` on `dev`, `v1.9` and `v2.0` on `staging`, etc.) and this dictionary contains configuration values that are not the same for all those clusters.\nWithout providing configuration values there you won't be able to generate Kubernetes cluster configuration or use Ansible playbooks to deploy to the cluster.\n\n``` bash\ncd concent-deployment/concent-builder/\nansible-playbook install-repositories.yml                          \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n\nansible-playbook build-cluster-configuration.yml                   \\\n    --extra-vars cluster=$cluster                                  \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\nansible-playbook build-test-and-push-containers.yml                \\\n    --extra-vars cluster=$cluster                                  \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\n\n### Deploying secrets\nBefore you can deploy containers, you need to make sure that certificates, keys and passwords used to configure those containers are available on the cluster.\nTo perform this step you need to have the .vault files with encrypted secrets in your local `concent-secrets/` directory.\nOnly cluster secrets are required in this case.\nAnsible will prompt you for password required to decrypt them.\nDeploy them with:\n\n``` bash\ncd concent-deployment/cloud/\nansible-playbook cluster-deploy-secrets.yml                        \\\n    --extra-vars cluster=$cluster                                  \\\n    --ask-vault-pass                                               \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\nThis step is necessary only when deploying the cluster for the first time or when the secrets change.\nSecret deployment is separate from deployment of the application specifically so that they can be performed separately, possibly from different machines.\n\n### Deploying to the cluster\n\n``` bash\ncd concent-deployment/concent-builder/\nansible-playbook deploy.yml                                        \\\n    --extra-vars cluster=$cluster                                  \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\n\n### Update configuration of nginx-proxy\n\nThis step is necessary only when we move persistent disks or IP address from nginx-proxy on one cluster to another (as specified in `var-concent-\u003ccluster\u003e.yml` file in `concent-deployment-values`).\nThat's because when you update the `var` file and deploy to a new cluster, the previous cluster still has the disks or the IP attached.\nRunning this playbook updates the configuration of the old cluster so that the disks and IPs are released.\nThe new cluster will then claim them automatically.\n\n``` bash\ncd concent-deployment/concent-builder/\nansible-playbook redeploy-nginx-proxy-router.yml                   \\\n    --extra-vars cluster=$cluster                                  \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\n\n### Initializing or migrating the database\n\n`concent-api` and other Django apps will try to connect a CloudSQL database configured in their settings.\nControl and storage clusters have separate databases that need to be created and migrated individually.\nSet `$cluster_type` to `control` or `storage` before proceeding.\nThese commands are meant to be executed on every cluster separately.\n\n#### Initialization\n\nInitialization must only be performed on a newly created cluster or if we want to clear the data and start from scratch:\n\n``` bash\ncd concent-deployment/concent-builder/\nansible-playbook job-cleanup.yml                                   \\\n    --extra-vars cluster=$cluster                                  \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n\nansible-playbook reset-db.yml                                      \\\n    --extra-vars \"cluster=$cluster cluster_type=$cluster_type\"     \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\n\n**WARNING**: This operation removes all the data from an existing database.\n\n#### Migration\n\nFrom time to time a Concent update may require making changes to the database schema.\nThis is done using Django migrations.\nMigrations should be executed after containers with new version have been deployed and all the containers running the old version deleted.\n\n``` bash\ncd concent-deployment/concent-builder/\nansible-playbook job-cleanup.yml                                   \\\n    --extra-vars cluster=$cluster                                  \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n\nansible-playbook migrate-db.yml                                    \\\n    --extra-vars \"cluster=$cluster cluster_type=$cluster_type\"     \\\n    --inventory  ../../concent-deployment-values/ansible_inventory \\\n    --user       $user\n```\n\nIt's safe to run migrations even if there are no changes - Django will detect that and simply leave the schema as is.\n\n#### Helper script for deployment process\n\n###### `build-and-deploy-to-dev.sh`\nThis script provides a streamlined way to deploy a development version of the code to the `concent-dev` cluster.\nThe use case could be automated because it does not require as much flexibility as the usual deployment procedure.\nThe `concent-dev` cluster does not hold any production data and when something goes wrong it's fine to just remove everything and start from scratch.\n\nThe `configure.yml` playbook installs the script on the build server where it can use secrets necessary to access the cluster.\nThen it can be run from that server by any user.\n\nThe script deploys the code from the `dev` branch in the `concent-deployment` repository.\nVersion of the `concent` repository is specified in `concent_version` in `containers/versions.yml`.\nTo deploy any other version you must change this value in your local working copy, push the commit to the repository and update the `dev` branch to point at it.\nThe value can be a commit ID, tag, branch.\n\nThe script performs its job by running the Ansible playbooks listed above in the right order and with the parameters suitable for the `concent-dev` cluster.\n\nCommand-line options:\n1. **database_operation**: _migrate_ (default) or _reset_.\n    _migrate_ does not clear the database runs the migrations.\n    _reset_ clears the database and reinitializes it.\n2. **deployment_dir**: the directory to store a clone of the `concent-deployment` repository.\n    `~/deployment` by default.\n    This working copy should not be modified manually.\n    The script will recreate it if it does not exist or update it if it does but it will fail if a checkout is not possible (e.g. because there are uncommitted changes).\n\n### Building nginx-storage locally\n\nStep-by-step instructions to building and running `nginx-storage` locally.\n\n\n- Install dependencies:\n\n```bash\napt-get update\napt-get install make python3 python-pip python3-yaml docker.io\npip install yasha\n```\n\n\n- Ensure that your user is in the `docker` group and that this group exists.\n  This is necessary to be able to run docker commands without `root` privileges.\n  Note that the change will not take effect until you log out of the current shell session.\n\n```bash\nsudo groupadd docker\nsudo usermod --all --groups docker \u003cuser\u003e\n```\n\n\n- Go to `concent-deployment` repository and run makefile to build `nginx-storage` image:\n\n```bash\ncd \u003cpath to concent deployment repository\u003e/containers/\nmake nginx-storage\n```\n\n\n- Run nginx-storage:\n\n```bash\ndocker run                         \\\n   --rm                            \\\n   --hostname nginx-storage-server \\\n   --network  host                 \\\n   --name     nginx-storage        \\\n   nginx-storage\n```\n\n\n- If everything went OK, you should now be able to reach `nginx-storage` on localhost:\n\n```bash\ncurl http://localhost:8001/\n```\n\n## Concent Signing Service\nIn addition to Concent itself, this repository contains files necessary to build the Concent Signing Service.\n`Makefile` builds a Docker container but also produces a source package that includes `Dockerfile` and all source files needed to build it.\nThe package can be used to build the container without having to set up `concent-deployment`.\n\n### Building the Signing Service package\nThis is only needed if you want to build the package yourself.\nIf you have received the package and only want to build and run the docker container, you can skip this section.\n\n1. Install dependencies needed to build containers and render configuration files from templates.\n\n    Example for Ubuntu:\n\n    ``` bash\n    apt-get update\n    apt-get install make python3 python-pip python3-yaml\n    apt-get install docker.io\n    pip install yasha\n    ```\n\n2. Ensure that you can start docker containers.\n    It's recommended to add your user to the `docker` group (make sure that this group exists) so that you don't have to do this as root.\n    Note that the change will not take effect until you log out of the current shell session.\n\n    Example for Ubuntu:\n\n    ``` bash\n    sudo groupadd docker\n    sudo usermod --all --groups docker \u003cyour user name\u003e\n    ```\n\n3. Run make\n\n    ```bash\n    cd containers/\n    make concent-signing-service-package\n    ```\n\n### Building the Signing Service from the package\n1. Extract the package\n2. Go to `signing_service/` directory inside the package and build the Docker image\n\n    ``` bash\n    cd signing_service/\n    docker build --tag concent-signing-service:$(cat signing-service/RELEASE-VERSION) .\n    docker tag                                                         \\\n        concent-signing-service:$(cat signing-service/RELEASE-VERSION) \\\n        concent-signing-service:latest\n    ```\n\n### Generating a key pair\nSigning Service needs a key pair to secure its communication with Concent.\nThese are the same keys that are used for signing messages generated by golem-messages.\nThey're just base64-encoded so that they can be easily passed on the command line or in shell variables.\n\nThe public key needs to be known to Concent.\nThe private one must not - this is the whole point of having a separate service for signing.\n\nWhen setting up a new instance of the service, you can easily generate a new pair using the `generate-ecc-key-pair.sh` script included in the Docker image:\n\n```bash\ndocker run                      \\\n    --interactive               \\\n    --tty                       \\\n    --entrypoint /bin/bash      \\\n    concent-signing-service     \\\n    generate-ecc-key-pair.sh\n```\n### Running the Signing Service\nTo run it in a docker container with access to your local network interface, run:\n\n```bash\ndocker run                                                                                                      \\\n    --detach                                                                                                    \\\n    --env         ETHEREUM_PRIVATE_KEY                                                                          \\\n    --env         SIGNING_SERVICE_PRIVATE_KEY                                                                   \\\n    --env         SENTRY_DSN                                                                                    \\\n    --network     host                                                                                          \\\n    --hostname    signing-service                                                                               \\\n    --name        signing-service                                                                               \\\n    --volume      /var/log/concent/daily_thresholds:/usr/lib/signing_service/signing-service/daily_thresholds   \\\n    --restart     on-failure                                                                                    \\\n    concent-signing-service                                                                                     \\\n        --concent-cluster-host                  concent.golem.network                                           \\\n        --concent-public-key                    85cZzVjahnRpUBwm0zlNnqTdYom1LF1P1WNShLg17cmhN2Us                \\\n        --concent-cluster-port                  9055                                                            \\\n        --ethereum-private-key-from-env                                                                         \\\n        --signing-service-private-key-from-env                                                                  \\\n        --sentry-dsn-from-env                                                                                   \\\n        --sentry-environment                    mainnet\n```\n\nThis assumes that:\n- The service can connect to a Concent cluster at `concent.golem.network:9055`.\n- `85cZzVjahnRpUBwm0zlNnqTdYom1LF1P1WNShLg17cmhN2Us` is Concent's public key encoded in base64.\n- There's a shell variable called `ETHEREUM_PRIVATE_KEY` and it contains base64-encoded private key of the Ethereum account authorized to move funds owned by the deposit contract.\n- There's a shell variable called `SIGNING_SERVICE_PRIVATE_KEY` and it contains base64-encoded private key for signing Golem Messages created by the Signing Service.\n- There's a shell variable called `SENTRY_DSN` that contains the secret ID that allows submitting crash report to a [Sentry](https://sentry.io) project.\n    The one given above is just an example.\n    If so, you need to provide a valid DSN for the project that should receive the reports.\n    Otherwise just skip the `--sentry-dsn-from-env` parameter and the `SENTRY_DSN` variable.\n- `mainnet` is the name of the Sentry environment that should be included in error reports.\n- `/var/log/concent/daily_thresholds/` is location in the host system where the Signing Service can create daily threshold reports.\n    It must be writable by a user with UID of 999 which is the UID under which the application runs in the container.\n\nNote that the service will crash on errors.\nThe host system is responsible for restarting it in that case.\nIf it's running in a Docker container you can easily achieve this with the `--restart on-failure` option.\n\n### Concent Virtual Machine\nThe `concent-vm/` directory contains a Vagrant configuration that creates a virtual machine with Concent set up for development.\nThe machine has multiple purposes:\n- It can be used to run and debug Concent tests in a reproducible environment.\n- It serves as a reference for setting up Concent development environment.\n- It can be set up to build and run Golem from source.\n\n#### Prerequisites\n##### Vagrant\nYou need Vagrant \u003e= 2.2.0.\nInstall it with your system package manager.\n\n##### Ansible\nYou need Asible \u003e= 2.7.0 for Vagrant to run configuration playbooks.\nInstall it with your system package manager.\n\n##### VirtualBox\nThe machine runs on VirtualBox.\nInstall it with your system package manager.\n\nVirtualBox provides several kernel modules and requires them to be loaded before you can start any virtual machine.\nThese modules need to be built for your specific kernel version and rebuilt again whenever you update your kernel.\nIt's recommended to use [DKMS](https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support) to do this automatically.\nMost distributions provide a package named `virtualbox-dkms` (Ubuntu) or `virtualbox-host-dkms` that provides module sources and configures your system to build them.\n\nTo be able to build the modules you also need kernel headers.\nThe package containing them is called `linux-headers` on Arch Linux; the name contains kernel version on other distributions.\nMake sure that it matches the version of the kernel you're currently running (you can check that with `uname --kernel-release`).\n\n``` bash\nsudo apt-get install virtualbox-dkms linux-headers-x.y.z-w\n```\n\nOn some systems the modules are not loaded automatically after the installation.\nIf you can't start a machine, try to load `vboxdrv` kernel module manually first:\n```bash\nsudo modprobe vboxdrv\n```\nThese modules are loaded automatically when the system starts so you should no longer have to do this after the next reboot.\n\n###### VirtualBox Guest Additions\nVirtualBox provides a set of additional drivers and software that can be installed in the virtual machine to improve performance and usability.\nThis includes: sharing folders with the machine, clipboard integration or better video support.\nIt's all provided in the form of a [VirtualBox Guest Additions](https://www.virtualbox.org/manual/ch04.html) ISO.\n\nVirtualBox will download the ISO automatically but it will do it every time a new machine is built which significantly slows down the process.\nMany distributions provide a package which stores the ISO locally.\nIt's recommended to install it with your package manager.\n\nThe package is called `virtualbox-guest-additions-iso` on Ubuntu and `virtualbox-guest-iso` on Arch Linux.\n\n``` bash\nsudo apt-get install virtualbox-guest-additions-iso\n```\n\n##### Vagrant plugins\nInstall `vagrant-vbguest` plugin:\n```\nvagrant plugin install vagrant-vbguest\n```\n\n#### Building the machine\n##### Quick setup examples\nThe examples below assume that you already have a clone of the `concent-deployment` repository and you're in the `concent-vm/` directory.\n\n###### VM with Golem\nYou can build the machine by running:\n``` bash\nexport CONCENT_VM_INSTALL_GOLEM=true\nexport CONCENT_VM_INSTALL_GOLEM_ELECTRON=true\nexport CONCENT_VM_GOLEM_ELECTRON_VERSION=b0.19.0\nexport ANSIBLE_STDOUT_CALLBACK=debug\n\nvagrant up\nvagrant reload\n```\n\nThis installs:\n- Docker\n- PosgtgreSQL server\n- Local `nginx-storage` instance.\n- `golem` from the branch/commit/tag listed under `golem_version` in `containers/versions.yml`.\n- `golem-electron` from the `b0.19.0` branch\n- Lightweight desktop environment (XFCE)\n\n\nYou can stop the machine at any moment with `vagrant halt`.\nNote that to start the machine later you need to have the environment variables defined in your shell.\nIt's recommended to put them into a script and always source it before running any Vagrant commands.\n\n###### VM with Concent\nThis step requires two configuration files.\n\n`concent-vm/extra_settings.py` is a Python script that will be imported into the automatically generated `local_settings.py` in the machine.\nYou can use it to provide secrets or override default settings.\nIt can be empty if you're fine with defaults.\n\n`concent-vm/signing-service-env.sh` is a shell script meant to be sourced immediately before starting an instance of Concent's Signing Service and can define values of environment variables used by it.\nAt minimum it should define the following variables:\n\n``` bash\nexport ETHEREUM_PRIVATE_KEY=\"...\"\nexport SIGNING_SERVICE_PRIVATE_KEY=\"...\"\n```\n\nNow you can build the machine by running:\n``` bash\nexport CONCENT_VM_INSTALL_CONCENT=true\nexport CONCENT_VM_CONCENT_DEPLOYMENT_VERSION=$(git describe)\nexport ANSIBLE_STDOUT_CALLBACK=debug\n\nvagrant up\n```\n\nThe above assumes that the commit you have checked out has already been pushed to Github.\nThe machine always downloads all the repositories (including `concent-deployment`) from there.\n\nThis installs:\n- Docker\n- PosgtgreSQL server\n- Local `nginx-storage` instance.\n- RabbitMQ server running in a Docker container.\n- `concent` from the branch/commit/tag listed under `concent_version` in `containers/versions.yml`.\n\n\n###### VM with everything\nYou can build the machine by running:\n``` bash\nexport CONCENT_VM_INSTALL_CONCENT=true\nexport CONCENT_VM_INSTALL_GOLEM=true\nexport CONCENT_VM_INSTALL_GOLEM_ELECTRON=true\nexport CONCENT_VM_CONCENT_DEPLOYMENT_VERSION=$(git describe)\nexport CONCENT_VM_CONCENT_VERSION=master\nexport CONCENT_VM_GOLEM_VERSION=develop\nexport CONCENT_VM_GOLEM_ELECTRON_VERSION=b0.19.0\nexport CONCENT_VM_HYPERG_PORT=3282\nexport CONCENT_VM_GOLEM_START_PORT=40102\nexport CONCENT_VM_SHOW_GUI=true\nexport CONCENT_VM_MEMORY=3072\nexport CONCENT_VM_CPUS=3\nexport ANSIBLE_STDOUT_CALLBACK=debug\n\nvagrant up\nvagrant reload\n```\n\nThis installs:\n- Docker\n- PosgtgreSQL server\n- Local `nginx-storage` instance.\n- RabbitMQ server running in a Docker container.\n- `concent` from the `master` branch\n- `golem` from the `develop` branch\n- `golem-electron` from the `b0.19.0` branch\n- Lightweight desktop environment (XFCE)\n\nThis example also shows how to customize the machine.\n- The machine runs with 3 GB RAM and 2 CPU cores.\n- Vagrant will forward ports 3282, 40102 and 40103 of the machine to the host.\n     Note that these ports may still not be available to other clients if your host is behind a NAT (e.g. you're using a home router).\n- Golem will be configured to use 40102 as `start port` and `seed port` in `app_cfg.ini`.\n- The helper script for starting Golem in the machine (`golem-run-console-mode.sh`) will run `hyperg` at port 3282.\n\n##### Configuration variables\n| Variable                                | Value                                                                       | Default                                        | Description                                |\n|-----------------------------------------|-----------------------------------------------------------------------------|------------------------------------------------|--------------------------------------------|\n| `CONCENT_VM_INSTALL_CONCENT`            | `true` or `false`                                                           | `false`                                        | If `true`, Vagrant will download code from [`concent`](https://github.com/golemfactory/concent/) repository, build Concent and Signing Service, initialize the database and prepare the environment for running them.\n| `CONCENT_VM_INSTALL_GOLEM`              | `true` or `false`                                                           | `false`                                        | If `true`, Vagrant will download code from [`golem`](https://github.com/golemfactory/golem/) repository, build Golem, install `hyperg` and prepare the environment for running them.\n| `CONCENT_VM_INSTALL_GOLEM_ELECTRON`     | `true` or `false`                                                           | `false`                                        | If `true`, Vagrant will download code from [`golem-electron`](https://github.com/golemfactory/golem-electron/) repository, install XFCE desktop environment, build Golem's GUI and prepare the environment for running it.\u003cbr\u003e\u003cbr\u003eIf `CONCENT_VM_SHOW_GUI` is not defined, this variable also determines whether virtual machine screen is shown when the machine starts.\n| `CONCENT_VM_CONCENT_DEPLOYMENT_VERSION` | tag/branch/commit present in the `concent-deployment` repository on Github  | `master`                                       | Using version matching your local working copy (i.e. `CONCENT_VM_CONCENT_DEPLOYMENT_VERSION=$(git describe)`) is recommended.\u003cbr\u003e\u003cbr\u003eEven if `CONCENT_VM_INSTALL_CONCENT` is not enabled, Vagrant builds and installs helper services like `nginx-storage`. The source of these services is always downloaded from the `concent-deployment` repository on Github. This variable tells `Vagrant` which commit from that repository to check out.\u003cbr\u003e\u003cbr\u003eNote that this has no effect on which Ansible playbooks and `Vagrantfile` are used to build the machine. These are always taken from your local working copy.\n| `CONCENT_VM_CONCENT_VERSION`            | tag/branch/commit present in the `concent` repository on Github             | version specified in `containers/versions.yml` | Which commit from `concent` repository to check out before build.\u003cbr\u003e\u003cbr\u003eHas no effect if `CONCENT_VM_INSTALL_CONCENT` is `false`.\n| `CONCENT_VM_GOLEM_VERSION`              | tag/branch/commit present in the `golem` repository on Github               | version specified in `containers/versions.yml` | Which commit from `golem` repository to check out before build.\u003cbr\u003e\u003cbr\u003eHas no effect if `CONCENT_VM_INSTALL_GOLEM` is `false`.\n| `CONCENT_VM_GOLEM_ELECTRON_VERSION`     | tag/branch/commit present in the `golem-electron` repository on Github      | `dev`                                          | Which commit from `golem-electron` repository to check out before build.\u003cbr\u003e\u003cbr\u003eHas no effect if `CONCENT_VM_INSTALL_GOLEM_ELECTRON` is `false`.\n| `CONCENT_VM_HYPERG_PORT`                | port number                                                                 | 3282                                           | If defined, Vagrant forwards all packets reaching this port on the host to the same port in the machine. It also configures the `run-golem-console-mode.sh` script to start `hyperg` on that port.\u003cbr\u003e\u003cbr\u003eIf not defined, the default port is used but port forwarding is **not enabled**.\n| `CONCENT_VM_GOLEM_START_PORT`           | port number                                                                 | 40102                                          | If defined, Vagrant forwards all packets reaching this port and the one with number one higher on the host to the same ports in the machine. It also configures Golem to use this value for `start port` and `seed port` settings in `app_cfg.ini`.\u003cbr\u003e\u003cbr\u003eIf not defined, the default port is used but port forwarding is **not enabled**.\n| `CONCENT_VM_SHOW_GUI`                   | `true` or `false`                                                           | equal to `CONCENT_VM_INSTALL_GOLEM_ELECTRON`   | By default Vagrant uses `CONCENT_VM_INSTALL_GOLEM_ELECTRON` to determine whether to show virtual machine screen or not. `CONCENT_VM_SHOW_GUI` allows the user to override that decision. When it's `true`, the screen is always shown on machine startup, when `false`, the screen is always hidden, no matter whether the installation of `golem-electron` is enabled or not.\n| `CONCENT_VM_MEMORY`                     | megabytes of RAM                                                            | 2048                                           | The amount of RAM to allocate for the machine.\n| `CONCENT_VM_CPUS`                       | number of CPU cores                                                         | 1                                              | The number of CPU cores to allocate for the machine.\u003cbr\u003e\u003cbr\u003e`2` or `3` recommended.\n\n##### Reconfiguring the machine\nIf you change any of the environment variables used to configure the machine, you may need to rerun the configuration playbooks.\nYou can do it with\n\n``` bash\nvagrant provision\n```\n\nIn some cases (e.g. if the change resulted in the graphical environment being installed) it may be necessary to restart the machine as well:\n``` bash\nvagrant reload\n```\n\n\n#### Using the machine\n##### Using Vagrant\nPlease read the [Getting Started](https://www.vagrantup.com/intro/getting-started/) page in Vagrant docs to get familar with basic operations like starting the machine, logging into it via ssh or destroying it.\n\n\n##### Helper scripts\nThe machine provides several scripts that automate common development tasks.\nThey're located in `/home/vagrant/bin/` which is in user's `PATH` so you can execute them from any location.\n\n###### `concent-env.sh`\nThis is a helper script that loads a Python virtualenv with all dependencies required to run Concent and enters the directory that contains a working coy of the `concent` repository.\n\nThis script is meant to be sourced rather than executed:\n```\nsource concent-env.sh\n```\n\nUse it when you want to be able to run scripts from the repository (start Concent, run unit tests, etc.).\nAll the helper scripts provided with the machine source this file automatically when needed.\n\n###### `concent-run.sh`\nThis script starts Concent, including:\n- A development Django server (`manage.py runserver`).\n- 3 Celery worker instances attached to the right queues.\n- Signing Service\n- Middleman\n\n###### `concent-migrate.sh`\nMigrates the databases, preserving their content.\n\n###### `concent-reset.sh`\nThis script reinitializes Concent, removing all data stored by it so that you can start from scratch:\n- Destroys and recreates the databases.\n- Migrates the databases.\n- Empties RabbitMQ queues.\n- Re-creates the superuser account.\n- Restarts all the services.\n- Does **not** remove blockchain data.\n\n###### `concent-update.sh`\nUpdates Concent to the version (tag/branch/commit) specified in the first parameter (`master` by default):\n- Fetches the latest code from git.\n- Checks out the specified version.\n- Destroys and recreates the virtualenv.\n- Installs Concent dependencies in the virtualenv.\n- Migrates the databases.\n\n###### `golem-env.sh`\nSimilar to `concent-env.sh`.\nPrepares your shell for work with the Golem working copy checked out in the machine:\n- Loads the virtualenv with Golem's dependencies.\n- Changes the directory to `golem`.\n\nThis script is meant to be sourced rather than executed:\n```\nsource golem-env.sh\n```\n\n###### `golem-run-console-mode.sh`\nStarts Golem without GUI.\n- Sources `golem-env.sh`.\n- Starts `golemapp` in console mode and passes all the command-line arguments to it.\n\nYou can use it to start Golem like this:\n``` bash\ngolem-run-console-mode.sh \\\n    --accept-terms        \\\n    --password $password\n```\n\n`$password` needs to contain your Golem password.\n\nYou can see all the available `golemapp` options by running:\n\n``` bash\ngolem-run-console-mode.sh --help\n```\n\n###### `golem-run-gui-mode.sh`\nStarts the Electron app that provides a GUI for Golem.\nDoes **not** start the console app.\nThe console app should already be running in a separate terminal.\n\nNow you can launch the GUI from a terminal:\n``` bash\ngolem-run-gui-mode.sh\n```\n\nThis does not need to be a graphical terminal.\nYou can do it from a shell started with `vagrant ssh`.\n\nIf you have configured your machine with `CONCENT_VM_INSTALL_GOLEM_ELECTRON=true` or `CONCENT_VM_SHOW_GUI=true` and restarted it, you should be able to see the desktop of the machine.\nIf that's not the case you can always launch VirtualBox GUI, select the machine and use the \"Show\" option.\n\n###### `golem-run-all.sh`\nStarts both Golem GUI and the console app on a single terminal.\n- Runs `golem-run-console-mode.sh` in console mode and passes all the command-line arguments to it.\n- Runs `golem-run-gui-mode.sh`\n\nAll remarks about `golem-run-gui-mode.sh` apply here too.\n\n##### What's inside the machine\nHere's some extra information you should be aware of when using the machine:\n- The following services are automatically started inside the machine when the it boots:\n    - Docker\n    - PostgreSQL (accepts connections from within the machine without a password)\n    - RabbitMQ (runs in a Docker container)\n    - Geth (runs in a Docker container)\n    - nginx (configured to act as `nginx-storage`, built from `concent-deployment`)\n- The initialization playbook automatically creates PostgreSQL databases required by Concent\n- Concent, Signing Service and Golem do not start automatically.\n    Since you may want to run only one or the other, you need to start them using the helper scripts listed above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgolemfactory%2Fconcent-deployment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgolemfactory%2Fconcent-deployment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgolemfactory%2Fconcent-deployment/lists"}