{"id":20632478,"url":"https://github.com/citusdata/test-automation","last_synced_at":"2025-04-15T19:00:02.415Z","repository":{"id":14515790,"uuid":"76336560","full_name":"citusdata/test-automation","owner":"citusdata","description":"Tools for making our tests easier to run","archived":false,"fork":false,"pushed_at":"2025-04-04T14:15:24.000Z","size":3545,"stargazers_count":14,"open_issues_count":78,"forks_count":5,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-04-15T18:59:46.135Z","etag":null,"topics":["citus","testing"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/citusdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-12-13T08:02:28.000Z","updated_at":"2025-01-21T20:05:45.000Z","dependencies_parsed_at":"2024-04-15T11:43:24.240Z","dependency_job_id":"90d999f9-5525-4a8e-8b8d-5ea66c6b9f33","html_url":"https://github.com/citusdata/test-automation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Ftest-automation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Ftest-automation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Ftest-automation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/citusdata%2Ftest-automation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/citusdata","download_url":"https://codeload.github.com/citusdata/test-automation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249135822,"owners_count":21218365,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["citus","testing"],"created_at":"2024-11-16T14:16:19.738Z","updated_at":"2025-04-15T19:00:02.383Z","avatar_url":"https://github.com/citusdata.png","language":"C","readme":"# test-automation\n\nTools for making our tests easier to run. Automates setting up a cluster with\nAzure/Cloudformation and installs a script which automates setting up citus and everything\nrequired for testing citus.\n\n## Table of Contents\n\n* [Azure](#azure)\n  * [Getting Started](#azure-getting-started)\n    * [Setup steps for each test](#azure-setup-steps)\n    * [Steps to delete a cluster](#azure-delete-cluster)\n  * [Under The Hood](#under-the-hood)\n* [AWS(Deprecated)](#AWS)\n  * [Getting Started](#getting-started)\n    * [Setup steps for each test](#aws-setup-steps)\n    * [Steps to delete a cluster](#aws-delete-cluster)\n  * [Detailed Configuration](#detailed-configuration)\n    * [Starting a Cluster](#start-a-cluster)\n    * [Connecting to the Master](#connect-to-master)\n* [Running Rests](#running-tests)\n  * [Running Automated Tests](#running-automated-tests)\n  * [Running Automated Hammerdb Benchmark](#running-automated-hammerdb-benchmark)\n  * [Running Automated Compatibility Tests](#running-automated-compatibility-tests)\n  * [Basic Cluster Setup](#basic-cluster-setup)\n  * [Running PgBench Tests](#pgbench)\n  * [Running Scale Tests](#scale)\n  * [Running Extension Tests](#extension)\n  * [Running PgBench Tests Against Hyperscale (Citus)](#pgbench-cloud)\n  * [Running TPC-H Tests](#tpch)\n  * [Running TPC-H Tests Against Hyperscale (Citus)](#tpch-cloud)\n  * [Running Valgrind Tests](#valgrind)\n* [Example fab Commands](#fab-examples)\n* [Tasks, and Ordering of Tasks](#fab-tasks)\n* [Task Namespaces](#task-namespaces)\n  * [`use`, Choose Exactly What to Install](#use)\n  * [`add`, Add add-ons (such as extensions) to a Citus Cluster](#add)\n  * [`pg`, Run Commands Involving pg_ctl and psql](#pg)\n  * [`run`, Run pgbench and tpch Rests Automatically](#run)\n* [Advanced fab Usage](#advanced-fab)\n  * [Using Multiple Citus Installations](#multiple-installs)\n* [TroubleShooting](#TroubleShooting-test-automation)\n\n## \u003ca name=\"azure\"\u003e\u003c/a\u003eAzure\n\n## \u003ca name=\"azure-getting-started\"\u003e\u003c/a\u003e Getting Started\n\nYou can find more information about every step below in other categories. This list of commands show how to get started quickly. Please see other items below to understand details and solve any problems you face.\n\n### Prerequisites\n1. You should have `az cli` in your local to continue. [Install instructions](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)\n2. Run `az login` to make the CLI log in to your account\n3. Make sure that your default subscription is the right one(Azure SQL DB Project Orcas - CitusData):\n    ```bash\n    # List your subscriptions\n    az account list\n    # Pick the correct one from the list and run\n    az account set --subscription {uuid-of-correct-subscription}\n    ```\n\nIf your subscriptions list doesn't contain `Azure SQL DB Project Orcas - CitusData`, to add it, contact someone who is authorized.\n\n4. You should use `ssh-agent` to add your ssh keys, which will be used to upload to the release-test-results repository. Note that your keys are kept only in memory, therefore this is a secure step.\n    ```bash\n    # start ssh agent\n    eval `ssh-agent -s`\n\n    # Add your Github ssh key to upload results to release-test-results repo\n    ssh-add\n    ```\n\n5. You should setup your VPN to be able to connect to Azure VM-s if your tests are not running on GHA. Doing this as of latest consists of:\n* Open your VPN.\n* Run `routes.ps1` (on Windows only, if you are developing on Mac you should probably ping smn from the team for help). The script requires\n`python` to be installed to run.\n\n### General info\nIn `azuredeploy.parameters.json` file, you will see the parameters that you can change. For example if you want to change the number of workers, you will need to change the parameter `numberOfWorkers`. You can change the type of coordinator and workers separately from the parameters file. Also by default for workers, memory intense vms are used(E type) while for coordinator CPU intense vms are used(D type).\n\nAfter you run tests, you can see the results in `results` folder. The `results` folder will have the name of the config used for the test.\n\nThe data will be stored on the attached disk, size of which can be configured in the parameters.\n\nIf you dont specify the region, a random region among `eastus`, `west us 2` and `south central us` will be chosen. This is to use resources uniformly from different regions.\n\n**Port 3456 is used for ssh, you can connect to any node via Port 3456, if you don't use this node, you will hit the security rules.**\n\n## \u003ca name=\"azure-setup-steps\"\u003e\u003c/a\u003e Setup Steps For Each Test\n\nYou will need to follow these steps to create a cluster and connect to it, on your local machine:\n\n```bash\n\n# in the session that you will use to ssh, set the resource group name\nexport RESOURCE_GROUP_NAME=give_your_name_citus_test_automation_r_g\n\n# if you want to configure the region\n# export AZURE_REGION=eastus2\n\n# Go to the azure directory to have access to the scripts\ncd azure\n\n# open and modify the instance types/discs as you wish\nless azuredeploy.parameters.json\n\n# Quickly start a cluster of with defaults. This will create a resource group and use it for the cluster.\n./create-cluster.sh\n\n# connect to the coordinator\n./connect.sh\n```\n\n## \u003ca name=\"azure-delete-cluster\"\u003e\u003c/a\u003e Steps to delete a cluster\n\nAfter you are done with testing you can run the following the delete a cluster and the relevant resource group:\n\n```bash\n# Delete the formation\n# It's a good practice to check deletion status from the azure console\n./delete-resource-group.sh\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eUnder the hood\u003c/summary\u003e\n\n## \u003ca name=\"under-the-hood\"\u003e\u003c/a\u003eUnder The Hood\n\n  Azure has ARM templates that can be used to deploy servers with ease. There are two main files for ARM templates, `azuredeploy.json` and `azuredeploy.parameters.json`. `azuredeploy.json` has the main template and `azuredeploy.parameters.json` contains the parameters that are used in the main template. For example if you want to change the number of workers, you would do that in the parameters. You shouldnt change anything in the template file for configuration.\n\n  The main template has 4 main parts:\n\n* Parameters\n* Variables\n* Resources\n* Outputs\n\nParameters can be configured from the parameters file. Variables are constants. Resources have all of the resource definitions such as VMs, network security groups. Outputs can be useful for displaying a connection string.\n\nWhen creating resources, we can specify the order so that if a resource depends on some other resource, it wont be created before the dependant is created. We can also specify how many instances of a resource to create with a `copy` command.\n\nThe first virtual machine with index 0 is treated as a coordinator. When all the virtual machines are ready, a custom script is installed to do initialization in vms. The initailization script is retrieved from the github with a url.\n\nThe initialization script also finds the private ip addresses of workers and puts them to the coordinator. The way this is done is with a storage account resource. This storage account resource is created within the template itself and all the vms upload their private ip addresses to the storage. After all are uploaded the coordinator downloads all the private ip addresses from the storage account and puts it to `worker-instances` file, which is then used when creating a citus cluster.\n\nWe have a special security group which blocks ssh traffic. The rule's priority is 103 and 100, 101, 102 are also taken by this security group.\n\nYou can use `connect.sh` which will connect to the coordinator for you on a custom ssh port (at the time of writing 3456).\n\nBefore starting the process you should set the environment variable `RESOURCE_GROUP_NAME`, which is used in all scripts.\n\n```bash\nexport RESOURCE_GROUP_NAME=give_your_name_citus_test_automation_r_g\n```\n\nif you want to configure the region, you can also set that:\n\n```bash\nexport AZURE_REGION=eastus2\n```\n\nYou should use a single session because the exported variable is only available in the current session and its children sessions. You should start `ssh-agent` and add your key with `ssh-add`.\n\nBy default, your public key from `~/.ssh/id_rsa.pub` will be used. This public key will be put to the virtual machines so that you can ssh to them.\n\nTo simplify this process, there is a script called `create-cluster.sh`, which:\n\n* creates a resource group from the environment variable `RESOURCE_GROUP_NAME`, and `AZURE_REGION`.\n* creates a cluster with the `azuredeploy.json` template in the resource group\n* prints the connection string to ssh\n\nthen you should run:\n\n```bash\n./connect.sh\n```\n\nAfter you are done with testing, you can delete the resource group with:\n\n```bash\n./delete-resource-group.sh\n```\n\nCurrently the default time for tests 300 seconds, and as we have many tests it might take a while to run all the tests. So when testing a change, it is better to change the test times to something short such as 5 seconds. The time can be changed with the -T parameter:\n\n```bash\npgbench_command: pgbench -c 32 -j 16 -T 600 -P 10 -r\n\n-\u003e\n\npgbench_command: pgbench -c 32 -j 16 -T 600 -P 10 -r\n\n```\n\nIf you want to add different vm sizes, you should change the allowed values for `coordinatorVMSize` and `workerVMSize` in `azuredeploy.json`.\n\nWe run a custom script to initialize the vms. The script is downloaded to `/var/lib/waagent/custom-script/download/0 `. You can find the script logs\nin this file.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eAWS(Deprecated)\u003c/summary\u003e\n\n## \u003ca name=\"aws\"\u003e\u003c/a\u003eAWS\n\n## \u003ca name=\"getting-started\"\u003e\u003c/a\u003e Getting Started\n\nYou can find more information about every step below in other categories. This list of commands show how to get started quickly. Please see other items below to understand details and solve any problems you face.\n\n\n## \u003ca name=\"aws-setup-steps\"\u003e\u003c/a\u003e Setup Steps For Each Test\n\nYou will need to follow these steps to create a cluster and connect to it, on your local machine:\n\n```bash\n\n# start ssh agent\neval `ssh-agent -s`\n\n# Add your EC2 keypair's private key to your agent\nssh-add path_to_keypair/metin-keypair.pem\n\n# Add your Github ssh key for release-test-results repo\nssh-add\n\n# Quickly start a cluster of (1 + 3) c3.4xlarge nodes\ncloudformation/create-stack.sh -k metin-keypair -s PgBenchFormation -n 3 -i c3.4xlarge\n\n# When your cluster is ready, it will prompt you with the connection string, connect to master node\nssh -A ec2-user@ec2-35-153-66-69.compute-1.amazonaws.com\n```\n\n## \u003ca name=\"aws-delete-cluster\"\u003e\u003c/a\u003e Steps to delete a cluster\n\nOn your local machine:\n'\n```bash\n# Delete the formation\n# It's a good practice to check deletion status from the cloud formation console\naws cloudformation delete-stack --stack-name \"ScaleFormation\"\n```\n\n# \u003ca name=\"detailed-configuration\"\u003e\u003c/a\u003e Detailed Configuration\n\n## \u003ca name=\"start-a-cluster\"\u003e\u003c/a\u003e Starting a Cluster\n\nYou'll need to have installed the [AWS CLI](https://aws.amazon.com/cli/). Once that's\ninstalled you should configure it with `aws configure`. Once it's configured you can run\nsomething like:\n\n`cloudformation/create-stack.sh -k [your keypair name] -s MyStack`\n\nThis will take some time (around 7 minutes) and emit some instructions for connecting to\nthe master node.\n\nThe name you pass `-s` must be unique. There are more parameters you can pass such as\n`-n`, which changes the number of worker nodes which are launched. If you run\n`create-stack` with no parameters it will tell you more.\n\nIf you forget the name of your cluster you can get the list of active clusters by running:\n\n`aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE CREATE_IN_PROGRESS CREATE_FAILED --query \"StackSummaries[*].{StackName:StackName,StackStatus:StackStatus}\"`\n\nThis will only list clusters which are in your default region. You can specify a region\nwith the `--region` flag.\n\nTo get the hostname of the master you can run:\n\n`aws cloudformation describe-stacks --stack-name MyStack --query Stacks[0].Outputs[0].OutputValue`\n\n## \u003ca name=\"connect-to-master\"\u003e\u003c/a\u003e Connecting to the Master\n\n1. **Make sure you have a running ssh-agent**\n\n   If you are running linux you either have a running ssh-agent or know that you don't ;).\n   If you're using OSX you probably have a working ssh-agent but it's possible Apple has\n   pushed an update and broken things.\n\n   If it is running then the `SSH_AUTH_SOCK` environment variable will be set:\n\n   ```bash\n   brian@rhythm:~$ echo $SSH_AUTH_SOCK\n   /tmp/ssh-s1OzC5ULhRKg/agent.1285\n   ```\n\n   If your `SSH_AUTH_SOCK` is empty then Google is your friend for getting an ssh-agent to\n   start automatically when you login. As a temporary fix you can run `exec ssh-agent bash`:\n\n   ```bash\n   brian@rhythm:~$ echo $SSH_AUTH_SOCK\n\n   brian@rhythm:~$ exec ssh-agent bash\n   brian@rhythm:~$ echo $SSH_AUTH_SOCK\n   /tmp/ssh-et5hwGiqPxUn/agent.31580\n   ```\n\n2. **Add your keypair's private key to your ssh agent**\n\n   When you created your EC2 keypair it gave you a `.pem` file for safekeeping. Add it to\n   your agent with:\n\n   ```bash\n   brian@rhythm:~$ ssh-add Downloads/brian-eu.pem\n   Identity added: Downloads/brian-eu.pem (Downloads/brian-eu.pem)\n   ```\n\n   `find ~ -name '*.pem'` can help you find your key. Running `ssh-add ~/.ssh/id_rsa` will\n   not work, you must use the keypair you received from EC2.\n\n3. **If you plan on checking out private repos, add that private key to your agent as well**\n\n   When you check out a private repo on the master/workers, they will reach back and talk\n   to your ssh-agent in order to authenticate as you when talking to Github. You can find\n   your list of keys [here](https://github.com/settings/keys). One of them should be added\n   to ssh-agent with another `ssh-add` command. Personally, I run:\n\n   ```\n   brian@rhythm:~$ ssh-add\n   Identity added: /home/brian/.ssh/id_rsa (/home/brian/.ssh/id_rsa)\n   ```\n\n   Note: If you use 2-factor authentication that will not change these instructions in any\n   way. For the purposes of ssh access your key counts as a factor. If you have a\n   passphrase on your key you will be prompted for it when you run `ssh-add`, that's the\n   second factor. If you do not have a passphrase on your key, well, Github has no way of\n   knowing that.\n\n4. **ssh into the master in a way which allows it to use your local ssh-agent**\n\n   The `create-stack.sh` script should have given you a connection string to use. You can\n   also find the hostname of the master node in the cloudformation control panel in the\n   \"outputs\" section. There's a command above which tells you how to get the hostname\n   without going to the control panel.\n\n   You'll connect using a string like:\n\n   ```bash\n   ssh -A ec2-user@ec2-35-156-235-11.eu-central-1.compute.amazonaws.com\n   ```\n\n   The `-A` is not optional. It is required so the master node can ssh into the worker\n   nodes and so the nodes can checkout private repos.\n\n   That means you should not pass the `-i` flag into ssh:\n\n   ```bash\n   # careful, this is wrong!\n   ssh -i Downloads/brian2.pem ec2-user@[hostname]\n   ```\n\n   If you need to pass the `-i` flag in order to connect this means the key is not in your\n   agent. That means the master node will not be able to ssh into the worker nodes when you\n   later run `fab`.\n\nIt's unfortunate that you have no flexibility here. This restriction which will hopefully\nbe lifted in the future.\n\n\u003c/details\u003e\n\n# \u003ca name=\"running-tests\"\u003e\u003c/a\u003e Running Tests\n## \u003ca name=\"running-automated-tests\"\u003e\u003c/a\u003eRunning Automated Tests\n\n**Depending of the tests you trigger here, you can block at most 3 jobs slots in GHA for around 3 hours. Choose wisely the time you want to run the tests to not block development**\n\nIf you want, you can run trigger a job which can run pgbench, scale, tpch and extension tests. What the job does is:\n\n* It creates a cluster with the test resource group name\n* It connects to the coordinator\n* It runs the corresponding test for the job\n* It deletes the cluster.\n\nThere is a separate job for each test and you can run any combinations of them. To trigger a job, you should create a branch which has specific prefixes.\n\n* If the branch has a prefix `pgbench/`, then pgbench job will be triggered.\n* If the branch has a prefix `scale/`, then scale job will be triggered.\n* If the branch has a prefix `tpch/`, then tpch job will be triggered.\n* If the branch has a prefix `all_performance_test/`, then all jobs will be triggered.\n* If the branch has a prefix `extension/`, then extension job will be triggered.\n\nYou should push your branch to Github so that the GHA job will be triggerred.\n\nEach job uses a specific resource group name so that there will be at most 3 resource groups for these jobs. If there is already a resource group, then you should make sure that:\n\n* Someone else is currently not running the same test as you\n\nIf not, then you can delete the resource group name from portal, you can find it by search the prefix `citusbot`. Under normal circumstances the resource group will already be deleted at the end of the test\neven if it fails.\n\nYou can find your test results in https://github.com/citusdata/release-test-results under `periodic_job_results` folder. Test results will be pushed to a branch which is in the format ${rg_name}/${month_day_year_uniqueID}.\n\nBy default the tests will be run against `release-9.2` and the latest released version. If you want to test on a custom branch you should change the config files of relevant tests with your custom branch name in:\n\n```text\npostgres_citus_versions: [('12.1', 'your-custom-branch-name-in-citus'), ('12.1', 'release-9.1')]\n```\n\n*Note*: While you can run multiple tests by adding more elements to the array above, the results of the tests after the first might\nbe inflated due to cache hits (this depends on the tests being run and the type of disks being used by the VM-s). For the fairest\npossible comparisons, consider running the tests seperately.\n\nYou can change all the settings in these files, the config files for tests are located at:\n\n* pgbench: https://github.com/citusdata/test-automation/tree/master/fabfile/pgbench_confs\n* scale: https://github.com/citusdata/test-automation/tree/master/fabfile/pgbench_confs\n* tpch: https://github.com/citusdata/test-automation/tree/master/fabfile/tpch_confs\n* extension: https://github.com/citusdata/test-automation/tree/master/fabfile/extension_confs\n\nBy default, the following tests will be run for each test:\n\n* pgbench: `pgbench_default.ini` and `pgbench_default_without_transaction.ini`\n* scale: `scale_test.ini`\n* tpch: `tpch_default.ini`\n* extension: `extension_default.ini`\n\nIf you dont want to use default cluster settings(instance types etc), you can change them in https://github.com/citusdata/test-automation/blob/master/azure/azuredeploy.parameters.json.\n\nIf you want to change how long each test will be run, you can change the times with the `-T` parameter. https://github.com/citusdata/test-automation/blob/master/fabfile/pgbench_confs/pgbench_default.ini#L33\n\n```\npgbench_command: pgbench -c 32 -j 16 -T \u003ctest time in seconds\u003e -P 10 -r\n```\n\n## \u003ca name=\"running-automated-hammerdb-benchmark\"\u003e\u003c/a\u003eRunning Automated Hammerdb Benchmark\n\n**Important:** Push your branch to the github repo even though the HammerDb tests are run from your local.\nThe initiliazer script used to setup the Azure VM-s will pull your branch from github and not from your local.\n\nHammerdb tests are run from a driver node. Driver node is in the same virtual network as the cluster.\nYou can customize the hammerdb cluster in the `hammerdb` folder using `hammerdb/azuredeploy.parameters.json`.\nNote that this is the configuration for the cluster, which is separate than benchmark configurations(`fabfile/hammerdb_confs/`)\n\nIn `fabfile/hammerdb_confs` you can add more configs to this folder:\n\n* change postgres version\n* use a custom branch (You can also use git refs instead of branch names)\n* change/add postgres/citus settings\n\nYou can add as many configs as you want to `fabfile/hammerdb_confs` folder and the automation tool will\nrun the benchmark for each config. So if you want to compare two branches, you can create two identical config files with two different branches. (Note that you can also use git refs instead of branch names)\nEven though the script will vacuum the tables in each iteration to get more accurate results, the disk\ncache is likely to inflate the results of the tests running after the first file so for the most unbiased results\ntest the setups seperately (repeat this produre twice).\nThe result logs will contain the config file so that it is easy to know which config was used for a run.\n\nAfter adding the configs `fabfile/hammerdb_confs` could look like:\n\n* ./master.ini\n* ./some_branch.ini\n* ./some_other_branch.ini\n\nIn order to run hammerdb benchmark:\n\n```bash\neval `ssh-agent -s`\nssh-add\nexport RESOURCE_GROUP_NAME=\u003cyour resource group name\u003e\nexport GIT_USERNAME=\u003cYour github username\u003e\nexport GIT_TOKEN=\u003cYour github token with repo, write:packages and read:packages permissions\u003e # You can create a github token from https://github.com/settings/tokens\ncd hammerdb\n# YOU SHOULD CREATE A NEW BRANCH AND CHANGE THE SETTINGS/CONFIGURATIONS IN THE NEW BRANCH\n# AND PUSH THE BRANCH SO THAT WHEN THE TOOL CLONES THE REPOSITORY\n# IT CAN DOWNLOAD YOUR BRANCH.\nvim fabfile/hammerdb_confs/\u003cbranch_name\u003e.ini # verify that your custom config file is correct\n./create-run.sh\n# you will be given a command to connect to the driver node and what\n# to run afterwards.\n```\n\n**After running ./create-run.sh you do not have to be connected to the driver node at all, it will take care of the rest for you.**\n\nThe cluster deployment is flaky and sometimes it will fail. This behaviour is somewhat rare so it is not a\nbig problem. In that case, simply delete the previous resource group, and try again.\nYou can do that with:\n```bash\n# running from the same shell where you called create-run.sh to start the test\n../azure/delete-resource-group.sh\n./create-run.sh\n```\n\nIf it is persistent, some policy might have been changed on Azure so either consider debugging the issue,\nor opening an issue in test-automation.\n\n**The cluster will be deleted if everything goes okay, but you should check if it is deleted to be on the safe side.(If it is not, you can delete that with azure/delete-resource-group.sh or from the portal).**\n\nIn order to see the process of the tests, from the driver node:\n\n```bash\n./connect-driver.sh\nscreen -r\n```\n\nYou can see the screen logs in `~/screenlog.0`.\n\nYou will see the results in a branch `hammerdb_date_id` in https://github.com/citusdata/release-test-results.\nYou won't get any notifications for the results, so you will need to manually check it.\nWhat files are pushed to github:\n\n* build.tcl (This is the configuration file used for building hammerdb tables)\n* run.tcl (This is the configuration file used for running hammerdb tpcc benchmark)\n* build_\u003cconfig_file_name\u003e.log (These are the outputs of building the hammerdb tables for the 'config_file_name')\n* run_\u003cconfig_file_name\u003e.log (These are the outputs of running hammerdb tpcc benchmark for the 'config_file_name')\n* ch_benchmarks.log (This is the log file that is generated from ch-benCHmark script)\n* ch_results.txt (This is the file that contains the results of ch benchmark, each config file's result is saved in a new line)\n* \u003cconfig_file_name\u003e.NOPM.log (These are the files that contains the NOPM for the given config file name.)\n\n`hammerdb/build.tcl` creates and fills hammerdb tpcc tables. You should have at least 1:5 ratio for vuuser:warehouse_count otherwise the build.tcl might get stuck.\n\n`hammerdb/run.tcl` runs tpcc benchmark. You can configure things such as test duration here.\n\nNote that running a benchmark with a single config file with a vuuser of 250 and 1000 warehouses could\ntake around 2-3 hours. (the whole process)\n\nIf you want to run only the tpcc benchmark or the analytical queries, you should change the `is_tpcc` and `is_ch` variables in `create-run.sh`. For example if you want to run only tpcc benchmarks, you should set `is_tpcc` to `true` and `is_ch` to `false` (Alternatively you can see `IS_CH` and `IS_TPCC` environment variables). When you are only running the analytical queries, you can also specify how long you want them to be run by changing the `DEFAULT_CH_RUNTIME_IN_SECS` variable in `build-and-run.sh`. By default it will be run 3600 seconds.\n\nYou can change the thread count and initial sleep time for analytical queries from `build-and-run.sh` with `CH_THREAD_COUNT` and `RAMPUP_TIME` variables respectively.\n\nIf you want to run hammerdb4.0 change `hammerdb_version` to `4.0` in `create-run.sh`.\n\nBy default a random region will be used, if you want you can specify the region with `AZURE_REGION` environment variable prior to running `create-run.sh` such as `export AZURE_REGION=westus2`.\n\n## \u003ca name=\"running-automated-compatibility-tests\"\u003e\u003c/a\u003eRunning Automated Copatibility Tests\nCurrently, only testing compatibility with jdbc is automated.\nTo run, create a branch called `jdbc/{whatever-you-want}` and push to origin.\nThe citus branch and jdbc version can be configured from [JDBC Config](jdbc/jdbc_config.json)\n\nFor more details read: [JDBC README](jdbc/README.md)\n\n## \u003ca name=\"basic-cluster-setup\"\u003e\u003c/a\u003e Basic Cluster Setup\n\nOn the coordinator node:\n\n```bash\n# Setup your test cluster with PostgreSQL 12.1 and Citus master branch\nfab use.postgres 12.1 use.citus master setup.basic-testing\n\n# Lets change some conf values\nfab pg.set-config max_wal_size \"'5GB'\"\nfab pg.set-config max_connections 1000\n\n# And restart the cluster\nfab pg.restart\n```\n\nIf you want to add the coordinator to the cluster, you can run:\n\n```bash\nfab add.coordinator-to-metadata\n```\n\nIf you want the coordinator to have shards, you can run:\n\n```bash\nfab add.shards-on-coordinator\n```\n\n## \u003ca name=\"pgbench\"\u003e\u003c/a\u003e Running PgBench Tests\n\nOn the coordinator node:\n\n```bash\n# This will run default pgBench tests with PG=12.1 and Citus 9.2 and 8.3 release branches\n# and it will log results to pgbench_results_{timemark}.csv file\n# Yes, that's all :) You can change settings in fabfile/pgbench_confs/pgbench_default.ini\nfab run.pgbench-tests\n\n# It's possible to provide another configuration file for tests\n# Such as with this, we run the same set of default pgBench tests without transactions\nfab run.pgbench-tests --config-file=pgbench_default_without_transaction.ini\n```\n\n## \u003ca name=\"scale\"\u003e\u003c/a\u003e Running Scale Tests\n\nOn the coordinator node:\n\n```bash\n# This will run scale tests with PG=12.1 and Citus 9.2 and 8.3 release branches\n# and it will log results to pgbench_results_{timemark}.csv file\n# You can change settings in files under the fabfile/pgbench_confs/ directory\nfab run.pgbench-tests --config-file=scale_test.ini\nfab run.pgbench-tests --config-file=scale_test_no_index.ini\nfab run.pgbench-tests --config-file=scale_test_prepared.ini\nfab run.pgbench-tests --config-file=scale_test_reference.ini\nfab run.pgbench-tests --config-file=scale_test_foreign.ini\nfab run.pgbench-tests --config-file=scale_test_100_columns.ini\n```\n\n## \u003ca name=\"extension\"\u003e\u003c/a\u003e Running Extension Tests\n\nYou can execute a PG extension's regression tests with any combination of other extensions created in the database. The purpose of those tests is to figure out if any test fails due to having those extensions together. Currently we only support extensions whose tests can be run by pg_regress. We do not support any other extensions whose tests are run by some other tools. e.g. tap tests\n\nHere is the schema for main section:\n\n```\n[main]\npostgres_versions: [\u003cstring\u003e]   specifies Postgres versions for which the test should be repeated\nextensions: [\u003cstring\u003e]          specifies the extensions for which we give information\ntest_count: \u003cinteger\u003e           specifies total test scenarios which uses any extensions amongst the extension definitions\n\n\n[main]\npostgres_versions: ['14.5']\nextensions: ['citus', 'hll', 'topn', 'tdigest', 'auto_explain']\ntest_count: 4\n```\n\nHere is the schema for an extension definition:\n\n```\n[\u003cstring\u003e]                    specifies the extension name (that should be the same name with the extension name used in 'create extension \u003cextension_name\u003e;')\ncontrib: \u003cbool\u003e               specifies if the extension exists in contrib folder under Postgres (we do not install if it is a contrib extension because it is bundled with Postgres)\npreload: \u003cbool\u003e               specifies if we should add the extension into shared_preload_libraries\ncreate: \u003cbool\u003e                specifies if we should create extension in database (for example: 'create extension auto_explain;' causes error because it does not add any object)\nconfigure: \u003cbool\u003e             specifies if the installation step has a configure step (i.e. ./configure)\nrepo_url: \u003cstring\u003e            specifies repo url for non-contrib extension\ngit_ref: \u003cstring\u003e             specifies repo branch name for non-contrib extension\nrelative_test_path: \u003cstring\u003e  specifies relative directory in which pg_regress will run the tests\nconf_string: \u003cstring\u003e         specifies optional postgres.conf options\npost_create_hook: \u003cstring\u003e    specifies optional method name to be called after the extension is created. You should implement the hook in fabfile/extension_hooks.py.\n\n\n[tdigest]\ncontrib: False\npreload: False\ncreate: True\nconfigure: False\nrepo_url: https://github.com/tvondra/tdigest.git\ngit_ref: v1.4.0\nrelative_test_path: .\n```\n\nHere is the schema for a test case:\n\n```\n[test-\u003cinteger\u003e]                  specifies the test name\next_to_test: \u003cstring\u003e             specifies the extension to be tested\ndep_order: \u003cstring\u003e               specifies the shared_preload_libraries string order\ntest_command: \u003cstring\u003e            specifies the test command\nconf_string: \u003cstring\u003e             specifies the postgres configurations to be used in the test\n\n\n[test-4]\next_to_test: citus\ndep_order: citus,auto_explain\ntest_command: make check-vanilla\nconf_string: '''\n    auto_explain.log_min_duration=0\n    auto_explain.log_analyze=1\n    auto_explain.log_buffers=1\n    auto_explain.log_nested_statements=1\n    '''\n```\n\nOn the coordinator node:\n\n```bash\n# This will run default extension tests with PG=14.5\n# Yes, that's all :) You can change settings in fabfile/extension_confs/extension_default.ini\nfab run.extension-tests\n\n# It's possible to provide another configuration file for tests\nfab run.extension-tests --config-file=[other_config.ini]\n```\n\n## \u003ca name=\"extension\"\u003e\u003c/a\u003e Running Extension Tests\n\nYou can execute a PG extension's regression tests with any combination of other extensions created in the database. The purpose of those tests is to figure out if any test fails due to having those extensions together. Currently we only support extensions whose tests can be run by pg_regress. We do not support any other extensions whose tests are run by some other tools. e.g. tap tests\n\nHere is the schema for main section:\n\n```\n[main]\npostgres_versions: [\u003cstring\u003e]   specifies Postgres versions for which the test should be repeated\nextensions: [\u003cstring\u003e]          specifies the extensions for which we give information\ntest_count: \u003cinteger\u003e           specifies total test scenarios which uses any extensions amongst the extension definitions\n\n\n[main]\npostgres_versions: ['14.5']\nextensions: ['citus', 'hll', 'topn', 'tdigest', 'auto_explain']\ntest_count: 4\n```\n\nHere is the schema for an extension definition:\n\n```\n[\u003cstring\u003e]                    specifies the extension name (that should be the same name with the extension name used in 'create extension \u003cextension_name\u003e;')\ncontrib: \u003cbool\u003e               specifies if the extension exists in contrib folder under Postgres (we do not install if it is a contrib extension because it is bundled with Postgres)\npreload: \u003cbool\u003e               specifies if we should add the extension into shared_preload_libraries\ncreate: \u003cbool\u003e                specifies if we should create extension in database (for example: 'create extension auto_explain;' causes error because it does not add any object)\nconfigure: \u003cbool\u003e             specifies if the installation step has a configure step (i.e. ./configure)\nrepo_url: \u003cstring\u003e            specifies repo url for non-contrib extension\ngit_ref: \u003cstring\u003e             specifies repo branch name for non-contrib extension\nrelative_test_path: \u003cstring\u003e  specifies relative directory in which pg_regress will run the tests\nconf_string: \u003cstring\u003e         specifies optional postgres.conf options\npost_create_hook: \u003cstring\u003e    specifies optional method name to be called after the extension is created. You should implement the hook in fabfile/extension_hooks.py.\n\n\n[tdigest]\ncontrib: False\npreload: False\ncreate: True\nconfigure: False\nrepo_url: https://github.com/tvondra/tdigest.git\ngit_ref: v1.4.0\nrelative_test_path: .\n```\n\nHere is the schema for a test case:\n\n```\n[test-\u003cinteger\u003e]                  specifies the test name\next_to_test: \u003cstring\u003e             specifies the extension to be tested\ndep_order: \u003cstring\u003e               specifies the shared_preload_libraries string order\ntest_command: \u003cstring\u003e            specifies the test command\nconf_string: \u003cstring\u003e             specifies the postgres configurations to be used in the test\n\n\n[test-4]\next_to_test: citus\ndep_order: citus,auto_explain\ntest_command: make check-vanilla\nconf_string: '''\n    auto_explain.log_min_duration=0\n    auto_explain.log_analyze=1\n    auto_explain.log_buffers=1\n    auto_explain.log_nested_statements=1\n    '''\n```\n\nOn the coordinator node:\n\n```bash\n# This will run default extension tests with PG=14.5\n# Yes, that's all :) You can change settings in fabfile/extension_confs/extension_default.ini\nfab run.extension-tests\n\n# It's possible to provide another configuration file for tests\nfab run.extension-tests:[other_config.ini]\n```\n\n**Note**: You should `export EXTENSION_TEST=1` before running `create-cluster.sh` if you plan to run extension tests.\n\n## \u003ca name=\"pgbench-cloud\"\u003e\u003c/a\u003e Running PgBench Tests Against Hyperscale (Citus)\n\nOn the coordinator node:\n\n```bash\n\n# Use pgbench_cloud.ini config file with connection string of your Hyperscale (Citus) cluster\n# Don't forget to escape `=` at the end of your connection string\nfab run.pgbench-tests --config-file=pgbench_cloud.ini --connectionURI='postgres://citus:HJ3iS98CGTOBkwMgXM-RZQ@c.fs4qawhjftbgo7c4f7x3x7ifdpe.db.citusdata.com:5432/citus?sslmode\\=require'\n```\n\n**Important Note**\n- If an extension is a contrib module, then any url and branch will be discarded. If it is not a contrib module, url and branch is required.\n- `conf_string` is optional both for extension and test case definitions.\n\n## \u003ca name=\"tpch\"\u003e\u003c/a\u003e Running TPC-H Tests\n\nOn the coordinator node:\n\n```bash\n# This will run TPC-H tests with PG=12.1 and Citus 9.2 and 8.3 release branches\n# and it will log results to their own files on the home directory. You can use diff to\n# compare results.\n# You can change settings in files under the fabfile/tpch_confs/ directory\nfab run.tpch-automate\n\n# If you want to run only Q1 with scale factor=1 against community master,\n# you can use this config file. Feel free to edit conf file\nfab run.tpch-automate --config-file=tpch_q1.ini\n```\n\n## \u003ca name=\"tpch-cloud\"\u003e\u003c/a\u003e Running TPC-H Tests Against Hyperscale (Citus)\n\nOn the coordinator node:\n\n```bash\n# Provide your tpch config file or go with the default file\n# Don't forget to escape `=` at the end of your connection string\nfab run.tpch-automate --config-file=tpch_q1.ini --connectionURI='postgres://citus:dwVg70yBfkZ6hO1WXFyq1Q@c.fhhwxh5watzbizj3folblgbnpbu.db.citusdata.com:5432/citus?sslmode\\=require'\n```\n\n## \u003ca name=\"valgrind\"\u003e\u003c/a\u003e Running Valgrind Tests\n\nTL;DR\n\n```bash\n# set the appropriate az account subscription\naz account set --subscription \u003csubscriptionId\u003e\n\n# setup the ssh-agent and pass your credentials to it so the azure VM-s\n# will be setup to allow ssh connection requests with your public key\neval `ssh-agent -s`\nssh-add\n\n# 1 # start valgrind test\n\n# create valgrind instance to run\nexport RESOURCE_GROUP_NAME='your-valgrind-test-rg-name-here'\nexport VALGRIND_TEST=1\ncd azure\n./create-cluster.sh\n\n# connect to coordinator\n./connect.sh\n\n# run fab command in coordinator in a detachable session\n#\n# Note that you can use any valid schedule name for regression, isolation or failure tests here\ntmux new -d \"fab use.postgres 15.2 use.citus release-11.1 run.valgrind multi_1_schedule\"\n\n# simply exit from coordinator after detaching\n\n# 2 # finalize valgrind test\n\n# reconnect to coordinator after 9.5 hours (if you preferred default coordinator configuration)\nexport RESOURCE_GROUP_NAME='your-valgrind-test-rg-name-here'\n\neval `ssh-agent -s`\nssh-add\ncd azure\n./connect.sh\n\n# you can first check if valgrind test is finished by attaching to tmux session\ntmux a\n# then you should detach from the session before moving forward\nCtrl+b d\n\n# run push results script\ncd test-automation/azure\n./push-results.sh \u003cbranch name you prefer to push results\u003e\n\n# simply exit from coordinator after pushing the results\n\n# delete resource group finally\ncd azure\n./delete-resource-group.sh\n```\n\nDETAILS:\n\nTo create a valgrind instance, following the steps in [Setup Steps For Each Test](#azure-setup-steps), do the following before executing `create-cluster.sh`:\n\n```bash\neval `ssh-agent -s`\nssh-add\n\nexport VALGRIND_TEST=1\n```\n\n, which makes `numberOfWorkers` setting useless.\nThis is because we will already be using our regression test structure and it creates a local cluster\nitself. Also, as we install `valgrind` only on coordinator, if we have worker nodes, then we cannot build\nPostgreSQL as we require `valgrind` on workers and get error even if we do not need them.\nAlso, the `create-cluster.sh` uses the first public key it finds in the ssh-agent to setup the ssh authentication\nfor the Azure VM-s so if the ssh-agent is not up or it doesn't have your credentials, you won't be able to ssh\ninto the VM-s.\n\nOn the coordinator node:\n\n```bash\n# an example usage: Use PostgreSQL 15.2 and run enterprise failure tests with valgrind support on citus/release-11.1\nfab use.postgres 15.2 use.citus release-11.1 run.valgrind enterprise_failure_schedule\n```\n\nHowever as valgrind tests take too much time to complete, we recommend you to run valgrind tests in a detached session:\n```bash\n# Note that you can use any valid schedule name for regression, isolation or failure tests here\ntmux new -d \"fab use.postgres 15.2 use.citus release-11.1 run.valgrind multi_1_schedule\"\n```\n\nAfter the tests are finished (takes up to 9 hours with default coordinator size), re-connect to the coordinator.\nResult can be found under `$HOME/results` directory.\n\nTo push the results to `release_test_results` repository, run the below command in coordinator node:\n\n```bash\nsh $HOME/test-automation/azure/push-results.sh \u003cbranch_name_to_push\u003e\n```\n\nFinally, delete your resource group.\n\n## \u003ca name=\"fab-examples\"\u003e\u003c/a\u003e Example fab Commands\n\nUse `fab --list` to see all the tasks you can run! This is just a few examples.\n\nOnce you have a cluster you can use many different variations of the \"fab\" command to\ninstall Citus:\n\n- `fab --list` will return a list of the tasks you can run.\n- `fab setup.basic-testing`, will create a vanilla cluster with postgres and citus. Once this has run you can simply run `psql` to connect to it.\n- `fab use.citus v7.1.1 setup.basic-testing` will do the same, but use the tag `v7.1.1` when installing Citus. You can give it any git ref, it defaults to `master`.\n- `fab use.postgres 10.1 setup.basic-testing` lets you choose your postgres version.\n- `fab use.citus release-9.2 setup.citus` will install postgres and the `release-9.2` branch of the citus repo.\n\n## \u003ca name=\"fab-tasks\"\u003e\u003c/a\u003e Tasks, and Ordering of Tasks\n\nWhen you run a command like `fab use.citus v7.1.1 setup.basic-testing` you are running two\ndifferent tasks: `use.citus` with a `v7.1.1` argument and `setup.basic-testing`. Those\ntasks are always executed from left to right, and running them is usually equivalent to\nrunning them as separate commands. For example:\n\n```\n# this command:\nfab setup.basic-testing add.tpch\n# has exactly the same effect as this series of commands:\nfab setup.basic-testing\nfab add.tpch\n```\n\nAn exception is the `use` namespace, tasks such as `use.citus` and `use.postgres` only\nhave an effect on the current command:\n\n```\n# this works:\nfab use.citus v7.1.1 setup.basic-testing\n# this does not work:\nfab use.citus v7.1.1  # tells fabric to install v7.1.1, but only works during this command\nfab setup.basic-testing  # will install the master branch of citus\n```\n\n`use` tasks must come before `setup` tasks:\n\n```\n# this does not work!\n# since the `setup` task is run before the `use` task the `use` task will have no effect\nfab setup.basic-testing use.citus v.7.1.1\n```\n\nFinally, there are tasks, such as the ones in the `add` namespace, which asssume a cluster\nis already installed and running. They must be run after a `setup` task!\n\n\n# \u003ca name=\"task-namespaces\"\u003e\u003c/a\u003e Task Namespaces\n\n## \u003ca name=\"use\"\u003e\u003c/a\u003e `use` Tasks\n\nThese tasks configure the tasks you run after them. When run alone they have no effect.\nSome examples:\n\n```\nfab use.citus v7.1.1 setup.basic-testing\nfab use.citus release-9.2 setup.citus\nfab use.debug-mode use.postgres 10.1 use.citus v7.1.1 setup.basic-testing\n```\n\n`use.debug-mode` passes the following flags to postges' configure: `--enable-debug --enable-cassert CFLAGS=\"-ggdb -Og -g3 -fno-omit-frame-pointer\"`\n\n`use.asserts` passes `--enable-cassert`, it's a subset of `use.debug-mode`.\n\n## \u003ca name=\"add\"\u003e\u003c/a\u003e `add` Tasks\n\nIt is possible to add extra extensions and features to a Citus cluster:\n\n- `fab add.tpch --scale-factor=1 --partition-type=hash` will generate and copy tpch tables.\n\n  The default scale factor is 10. The default partition type is reference for nation, region and supplier and hash for remaining. If you set partition type to 'hash' or 'append', all the tables will be created with that partition type.\n- `fab add.session_analytics` will build and install the session_analytics package (see the instructions above for information on how to checkout this private repo)\n\nFor a complete list, run `fab --list`.\n\nAs described [above](#fab-tasks), you can run these at the same time as you run `setup` tasks:\n\n- `fab use.citus v7.1.1 setup.citus add.shard_rebalancer` does what you'd expect.\n\n## \u003ca name=\"pg\"\u003e\u003c/a\u003e `pg` Tasks\n\nThese tasks run commands which involve the current postgres instance.\n\n- `fab pg.stop` will stop postgres on all nodes\n- `fab pg.restart` will restart postgres on all nodes\n- `fab pg.start` guess what this does :)\n- `fab pg.read-config [parameter]` will run `SHOW [parameter]` on all nodes. For example:\n- `fab pg.read-config max_prepared_transactions`\n\nIf you want to use a literal comma in a command you must escape it (this applies to all\nfab tasks)\n\n- `fab pg.set-config shared_preload_libraries 'citus\\,cstore_fdw'`\n\nUsing `pg.set-config` it's possible to get yourself into trouble. `pg.set-config` uses\n`ALTER SYSTEM`, so if you've broken your postgres instance so bad it won't boot, you won't\nbe able to use `pg.set-config` to fix it.\n\nTo reset to a clean configuration run this command:\n\n- `fab -- rm pg-latest/data/postgresql.auto.conf`\n\n## \u003ca name=\"run\"\u003e\u003c/a\u003e `run` Tasks\n\nIn order to run pgbench and tpch tests automatically, you can use `run.pgbench_tests` or `run.tpch_automate`. If you want to use default configuration files, running commands without any parameter is enough.\n\nTo change configuration file for pgbench tests, you should prepare configuration file similar to fabfile/pgbench_confs/pgbench_config.ini.\n\nTo change the configuration file for tpch tests, you should prepare configuration file similar to fabfile/tpch_confs/tpch_default.ini.\n\n# \u003ca name=\"advanced-fab\"\u003e\u003c/a\u003e Advanced fab Usage\n\nBy default your fab commands configure the entire cluster, however you can target individual machines.\n\n- `fab -H 10.0.1.240 pg.start` will start pg on that specific node.\n\nYou can also ask to run arbitrary commands by adding them after `--`.\n\n- `fab -H 10.0.1.240 -- cat \"max_prepared_transactions=0\" \u003e\u003e pg-latest/data/postgresql.conf` will modify the postgresql.conf file on the specified worker.\n- `fab -- 'cd citus \u0026\u0026 git checkout master \u0026\u0026 make install'` to switch the branch of Citus you're using. (This runs on all nodes)\n\n## \u003ca name=\"multiple-installs\"\u003e\u003c/a\u003e Using Multiple Citus Installations, `pg-latest`\n\nSome kinds of tests (such as TPC-H) are easier to perform if you create multiple\nsimultanious installations of Citus and are able to switch between them. The fabric\nscripts allow this by maintaining a symlink called `pg-latest`.\n\nMost tasks which interact with a postgres installation (such as `add.cstore` or `pg.stop`)\nsimply use the installation in `pg-latest`. Tasks such as `setup.basic-testing` which\ninstall postgres will overwrite whatever is currently in `pg-latest`.\n\nYou can change where `pg-latest` points by running `fab set-pg-latest some-absolute-path`. For\nexample: `fab set-pg-latest $HOME/postgres-installation`. Using multiple\ninstallations is a matter of changing your prefix whenever you want to act upon or create\na different installation.\n\nHere's an example:\n\n```bash\nfab set-pg-latest $HOME/pg-960-citus-600\nfab use.postgres 9.6.0 use.citus v6.0.0 setup.basic-testing\nfab set-pg-latest $HOME/pg-961-citus-601\nfab use.postgres 9.6.1 use.citus v6.0.1 setup.basic-testing\n# you now have 2 installations of Citus!\nfab pg.stop  # stop the existing Citus instance\nfab set-pg-latest $HOME/pg-960-citus-600  # switch to using the new instance\nfab pg.start  # start the new instance\n# now you've switched back to the first installation\n\n# the above can be abbreviated by writing the following:\nfab pg.stop set-pg-latest $HOME/pg-960-citus-600 pg.start\n```\n\n## \u003ca name=\"TroubleShooting-test-automation\"\u003e\u003c/a\u003e TroubleShooting\n\nCurrently test automation has a lot of dependencies such as fabfile, azure and more. In general failures are temporary, which may be as long as a few days(If the problem is on azure service). In that case there is nothing we can do, but sometimes there are other problems that we can fix, and it is useful to try some of the following steps in that case:\n\n- Even if a creation of a cluster fails, you can still see the logs and what caused the problem:\n\n  * Find the public ip address of any instance (connect scripts might not be available if the cluster is in an incorrect state)\n  * Connect to the machine `ssh pguser@\u003cpublic_ip\u003e`\n  * switch to the root user(since `pguser` doesn't have the access to the logs) `sudo su root`\n  * cd into the log directory `/var/lib/waagent/custom-script/download/0`\n  * Now you can look at the `stderr` or `stdout` to see what went unexpected.\n\n- Updating `az cli` is also mostly a good option, follow the installation instructions in https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-linux to update your local `az cli` installation.\n\n- If you suspect if a particular `az foo bar` command doesn't work as expected, you could also insert `--debug` to have a closer look.\n\n-  If you're consistently having connection timeout errors (255) when trying to connect to a VM, then consider setting `AZURE_REGION` environment variable to `eastus`. This error will likely occur due to connection policy issues. As of latest, setting up your VPN properly should fix this issue.\n\n- While running on Azure VM-s there might be deployment errors (go to your resource group overview in the portal). This might be caused due to changing\nnetwork security policies in Azure. The error message of the deployment failure should show the conflicting policies. You can then go to the `azuredeploy.json` file for your test and try to change the priority of the custom policies (search priority in the file) until there are no conflicts.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcitusdata%2Ftest-automation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcitusdata%2Ftest-automation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcitusdata%2Ftest-automation/lists"}