{"id":18355827,"url":"https://github.com/perlin-network/wavelet-stack","last_synced_at":"2025-04-10T01:46:03.505Z","repository":{"id":98777980,"uuid":"206355273","full_name":"perlin-network/wavelet-stack","owner":"perlin-network","description":null,"archived":false,"fork":false,"pushed_at":"2019-12-18T07:52:00.000Z","size":187,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-02-15T17:36:55.480Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perlin-network.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-04T15:43:41.000Z","updated_at":"2019-12-18T07:52:00.000Z","dependencies_parsed_at":"2023-03-13T15:54:57.286Z","dependency_job_id":null,"html_url":"https://github.com/perlin-network/wavelet-stack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perlin-network%2Fwavelet-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perlin-network%2Fwavelet-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perlin-network%2Fwavelet-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perlin-network%2Fwavelet-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perlin-network","download_url":"https://codeload.github.com/perlin-network/wavelet-stack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248142529,"owners_count":21054665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T22:08:04.384Z","updated_at":"2025-04-10T01:46:03.477Z","avatar_url":"https://github.com/perlin-network.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wavelet Stack and Swarm Manager\n## Introduction\nWavelet Stack and Swarm manager lets you run Wavelet Stacks on\nDocker Swarms.  Additionally, it helps manage Docker swarms using\nDocker Machine.\n\nThere are three provided tools:\n\n  1. `manage-swarm` - Manage Docker Swarm instances\n  2. `manage-stack` - Manage Wavelet Stack on a Docker Swarm\n  3. `build-all-nodes` - Builds all the nodes required to run a stack\n\nThese tools will look at configuration files in the `config/` directory\nas well as on `/etc/wavelet-stack` on remote Docker Machine hosts.\n\nThe configuration files are:\n\n  1. `config/default` - Always loaded;  Useful for setting \"`stackName`\" to set the default stack\n  2. `config/swarm/\u003cswarmName\u003e` - Loaded when using a particular swarm\n  3. `config/stack/\u003cstackName\u003e` - Loaded when using a particular stack, should not be used for remote stacks because other developers may not have the same configuration.  It is usually better to use `manage-stack edit-config` to edit this.\n\nYou may need to run \"[docker login](https://docs.docker.com/engine/reference/commandline/login/)\" to\nlogin to your Docker registry prior to using or updating images on that Docker registry.\n\n## manage-swarm\nThe `manage-swarm` tool helps create and manage remote Docker swarms using\nDocker Machine.\n\n### manage-swarm create\n`manage-swarm create \u003cprovider\u003e \u003cswarmName\u003e ...`\n\nCreates a new Swarm.  Currently the only provider supported is\ndigitalocean.  The Swarm will be named _\u003cswarmName\u003e_.  This name\nwill be the basis of serveral [docker machine](https://docs.docker.com/machine/)\nhosts added to the local system.\n\n`manage-swarm create digitalocean \u003cswarmName\u003e \u003capiToken\u003e [--count \u003cn\u003e] [--size \u003cn\u003e] [--region \u003cname\u003e]`\n\nCreates a new Swarm on DigitalOcean using the specified [API Token](https://www.digitalocean.com/docs/api/create-personal-access-token/).\nThe `--count` argument specifies the number of hosts to create.  The `--size`\nargument specifies the DigitalOcean droplet size to use for each host.  The\n`--region` argument specifies the DigitalOcean region name to place all the\ndroplets in.  You may specify the value \"`ask`\" for the arguments `--size`\nand `--region` to be interactively prompted for a list of options.\n\n### manage-swarm import\n`manage-swarm import \u003cip\u003e`\n\nImport the configuration for a swarm by SSHing into one of the machines\nmanaged.  Another administrative user must have already added your SSH\nkey to the machine's `~/.ssh/authorized_keys` file.  This can be done\nusing `docker-machine ssh` to login to the system.\n\n### manage-swarm expand\n`manage-swarm expand \u003cswarmName\u003e [\u003ccount\u003e [...]]`\n\nExpands a swarm by _\u003ccount\u003e_ hosts (default 1).  Takes the same arguments as\n`manage-swarm create` for the provider the swarm was created on.\n\n### manage-swarm destroy\n`manage-swarm destroy \u003cswarmName\u003e`\n\nDestroy a swarm -- USE WITH EXTREME CAUTION.\n\n### manage-swarm status \u003cswarmName\u003e\n`manage-swarm status \u003cswarmName\u003e`\n\nGet the status of a swarm.\n\n### manage-swarm list\n`manage-swarm list`\n\nList known swarms from the local configuration database.\n\n## manage-stack\nThe `manage-stack` tool is the most commonly used tool.  It helps manage a specified\nstack.\n\nStack configurations are stored in `./config/stacks/` locally, or in `/etc/wavelet-stack`\non the Docker Machine host if using a Swarm.\n\nConfiguration options are:\n\n  1. `REGISTRY` - Docker Registry to use (defaults to `localhost:5000`)\n  2. `WAVELET_GENESIS` - Wavelet Genesis block descriptor\n  3. `WAVELET_KEYS` - CSV list of private keys and public keys\n  4. `WAVELET_NODES` - Number of Wavelet nodes to run (defaults to `3`)\n  5. `WAVELET_RICH_WALLETS` - Number of Rich wallets to create if generating the Genesis block (that is, if `WAVELET_GENEISIS` is not supplied; defaults to `3`)\n  6. `WAVELET_SNOWBALL_K` - Wavelet Snowball K\n  7. `WAVELET_SNOWBALL_BETA` - Wavelet Snowball Beta\n  8. `WAVELET_MEMORY_MAX` - Max amount of memory to terminate the node after (in MiB)\n  9. `WAVELET_NO_RPC` - Boolean to indicate whether not RPC ports are exposed (if not specified as true, random port)\n  10. `WAVELET_RPC_PORT` - Port to listen for the first node for RPC requests\n  11. `WAVELET_TAG` - Tag of the wavelet image to pull down (defaults to `latest`)\n  12. `WAVELET_CLEAN_VOLUMES` - Boolean to indicate whether or not the volumes are removed on `stop` or `reset`\n  13. `WAVELET_API_HOST` - Hostname, if supplied, HTTPS support is enabled on port 443/tcp\n  14. `WAVELET_API_PORT` - Port to listen on for API requests (HTTP-only) (if not specified, random port)\n  15. `WAVELET_API_ACME_ACCOUNT_KEY` - PEM encoded ACME account key for autocert.  Generally if `WAVELET_API_HOST` is provided, this should be provided also.\n  16. `WAVELET_BACKUP_DB` - Boolean to indicate whether database backups are automatically taken for wavelet nodes\n  17. `WAVELET_BUILD_DIR` - Directory to rebuild the \"wavelet\" container from when building all images using `build-all-nodes`\n  18. `WAVELET_REBUILD_ON_START` - Indicate that the `build-all-nodes` script should be run for the given stack when it is started (defaults to false)\n\n```\nUsage: manage-stack [-s \u003cstackName\u003e] {stop|start|update|restart-wavelet|reset|status}\n       manage-stack [-s \u003cstackName\u003e] benchmark [\u003ccount\u003e]\n       manage-stack [-s \u003cstackName\u003e] nobenchmark\n       manage-stack [-s \u003cstackName\u003e] {attach|shell|logs|debug-wavelet} \u003cnodeId\u003e\n       manage-stack [-s \u003cstackName\u003e] {config|edit-config}\n       manage-stack [-s \u003cstackName\u003e] dump-db \u003cnodeId\u003e \u003coutputFile\u003e\n       manage-stack [-s \u003cstackName\u003e] cp \u003csrc\u003e... \u003cdest\u003e\n       manage-stack [-s \u003cstackName\u003e] duplicate-stack \u003cnewStackName\u003e\n```\n\n### manage-stack stop\n`manage-stack stop`\n\nStops the given stack -- this will not cleanup the volumes unless the\nstack configuration option `WAVELET_CLEAN_VOLUMES` is set to the value\n`yes`.\n\n### manage-stack start\n`manage-stack start`\n\nStarts the given stack.\n\n### manage-stack update\n`manage-stack update`\n\nUpdates the stack using any changed parameters.\n\n### manage-stack restart-wavelet\n`manage-stack restart-wavelet`\n\nRestart the wavelet process on all nodes\n\n### manage-stack reset\n`manage-stack restart [--hard]`\n\nRestarts (stops, then starts) a given stack.  If `--hard` is specified then the stack's\ndatabase volumes are wiped before starting up, even if `WAVELET_CLEAN_VOLUMES` is not\nspecified as `yes`.\n\n### manage-stack status\n`manage-stack status`\n\nDumps the stack status to stdout.\n\n### manage-stack benchmark\n`manage-stack benchmark [\u003ccount\u003e]`\n`manage-stack nobenchmark`\n\nStarts (or stops, in the case of `nobenchmark`) the benchmarking node.  If `\u003ccount\u003e` is specified\nthe that number of benchmarking nodes will be started, if not specified then one benchmarking\nnode will be started.\n\n### manage-stack attach\n`manage-stack attach \u003cnodeId\u003e`\n\nAttach to the console of the specified node.  Node IDs are numeric to indicate which Wavelet\nnode to attach to, or can be \"`sync`\", \"`loadbalancer`\", or \"`benchmark`\" to specify that the\n\"sync\" node, the \"loadbalancer\" node, or the \"benchmark\" node should be attached to, respectively.\n\nIf multiple benchmark nodes are running you can append the \"`\u003cnodeId\u003e`\" with a dot and index.  For\nexample:\n\n    manage-stack attach benchmark.2\n\n### manage-stack shell\n`manage-stack shell \u003cnodeId\u003e [\u003ccmd...\u003e]`\n\nExecute a command or shell on a given node.  See `manage-stack attach` for the format of the\nnodeId parameter.\n\n### manage-stack logs\n`manage-stack logs \u003cnodeId\u003e [\u003cargs...\u003e]`\n\nGet the logs for a given node.  See `manage-stack attach` for the format of the\nnodeId parameter.\n\n### manage-stack debug-wavelet\n`manage-stack debug-wavelet \u003cnodeId\u003e`\n\nAttach the \"delve\" debugger to the wavelet process of a given node.  See `manage-stack attach`\nfor the format of the nodeId parameter.\n\n### manage-stack config\n`manage-stack config`\n\nDump the configuration for a given stack to stdout.\n\n### manage-stack edit-config\n`manage-stack edit-config`\n\nEdits the given stack's configuration using the editor specified by the environment\nvariable `VISUAL` (defaulting to `vi`).\n\n### manage-stack dump-db\n`manage-stack dump-db \u003cnodeId\u003e \u003coutputFile\u003e`\n\nCreates a local tarball named `\u003coutputFile\u003e` with the database configuration of the specific\nnumeric `\u003cnodeId\u003e`.\n\n### manage-stack cp\n`manage-stack cp \u003csrc\u003e... \u003cdest\u003e`\n\nCopies files either to or from a remote container.  Only one remote container can be\nspecified (that is, you cannot copy to AND from a remote container).  Containers are\nidentified by their nodeId (see `manage-stack attach` for the format of nodeIds).\n\nExample:\n\n    $ manage-stack cp 1:/tmp/test local-test\n\n### manage-stack duplicate-stack\n`manage-stack duplicate-stack \u003cnewStackName\u003e`\n\nDuplicates the given stack to the specified `\u003cnewStackName\u003e`.  This includes the configuration\nand database for the current stack.\n\n## build-all-nodes\n`build-all-nodes [\u003cstackName\u003e]`\n\nBuilds all the node images (`wavelet-stack-lb`, and `wavelet-stack-node`) if needed, and\npushes them to the remote registry.  If a `\u003cstackName\u003e` is provided then the configuration\n(`REGISTRY` and `WAVELET_TAG`) for that stack are used to build the images.\n\nThat is `REGISTRY/wavelet-stack-lb:latest` will be created from the files in the\n`nodes/wavelet-stack-lb` directory and `REGISTRY/wavelet-stack-node:WAVELET_TAG`\nwill be crated from `REGISTRY/wavelet:WAVELET_TAG` and the files in the\n`nodes/wavelet-stack-node` directory.\n\nIf `WAVELET_TAG` is not specified it will default to `latest`.\n\n## Examples\n### Create a new Swarm\n\nThe following example creates a Docker Swarm on 3 newly provisioned Droplets on DigitalOcean.  It prompts\nthe user for the size of those droplets and which region they will be provisioned in.\n\n```\n$ ./manage-swarm create digitalocean demoswarm \u003capiKey\u003e --size ask --region ask --count 3\nLoading...\nPlease select one of the following for the size:\n1) 1gb             6) c-2           11) m-1vcpu-8gb   16) s-2vcpu-2gb\n2) 2gb             7) c-4           12) m-2vcpu-16gb  17) s-2vcpu-4gb\n3) 4gb             8) g-2vcpu-8gb   13) s-1vcpu-1gb   18) s-3vcpu-1gb\n4) 512mb           9) gd-2vcpu-8gb  14) s-1vcpu-2gb   19) s-4vcpu-8gb\n5) 8gb            10) m-16gb        15) s-1vcpu-3gb   20) s-6vcpu-16gb\n#? 17\nLoading...\nPlease select one of the following for the region:\n1) ams3  3) fra1  5) nyc1  7) sfo2  9) tor1\n2) blr1  4) lon1  6) nyc3  8) sgp1\n#? 5\nCreating machine...\nCreating machine...\nCreating machine...\n...\nComplete !\n$\n```\n\n### Create a new Stack\n\nThe following example creates a Stack named \"`demoswarm-demostack`\" on the Swarm named \"`demoswarm`\".  By default,\nStacks are created on Swarms named before the first dash (`-`).  This can be changed by creating a local stack configuration\nin `config/stacks/\u003cstackName\u003e` but every user of the stack will need that local configuration so it is not recommended.\n\n```\n$ ./manage-stack -s demoswarm-demostack edit-config\nREGISTRY=rkeene\nWAVELET_BUILD_DIR=/home/rkeene/devel/perlin-dev/wavelet-clean\nWAVELET_REBUILD_ON_START=yes\nWAVELET_CLEAN_VOLUMES=no\nWAVELET_MEMORY_MAX=2048\nWAVELET_NODES=3\nWAVELET_NO_RPC=true\nWAVELET_SNOWBALL_BETA=150\nWAVELET_SNOWBALL_K=2\nWAVELET_TAG=benchmark\n:wq\n$ docker login\n$ ./manage-stack -s demoswarm-demostack start\n...\nCreating network demoswarm-demostack_default\nCreating service demoswarm-demostack_wavelet\nCreating service demoswarm-demostack_benchmark\nCreating service demoswarm-demostack_sync\nCreating service demoswarm-demostack_loadbalancer\n$ ./manage-stack -s demoswarm-demostack status\ndemoswarm-demostack (on demoswarm):\n  - RUNNING\n  - VOLUMES\n  - API (main): http://\u003cip\u003e:30000/\n  - API (all): http://\u003cip\u003e:30001/\n  - RPC: disabled\n----\nID                  NAME                                 IMAGE                                 NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS\nthan2lo5h0jk        demoswarm-demostack_loadbalancer.1   rkeene/wavelet-stack-lb:latest        demoswarm-2         Running             Running 40 seconds ago\nxzlxbn18rvul        demoswarm-demostack_sync.1           elcolio/etcd:latest                   demoswarm-1         Running             Running 47 seconds ago\npj6rx1eznf0g        demoswarm-demostack_wavelet.1        rkeene/wavelet-stack-node:benchmark   demoswarm-2         Running             Running 14 seconds ago\njwbjqznbqyh6        demoswarm-demostack_wavelet.2        rkeene/wavelet-stack-node:benchmark   demoswarm-3         Running             Running 13 seconds ago\nkhog9vro233v        demoswarm-demostack_wavelet.3        rkeene/wavelet-stack-node:benchmark   demoswarm-1         Running             Running 18 seconds ago\n$\n```\n\n(n.b., if you see the error \"scp: /etc/wavelet-stack/demoswarm-demostack: No such file or directory\" when creating your first stack on a swarm, it may be safely ignored.)\n\n### Run a Benchmark\n\nThe following example runs a single benchmark instance and then stops it:\n\n```\n$ ./manage-stack -s demoswarm-demostack benchmark\ndemoswarm-demostack_benchmark scaled to 1\noverall progress: 1 out of 1 tasks\n1/1: running   [==================================================\u003e]\nverify: Service converged\n11:03PM INF Benchmarking... accepted_tps: 0.1942050180078577 downloaded_tps: 0 gossiped_tps: 0 queried_bps: 0.08597350826813725 query_latency_max_ms: 6 query_latency_mean_ms: 0.3055822267509728 query_latency_min_ms: 0 received_tps: 0\n11:03PM INF Benchmarking... accepted_tps: 0.28440662295239344 downloaded_tps: 0 gossiped_tps: 0 queried_bps: 0.091853589683795 query_latency_max_ms: 6 query_latency_mean_ms: 0.3169363319066148 query_latency_min_ms: 0 received_tps: 0    \n^C\n$ ./manage-stack -s demoswarm-demostack nobenchmark\ndemoswarm-demostack_benchmark scaled to 0\noverall progress: 0 out of 0 tasks\nverify: Service converged\n$ \n```\n\n### Update Wavelet on a Stack\n\nExample 1: Update to include any changes from a local checkout of Wavelet\n```\n$ ./manage-stack -s demoswarm-demostack edit-config\nREGISTRY=rkeene\nWAVELET_TAG=benchmark\nWAVELET_BUILD_DIR=/home/rkeene/devel/perlin-dev/wavelet-clean\nWAVELET_REBUILD_ON_START=yes\n...\n:wq\n$ ( cd /home/rkeene/devel/perlin-dev/wavelet-clean \u0026\u0026 git update/pull/etc )\n$ docker login\n$ ./manage-stack -s demoswarm-demostack update\n$ ./manage-stack -s demoswarm-demostack status watch\n```\n\nExample 2: Update a Stack to run a specific tag -- in this case `perlin/*:v0.3.0`\n```\n$ ./manage-stack -s demoswarm-demostack edit-config\nREGISTRY=perlin\nWAVELET_TAG=v0.3.0\nWAVELET_BUILD_DIR=/home/rkeene/devel/perlin-dev/wavelet-clean\nWAVELET_REBUILD_ON_START=no\n...\n:wq\n$ ./manage-stack -s demoswarm-demostack update\n$ ./manage-stack -s demoswarm-demostack status watch\n```\n\nExample 3: Create a group of Docker images with a particular tag from a particular source tree\n```\n$ docker login\n$ REGISTRY=perlin \\\n      WAVELET_TAG=v0.3.0 \\\n      WAVELET_BUILD_DIR=/home/rkeene/devel/perlin-dev/wavelet-clean \\\n      ./build-all-nodes\n```\n\n### Import an existing Swarm\n\nThe following example imports a Swarm by SSH'ing as root the the IP specified and downloading the configuration\ninto the user's home directory for Docker Machine as well creating `config/swarms/\u003cswarmName\u003e`.  Another user\nof the swarm must have previously configured the SSH `~/.ssh/authorized_keys` on the host before it can be imported.\n\n```\n$ ./manage-swarm import \u003cip\u003e\nwavelet-swarm.tar.gz                                                                            100%   15KB 244.4KB/s   00:00    \n$ ./manage-swarm list\ndemoswarm\ntestnet\ntradenet\n$ ./manage-swarm status demoswarm\nID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION\nuq5lc5qh9iahucmlrlz96kudo *   demoswarm-1         Ready               Active              Leader              19.03.5\np61qvckkxyp6cgqs4xvs36c50     demoswarm-2         Ready               Active                                  19.03.5\npm3rzbv5qlzy4l6tpxlqd3b22     demoswarm-3         Ready               Active                                  19.03.5\nNAME                  SERVICES            ORCHESTRATOR\ndemoswarm-demostack   4                   Swarm\n$\n```\n\n### Create a Geographically Diverse Swarm\n\nThe following example creates a Docker Swarm that has Docker hosts in 9 different DigitalOcean regions.\nThis is useful for testing Wavelet with a wide array of latencies.\n\n```\n$ ./manage-swarm create digitalocean bignet \u003capiKey\u003e --count 1 --region nyc1 --size s-4vcpu-8gb\nfor region in sgp1 lon1 nyc3 ams3 fra1 tor1 sfo2 blr1; do\n        ./manage-swarm expand bignet 1 --region \"${region}\"\ndone\n```\n\n### Duplicating a Stack\n\nThe following example duplicates an existing running Stack's configuration\nand database into a new Stack named \"`demoswarm-duplistack`\"\n\n```\n$ ./manage-stack -s demoswarm-demostack duplicate-stack demoswarm-duplistack\n...\n$ ./manage-stack -s demoswarm-duplistack status\ndemoswarm-duplistack (on demoswarm):\n  - RUNNING\n  - VOLUMES\n  - API (main): http://\u003cip\u003e:30097/\n  - API (all): http://\u003cip\u003e:30098/\n  - RPC: \u003cip\u003e:30146\n----\nID                  NAME                                  IMAGE                                 NODE                DESIRED STATE       CURRENT STATE                ERROR               PORTS\no57biard8uwu        demoswarm-duplistack_loadbalancer.1   rkeene/wavelet-stack-lb:latest        demoswarm-1         Running             Running about a minute ago\nn07rpaayhy6b        demoswarm-duplistack_sync.1           elcolio/etcd:latest                   demoswarm-3         Running             Running about a minute ago\nyn3n0fve6fbs        demoswarm-duplistack_wavelet.1        rkeene/wavelet-stack-node:benchmark   demoswarm-3         Running             Running 39 seconds ago\nm99xow4evz85        demoswarm-duplistack_wavelet.2        rkeene/wavelet-stack-node:benchmark   demoswarm-2         Running             Running 36 seconds ago\nypq5dk7lv4rf        demoswarm-duplistack_wavelet.3        rkeene/wavelet-stack-node:benchmark   demoswarm-1         Running             Running 37 seconds ago\n```\n\n### Rebuilding a Stack from a Backup\n\nThe following example recreates a stack from a database backup (taken from `manage-stack dump-db`) and configuration\n(taken from `manage-stack config`).\n\nFirst we make the backup and destroy the existing stack.\n\n```\n$ ./manage-stack -s demoswarm-demostack dump-db 1 backups/demo-backup-db.tar.gz\n$ ./manage-stack -s demoswarm-demostack config\nREGISTRY=rkeene\nWAVELET_CLEAN_VOLUMES=no\nWAVELET_MEMORY_MAX=2048\nWAVELET_NODES=3\nWAVELET_NO_RPC=true\nWAVELET_SNOWBALL_BETA=150\nWAVELET_SNOWBALL_K=2\nWAVELET_TAG=benchmark\n$ ./manage-stack -s demoswarm-demostack stop\n...\nNothing found in stack: demoswarm-demostack\n$ ./manage-stack -s demoswarm-demostack cleanVolumes --force\ndemoswarm-demostack_wavelet_wavelet_db_instance_1\ndemoswarm-demostack_sync_wavelet_db_instance_1\ndemoswarm-demostack_wavelet_wavelet_db_instance_2\ndemoswarm-demostack_wavelet_wavelet_db_instance_3\n$\n```\n\nSecond we rebuild the Stack using the previous configuration:\n\n```\n$ ./manage-stack -s demoswarm-copystack edit-config\n\u003cInsert Config Here\u003e\n### Note the number of nodes, temporarily change it to 1 so we only have\n### to upload the DB once\nWAVELET_NODES=1\n:wq\n$ ./manage-stack -s demoswarm-copystack start\nCreating network demoswarm-copystack_default\nCreating service demoswarm-copystack_benchmark\nCreating service demoswarm-copystack_sync\nCreating service demoswarm-copystack_loadbalancer\nCreating service demoswarm-copystack_wavelet\n$ ./manage-stack -s demoswarm-copystack cp backups/demo-backup-db.tar.gz 1:/tmp\n$ ./manage-stack -s demoswarm-copystack shell 1\nbash-4.4# cd /db\nbash-4.4# ls\n000001.log       CURRENT          LOCK             LOG              MANIFEST-000000\nbash-4.4# mkdir x\nbash-4.4# cd x\nbash-4.4# gzip -dc /tmp/demo-backup-db.tar.gz | tar -xf -\nbash-4.4# cd ..\nbash-4.4# rm -f *; mv x/* .; rmdir x; pkill -9 -x /wavelet\nrm: 'x' is a directory\nbash-4.4# ls\n000003.ldb       000005.log       CURRENT.bak      LOG\n000004.ldb       CURRENT          LOCK             MANIFEST-000006\nbash-4.4# exit\n$ ./manage-stack -s demoswarm-copystack edit-config\n\u003cExiting Config Here\u003e\nWAVELET_NODES=\u003cNumberOfNodesFromPreviousStep\u003e\n:wq\n$ ./manage-stack -s demoswarm-copystack update\nUpdating service demoswarm-copystack_benchmark (id: w8gin0o6aeisap86hsyagjrcy)\nUpdating service demoswarm-copystack_sync (id: yx3quhjpukhosmzgn3rp2m82i)\nUpdating service demoswarm-copystack_loadbalancer (id: nnbvghcw4rhr8sy4tqenlel5k)\nUpdating service demoswarm-copystack_wavelet (id: qqgh8663rgt0q1k9gvl1alu1d)\n$ ./manage-stack -s demoswarm-copystack attach 1\nl\n11:31PM INF Here is the current status of your node. balance: 9999999999999529446 block_height: 48 block_id: 0552dc9c6b4141d2f8e257cfd6da134cfea9d26995e87dfc0040688c7ea437ce client_block: 48 num_accounts_in_store: 0 num_missing_tx: 0 num_tx: 0 num_tx_in_store: 0 peers: [\"10.0.3.14:3001[68a5f19ea9c5e1e61836a189175a159f36dc273c07eb9e58448b3529957e3218]\",\"10.0.3.15:3002[ce55ce8e451272e042683766f02165b8ca18ce1aa989c77f48b87ff4fc29fcfa]\"] preferred_block_id: N/A preferred_votes: 0 reward: 268888 stake: 201666 sync_status: \"Node is fully synced\" user_id: e919a3626df31b6114ec79567726e9a31c600a5d192e871de1b862412ae8e4c0     \n```\n\n## Theory of Operation\n### Introduction\nThe two main components of this suite of tools are:\n  1. `manage-swarm`\n  2. `manage-stack`\n\n### `manage-swarm`\nManage Swarm is relatively simple and creates, destroys, imports, modifies, or examines a [Docker Swarm](https://docs.docker.com/engine/swarm/) on\nexternal providers using [Docker Machine](https://docs.docker.com/machine/).\n\nCurrently there is only one provider:\n  1. `digitalocean`\n  \nFurther providers can be added by modifying `manage-swarm` to add the following functions named after that provider:\n  1. `create_\u003cprovider\u003e()` which must create new Docker Swarm nodes, including a master, on the specified provider using `docker-machine` and join them to the same Docker Swarm using the commands returned by \"`docker swarm init`\" command run on the master node.\n  2. `expand_\u003cprovider\u003e()` which must create new Docker Swarm nodes on the specified provider using `docker-machine` and join them to an existing Swarm\n\nThe Docker Machine configuration is stored on every host in the Swarm, which can then be used to import the Swarm onto new systems\nusing the `manage-swarm import` command.\n\nThe IP of any host in the swarm can be used with the `manage-swarm import` command.\nThe `manage-swarm import` command will SSH into the IP specified and download the swarm configuration into the user's Docker Machine\ndirectory as well as place the Docker Machine information into `config/swarms/\u003cswarmName\u003e` so that the swarm can be used by\n`manage-stack` when appropriate.\n\nSwarms may also be destroyed using `manage-swarm destroy`, which uses `docker-machine` to deprovision the backing virtual machines\nand releases all resources.  This should be used with care.\n\n### `manage-stack`\nThe main utility in this suite is called `manage-stack` and it can create, configure, reconfigure, start, stop, update, attach,\nand various other actions to a Wavelet Stack.  A Wavelet Stack is a [Docker Stack](https://docs.docker.com/engine/reference/commandline/stack/)\nfor running [Wavelet](https://github.com/perlin/wavelet/) in a managed way on a [Docker Swarm](https://docs.docker.com/engine/swarm/).  It\nhandles things like gracefully upgrading between releases based on [Docker Images](https://docs.docker.com/engine/reference/commandline/images/),\nextending the number of Wavelet instances in a cluster, ensuring that nodes within a Wavelet cluster are communicating with\neach other, and configuring services so that the cluster can be accessed externally via either the HTTP/HTTPS API or via the\ngRPC RPC interface.\n\nThe main components of a Wavelet Stack are:\n   1. loadbalancer\n   2. sync\n   3. wavelet\n   4. benchmark\n   \nThe definition for these services within the Docker Stack live in the [docker-compose.yml](https://docs.docker.com/compose/compose-file/) file\nin the top-level directory for `wavelet-stack`.\n\nThe `loadbalancer` node is a container running HAProxy that acts as the frontend for all communications to the cluster.  It\nthen proxies requests to the correct destination.\n\nThe `sync` node is a container based on etcd that acts as a way for the Wavelet nodes to advertise their address and port\nto each other as well as to the `loadbalancer` node.\n\nThe `wavelet` node is a container that runs Wavelet.  It reaches out to the `sync` node to register the RPC port it will listen\non as well as pulls in from the `sync` node the RPC IP and RPC port that are exposed externally, as these need to be specified\nby the listening node.  Many instances of this container may be run in a stack and they will coordinate (via the `sync` node)\ntheir activities to ensure that they are communicating.\n\nThe `benchmark` node is a container that runs the `benchmark` command from Wavelet.  Many instaces of this node may be run\nsimultaneously and they will each interact with a different Wavelet instance from the `wavelet` component over the HTTP API.\n\nStack configuration is managed as a set of key-value pairs, identified in an early section.  The stack configuration file is kept\non the Swarm host in the `/etc` directory.  This is so that the configuration may be shared among multiple users of a stack.\n\nOnce a configuration for a stack has been created (refer to examples and references above) using the `manage-stack edit-config`\ncommand the stack may be started using the `manage-stack start` command.\n\nIf the configuration is updated, again using the `manage-stack edit-config` command, the `manage-stack update` command may be\nused.  Both `start` and `update` perform the same action and they are synonyms.\n\nWavelet instances store their data in [Docker Volumes](https://docs.docker.com/storage/volumes/) which are backed by host storage.\nEach instance in a stack gets their own volume.  As long as the stack and host are running, Docker maintains an affinity for\nthe container on the host holding its storage so nodes will be restarted attached to their database when updating.  If the\ncontainer is stopped (e.g., using `manage-stack stop` or `manage-stack reset`) then this affinity is lost and when the node\ncontainers are restarted they may not be on the same host and thus may not get the same volume as before they were stopped or\nreset.  To avoid this case it is usually best to avoid using `manage-stack stop` or `manage-stack reset` unless the data\nis no longer needed.\n\nBy default the volumes are not deleted when the Wavelet Stack is stopped.  This behavior can be changed by setting the\n`WAVELET_CLEAN_VOLUMES` variable to a true value in the stack configuration.  Volumes will also be removed when using\n`manage-stack reset --hard`.  Volumes can also be manually cleaned up for a stopped stack using the command\n`manage-stack cleanVolumes --force`.\n\n## Questions and Answers\n#### Q1. What exact key/value pairs are being stored in etcd, and how are they used?\n  A1. The etcd process on the `sync` instance stores stack-wide configuration details.  This includes a list of what IP and Ports each node is listening (and advertising) on for Wavelet and the Docker port mappings for the stack.  The root of these trees are called \"`peers`\" and \"`mapping`\" within the etcd instance.\n  \n#### Q2. What is the process explicitly step-by-step? Given N nodes and a specific configuration file, how does wavelet-stack setup the cluster from start to finish?\n  A2. When a stack is created \"[docker deploy](https://docs.docker.com/engine/reference/commandline/stack_deploy/)\" will create a Docker Stack, which consists of several docker services, defined in the \"[docker-compose.yml](https://docs.docker.com/compose/compose-file/)\" file.  This file is modified slightly to handle various port configuration options.  Each of those services will create 0 or more containers per service.  The \"benchmark\" service initially starts with 0 containers, the \"sync\" and \"loadbalancer\" services initially start with 1 container, while the \"wavelet\" service starts with the number of containers specified by `WAVELET_NODES` in the stack configuration.  Additionally, the \"manage-stack\" script pushes the generated port mapping details from Docker into keys on etcd running on the \"sync\" instance as soon as its available.  After that, each Wavelet instance starts up and registers the IP address and port is accessible via in etcd on the \"sync\" instance and tries to connect to each other peer registered in etcd on the \"sync\" instance.\n\n#### Q3. What does every single value in a cluster configuration settings file mean, and how does changing each and every single value affect the cluster as a whole? What are preconditions and postconditions for configuring each and every value?\n  A3. Configuration options are:\n\n  1. `REGISTRY` - Docker Registry to use (defaults to `localhost:5000`);  Changing this affects where images are pushed to and pulled from.\n  2. `WAVELET_GENESIS` - Wavelet Genesis block descriptor;  Supplying a genesis block descriptor inhibits generation of one internally, and provides this variable to Wavelet.\n  3. `WAVELET_KEYS` - CSV list of private keys and public keys;  Supplying keys provides a way to specify wallets per run node that are not well-known.  If keys are not specified, the default keys will be used, and they are public.\n  4. `WAVELET_NODES` - Number of Wavelet nodes to run (defaults to `3`)\n  5. `WAVELET_RICH_WALLETS` - Number of Rich wallets to create if generating the Genesis block (that is, if `WAVELET_GENEISIS` is not supplied; defaults to `3`)\n  6. `WAVELET_SNOWBALL_K` - Wavelet Snowball K;  Supply Snowball K parameter to every Wavelet instance\n  7. `WAVELET_SNOWBALL_BETA` - Wavelet Snowball Beta;  Supply Snowball Beta parameter to every Wavelet instance\n  8. `WAVELET_MEMORY_MAX` - Max amount of memory to terminate the node after (in MiB);  Supply the parameter to every Wavelet instance\n  9. `WAVELET_NO_RPC` - Boolean to indicate whether not RPC ports are exposed (if not specified as true, random port)\n  10. `WAVELET_RPC_PORT` - Port to listen for the first node for RPC requests\n  11. `WAVELET_TAG` - Tag of the wavelet image to pull down (defaults to `latest`);  Similar to the `REGISTRY` option this affects the tag on images, both pushed and pulled\n  12. `WAVELET_CLEAN_VOLUMES` - Boolean to indicate whether or not the volumes are removed on `stop` or `reset`\n  13. `WAVELET_API_HOST` - Hostname, if supplied, HTTPS support is enabled on port 443/tcp;  Supply the parameter to every Wavelet instance\n  14. `WAVELET_API_PORT` - Port to listen on for API requests (HTTP-only) (if not specified, random port)\n  15. `WAVELET_API_ACME_ACCOUNT_KEY` - PEM encoded ACME account key for autocert.  Generally if `WAVELET_API_HOST` is provided, this should be provided also.\n  16. `WAVELET_BACKUP_DB` - Boolean to indicate whether database backups are automatically taken for wavelet nodes\n  17. `WAVELET_BUILD_DIR` - Directory to rebuild the \"wavelet\" container from when building all images using `build-all-nodes`\n  18. `WAVELET_REBUILD_ON_START` - Indicate that the `build-all-nodes` script should be run for the given stack when it is started (or reset) (defaults to false)\n\n#### Q4. What exact environment variables are used on each node, and how are they generated/set? At what part of a clusters lifecycle?\n  A4. This is documented as part of the [docker-compose.yml](https://github.com/perlin-network/wavelet-stack/blob/master/docker-compose.yml) as part of the `environment` tag within the YAML file.  The parameters set there come from the stack configuration, with some parameters having default values.\n\n#### Q5. What exact settings is HAProxy configured with? What about for each and every single other component like Docker Machine/Docker Swarm/etc?\n  A5. The HAProxy configuration is set based on [a template](https://github.com/perlin-network/wavelet-stack/blob/901c758974cea19dd69be3b0728d1dbbaf10ab7d/nodes/wavelet-stack-lb/etc/haproxy.cfg.in) and then the details for the API and RPC are added based on exactly how the stack is configured (e.g., how many instances of the `wavelet` node are running, if RPC is open to the outside, if particular RPC or API ports have been specified) by [a script](https://github.com/perlin-network/wavelet-stack/blob/901c758974cea19dd69be3b0728d1dbbaf10ab7d/nodes/wavelet-stack-lb/bin/create-haproxy-cfg).  This script will regenerate the HAProxy configuration file 10 seconds.  If it detects any changes then a new configuration file is written and HAProxy is signalled to gracefully reload.\n\n#### Q6. What happens when a node fails? If a node fails to start up? If a node becomes unresponsive? What are the restart policies?\n  A6. The Docker nodes are all using the default Docker configuration for node management.  Please refer to the [Docker documentation](https://docs.docker.com/engine/reference/commandline/node/)\n\n#### Q7. How do you determine whether or not a node is healthy? How often is this check performed?\n  A7. The Docker health check script is in `nodes/wavelet-stack-node/bin/health` and is run by the `healthcheck` directive, which is documented as part of the [HEALTHCHECK](https://docs.docker.com/engine/reference/builder/#healthcheck) instruction.  The current health check script just validates that the local node has peers.\n   \n#### Q8. If a developer were to go into a cluster and manually stop certain components of the cluster, is there anything they would have to reset to ensure that wavelet-stack wouldn't exhibit strange behavior?\n  A8. Running \"`manage-stack update`\" will ensure that the stack reaches its configured state once again, which will start any [services](https://docs.docker.com/engine/reference/commandline/service/).\n  \n#### Q9. In what setting would I want to use WAVELET_CLEAN_VOLUMES ?\n  A9. You would use `WAVELET_CLEAN_VOLUMES` if you wanted to have the volumes cleaned after every `stop` (which includes in the middle of a `reset` which is a `stop` then `start`).  For example if you were testing building a stack from scratch and needed to wipe the database and all persistent storage after you are done with it.\n\n#### Q10. What is a swarm's lifecycle?  What is a stack's lifecycle?\n  A10. Swarms are created, and may be upgraded, expanded, or destroyed any time after they have been created.  Stacks may be defined (by creating a configuration for that stack using \"`manage-stack edit-config`\"), after being created they can be stopped, reset, have Wavelet on every Wavelet container instance be restarted simultaneously using the \"`manage-stack restart-wavelet`\" command, updated (to apply any changed configuration to the running stack, this will only apply any thing that has changed, if nothing has changed then this will be a no-op), attached to, shelled into, logs inspected, benchmarked, duplicated, or have any Wavelet instances database saved.  Once a benchmark has been started, all of those same things can be done, as well as being able to stop the benchmark. \n\n#### Q11. Could a wavelet instance get stuck in a bootloop when the cluster is being bootstrapped? What are the possible reasons why?\n  A11. Yes, if the stack is misconfigured in many ways Wavelet may not be able to function.  Please refer to the Wavelet documentation to determine under what conditions this could occur.\n\n#### Q12. How exactly are cluster configurations (devnet, mainnet, etc.) configured/stored/replicated/maintained in wavelet-stack?\n  A12. If using an remote swarm: Every time you use `manage-stack edit-config` the new configuration file is uploaded to `/etc/wavelet-stack/\u003cstackName\u003e` on the manager of the Swarm.  If not using a remote swarm: The files are stored in `config/stacks/\u003cstackName\u003e`\n\n#### Q13. What exact HTTP APIs does wavelet-stack call on a wavelet node, what parameters are the calls performed with, and at what part of a clusters lifecycle are they called?\n  A13. It calls `/node/connect`, `/node/disconnect`, and `/ledger`.  The `/node/connect` and `/node/disconnect` API calls use the IP and port of a peer to connect to or disconnect from (respectively).  The `/ledger` API call is done with no parameters.  The `/node/connect` and `/node/disconnect` APIs are called when a Wavelet instance has fewer than `WAVELET_SNOWBALL_K` peers and there are additional peers in the peer registration keys within etcd on the \"sync\" node.  The `/ledger` API call is used to determine how many peers are currently connected to the Wavelet instance as part of determing whether new peers need to be added, as well as part of the Wavelet health check.\n\n#### Q14. Is there any local state stored via wavelet-stack? Or is all state stored remotely?\n  A14. The `docker-machine` information is stored for remote Docker Swarms.  If a remote swarm is used then stack information is also stored on the swarm node, otherwise stack information is stored locally.\n\n#### Q15. What are the preconditions for running each and every single command? What errors can each command possibly produce and what can be done to resolve these errors?\n  A15. There are only 3 commands: `manage-swarm`, `manage-stack`, and `build-all-nodes`.  You only really need to run `build-all-nodes` to build the Docker images before running `manage-stack`.  You can use an remote swarm, in which case you will need to use `manage-swarm` to create or import an remote swarm.  By default the local machine is used for the swarm.\n\n#### Q16. What exactly is the upgrade process for upgrading to a new version of Wavelet? If the auto-updater is not working, what could be done instead?\n  A16. This is handled by upgrading to a new Docker image, and calling `docker stack deploy`, which is documented by [Docker stack deploy](https://docs.docker.com/engine/reference/commandline/stack_deploy/), there's nothing special about `manage-stack` apart from modifying the docker-compose.yml file if needed based on the Stack configuration -- it's the same as a start\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperlin-network%2Fwavelet-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperlin-network%2Fwavelet-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperlin-network%2Fwavelet-stack/lists"}