{"id":17383103,"url":"https://github.com/voronenko/hashi_vault_utils","last_synced_at":"2025-04-15T09:52:33.324Z","repository":{"id":145345503,"uuid":"42526477","full_name":"Voronenko/hashi_vault_utils","owner":"Voronenko","description":"handy batches to execute common tasks with vault","archived":false,"fork":false,"pushed_at":"2018-07-04T10:59:05.000Z","size":30,"stargazers_count":80,"open_issues_count":0,"forks_count":29,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-28T19:07:09.602Z","etag":null,"topics":["hashicorp","shell-scripts","vault","vault-010x","vault-06x","vault-09x"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Voronenko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-15T14:58:14.000Z","updated_at":"2025-02-13T19:47:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"c834b9f7-4a03-46bd-94eb-1cf18dea4662","html_url":"https://github.com/Voronenko/hashi_vault_utils","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Voronenko%2Fhashi_vault_utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Voronenko%2Fhashi_vault_utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Voronenko%2Fhashi_vault_utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Voronenko%2Fhashi_vault_utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Voronenko","download_url":"https://codeload.github.com/Voronenko/hashi_vault_utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249048711,"owners_count":21204305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hashicorp","shell-scripts","vault","vault-010x","vault-06x","vault-09x"],"created_at":"2024-10-16T07:40:36.816Z","updated_at":"2025-04-15T09:52:33.305Z","avatar_url":"https://github.com/Voronenko.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"Using Vault by HashiCorp to secure your deployment secrets\n==========================================================\n\nEducational repository demonstrating approaches for safe secure deployment of passwords, api keys etc\n\n\n[![Build Status](https://travis-ci.org/Voronenko/hashi_vault_utils.svg?branch=master)](https://travis-ci.org/Voronenko/hashi_vault_utils)\n\n# Updates\n\nBatches were updated to conform with vault API versions 0_9 and 0_10 ;\nIf your vault is less than 0.10.0 , - you need to checkout 0.9.0 branch.\n\n# Background\n\"Don’t Check Passwords into Source Control or Hard-Code Them\nin Your Application\nOperations staff will remove your eyes with a spoon if they catch you doing this.\nDon’t give them the pleasure Passwords should always be entered by the user performing the deployment. There are several acceptable ways to handle authentication for a multilayer system. You could use certificates, a directory service, or a single sign-on system\". This quote is taken from Chapter 2 of the Continuous Delivery: Reliable Software Releases Through Build, Test, And Deployment Automation (Addison-Wesley Signature Series (Fowler)) book by and David Farley, Jez Humble\n\nVault by HashiCorp is one of the tools that might provide acceptable level of security for devops engineers for enterprise scenarios as well as for smaller teams like startups.\n\n## Challenges to address\n  At the end of the article we should be able\n\n- install vault on a ubuntu 14.04 :TS server\n- initialize vault\n- store secrets in vault\n- access secrets in vault\n\n# Installing Manually\n\nFormal installation steps are covered by this article: [https://vaultproject.io/docs/install/](https://vaultproject.io/docs/install/)\nFor purposes of the demo article let me provide semi automated script, that installs vault *0.1.2* into /opt/vault_0.1.2 folder , configures it to listen on localhost port 8200 and registers it as a service called vault-server\n\u003cpre\u003e\n#!/bin/sh\n\nVAULT_VERSION=${VAULT_VERSION-0.5.2}\nVAULT_PATH=/opt/vault_$VAULT_VERSION\nUNAME=`uname -m`\n\nif [ \"$UNAME\" != \"x86_64\" ]; then\n  PLATFORM=386\nelse\n  PLATFORM=amd64\nfi\n\n\nif [ \"$(id -u)\" != \"0\" ]; then\n\techo \"Installation must be done under sudo\"\n\texit 1\nfi\n\ntest -x $VAULT_PATH/vault\nif [ $? -eq 0 ]; then\n    echo vault already installed\n    exit 1\nfi\n\napt-get install -y curl unzip\n\nrm /opt/vault_${VAULT_VERSION}_linux_${PLATFORM}.zip\n\ncurl -L \"https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_${PLATFORM}.zip\" \u003e /opt/vault_${VAULT_VERSION}_linux_${PLATFORM}.zip\n\nmkdir -p $VAULT_PATH\n\nunzip /opt/vault_${VAULT_VERSION}_linux_${PLATFORM}.zip -d $VAULT_PATH\n\nchmod 0755 $VAULT_PATH/vault\nchown root:root $VAULT_PATH/vault\n\necho create config\n\ncat \u003c\u003cEOF \u003e$VAULT_PATH/vault-config.hcl\nbackend \"file\" {\n  path = \"$VAULT_PATH/storage\"\n}\n\nlistener \"tcp\" {\n  address = \"127.0.0.1:8200\"\n  tls_disable = 1\n}\nEOF\n\necho create run script\ncat \u003c\u003cEOF \u003e$VAULT_PATH/vault-server\n#!/bin/sh\nif [ -z \\$1 ]\nthen\n  echo syntax: vault-server /PATH/TO/VAULT/HCL/CONFIG optional_flags\n  exit 1\nfi\nBASEDIR=\\$(dirname \\$0)\ncd \\$BASEDIR\n./vault server -config=\\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9\nEOF\n\nchmod 0755 $VAULT_PATH/vault-server\nchown root:root $VAULT_PATH/vault-server\n\necho create upstart script\ncat \u003c\u003cEOF \u003e/etc/init/vault-server.conf\ndescription \"Vault server\"\nstart on runlevel [2345]\nstop on runlevel [!2345]\nrespawn\nscript\n  # Make sure to use all our CPUs, because Vault can block a scheduler thread\n  export GOMAXPROCS=`nproc`\n  exec $VAULT_PATH/vault-server ${VAULT_PATH}/vault-config.hcl \u003e\u003e/var/log/vault.log 2\u003e\u00261\nend script\nEOF\n\necho linking vault binary to /usr/local/bin/vault\nsudo ln -s $VAULT_PATH/vault /usr/local/bin/vault\n\nservice vault-server start\ncat /var/log/vault.log\n\n\u003c/pre\u003e\n\n\n# Installing automatically with Ansible\n\nSee [https://github.com/softasap/sa-hashicorp-vault](https://github.com/softasap/sa-hashicorp-vault)  Ansible Role for unattended Vault installation\n\n# Check the installation:\n\u003cpre\u003e\n./vault_status.sh\nError checking seal status: Error making API request.\n\nURL: GET http://localhost:8200/v1/sys/seal-status\nCode: 400. Errors:\n\n* server is not yet initialized\n\u003c/pre\u003e\nMessage means, that vault was installed and configured correctly, but needs to be initialized. Initialization happens once when the server is started against a new backend that has never been used with Vault before. During initialization, the encryption keys are generated, unseal keys are created, and the initial root token is setup. To initialize Vault use vault init. This is an unauthenticated request, but it only works on brand new Vaults with no data\n\nLet's init. Important influence on security has number of key shares to generate and number of key shares provided to unlock the seal.\n\nHow does it work: the key used to encrypt the data is also encrypted using 256-bit AES in GCM mode. This is known as the master key. The encrypted encryption key is stored in the backend storage. The master key is then split using Shamir's Secret Sharing. Shamir's Secret Sharing ensures that no single person (including Vault) has the ability to decrypt the data. To decrypt the data, a threshold number of keys (by default three, but configurable) are required to unseal the Vault. Thesekeys are expected to be with three different places / individuals.\n\nIt has full analogy to secure bank cell where one key has bank personnel and one is yours.In case of vault you might have much higher level of security.\n\nFor demo purposes I will use single key only.\n\n\u003cpre\u003e\n./vault_init.sh\nThe number of key shares to split the master key into: 1\nThe number of key shares required to reconstruct the master key 1\nKey 1: af29615803fc23334c3a93f8ad58353b587f50eb0399d23a6950721cbae94948\nInitial Root Token: 98df443c-65ee-d843-7f4b-9af8c426128a\n\nVault initialized with 1 keys and a key threshold of 1!\n\nPlease securely distribute the above keys. Whenever a Vault server\nis started, it must be unsealed with 1 (the threshold) of the\nkeys above (any of the keys, as long as the total number equals\nthe threshold).\n\nVault does not store the original master key. If you lose the keys\nabove such that you no longer have the minimum number (the\nthreshold), then your Vault will not be able to be unsealed.\n\u003c/pre\u003e\n\nInitial Root Token must be immediately saved in a secure location.\n\n# Using vault\n\n## Unsealing\nWhen a Vault server is started, it starts in a sealed state. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data, thus prior to unsealing, almost no operations are possible with Vault.\n\nLet's unseal:\n\u003cpre\u003e\n./vault_ unseal af29615803fc23334c3a93f8ad58353b587f50eb0399d23a6950721cbae94948\nSealed: false\nKey Shares: 1\nKey Threshold: 1\nUnseal Progress: 0\n\u003c/pre\u003e\n\nNote, if you had higher threshold set, all the key holders would need to perform unseal operation with their parts of the key.  That's provides additional level of security for accessing the data\n\n## Authorization\nIn order to continue working with vault, you should first identify yourself.\nLet's use auth command to do this by providing our initial root token\n\n\u003cpre\u003e\n./vault_ auth 98df443c-65ee-d843-7f4b-9af8c426128a\nSuccessfully authenticated! The policies that are associated\nwith this token are listed below:\n\nroot\n\n\u003c/pre\u003e\n\n## Policies\n\nAccess control policies in Vault control what a user can access.When initializing Vault, only the \"root\" policy is present. It gives superuser access to everything in Vault.\n\nAs we plan to store secrets for saying multiple projects, we should be able to clearly separate access to secrets that belong to different projects. And this is where policies do their job.\n\n\nPolicies in Vault are formatted with HCL. HCL is a human-readable configuration format that is also JSON-compatible, so you can use JSON as well. An example policy is shown below:\n\n\u003cpre\u003e\npath \"secret/project/name\" {\n  policy = \"read\"\n}\n\u003c/pre\u003e\n\nIt specify path, like we have in some tree structure, wildcards are supported.\nIf you provide access to specific part of the tree, you also provide the same access to all subnodes, unless you override it.\n\nPolicy is registered with *policy-write* command\n\u003cpre\u003e\n./vault_ policy-write demo demo.hcl\nvault policy-write -address=http://localhost:8200 demo demo.hcl\nPolicy 'demo' written.\n\u003c/pre\u003e\n\n## Deployment tokens\n\nNow it is time to create deployment token. In our case, this is token that would allow us to read the secret deployment value from vault, and does not have any additional privileges except this.\n\nIn order to do so, we are using creating token with policy command\n\n\u003cpre\u003e\n./vault_create_token_with_policy.sh demo\nvault token-create -address=http://localhost:8200 -policy=demo\n4d79adad-a4ec-de8b-3f85-5467b3e8536a\n\u003c/pre\u003e\n\n\n## Storing data\n\nNow it is time to store some secrets for deployment. For purposes of the demo, let it be some api key and private key used for deployment.\n\nCommand write is used to write the secrets\n\u003cpre\u003e\n./vault_write.sh secret/project/name/apikey BLABLABLA\nvault write -address=http://localhost:8200 secret/project/name/apikey value=BLABLABLA\nSuccess! Data written to: secret/project/name/apikey\n\n./vault_write_file.sh secret/project/name/id_rsa ./demo_rsa\nSuccess! Data written to: secret/project/name/id_rsa\n\u003c/pre\u003e\n\n### Important\nBinary file storing is not supported as for now, but you always can store base64 encoded file, like the MIME attachments are stored in mails.\nFortunately, for most deployments we have api keys and private keys that are text files.\n\n## Retrieving the data\n\nThere are two ways to access your data.\nFirst is using vault client itself\n\n\u003cpre\u003e\n./vault_read.sh secret/project/name/apikey\nvault read -address=http://localhost:8200 secret/project/name/apikey\nKey            \tValue\nlease_id       \tsecret/project/name/apikey/a74dd189-de4b-1c98-ba24-6b29258c511b\nlease_duration \t2592000\nlease_renewable\tfalse\nvalue          \tBLABLABLA\n\n./vault_read.sh secret/project/name/id_rsa\nvault read -address=http://localhost:8200 secret/project/name/id_rsa\nKey            \tValue\nlease_id       \tsecret/project/name/id_rsa/204ba657-9648-4fa5-8f82-ede992a054b4\nlease_duration \t2592000\nlease_renewable\tfalse\nvalue          \t-----BEGIN RSA PRIVATE KEY-----\nMIIEpgIBAAKCAQEApiLCR2sgf5unedMk1a2maL22PsoPwQWpGTDFZgCvhSVWvnBs\n...\n\u003c/pre\u003e\n\nsecond is using http based API. For that scenario you will need to authorize via deployment token we allocated previously.\n\u003cpre\u003e\n./vault_curl.sh 4d79adad-a4ec-de8b-3f85-5467b3e8536a secret/project/name/apikey\ncurl -H X-Vault-Token: 4d79adad-a4ec-de8b-3f85-5467b3e8536a -X GET http://localhost:8200/v1/secret/project/name/apikey\n{\"lease_id\":\"secret/project/name/apikey/2189c6c4-1fa7-0f4d-2598-bded29a4ce6b\",\"renewable\":false,\"lease_duration\":2592000,\"data\":{\"value\":\"BLABLABLA\"},\"auth\":null}\n\n./vault_curl.sh 4d79adad-a4ec-de8b-3f85-5467b3e8536a secret/project/name/id_rsa\ncurl -H X-Vault-Token: 4d79adad-a4ec-de8b-3f85-5467b3e8536a -X GET http://localhost:8200/v1/secret/project/name/id_rsa\n{\"lease_id\":\"secret/project/name/id_rsa/ec509e1f-09a7-6aee-54e2-f3364720c7de\",\"renewable\":false,\"lease_duration\":2592000,\"data\":{\"value\":\"-----BEGIN RSA PRIVATE KEY-----\\nMIIEpgI......-----END RSA PRIVATE KEY-----\"},\"auth\":null}\n\u003c/pre\u003e\n\n\n# Securing vault http api\nVault supports https itself, but I believe for production deployment it would be better to hide it behind web server as a proxy.\n\nBelow is an example of nginx configuration\n\u003cpre\u003e\nserver {\n  listen 443 ssl;\n  server_name vault.YOURDOMAIN.COM;\n\n  ssl_certificate YOUR_SSL_CERTIFICATE.crt;\n  ssl_certificate_key YOUR_SSL_CERTIFICATE_KEY.key;\n\n  location / {\n    proxy_pass http://127.0.0.1:8200;\n    proxy_set_header Host $host;\n    expires -1;\n  }\n\n  #ssl config per https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html\n  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;\n\n  ssl_ciphers \"EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA256:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EDH+aRSA+AESGCM:EDH+aRSA+SHA256:EDH+aRSA:EECDH:!aNULL:!eNULL:!MEDIUM:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!SEED\";\n  ssl_prefer_server_ciphers on;\n\n  ssl_dhparam dhparam.pem;\n\n  #only supported since 1.3.7\n  ssl_stapling on;\n  ssl_stapling_verify on;\n\n  # Optimize SSL by caching session parameters for 10 minutes. This cuts down on the number of expensive SSL handshakes.\n  # The handshake is the most CPU-intensive operation, and by default it is re-negotiated on every new/parallel connection.\n  # By enabling a cache (of type \"shared between all Nginx workers\"), we tell the client to re-use the already negotiated state.\n  # Further optimization can be achieved by raising keepalive_timeout, but that shouldn't be done unless you serve primarily HTTPS.\n  ssl_session_cache    shared:SSL:10m; # a 1mb cache can hold about 4000 sessions, so we can hold 40000 sessions\n  ssl_session_timeout  10m;\n\n  add_header Strict-Transport-Security max-age=63072000;\n  add_header X-Frame-Options DENY;\n  add_header X-Content-Type-Options nosniff;\n}\n\u003c/pre\u003e\n\n# Code in action\n\nCode can be downloaded from repository [https://github.com/Voronenko/hashi_vault_utils](https://github.com/Voronenko/hashi\\_vault\\_utils \"https://github.com/Voronenko/hashi\\_vault\\_utils\")\n\nSome files just help using existing vault functionality in a more handy way:\n\n- vault_status.sh - gets status of the vault\n- vault_policy.sh - lists known policies, or shows details of the policy provided as a first parameter\n- vault_create_token_with_policy.sh creates and returns token with policy provided as a first parameter.\n- vault_read.sh reads secret by key (first parameter)\n- vault_write.sh writes secret by key (first parameter) and set's it's value (second parameter)\n- vault_write\\_file.sh writes secret by key (first parameter) and stores content's of text file provided as second parameter\n- vault_curl.sh can be used to test http api. First parameter - access token, second parameter secret key to read\n\n\n# Devops: usage with ansible\n\nLookup plugin:\n[github.com/Voronenko/ansible-developer_recipes/blob/master/ansible_extras/lookup_plugins/sa_hashi_vault.py](https://github.com/Voronenko/ansible-developer_recipes/blob/master/ansible_extras/lookup_plugins/sa_hashi_vault.py)\n\n\nAction module:\n[github.com/Voronenko/ansible-developer_recipes/blob/master/ansible_extras/action_plugins/sa_hashi_vault.py](https://github.com/Voronenko/ansible-developer_recipes/blob/master/ansible_extras/action_plugins/sa_hashi_vault.py)\n\nGalaxy role:\n[softasap/sa-hashicorp-vault](https://galaxy.ansible.com/softasap/sa-hashicorp-vault/)\n\nStandalone ansible deployment:\n[sa-hashicorp-vault box example](https://github.com/softasap/sa-hashicorp-vault/tree/master/box-example)\n\n\n\n# Points of interest\n\nThis covers only very basic aspects to start using vault in your organization, but could be a  nice first step to move forward.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoronenko%2Fhashi_vault_utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvoronenko%2Fhashi_vault_utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoronenko%2Fhashi_vault_utils/lists"}