{"id":38292155,"url":"https://github.com/shieldproject/shield-boshrelease","last_synced_at":"2026-01-17T02:11:15.199Z","repository":{"id":39632695,"uuid":"46288786","full_name":"shieldproject/shield-boshrelease","owner":"shieldproject","description":"BOSH Release for shield","archived":false,"fork":false,"pushed_at":"2025-08-23T04:30:13.000Z","size":806,"stargazers_count":11,"open_issues_count":3,"forks_count":18,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-08-24T07:58:10.314Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shieldproject.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-11-16T16:55:45.000Z","updated_at":"2025-08-23T04:25:16.000Z","dependencies_parsed_at":"2025-08-05T02:18:59.111Z","dependency_job_id":"7aee64f1-b2a7-40c4-8609-7b16705db06c","html_url":"https://github.com/shieldproject/shield-boshrelease","commit_stats":null,"previous_names":[],"tags_count":91,"template":false,"template_full_name":null,"purl":"pkg:github/shieldproject/shield-boshrelease","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shieldproject%2Fshield-boshrelease","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shieldproject%2Fshield-boshrelease/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shieldproject%2Fshield-boshrelease/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shieldproject%2Fshield-boshrelease/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shieldproject","download_url":"https://codeload.github.com/shieldproject/shield-boshrelease/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shieldproject%2Fshield-boshrelease/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28492057,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T00:50:05.742Z","status":"online","status_checked_at":"2026-01-17T02:00:07.808Z","response_time":85,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-17T02:11:15.092Z","updated_at":"2026-01-17T02:11:15.186Z","avatar_url":"https://github.com/shieldproject.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"SHIELD Data Protection\n======================\n\nQuestions? Pop in our [slack channel][slack-help]!\n\n[SHIELD][home] provides an easy-to-use, secure, and rock-solid\ndata protection system for performing backup and recovery\noperations, for both operations and application delivery teams.\n\nThis repository packages SHIELD in a BOSH release for protecting\nthe sensitive data in your BOSHified environment.  It provides the\nSHIELD Core and a local SHIELD Agent.  Optionally, you may deploy\nalongside SHIELD a small WebDAV cloud storage system, complete\nwith BOSH persistent disks.\n\nBOSH Deployment Manifests \u0026 Operator Files\n------------------------------------------\n\nThis release comes with a BOSH (v2) base manifest and a collection\nof operator files for enabling additional features.\n\n### Operator Files\n\n\n\nParts of a SHIELD Deployment\n----------------------------\n\nSHIELD provides the following jobs:\n\n### core\n\nThis is the SHIELD Core.  It provides the heart of SHIELD,\nincluding the API, the database, the job scheduler, the encrypted\nvault, the Web UI, and an HTTPS proxy to run it all.\n\nEvery SHIELD deployment requires exactly one instance of this job.\n\n\n### shield-agent\n\nSome data systems can only be backed up from a local process;\nRedis works this way, since it dumps the backup to local disk.\nFor those systems, you must set up a SHIELD agent, and then\nconfigure SHIELD to initiate the backup via that agent.\n\nThe `shield-agent` job provides this functionality.  Just colocate\nit on the deployment in question, and consume the `shield` link\nfrom your `core` job.\n\n```yaml\ninstance_groups:\n  - name: whatever\n    # etc...\n    jobs:\n      - release:  shield\n        name:     shield-agent\n        consumes:\n          shield: { from: shield, deployment: shield-itself }\n```\n\n(assuming you named your SHIELD Core instance_group \"shield\")\n\n\n### store\n\nAn add-on HTTPS WebDAV installation that provides \"cloud storage\"\non-site but off-VM.  Useful for testing / tinkering, or when you\ndon't have an in-house storage solution and won't (or can't) use\nS3, GCP, or Azure storage.\n\n\n\nUpgrading from v6.x / v7.x\n--------------------------\n\nThis section details deployment manifest changes that operators\nwill need to apply in order to migate from v6 / v7 of SHIELD to\nthis (v8) release.\n\n### Changes to the `shield-daemon` job (now `core`)\n\nThe `shield-daemon` job is now just `core`.\n\nThe `name` property is gone.  In its place are the following\nproperties for identifying your SHIELD:\n\n  - `core.env` - The name of the environment, like \"sandbox\" or\n      \"production\", or \"a testing shield instance\", or \"fred\".\n  - `core.color` - A CSS color name, or hexadecimal RGB color to\n      use for the environment name in the new web UI.  `yellow`\n      and `green` look nice.\n  - `core.motd` - A (possible multi-line) message that will be\n      displayed to users logging into SHIELD.  Useful for whatever\n      messages of the day are generally useful for (compliance,\n      advertising maintenance windows, etc.)\n\n`workers` has been renamed to `core.workers`, but otherwise retains\nits semantic meaning.\n\n`max_timeout` has been renamed to `core.task-timeout`, but\notherwise retains its semantic meaning.\n\n`ssl.key`, `ssl.crt`, and `ssl.timeout` have been renamed to\n`tls.key`, `tls.certificate`, and `tls.reuse-after`, respectively.\nThe default value of `tls.reuse-after` was dropped from 12 (hours)\nto 2 (hours).\n\nThe `ssh_private_key` has been renamed to `agent.key`, because\nit's not used for SSH in the same sense as most SSH (RSA) keys.\nIts value should stay the same for a smooth upgrade.\n\nThe `database.*` properties have been removed; SHIELD v8 has its\nown internal data store that does not need to be configured by the\noperator.  See the **Database Migration** subsection, later, for\ndetails on migrating your data into this new data store.\n\nThe `auth.oauth.*` properties have been removed;  SHIELD v8\nsupports multiple (possibly OAuth2-based) authentication\nproviders.  These are configured under the new top-level\n`authentication` key.\n\nThe `auth.username` and `auth.password` properties have been\nremoved;  SHIELD v8 no longer supports simple HTTP Basic\nAuthentication.  Instead, it features a robust user authentication\nsystem backed by an internal local user database.  Two new\nproperties, `failsafe.username` and `failsafe.password` kind of\ntake over for these deprecated properties.  They specify a\nusername and (cleartext) password that SHIELD will insert into the\nlocal user database if it boots up and finds that no users exist\nyet.  This \"failsafe\" is intended to provide a secure way of\nbootstrapping a SHIELD environment, without being stuck with a\nuser whose password is in a BOSH manifest somewhere.\nAdministrators are free to delete the failsafe user once they have\nset up other accounts.\n\nThe `auth.api_keys` property has been removed; SHIELD v8 does not\nsupport API Keys in the same fashion as its predecessors.\nInstead, user accounts are free to issue _Auth Tokens_ that behave\na stand-ins for their issuer (not unlike Github Personal Access\nTokens).\n\n`nginx.worker_processes` has been shortened to `nginx.workers`.\n\n`nginx.worker_connections` has been shortened to\n`nginx.connections`.\n\n`nginx.keepalive_timeout` has been shortened to `nginx.keepalive`.\n\nThe `log_level` property has been renamed to `log-level`.\n\n### Changes to the `shield-agent` job\n\nThis job is still called `shield-agent`, since it needs to be\nunique across a wide variety of other deployments.\n\n`name` is a new property for specifying the name this agent will\nuse when registering with the SHIELD Core.\n\n`autoprovision` has been removed.  Its usage was always\nproblematic, and with the introduction of proper BOSH links, we\nonly need to specify where and how to talk to the SHIELD Core in\nthe event that our Core lives on another BOSH director (which is\nrare).\n\n`shield-url` is a new property that kind of takes the place of\n`autoprovision`, by allowing operators to identify where their\nSHIELD Core lives, as a full URL (i.e.\n\"https://shield.example.com\")\n\n`require-shield-core` is a new property that lets operators ignore\nan error condition whereby an agent is unable to communicate with\nthe SHIELD Core.  In theory, that is a show-stopping problem, but\nin practice, we found that it held up too many deployments for\nlegitimate reasons, ranging from simple network connectivity\nissues and firewalling to more mundane problems like \"we haven't\ndeployed SHIELD itself yet.\"\n\nThe `daemon_public_key` property has been removed.  In its place\nis the new `agent.key` property.  The meaning of the property is\nstill the same, i.e. you should specify the\nauthorized_keys-formatted public key (i.e. `ssh-rsa AAA...`).\n\nNote that if the `shield` link is in use, you don't need to\nexplicitly set `agent.key` -- the agent startup scripts will just\nretrieve the public key from the SHIELD Core automagically.  This\nallows SHIELD site operators to rotate that key with minimal fuss.\n\nThe `recovery.*` properties have been removed, since SHIELD v8's\nnew encryption feature makes it difficult to restore backups\noutside of the watchful eye of a running SHIELD Core.\n\nFor SHIELD Agents that need to operate behind HTTP proxies, three\nnew `env.*` properties were added.  `env.http_proxy` and\n`env.https_proxy` allow you to specify the full URL for an\nupstream proxy that will handle (respectively) cleartext HTTP\nrequests and TLS-encrypted HTTPS requests.  The `env.no_proxy`\nproperty is a list of FQDNs, domain fragments, and IP addresses\nthat will be flattened and joined by commas to fashion an\nexclusion list to put in the `$no_proxy` environment variable.\n\nThe new `env.path`, `env.libs`, and `env.auto` properties control\nhow the SHIELD agent process will set up its environment, for the\nbenefit of executed plugins.\n\n`env.path` is a list of auxiliary paths to bin/ and sbin/\ndirectories that you want to manually inject into the `$PATH` of\nthe running shield-agent / plugins.\n\n`env.libs` is a list of auxiliary paths to lib/ directories that\nyou want / need in your `$LD_LIBRARY_PATH` for dynamic shared\nobject runtime loading.\n\n`env.auto` is a boolean; if set, the shield-agent job will go\nlooking for installed BOSH packages named `shield-addon-*`, add\nany bin/ and sbin/ directories to `$PATH`, and add any lib/\ndirectories to `$LD_LIBRARY_PATH`.  This allows you to augment an\nagent with additional command-line tools it might need, like\nspecific versions of `psql`, or `xtrabackup`.  `env.auto` is on by\ndefault.\n\nThe auto-provisioning properties `stores`, `targets`,\n`retention-policies`, and `jobs` have all been removed, in favor\nof the new `shield import`-based `import` errand.\n\nThe `log_level` property has been renamed to `log-level`.\n\n### Removed Jobs\n\nThe `agent-mysql` and `xtrabackup` jobs have been removed.  If you\nneed to augment a SHIELD agent with MySQL / MariaDB tools, you can\ntry the nee [SHIELD MySQL Addon][mysql-addon], which contains all\nof these packages.\n\nThe `agent-pgtools` job has been removed.  If you need to augment\na SHIELD agent with PostgreSQL tools, you can try the new [SHIELD\nPostgreSQL Addon][postgres-addon].\n\nThe `mongo-tools3.2` and `mongo-tools3.4` jobs have been removed.\nThey too have moved into a separate BOSH release, the [SHIELD\nMongoDB Addon][mongodb-addon].\n\nThe `postgres` and `mariadb` jobs have been removed.  SHIELD v8\nnow leverages a standalone, dedicated database that is baked into\nthe new `core` job.  See the subsection **Database Migration**,\nbelow, for details on migrating your SHIELD data.\n\nThe `nginx` job has been removed. It is now integrated into core.\n\n### The New Import Errand\n\nPrevious versions of the SHIELD BOSH release used a post-start\nscript and `shield-agent` properties to facilitate a form of\nconfiguration auto-provisioning.\n\nIn v8, this has all been replaced by the new `import` errand,\nwhich drives the much more powerful and flexible `shield import`\ncommand-line tool.\n\nThe `import` errand takes a single property, `import`, which is a\nfull recipe of things to import into SHIELD, as understood by the\n`shield` tool's `import` sub-command.\n\nHere's an example that sets up a bunch of stuff:\n\n```yaml\n- name: import\n  lifecycle: errand\n  instances: 1\n  azs: [z1]\n  vm_type:         default\n  stemcell:        default\n  networks: {name: default}\n  jobs:\n    - name:    import\n      release: shield\n      properties:\n        import:\n          core:  https://shield.example.com\n          token: ... # an auth token, from `shield create-auth-token`\n\n          global:\n            storage:\n              - name: S3 Cloud Storage\n                summary: |\n                  Public S3 cloud storage for all SHIELD tenants to use\n                agent:  127.0.0.1:5444\n                plugin: s3\n                config:\n                  access_key_id:     AKI12\n                  secret_access_key: secret\n\n            policies:\n              - name: Long-Term Storage\n                days: 90\n\n          users:\n            - name:     James Hunt\n              username: jhunt\n              password: sekrit\n              sysrole:  admin\n              tenants:\n                - name: Stark \u0026 Wayne\n                  role: admin\n\n          tenants:\n            - name: CF Community\n              members:\n                - user: jhunt@local\n                  role: admin\n              storage:\n                - name: Scality\n                  agent:  10.32.45.10:5444\n                  plugin: scality\n                  config:\n                    s3_host: 10.32.45.1\n                    s3_port: 8200\n                    bucket:  my-bucket\n\n              policies:\n                - name: Short-Term\n                  days: 7\n                - name: Long-Term\n                  days: 90\n\n              systems:\n                - name:   BOSH\n                  agent:  10.4.0.1:5444\n                  plugin: postgres\n                  config: {}\n                  jobs:\n                    - name:    Daily\n                      when:    daily 4am\n                      policy:  Short-Term\n                      storage: Scality\n                      paused:  true\n\n                    - name:    Monthly\n                      when:    every month on the 1st at 3am\n                      policy:  Long-Term\n                      storage: Scality\n```\n\n\n### New Encryption Vault\n\nSHIELD v8 encrypts all backup archives, and it uses a unique,\nrandomly generated initialization vector and encryption key for\neach new archive.  These secrets are required for restoration, and\nthey have to be stored somewhere safe, so we store then in an\nencrypted vault.\n\nFor the most part, the care and feeding of this vault is entirely\nhandled for you.  However, the deployment needs to configure an\nX.509 Certificate Authority, and issue an X.509 Certificate for\nthe IP SAN 127.0.0.1.\n\n\n### Database Migration\n\nSHIELD v8 introduces several new features, including a new\nbuilt-in data store.  Chances are if you are upgrading from a\nprevious version of SHIELD (either v6 or v7), you are going to\nwant to migrate all that data.  To do so safely and effectively,\nyou just need to specify the `migrate-from.type` and\n`migrate-from.dsn` properties in your SHIELD deployment manifest.\n\nFor example, if you had a v6 SHIELD BOSH deployment manifest with\n`shield-daemon` properties that looked like this:\n\n```yaml\n# old school\nproperties:\n  database:\n    type:     postgres\n    host:     10.5.6.7\n    port:     5524\n    username: dba\n    password: sekrit\n    database: shield1\n```\n\nThen your `migrate-from.type` should be \"postgres\", and\n`migrate-from.dsn` should roll up all that connectivity\ninformation in a PostgreSQL data source name, like this:\n\n```yaml\n# new school\nproperties:\n  migrate-from:\n    type: postgres\n    dsn: postgres://dba:sekrit@10.5.6.7:5524/shield1?sslmode=disable\n```\n\nLikewise, if you used to use MySQL for SHIELD, and had this in\nyour manifest:\n\n```yaml\n# old school\nproperties:\n  type:     mysql\n  host:     172.15.3.4\n  port:     3316\n  username: scyld\n  password: sekrit\n  database: shielddb\n```\n\nYou would want to specify this to engage data migration:\n\n```yaml\n# new school\nproperties:\n  type: mysql\n  dsn:  scyld:sekrit@tcp(172.15.3.4:3316)/shielddb\n```\n\nRefer to the [lib/pq documentation][pq-dsn] and the\n[go-sql-driver/mysql documentation][my-dsn] for more details.\n\nNote that database migration is a once-only affair.  If the\ninternal database file exists, the release will skip migration\naltogether.  This takes some of the urgency out of needing to\n\"clean up\" your deployment manifest to remove the `migrate-from.*`\nproperties.\n\n### New Deprovision Errand\n\nThe new `deprovision` errand is used to delete a tenant and clean up existing\nconfiguration under that tenant. Specified tenants will be deleted along with\ntheir tasks, jobs, stores, targets, memberships, as well as have their tasks purged.\n\nAnd example deprovision job under the shield instance\n```yaml\n  - name: deprovision\n      release: shield\n      properties:\n        domain: shield.example.com\n        tenants_to_delete:\n          - tenant1\n          - tenant2\n        generate_token: true\n```\n\nMore Documentation\n------------------\n\nTo brush up on SHIELD, you'll want to check out\n[shieldproject.io][home], specifically the [documentation\nsection][docs].\n\n\n\nGetting Help\n------------\n\nIf you just need help getting things up and running, or have a\nquestion about how SHIELD works, how to backup and restore\nsomething with it, or just want to chat, we have a Slack Support\nOrganization, `shieldproject.io` - it's open-invitation, you can\n[join for free today!][slack-join]\n\n\nIf you've found a bug, please visit our [Github Issue\nTracker][github].\n\n\n\n\n\n[slack-help]:     https://shieldproject.slack.com/messages/help\n[slack-join]:     https://shieldproject.io/community#slack\n[home]:           https://shieldproject.io\n[docs]:           https://shieldproject.io/docs/latest/ops/\n[github]:         https://github.com/shieldproject/shield/issues\n\n[mysql-addon]:    https://github.com/shieldproject/shield-addon-mysql-boshrelease\n[postgres-addon]: https://github.com/shieldproject/shield-addon-postgres-boshrelease\n[mongodb-addon]:  https://github.com/shieldproject/shield-addon-mongodb-boshrelease\n\n[pq-dsn]:         https://godoc.org/github.com/lib/pq\n[my-dsn]:         https://github.com/go-sql-driver/mysql#dsn-data-source-name\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshieldproject%2Fshield-boshrelease","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshieldproject%2Fshield-boshrelease","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshieldproject%2Fshield-boshrelease/lists"}