{"id":19040934,"url":"https://github.com/cloudfoundry/exemplar-release","last_synced_at":"2025-07-29T21:44:17.624Z","repository":{"id":18353803,"uuid":"83743761","full_name":"cloudfoundry/exemplar-release","owner":"cloudfoundry","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-28T16:23:45.000Z","size":97,"stargazers_count":24,"open_issues_count":0,"forks_count":9,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-04-18T06:51:00.371Z","etag":null,"topics":["bosh"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudfoundry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-03T01:37:50.000Z","updated_at":"2025-03-15T15:24:42.000Z","dependencies_parsed_at":"2024-11-08T22:27:18.339Z","dependency_job_id":"c27b8d34-d4a1-4f40-9595-ed7077556f91","html_url":"https://github.com/cloudfoundry/exemplar-release","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudfoundry%2Fexemplar-release","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudfoundry%2Fexemplar-release/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudfoundry%2Fexemplar-release/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudfoundry%2Fexemplar-release/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudfoundry","download_url":"https://codeload.github.com/cloudfoundry/exemplar-release/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250516152,"owners_count":21443564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bosh"],"created_at":"2024-11-08T22:26:10.823Z","updated_at":"2025-04-23T21:25:50.958Z","avatar_url":"https://github.com/cloudfoundry.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Exemplar Release\n\nThis exemplar [BOSH][bosh] release is intended as a collection of recommended practices for authoring a BOSH release,\nwith a particular focus on jobs. Not all advice here should be taken verbatim for all releases, so we have made an\nattempt to explain the reasoning behind all recommendations so that the conscientious release author can adapt the\nrecommendations to their unique circumstances.\n\n[bosh]: https://bosh.io\n\n\u003c!-- toc --\u003e\n\n- [Order Independence](#order-independence)\n- [BOSH Job Lifecycle](#bosh-job-lifecycle)\n  * [Starting](#starting)\n    + [Pre-Start](#pre-start-docs)\n    + [Monit Start](#monit-start)\n      - [Logging](#logging)\n    + [Post-Start](#post-start-docs)\n    + [Post-Deploy](#post-deploy-docs)\n  * [Stopping](#stopping)\n    + [Pre-stop](#pre-stop-docs)\n    + [Drain](#drain-docs)\n    + [Monit Stop](#monit-stop)\n    + [Post-stop](#post-stop-docs)\n- [Advice for authoring `spec` files](#advice-for-authoring-spec-files)\n  * [Properties](#properties)\n  * [Links](#links-docs)\n- [Template Advice](#template-advice)\n  * [ERB](#erb)\n    + [Encoding](#encoding)\n    + [Testing](#testing)\n  * [bash](#bash)\n- [Monit Advice](#monit-advice)\n- [Exporting Logs](#exporting-logs)\n  * [Metron Agent](#metron-agent)\n  * [The Ol' `tee` and `logger` Approach](#the-ol-tee-and-logger-approach)\n- [Bosh DNS](#bosh-dns)\n  * [Aliases](#aliases)\n  * [Readiness with `bin/dns/healthy`](#readiness-with-bin-dns-healthy)\n- [Backup and Restore features](#backup-and-restore-features)\n\n\u003c!-- tocstop --\u003e\n\n## Order Independence\n\nThe primary goals of a BOSH job are to maximize availability, remove the need for manual operation, and contribute to\nquick deployments. To accomplish this, it is important to understand the BOSH deployment lifecycle and what your job\nshould be doing to take advantage of it.\n\nA process should be able to start and stay running without dependencies being ready. Only once its required dependencies\n(those needed to respond to requests) are reporting as healthy should it report as healthy. This allows jobs to be\nstarted in any order and then propagate health upwards through their dependents when they're ready. We think removing\nthis burden of job ordering from the operator reduces complexity and increases system reliability.\n\nOn the other hand, stopping can follow a similar but reversed pattern. If your service is being load-balanced then you\ncan use a signal in your `drain` script to mark yourself as unhealthy. After a period of time or after no more requests\nare detected you can stop the process completely. If you aren't behind a load balancer then you will have to find\nanother way to tell upstream components about your imminent demise.\n\n\n## New BOSH Job Lifecycle with BPM\n\nBPM stands for BOSH Process Manager. You can get more information about it in the [BPM chapter](bpm-chapter) of the BOSH\ndocumentation. The [BPM release repository](bpm-release) is also a great place where to find more information about BPM.\n\nWe provide here a [`sample-app` job](./jobs/sample-app) that shows how simple things can be when delegating daemons\nmanagement to BPM. That said, BPM doesn't provide an answer to all use cases yet. Classical BOSH `pre-start` and `drain`\nscripts are still necessary for the use cases documented below.\n\nOne note about BPM `pre_start` hooks: contrarily to BOSH `pre-start`, BPM `pre_start` scripts are run at every\n`monit start`, i.e. even when the BOSH instance gets back up after being evicted by the underlying Cloud\ninfrastructure. Actually, you should put in your BPM `pre_start` what you would have put in your `start` script before\n`exec`-ing your process.\n\n[bpm-chapter]: https://bosh.io/docs/bpm/bpm/\n[bpm-release]: https://github.com/cloudfoundry-incubator/bpm-release\n\n## BOSH Job Lifecycle\n\nEvery BOSH job goes through a specific lifecycle for starting and stopping. The BOSH website has a great\n[overview](https://bosh.io/docs/job-lifecycle/) of the lifecycle. You may wish to review this before reading on to\nour recommendations for the individual parts of the lifecycle.\n\n### Starting\n\n#### Pre-Start ([docs](https://bosh.io/docs/pre-start/))\n\nPre-start scripts are run before BOSH hands off control to Monit, so will not necessarily run every time a job starts\n(e.g. if a VM is rebooted outside of BOSH's control). However, `pre-start` scripts will run at least once for every new\nrelease version that is deployed, and have _no timeout_ (in contrast to a `start` script). Therefore, `pre-start`\nscripts can best be used for performing lengthy set-up of persistent state that must be done for a new version of a\nrelease, such as database migrations. However, because the `pre-start` script does not necessarily run in every VM your\njob will run in, do not perform any work in temporary directories such as `/var/vcap/sys/run` (see [VM Configuration\nLocations][vm-config-loc] for the list of temporary directories).\n\n[vm-config-loc]: https://bosh.io/docs/vm-config/\n\nIn general, a `pre-start` script should not be necessary and the start script should be sufficient. If you find a\n`pre-start` script is necessary, keep the above caveats in mind.\n\n#### Monit Start\n\nThe `start` script has two main responsibilities: writing the process ID (PID) to a `.pid` file and starting the main\nprocess. All setup required for the process to run (that has not been done in the `pre-start` script) must be done at\nthis time. Monit places a short timeout on the `.pid` file being written, so the work done in the `start` script should\nbe focused on executing your process and getting it healthy quickly.\n\n**Note**: This script will likely run many times, so ensure that it is idempotent (that it can be run repeatedly without\ncausing the system to enter a bad state)\n\nThe general workflow of a start script:\n\n1. Create a log directory for log persistence in `/var/vcap/sys/log/\u003cjob name\u003e`\n   - Idempotency recommendation: Do not fail if the log directory already exists\n\n2. Create a run directory to contain the pidfile in `/var/vcap/sys/run/\u003cjob name\u003e`\n   - Idempotency recommendation: Do not fail if the run directory already exists\n\n3. Change ownership permissions on the `log/` and `run/` directories to user `vcap` and group `vcap`\n\n4. Run your process with `start-stop-daemon`, which will manage your process's pidfile, ensuring multiple instances\n   of the process are not run simultaneously. A recommended usage of `start-stop-daemon` looks like this:\n\n   ```\n   /sbin/start-stop-daemon \\\n\n     # Write the pidfile, and error if it exists\n     # and refers to an already-running process\n     --pidfile \"$PIDFILE\" \\\n     --make-pidfile \\\n\n     # Run the process as the less-privileged vcap user\n     --chuid vcap:vcap \\\n\n     # Start the given process and redirect its logs\n     --start \\\n     --exec /var/vcap/packages/paragon/bin/web \\\n        \u003e\u003e \"$LOG_DIR/web.stdout.log\" \\\n       2\u003e\u003e \"$LOG_DIR/web.stderr.log\"\n   ```\n\n   Refer to the start script of the pararagon job for a more-complete example.\n\n**Note**: The start script is executed as root. Do not assume you can only break your own process.\n\n**Note**: If your process cannot start quickly, consider moving long-running tasks to `pre-start` or `post-start`.\n\n**Note for Windows Releases**: The Windows BOSH Agent does not use Monit and manages starting the script directly. If\nyour `pre-start` and `post-start` scripts have been written following the guidance in this document, the Windows BOSH\nAgent will be able to start your process correctly.\n\n##### Logging\n\nJob logs should be put in the `/var/vcap/sys/log/\u003cjob\u003e` directory. You can redirect the logs from your server (if\nthey're output on `stdout` and `stderr`) by using standard redirection:\n\n```bash\nexec ... \\\n  \u003e\u003e \"/var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stdout.log\" \\\n  2\u003e\u003e \"/var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stderr.log\"\n```\n\nIf your process takes log locations as parameters, use that instead:\n\n```bash\nexec ... \\\n  --out-log-file \"/var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stdout.log\" \\\n  --err-log-file \"/var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stderr.log\"\n```\n\nWhat is important here, is that all log files _must_ end with the `.log` extention in order to be properly rotated by\nBOSH. If any other extension is used for files where daemon append logs, you might end up with full disks and experience\nfailing nodes. For those who are interested to know more, BOSH enforces a Logrotate config\n[in `/etc/logrotate.d/vcap`](https://bosh.io/docs/vm-config/#global).\n\nFor detail on forwarding logs to external locations, please see [Exporting Logs](#exporting-logs).\n\nLogging the activity of `start`, `stop` is not critical when leveraging `start-stop-daemon` because they become really\nsimple. Some `pre-start` or `post-start` script are not trivial though, and need logging for debug purpose. In such\ncase, we recommend you prepend some time indication to the logged lines. This is useful for debugging nodes that have\nbeen deployed for a long time, in order to distinguish old errors from recent errors.\n\n```bash\nfunction start_logging() {\n  exec \\\n    \u003e \u003e(prepend_datetime \u003e\u003e /var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stdout.log) \\\n    2\u003e \u003e(prepend_datetime \u003e\u003e /var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stderr.log)\n}\n\nfunction prepend_datetime() {\n  awk -W interactive '{ system(\"echo -n [$(date +%FT%T%z)]\"); print \" \" $0 }'\n}\n\nfunction main() {\n  start_logging\n  ...\n}\n\nmain \"$@\"\n```\n\nAs a good remark, you'll note that the above code starts multiple processes for every line of logs. Yes indeed, this way\nof doing is not to be applied to all cases. Here, starting multiple processes for every line of logs is acceptable\nthough, because scripts like `start`, `stop`, `pre- start`, `post-start` or `drain` are not supposed to be executed\noften or to produce massive amounts of logs.\n\n#### Post-Start ([docs](https://bosh.io/docs/post-start/))\n\n`post-start` is useful for custom health-checks to ensure you job has started correctly. For example, if your process\nstarts quickly, but takes time to discover services or connect to its backend, you may wish to use `post-start` to query\nthe readiness of your job to start handling requests. If a `post-start` script is provided, BOSH will not consider a job\nto be ready until it has exited successfully.\n\nWhen ensuring in a `post-start` that your job's is healthy and its dependencies are met, you are responsible for\nimplementing the proper timeouts. Most of the time, these timeouts will need to be configurable in order for your job to\nadapt to slow or loaded IaaS infrastructures. A good way to do this is to expose a timeout scale factor in your job's\nconfiguration properties.\n\n#### Post-Deploy ([docs](https://bosh.io/docs/post-deploy/))\n\n`post-deploy` is useful for tasks that must occur once all instances of a deployment are started, such as checking the\nhealth of an entire deployment, enabling a feature after a successful upgrade, or finishing database migrations for an\nentire cluster.\n\n### Stopping\n\nJob is unmonitored before any stop scripts can run, so you can safely exit in either `pre-stop`, `drain` or `monit stop`\nwithout the job becoming listed as unhealthy.\n\n#### Pre-Stop ([docs][pre_stop])\n\nThe feature for `pre-stop` hook scripts has been released [in April 2019][bosh_v269]. The `pre-stop` script runs right\nbefore `drain`, with no timeout, and is exposed more information to help release authors take proper decision about\nfuture state of the bosh instance, whether it is going away temporarily of permanently.\n\nWhen `pre-stop` scripts are executed, daemon processes are no more monitored by Monit, so `pre-stop` is the proper place\nfor draining client connections, and should be favored over using the `drain` script, that abides to legacy complex\nrules (related to “static” or “dynamic” draining modes, that should be avoided).\n\nThe tricks in `drain` scripts based on `BOSH_JOB_NEXT_STATE` (see below) should be abandonned in favor of the newer\n`BOSH_VM_NEXT_STATE`, `BOSH_INSTANCE_NEXT_STATE` and `BOSH_DEPLOYMENT_NEXT_STATE` environment variables, that are now\nexposed to `pre-stop` scripts. See the documentation for more details\n\n[bosh_v269]: https://github.com/cloudfoundry/bosh/releases/tag/v269.0.0\n[pre_stop]: https://bosh.io/docs/pre-stop/\n\n#### Drain ([docs](https://bosh.io/docs/drain/))\n\nDrain scripts are the legacy hooks into the BOSH job lifecycle that are run before stopping the job via Monit. They've\ntypically been used for services which must perform some work before being shut down, for example flushing a request\nqueue or evacuating containers from a Diego cell. Now such things should be implemented in the newer `pre-stop` script.\nFrom this point on, and until the end of this section, everything that is said about `drain` should be ported to\n`pre-stop` instead.\n\nAs a rule of thumb, if `monit stop`-ping your job could cause dropped connections or a lack of availability, a `drain`\nscript should be used to prevent this. Most commonly, your `drain` script will send a request to a drain endpoint on\nyour process and wait for it to return rather than implementing the drain behavior itself.\n\nA concrete example is the [gorouter][gorouter], which has a configurable [`drain_wait`][drain-wait] parameter. When\nnon-zero, gorouter's drain script will instruct gorouter to report itself as unhealthy to its load-balancer with the\nintent of being removed from the balanced instance group before shutting down and rejecting requests. When `monit stop`\nis called, the router will already be receiving no connections, so will not drop connections when it is shut down\nquickly. This is the [lame duck][lame-duck] pattern.\n\n[gorouter]: https://github.com/cloudfoundry/gorouter\n[drain-wait]: https://github.com/cloudfoundry/routing-release/blob/develop/jobs/gorouter/spec#L62-L67\n[lame-duck]: https://landing.google.com/sre/book/chapters/load-balancing-datacenter.html#robust_approach_lame_duck\n\nDrain scripts have no timeout, so should take whatever time necessary to block on any draining work. One may, however,\nrun the risk of writing a `drain` script that never finishes and blocks a deployment. A well-written drain script is\nguaranteed to finish, such as by adding your own sensible timeout around draining work (e.g. 10 minutes).\n\nWhen your draining process has completed, your drain script should output a \"0\" to STDOUT to inform BOSH that draining\nis complete.\n\nOpen question: If your job needs to wait for another job to drain first, what is the best way to block on that? If you\nhave any ideas, please [let us know][contact-us].\n\nIn a `drain` script, you might need to tell the difference between your process being _temporarily_ stopped, or\n_permanently_ decommissioned. This is especially relevant for cluster nodes that share a global state. A node that\npermanently leaves the cluster might have to re-distribute its data to the other nodes that stay in the cluster (this is\nfor instance the case with Cassandra nodes) or just tell the others that it's no use waiting for it to come back\n(MongoDB calls this “stepping down”).\n\nIn such case, BOSH proposed a hack-ish check on the permanent disk size being set to zero in the future. This\ninformation is provided by the JSON document exposed by the `BOSH_JOB_NEXT_STATE` environment variable, as detailed in\nthe [Environment Variables](https://bosh.io/docs/drain/#environment-variables) section of the `drain` script\ndocumentation.\n\nYou can see two examples of such scripts that follow this best practice:\n- For etcd, in [cloudfoundry-incubator/cfcr-etcd-release/jobs/etcd/templates/bin/drain.erb][etcd-drain]\n- For Cassandra, in [orange-cloudfoundry/cassandra-boshrelease/jobs/cassandra/templates/bin/drain][cassandra-drain]\n\n[etcd-drain]: https://github.com/cloudfoundry-incubator/cfcr-etcd-release/blob/master/jobs/etcd/templates/bin/drain.erb\n[cassandra-drain]: https://github.com/orange-cloudfoundry/cassandra-boshrelease/blob/master/jobs/cassandra/templates/bin/drain\n\n#### Monit Stop\n\nThe `stop program` in your job's `monit` file is executed after the `drain` script (if present) has finished running. It\ncan also be run directly by an operator if they execute `monit stop \u003cjob\u003e` on the machine (though operators should\nalways consider running `bosh stop \u003cinstance-group\u003e/\u003cindex\u003e` instead of using `monit stop` directly). By default, there\nis a 30 second timeout on this script completing. Monit will assume scripts taking longer than this have failed. We do\nnot recommend changing this value if you need more time. Instead, you should do all the work necessary in your `drain`\nscript such that your service can shutdown quickly (where “quickly” generally means in under 10 seconds).\n\nIf you're not using `drain`, then we recommend that you send `SIGTERM` to your process, which should cause it to start\nshutting down. If your process shuts down at this point then you're good to go. If for some reason your process locks up\nor is unable to exit for some other reason then you should send `SIGKILL` before the timeout. If your language runtime\nsupports dumping the stacks of all running threads on `SIGQUIT` (Go and Java do) then you can send that signal just\nbefore the `SIGKILL` to aid debugging why the process is stuck. If you are not using a `SIGKILL` respecting runtime then\nadding this functionality to your own program normally isn't difficult.\n\n`start-stop-daemon` can be used for this:\n\n```\n/sbin/start-stop-daemon \\\n\n  # Remove the pidfile after killing the process\n  --pidfile \"$PIDFILE\" \\\n  --remove-pidfile \\\n\n  # Send SIGTERM, wait for the process to die for 20 seconds\n  # If the process has not died, send SIGQUIT and wait for 1 second\n  # If the process has still not died, send SIGKILL\n  --retry TERM/20/QUIT/1/KILL \\\n\n  # If the process is already gone, do not error\n  --oknodo \\\n\n  --stop\n```\n\nRefer to the stop script of the pararagon job for a more-complete example.\n\nIf you're using drain to kill the process then your process may already be shut down by the time that `monit stop` is\ncalled. In this case we do not need to do anything further in the stop executable.\n\n**Note for Windows Releases**: The Windows BOSH Agent does not use Monit and manages starting the script directly. If\nyour `pre-start` and `post-start` scripts have been written following the guidance in this document, the Windows BOSH\nAgent will be able to stop your process correctly.\n\n#### Post-Stop ([docs](https://bosh.io/docs/post-stop/))\n\nThe feature for `post-stop` hook scripts has been released [in March 2018][bosh_v265]. When `post-stop` is run, the\nMonit process is gone. `post-stop` may possibly cleanup some data before the VM is shut down. `post-stop` is not exposed\nthe same information as `pre-stop` for taking decisionsuppon permanent deletion of the VM.\n\nWe haven't see so far any use-case that requires running code in the `post-stop` script specifically, but we'd lov to\nhear your feedback and get suggestions/contributions in PRs.\n\n[bosh_v265]: https://github.com/cloudfoundry/bosh/releases/tag/v265.1.0\n\n## Advice for authoring `spec` files\n\n### Properties\n\nThe properties you include in your `spec` file create the product surface for your job from the perspective of the\noperator component. Job properties can make it much easier or much harder to operate a deployment, so one should be\nmindful of the operator when making decisions about properties.\n\n- Properties should not have a “namespace” for the job itself (e.g. `my-job.port`, `my-job.hostname`), but should\n  only use namespace to create property groups that the job cares about (e.g. `database.*` and `blobstore.*`).\n- If a property does not need to be configured specially for every deployment and a reasonable default exists, it should\n  be provided. This lets an operator have a terser deployment manifest, which is easier to generate, read, and modify.\n- Include a description for every property whenever it is slightly likely that it would help with understanding. It\n  can be easy for an operator to misinterpret a property and use it incorrectly.\n\nHere is an example of properly written description, from the [ATC job of Concourse][atc-spec-example]\n\n```yaml\n  external_url:\n    description: |\n      Externally reachable URL of the ATCs. Required for OAuth. This will be\n      auto-generated using the IP of each ATC VM if not specified, however\n      this is only a reasonable default if you have a single instance.\n\n      Typically this is the URL that you as a user would use to reach your CI.\n      For multiple ATCs it would go to some sort of load balancer.\n    example: https://ci.concourse.ci\n```\n\nYou can find other good examples [for Cassandra](cassandra-spec-example) or [MongoDB](mongod-spec-example). Generally,\nusing the pipe `|` notation for the `description:` YAML property, and providing a text with lines wrapped around the\n80th column, is best for readability.\n\n[atc-spec-example]: https://github.com/concourse/concourse/blob/master/jobs/atc/spec#L78-L86\n[cassandra-spec-example]: https://github.com/orange-cloudfoundry/cassandra-boshrelease/blob/master/jobs/cassandra/spec#L254-L264\n[mongod-spec-example]: https://github.com/orange-cloudfoundry/mongodb-boshrelease/blob/master/jobs/mongod/spec#L80-L92\n\n### Links ([docs](https://bosh.io/docs/links/))\n\nLinks allow jobs to provide and consume configuration that needs to be shared between jobs, which can greatly reduce\nthe amount of configuration required in a deployment manifest. Instead, jobs can declare which information they need\nand BOSH will provide that information automatically. When links are used correctly, there are two main benefits:\n\n- IP address information (e.g. locating servers from other jobs) does not need to be provided at all by the operator,\n  so default manifests are intrinsically much more network-agnostic and portable.\n- Configuration can be provided in a manifest to only one job, which will then export it as a link consumed by\n  downstream jobs, making it much easier to modify that configuration and reduce human error. For example, if the port\n  a server listens on is provided as a link, a change to the server's configuration will automatically propagate to all\n  jobs that need to communicate with it.\n\nIf links are used whenever a job depends on configuration from another job, manifests can become much simpler and\ndeployments can become more reliable. When authoring your release, all properties you include in your `spec` file should\nchange the runtime behavior of your process and should not include IP addresses, domain names, credentials, or\ncertificates of other jobs. If you are including these properties, they are candidates for links instead. If the job\nyou depend on does not provide the information as a link, please consider submitting an issue or PR to the maintainer.\n\n#### Overriding links\n\nSometimes it is necessary to allow operators to override individual properties within a link. For example, if your job\nuses a link's IP information to find a dependent service, but a particular deployment may be using custom DNS for\nservice discovery, your job template could prefer the DNS property over the link. That template could look like this:\n\n```yaml\n\u003c%\n  require \"json\"\n\n  server = nil\n  if_p(\"database_location\") do |prop|\n    server = prop\n  end.else do\n    server = link(\"database\").instances[0].address\n  end\n-%\u003e\n\nserver: \u003c%= server.to_json %\u003e\n```\n\n## Template Advice\n\n### ERB\n\nAny template in your job can use ERB (even the `monit` files!). While this is extremely powerful it can be very\ndifficult to understand and maintain complex templates. Therefore, we advise avoiding any ERB in your control scripts\nwherever possible. The control flow of starting and stopping your program should be deterministic and simple. All ERB\nshould be relegated to static configuration files so that properties can be interpolated.\n\n#### Encoding\n\nWhen you inject configuration properties with the ERB syntax in shell scripts or configuration files, you should take\ngreat care for properly encoding special caracters that are meaningful to the target syntax. Failing at doing so could\ncause shell scripts to erase critical data, or mis-configuration in YAML files to cause data loss.\n\nWith Bash shell scripts, we recommend encoding properties using `shellwords`:\n\n```bash\n#!/usr/bin/env bash\n\u003c%\n  require \"shellwords\"\n\n  def esc(x)\n    Shellwords.shellescape(x)\n  end\n-%\u003e\n\nsome_property=\u003c%= esc(p('some_property')) %\u003e # \u003c- Please observe that you must not quote the value here\n```\n\nWith YAML config files, we recommend encoding properties with `json`:\n\n```yaml\n---\n\u003c% require \"json\" -%\u003e\n\n# The name of the cluster. This is mainly used to prevent machines in\n# one logical cluster from joining another.\ncluster_name: \u003c%= p(\"cluster_name\").to_json %\u003e # \u003c- Please observe that you must not quote the value here\n```\n\nFor complex logic in YAML templates, you can adopt [the full-Ruby option][complex-bosh-property-templating], where you\nfirst build a Ruby Hash for your YAML configuration, and then `YAML.dump()` it, which properly performs the expected\nencoding.\n\nDon't hesitate to [contact us][contact-us] and submit advices or best-practice encoding for other syntax schemes.\n\n[complex-bosh-property-templating]: https://bosh.io/docs/bpm/transitioning/#complex-bosh-property-templating\n\n#### Testing\n\nYour templates should be simple enough to not need testing, such as by doing a simple passthrough to property parsing\nin your own code, but you may find yourself with more-complex templating needs, such as when writing a release wrapping\nthird-party code. Testing BOSH templates is classically a hard topic, but it has been greatly simplified with a new\nversion of the `bosh-template` Gem, and a proper documentation of [unit testing][unit-testing] with it.\n\nAs it might still be difficult to test BOSH template rendering, you could also imagine having a simple template (e.g.\n`\u003c% p('my-properties').to_json %\u003e`) that you would transform with a custom executable, which you can test in your\nstandard unit-testing workflow.\n\n[unit-testing]: https://bosh.io/docs/job-templates/#unit-testing\n\n### Bash\n\nWe advise you to use the `#!/usr/bin/env bash` sheebang variant.\n\nThe [ShellCheck][shellcheck] tool can be used to find common errors in your `bash` scripts. This yet another reason to\nkeep ERB out of your scripts!\n\nYou can also possibly use [BATS][bats] unit tests, or similar testing frameworks.\n\n[shellcheck]: https://github.com/koalaman/shellcheck\n[bats]: https://github.com/sstephenson/bats\n\n## Monit Advice\n\nTry to keep your Monit configuration as simple as possible. The BOSH team is planning on removing Monit soon - do not\ncouple yourself to it! This includes such changes as monitoring memory and changing away from the default timeout.\n\nYou must specify `group vcap` in your `monit` file because the agent [uses this tag][agent-monit-group] to find\nprocesses it should be managing on the machine.\n\n[agent-monit-group]: https://github.com/cloudfoundry/bosh-agent/blob/5beacb106a67e403a15e9926b6ee39006d774d31/jobsupervisor/monit_job_supervisor.go#L114\n\n## Exporting Logs\n\nBOSH itself does not handle forwarding logs off-system. If you have written your logs appropriately as described in\nthe [Logging](#logging) section, operators can choose the correct log-forwarding mechanism for their deployment using\njob co-location in the deployment manifest or BOSH [addons][addons]. Operators may choose to use something like\n[google-fluentd][google-fluentd] or [syslog-release][syslog-release]. If you are also responsible for providing a\ndeployment manifest generation tool, you may wish to provide the option to add syslog-release forwarding to all\ncomponents.\n\n[addons]: https://bosh.io/docs/runtime-config/#addons\n[google-fluentd]: https://github.com/cloudfoundry-community/stackdriver-tools#deploying-host-logging\n[syslog-release]: https://github.com/cloudfoundry/syslog-release\n\n**Backwards Compatibility Warning**: If your release is currently responsible for forwarding logs off-system, by\nfollowing this guide and removing that functionality, you are putting the onus for logging on deployment authors\n(e.g. [cf-deployment][cf-deployment] or your local friendly closed-source proprietary offering) to configure logging.\nIf your release has traditionally done this, deployment authors may not know how to update the deployment and logs could\nbe lost. If you are removing this functionality, please coordinate appropriately with deployment authors and operators.\n\n[cf-deployment]: https://github.com/cloudfoundry/cf-deployment\n\n### Metron Agent\n\n`metron_agent` has functionality to forward syslogs, but this has been superseded by `syslog-release`. `metron_agent`\nshould only be used to forward logs to [loggregator][loggregator].\n\n[loggregator]: https://github.com/cloudfoundry/loggregator\n\n### The Ol' `tee` and `logger` Approach\n\nMany releases currently include complex setup to forward logs to both `/var/vcap/sys/log` as well as syslog. This\nis not necessary if operators use one of the log-forwarding options mentioned above. Drawbacks of the tee and logger\napproach include starting multiple processes for every line of logs, fairly subtle behavior that can easily break or\nsecurity vulnerabilities. If your release has any code like the following, please remove it and follow the\nrecommendations above:\n\n```bash\nDO NOT USE THIS CODE\n\nexec \u003e \\\n  \u003e(\n    tee -a \u003e(logger -p user.info -t vcap.$(basename $0).stdout) | \\\n      awk -W interactive '{ gsub(/\\\\n/, \"\"); system(\"echo -n [$(date +\\\"%Y-%m-%d %H:%M:%S%z\\\")]\"); print \" \" $0 }' \\\n      \u003e\u003e /var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stdout.log\n  )\nexec 2\u003e \\\n  \u003e(\n    tee -a \u003e(logger -p user.error -t vcap.$(basename $0).stderr) | \\\n      awk -W interactive '{ gsub(/\\\\n/, \"\"); system(\"echo -n [$(date +\\\"%Y-%m-%d %H:%M:%S%z\\\")]\"); print \" \" $0 }' \\\n      \u003e\u003e /var/vcap/sys/log/\u003cjob\u003e/\u003cprocess\u003e.stderr.log\n  )\n\nDO NOT USE THIS CODE\n```\n\n\u003csub\u003eIf you must use this, you should understand it. The `exec` calls redirect `STDOUT` and `STDERR` respectively,\nsending them to a sub-shell that calls `tee`. `tee` splits the output to 1) syslog via `logger` and 2) the BOSH log\ndirectory (but not before appending timestamps with `awk`).\u003c/sub\u003e\n\n\n## Bosh DNS\n\nBosh DNS is an important feature in Bosh. It basically brings consistent cloud-agnostic internal load-balancer features\nfor free.\n\nBosh DNS is also particularily powerful for implementing cloud-agnostic split-DNS features. Internally, Bosh DNS can\nmask public DNS names, and return private IP addresses instead. With this, the cluster can use public DNS names and\nstill get the traffic routed internally. Without it, the traffic would exit the cluster and immediately re-enter it,\nresulting in unnecessary network hops.\n\n### Aliases\n\nRelease jobs can define their own DNS aliases in a `dns/aliases.json` file. Here we\n[demonstrate](jobs/stateful-daemon/templates/dns/aliases.json.erb) how to write such file. Indeed, when referring to\ndefault Bopsh DNS names, you'll need to get the network name from Bosh Cloud Config, and there is a trick for this with\nthe `.methods(false)` helper on the `.networks` OpenStruct.\n\n### Readiness with `bin/dns/healthy`\n\nThe Bosh `dns/health` script is actually misnamed, as it is actually a “_readiness_” check script. Indeed, when a\n`bin/dns/healthy` script exits with non-zero (error) status, then the instance is taken out of DNS aliases. When you see\nDNS aliases as cheap load-balancers, then it means that you stop routing traffic to the job. This is precisely the\nconcept of a “readiness” check.\n\nInside a `bin/dns/healthy` script, you can do whatever check you need to. Run binaries, check open TCP ports or run\nhealthcheck HTTP(S) queries.\n\nParticularily interesting is the way you can coordinate job draining in `pre-stop` with `dns/healthy` for properly\ndraining connections to a network daemon that is about to shut down. Indeed, the `pre-stop` can “inform” `dns/healthy`\nthat no more traffic should be routed to the daemon, creating a specific file for example. Then, the `dns/healthy`\nscript can react and return non-zero exit status, resulting in the IP being removed from the DNS query results, and no\nmore new connection sent to the daemon.\n\nIf we detail a bit, the `pre-stop` starts draining connections. So, it ensures that Bosh DNS doesn't route any new\nconnection to it writing a file like `/var/vcap/data/\u003cjob-name\u003e/DRAINING`. The `dns/healthy` script detects this file,\nand exits unconditionally with a status of `1` in order to report the job as non-ready.\n\nSee [this basic exemple][gk_kong_dns_healthy] from the Kong Bosh Release.\n\n[gk_kong_dns_healthy]: https://github.com/gstackio/gk-kong-boshrelease/blob/master/jobs/kong/templates/bin/dns/healthy\n\n## Backup and Restore features\n\nThe BOSH Backup and Restore project (BBR) provides a framework for backing up and restoring BOSH deployments and BOSH\nDirectors.\n\nFor more information, go read the [BOSH Backup and Restore](https://docs.cloudfoundry.org/bbr/) chapter in Cloud Foundry\ndocumentation.\n\nAs a BOSH Release author, you shall also be interested in reading the\n[BOSH Backup and Restore Developer's Guide](https://docs.cloudfoundry.org/bbr/bbr-devguide.html).\n\nThe BBR project provides its own\n[Exemplar Backup and Restore Release](https://github.com/cloudfoundry-incubator/exemplar-backup-and-restore-release) for\nthe purpose of demonstrating best practice in implementing the\n[BBR contract](https://docs.cloudfoundry.org/bbr/bbr-devguide.html). We advise you to refer to this materials for\nlearning more about providing standard backup and restore features in your BOSH Releases. Here we just provide an\nexample [`stateful-daemon` job](./jobs/stateful-daemon) which implements very basic BBR features.\n\n\u003c!-- Global Links --\u003e\n\n[contact-us]: https://github.com/cloudfoundry/exemplar-release/issues\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudfoundry%2Fexemplar-release","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudfoundry%2Fexemplar-release","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudfoundry%2Fexemplar-release/lists"}