{"id":20263624,"url":"https://github.com/springerpe/go-graphite-boshrelease","last_synced_at":"2025-09-04T12:44:53.839Z","repository":{"id":80987260,"uuid":"53514677","full_name":"SpringerPE/go-graphite-boshrelease","owner":"SpringerPE","description":"Bosh release for graphite go-carbon stack and c-carbon-relay","archived":false,"fork":false,"pushed_at":"2020-02-14T10:14:32.000Z","size":672,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-14T04:44:36.871Z","etag":null,"topics":["bosh-release","graphite"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SpringerPE.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-09T16:50:19.000Z","updated_at":"2020-02-14T10:14:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"bdb3cb36-ca44-41bb-9f01-b08125b599ba","html_url":"https://github.com/SpringerPE/go-graphite-boshrelease","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fgo-graphite-boshrelease","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fgo-graphite-boshrelease/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fgo-graphite-boshrelease/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fgo-graphite-boshrelease/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SpringerPE","download_url":"https://codeload.github.com/SpringerPE/go-graphite-boshrelease/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241731744,"owners_count":20010781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bosh-release","graphite"],"created_at":"2024-11-14T11:35:44.970Z","updated_at":"2025-03-03T19:59:46.226Z","avatar_url":"https://github.com/SpringerPE.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BOSH Release for go-graphite\n\nThe aim of this bosh release is to deploy a go-graphite cluster consisting of the following components:\n\n* [carbon-c-relay](https://github.com/grobian/carbon-c-relay) for metrics relaying and fordwarding\n* [go-carbon](https://github.com/lomik/go-carbon) sever to storage metrics\n* [carbonzipper](https://github.com/dgryski/carbonzipper) to transparently merge graphite carbon backends\n* [carbonapi](https://github.com/dgryski/carbonapi) to provide the graphite API \n* [buckytools](https://github.com/jjneely/buckytools) to provide a management layer on top of go-carbon\n\n## Requirements\nWe structured the release around the concepts of availability zones and clusters. \nOne of the properties required by the `carbon-c-relay` job gives you a way of designing the architecture of your `go-graphite` metrics deployment.\nThe `go-carbon` instances are assigned automatically to a specific cluster based on the `az` value of their vm instance. This is achieved by taking advantage of [bosh links](https://bosh.io/docs/links.html) and of [first class AZs](https://bosh.io/docs/azs.html).\n\nLet's discuss this idea more in details with the help of one example:\n```\n    properties:\n      carbon-c-relay:\n        clusters:\n          - name: \"cluster1\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z1\"]\n          - name: \"cluster2\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z2\"]\n```\nIn this configuration snippet `clusters` is defined as list of dictionaries, and each entry is representing a `carbon-c-relay` cluster. The cluster instances are futhermore grouped and isolated by availability zone.\n\nA single cluster can be deployed across several `az`s, if needed, like in the following example:\n```\n    properties:\n      carbon-c-relay:\n        clusters:\n          - name: \"cluster1\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z1\", \"z2\"] #\u003c- A cluster can use as many azs as you'd like\n```\n\nhowever you cannot define different clusters within the same availability zone and this would cause a validation error to be raised at deployment time. Because of this limitation it is also impossible to define more `carbon-c-relay` clusters than the number of configured azs.\n\nSo in a typical `n` azs architecture you can create maximum `n` different clusters each of them belonging to a different az.\n\n##### Example of erroneous configuration\n```\n    properties:\n      carbon-c-relay:\n        clusters:\n          - name: \"cluster1\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z1\", \"z2\"]\n          - name: \"cluster2\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z2\", \"z3\"]   #\u003c- Error: \"z2 is already used by cluster `cluster1`\" \n```\n\n##### Warning\nYou need to deploy at least **#azs * 2** `go-carbon` instances, or equivalently at least 2 instances per az. This is due to a limitation in the current implementation of `buckytools` which does not support clusters formed by a single instance.\n\n## Usage\n\nTo use this bosh release, first upload it to your bosh directly from the releases\nsection, or using the `yml` file in the `releases` folder.\n\nThis release makes use of [Bosh links](https://bosh.io/docs/links.html) between\ninstance groups in order to setup automatically the carbon relays and the carbonzippers\n(for carbon API). This an example of V2 manifest (for google cloud):\n\n```\n---\nname: go-graphite-boshrelease\n\nreleases:\n- name: go-graphite-boshrelease\n  version: latest\n\nstemcells:\n- alias: trusty\n  name: bosh-google-kvm-ubuntu-trusty-go_agent\n  version: 3312.6\n\ninstance_groups:\n- name: smoke_tests\n  instances: 1\n  vm_type: common\n  stemcell: trusty\n  lifecycle: errand\n  vm_extensions: []\n  azs: [z1, z2]\n  networks:\n  - name: tools\n  jobs:\n  - name: smoke-tests\n    release: go-graphite-boshrelease\n    properties:\n      smoke_tests:\n        api_host: \"10.255.3.251\"\n        api_port: 10000\n        host: \"10.255.3.250\"\n        port: 2003\n        tcp_enabled: true\n        udp_enabled: false\n\n- name: graphite-backend\n  instances: 4\n  persistent_disk_pool: graphite-disks\n  vm_type: graphite-backend\n  stemcell: trusty\n  vm_extensions: [graphite-api-europe-west1-backend-service]\n  azs: [z1, z2]\n  networks:\n  - name: tools\n  jobs:\n  - name: go-carbon\n    release: go-graphite-boshrelease\n    properties:\n      go-carbon:\n        tcp_listen: 2030\n        schemas:\n        - name: stats\n          pattern: '^stats\\..*'\n          retentions: '10:8d'\n        - name: default\n          pattern: '.*'\n          retentions: '60s:14d'\n        aggregations:\n        - name: team1_stats_sum\n          pattern: '^stats\\.services\\.team1\\..*\\.live\\.event\\..*\\.sum$'\n          aggregationMethod: sum\n          xFilesFactor: 0.0\n        - name: team1_counts_stats_sum\n          pattern: '^stats_counts\\.services\\.team1\\..*\\.live\\.event\\..*\\.sum$'\n          aggregationMethod: sum\n          xFilesFactor: 0.0\n        carbonserver:\n          enabled: true\n          port: 8080\n          host: \"0.0.0.0\"\n\n  - name: carbonzipper\n    release: go-graphite-boshrelease\n    properties:\n      carbonzipper:\n        port: 9090\n\n  - name: carbonapi\n    release: go-graphite-boshrelease\n    properties:\n      carbonapi:\n        port: 10000\n\n  - name: statsd\n    release: go-graphite-boshrelease\n    properties:\n      statsd:\n        graphiteHost: \"10.255.3.250\"\n        graphitePort: 2003\n        percentThreshold: [90, 95, 99]\n\n- name: carbon-c-relay\n  instances: 2\n  vm_type: graphite-relay\n  vm_extensions: [graphite-europe-west1-backend-service]\n  stemcell: trusty\n  azs: [z1, z2]\n  networks:\n  - name: tools\n  jobs:\n  - name: carbon-c-relay\n    release: go-graphite-boshrelease\n    properties:\n      carbon-c-relay:\n        clusters:\n          - name: \"cluster1\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z1\"]\n          - name: \"cluster2\"\n            lb: \"jump_fnv1a_ch\"\n            replication: 1\n            azs: [\"z2\"]\n  - name: statsrelay\n    release: go-graphite-boshrelease\n    properties:\n      statsrelay:\n        port: 8125\n\n# recommend serial True if there are a lot of nodes (TODO)\nupdate:\n  canaries: 1\n  max_in_flight: 1\n  serial: true\n  canary_watch_time: 1000-60000\n  update_watch_time: 1000-60000\n\n```\n\nFor [bosh-lite](https://github.com/cloudfoundry/bosh-lite), you can quickly create a deployment manifest \u0026 deploy a cluster:\n\n```\ntemplates/make_manifest warden\nbosh -n deploy\n```\n\n## Development\n\nAs a developer of this release, you have to be aware of some requirements on your\nlocal computer, mainly regarding golang and VCS.\n\nThe golang projects `carbonzipper` and `carbonapi` do not provide a way to manage\nlibrary dependencies, they are just using `go get` which is a wrapper around git\nand it always fetchs from master, which makes difficult to create reproducible bosh\nreleases. The idea behind BOSH is making use of blobs which are the external source\ncomponents packed, so, in order to get a source code blob for those components,\nyou have to run `update-blobs.sh` (on the root folder), which will create them with all\nthe dependencies inside by executing `packages/\u003cpackage\u003e/prepare`.\n\nRequirements for `update-blobs.sh`:\n\n* golang (tested with golang 1.8)\n* mercurial VCS (hg backend for some golang libs)\n* git VCS\n\nThen, you can run `update-blobs.sh` to download all the source code blobs\n(and optionally upload them to the blobstore).\n\n\nSteps to build a new release:\n\n```\ngit clone https://github.com/SpringerPE/go-graphite-boshrelease.git\ncd go-graphite-boshrelease\n./update-blobs.sh\nbosh create-release --force \u0026\u0026 bosh upload-release\n```\n\n### Creating a new final release and publishing to GitHub releases:\n\nRun: `./create-final-public-release.sh [version-number]`\n\nKeep in mind you will need a Github token defined in a environment variable `GITHUB_TOKEN`.\nPlease get your token here: https://help.github.com/articles/creating-an-access-token-for-command-line-use/\nand run `export GITHUB_TOKEN=\"xxxxxxxxxxxxxxxxx\"`, after that you can use the script.\n\n`version-number` is optional. If not provided it will create a new major version\n(as integer), otherwise you can specify versions like \"8.1\", \"8.1.2\". There is a\nregular expresion in the script to check if the format is correct. Bosh client\ndoes not allow you to create 2 releases with the same version number. If for some\nreason you need to recreate a release version, delete the file created in \n`releases/cf-logging-boshrelease` and update the index file in the same location,\nyou also need to remove the release (and tags) in Github.\n\n\n### Smoke tests\n\nThe release provides a job to run smoke tests as a Bosh errand `bosh run errand smoke_tests`.\nSee the previous manifest for the configuration of the errand.\n\n\n## Authors\n\nSpringer Nature Platform Engineering,\n\nClaudio Benfatto\nJose Riguera\n\n\nCopyright 2017 Springer Nature\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspringerpe%2Fgo-graphite-boshrelease","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspringerpe%2Fgo-graphite-boshrelease","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspringerpe%2Fgo-graphite-boshrelease/lists"}