{"id":20263644,"url":"https://github.com/springerpe/cf-logging-boshrelease","last_synced_at":"2026-03-04T23:31:46.530Z","repository":{"id":80987055,"uuid":"136320730","full_name":"SpringerPE/cf-logging-boshrelease","owner":"SpringerPE","description":"Bosh release to setup logging on CF","archived":false,"fork":false,"pushed_at":"2020-04-28T11:06:31.000Z","size":314,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-03T19:59:39.974Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SpringerPE.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-06T11:43:42.000Z","updated_at":"2023-03-22T03:03:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"c366c923-ab30-4755-858f-b0854c3847ba","html_url":"https://github.com/SpringerPE/cf-logging-boshrelease","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/SpringerPE/cf-logging-boshrelease","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-logging-boshrelease","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-logging-boshrelease/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-logging-boshrelease/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-logging-boshrelease/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SpringerPE","download_url":"https://codeload.github.com/SpringerPE/cf-logging-boshrelease/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SpringerPE%2Fcf-logging-boshrelease/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30099369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T23:31:22.529Z","status":"ssl_error","status_checked_at":"2026-03-04T23:31:22.112Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T11:35:49.075Z","updated_at":"2026-03-04T23:31:46.502Z","avatar_url":"https://github.com/SpringerPE.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cloudfoundry logging Bosh release\n\nThis is a release to based on https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry\nto easily manage logging in Cloudfoundry deployments.\n\n* Able to define logstash pipelines\n* Not linked to ES\n* Logstash 6.x and up (no legacy versions)\n* Easy to manage and define logstash configuration files\n* Includes Prometheus logstash exporter: https://github.com/BonnierNews/logstash_exporter\n\n\n## Why?\n\nThis is a simple an small release, the core is just logstash, with the idea of\nhaving something easy to manage and maintain decoupled from ES. Also, it is\nfocused on processing/filtering logs, not on store then in a ES cluster. It allows to\ndefine logstash configuration pipelines in a flexible way (directly editing\nlogstash configuration files) and work with a simple endpoint which a 3rd party\nservice offered by somebody else (it can be ES offered by Elastic, BigQuery endpoint\nby Google, etc.)\n\n\n# Developing\n\nFirst of all, when do a git commit, try to use good commit messages; the release\nchanges on each release will be taken from the commit messages!\n\n\nSecond, if first time, probably you will need to initialize the upstream git submodules\n(for Prometheus logstash_exporter: https://github.com/BonnierNews/logstash_exporter):\n\n```\ngit submodule init\ngit submodule update\n```\n\nWhen you make changes in the packages (or add new ones), please use\n`./update-blobs.sh` to sync and upload the new blobs. This script reads the `spec` file \nof every package or looks for a `prepare` script (inside the folder of each package):\n\n* If there is a `packages/\u003cpackage\u003e/prepare`, it executes it and goes to the next package.\n* If the spec file of a package in `packages/\u003cpackage\u003e/spec` has a key `files` with this\nformat `- folder/src.tgz   # url`, for example:\n```\nfiles:\n- ruby-2.3/ruby-2.3.7.tar.gz      # https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.7.tar.gz\n- ruby-2.3/rubygems-2.7.7.tgz     # https://rubygems.org/rubygems/rubygems-2.7.7.tgz\n```\nIt will take the url, download the file to `blobs/ruby-2.3/ruby-2.3.7.tar.gz` and\nit will run `bosh add-blob` with the new src \"ruby-2.3.7.tar.gz\". Take into\naccount the script does not download a package if there is a file with the same\nname in the destination folder, so it the package was not properly downloaded\n(e.g. script execution interrupted), please delete the destination folder and try\nagain.\n\nThe idea is make it easy to update the version of the packages. Making a `packaging`\nscript flexible, not linked to version, updating a package is just a matter of \nupdating its `spec` file and run `./update-blobs.sh` and you have a new version\nready!. Extract of a ruby `packaging` script (just and example):\n```\n# Grab the latest versions that are in the directory\nRUBY_VERSION=`ls -r ruby-2.3/ruby-* | sed 's/ruby-2.3\\/ruby-\\(.*\\)\\.tar\\.gz/\\1/' | head -1`\nRUBYGEMS_VERSION=`ls -r ruby-2.3/rubygems-* | sed 's/ruby-2.3\\/rubygems-\\(.*\\)\\.tgz/\\1/' | head -1`\n\necho \"Extracting ruby-${RUBY_VERSION} ...\"\ntar xvf ruby-2.3/ruby-${RUBY_VERSION}.tar.gz\n\necho \"Building ruby-${RUBY_VERSION} ...\"\npushd ruby-${RUBY_VERSION}\n  LDFLAGS=\"-Wl,-rpath -Wl,${BOSH_INSTALL_TARGET}\" ./configure --prefix=${BOSH_INSTALL_TARGET} --disable-install-doc --with-opt-dir=${BOSH_INSTALL_TARGET}\n  make\n  make install\npopd\n```\n\nThe script does not process any args and it is safe to run as many times as you need\n(take into account if you create `prepare` scrips!).\n\n## Creating Dev releases (for testing)\n\nTo create a dev release -for testing purposes-, just run:\n\n```\n# Update or sync blobs\n./update-blobs.sh\n# Create a dev release\nbosh  create-release --force --tarball=/tmp/release.tgz\n# Upload release to bosh director\nbosh -e \u003cbosh-env\u003e upload-release /tmp/release.tgz\n```\n\nThen you can modify your manifest to include `latest` as a version (no `url` and `sha` \nfields are needed when the release is manually uploaded): \n\n```\nreleases:\n  [...]\n- name: cf-logging\n  version: latest\n```\n\nOnce you know that the dev version is working, you can generate and publish a final\nversion of the release (see  below), and remember to change the deployment manifest\nto use a url of the new final manifest like this:\n\n```\nreleases:\n  [...]\n- name: cf-logging\n  url: https://github.com/SpringerPE/cf-logging-boshrelease/releases/download/v8/cf-logging-8.tgz\n  version: 8\n  sha1: 12c34892f5bc99491c310c8867b508f1bc12629c\n```\n\nor much better, use an operations file ;-)\n\n\n\n## Creating a new final release and publishing to GitHub releases:\n\nRun: `./create-final-public-release.sh [version-number]`\n\nKeep in mind you will need a Github token defined in a environment variable `GITHUB_TOKEN`.\nPlease get your token here: https://help.github.com/articles/creating-an-access-token-for-command-line-use/\nand run `export GITHUB_TOKEN=\"xxxxxxxxxxxxxxxxx\"`, after that you can use the script.\n\n`version-number` is optional. If not provided it will create a new major version\n(as integer), otherwise you can specify versions like \"8.1\", \"8.1.2\". There is a\nregular expresion in the script to check if the format is correct. Bosh client\ndoes not allow you to create 2 releases with the same version number. If for some\nreason you need to recreate a release version, delete the file created in \n`releases/cf-logging-boshrelease` and update the index file in the same location,\nyou also need to remove the release (and tags) in Github.\n\nWhen creating a major release make sure not to use `.0` annotation.\nfor ex: Release 7, `./create-final-public-release.sh 7` (not 7.0)\n\n\n# Usage in a deployment manifest\n\n\n## Creating operarions file to define a new pipeline\n\nLogstash pipeline configuration files are defined in `manifest/pipelines/\u003cpipeline-name\u003e` folder.\nThre is a script `manifest/pipelines/generate-pipeline-operations.sh`:\n\n```\nUsage:\n   generate-pipeline-operations.sh [-m \u003cgenerated-operations-manifest] \u003cfolder\u003e [[config1] [config2] ...]\n\nGenerates a Bosh client operations file, next to \u003cfolder\u003e, with the name\n\"\u003cfolder\u003e.yml\" including all snippets if no extra arguments are provided,\notherwise it will use the snippets given as arguments.\n\nThe output is an operations file ready to be used by bosh.\n\nWhy? Because this allows us to split the logstash configuration in different\nfiles (snippets), making the logstash config easy to manage, and automatically\ngenerates an operations file including these snippets.\n```\n\nSo, if you generate/modify files in a pipeline folder, you can run:\n\n```\n./generate-pipeline-operations.sh \u003cfolder\u003e\n```\n\nand it will generate an operations \u003cfolder\u003e.yml with all the logstash files\ninside such folder. You can filter which operations files you wan to include, for\nexample:\n\n```\n./generate_pipeline_operations.sh cf-platform\n* Warning file 'cf-platform.yml' exists! Renaming to 'cf-platform.yml.old'\n* Generating cf-platform.yml with snippets from cf-platform:\n* Adding snippet 'cf-platform/filter-00-prefiltering.conf' ... ok\n* Adding snippet 'cf-platform/filter-10-syslog_rfc5424_parsing.conf' ... ok\n* Adding snippet 'cf-platform/filter-15-set_platform_index.conf' ... ok\n* Adding snippet 'cf-platform/filter-50-platform_sd_parsing.conf' ... ok\n* Adding snippet 'cf-platform/filter-51-platform_haproxy_parsing.conf' ... ok\n* Adding snippet 'cf-platform/filter-52-platform_uaa_parsing.conf' ... ok\n* Adding snippet 'cf-platform/filter-53-platform_vcap_and_json_parsing.conf' ... ok\n* Adding snippet 'cf-platform/filter-90-set_syslog_level.conf' ... ok\n* Adding snippet 'cf-platform/filter-91-rework_fields.conf' ... ok\n* Adding snippet 'cf-platform/filter-99-cleanup.conf' ... ok\n* Adding snippet 'cf-platform/input-10-syslog.conf' ... ok\n* Adding snippet 'cf-platform/output-10-es.conf' ... ok\n* Adding snippet 'cf-platform/output-99-debug.conf' ... ok\n* Exit=0\n\n```\n\nWill read all the configuration files in `cf-platform` and create a bosh operations\nfile `cf_platform.yml`\n\nLater on you can use the new operations file in a bosh deployment to include\nthe new settings with `bosh int logstash.yml -o manifest/pipelines/cf_platform.yml`\n\n\n\n# Deploying with operations files:\n\n\nFor example to deploy the Logstash pipeline for platform logs:\n \n\n```\nbosh -d logstash deploy logstash.yml \\\n    -o operations/pipelines/cf-platform-es.yml \\\n    -o operations/add-es-cloud-id.yml \\\n    -o operations/add-es-xpack.yml \\\n    -o operations/add-logstash-exporter.yml \\\n    -o operations/add-release-version.yml  --vars-file vars-release-version.yml \\\n    -o operations/add-iaas-parameters.yml  --vars-file vars-iaas-parameters.yml\n```\n\n\nBe aware you need to define this secrects in Credhub:\n\n```\nes:\n  cloud_id: \"cf-test:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  host: \"https://elastic.example.com:9000\"\n  user: \"elastic\"\n  password: \"pass\"\n  index_prefix: logs\n\ncf_api: \"my.cf.api.com\"\nsource_env: gcp\nsource_platform: cf\n```\n\n\n# Author\n\nOriginally based on: https://github.com/making/logstash-boshrelease\n\nA lot of things changed by SpringerNature Platform Engineering, José Riguera López (jose.riguera@springer.com)\n\n\n# License\n\nApache 2.0 License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspringerpe%2Fcf-logging-boshrelease","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspringerpe%2Fcf-logging-boshrelease","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspringerpe%2Fcf-logging-boshrelease/lists"}