{"id":26059705,"url":"https://github.com/orange-cloudfoundry/loghost-boshrelease","last_synced_at":"2026-03-11T00:35:03.583Z","repository":{"id":47963688,"uuid":"249170873","full_name":"orange-cloudfoundry/loghost-boshrelease","owner":"orange-cloudfoundry","description":"BOSH release gather, store and analyze logs generated by bosh VMs","archived":false,"fork":false,"pushed_at":"2025-12-18T08:08:17.000Z","size":131,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-12-19T22:36:32.338Z","etag":null,"topics":["bosh","grok","prometheus","syslog"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orange-cloudfoundry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-03-22T11:40:48.000Z","updated_at":"2025-12-18T08:07:00.000Z","dependencies_parsed_at":"2024-01-25T16:47:04.214Z","dependency_job_id":"4a3e4316-8ce5-4e2e-bc92-1fd85fa8179e","html_url":"https://github.com/orange-cloudfoundry/loghost-boshrelease","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/orange-cloudfoundry/loghost-boshrelease","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Floghost-boshrelease","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Floghost-boshrelease/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Floghost-boshrelease/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Floghost-boshrelease/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orange-cloudfoundry","download_url":"https://codeload.github.com/orange-cloudfoundry/loghost-boshrelease/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orange-cloudfoundry%2Floghost-boshrelease/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30364069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"ssl_error","status_checked_at":"2026-03-10T21:40:59.357Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bosh","grok","prometheus","syslog"],"created_at":"2025-03-08T13:27:57.275Z","updated_at":"2026-03-11T00:35:03.566Z","avatar_url":"https://github.com/orange-cloudfoundry.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --\u003e\n**Table of Contents**\n\n- [loghost-boshrelease](#loghost-boshrelease)\n    - [Introduction](#introduction)\n    - [Components](#components)\n        - [Concentrator](#concentrator)\n            - [Format](#format)\n            - [Generated files](#generated-files)\n            - [Rotation](#rotation)\n            - [Forwarding and clustering](#forwarding-and-clustering)\n        - [Dns](#dns)\n        - [Exporter](#exporter)\n        - [Alerts](#alerts)\n        - [Dashboards](#dashboards)\n    - [Usage](#usage)\n        - [Step 1: Deploy loghost](#step-1-deploy-loghost)\n        - [Step 2: Forward all logs to loghost instance](#step-2-forward-all-logs-to-loghost-instance)\n        - [Step 3: Add alerts and dashboard to prometheus](#step-3-add-alerts-and-dashboard-to-prometheus)\n    - [Reference](#reference)\n        - [Ops-files](#ops-files)\n        - [Metrics](#metrics)\n\n\u003c!-- markdown-toc end --\u003e\n\n# loghost-boshrelease\n\nThis is a BOSH release to gather, store and analyze syslog events forwarded by bosh VMs. It\ncurrently uses [RSyslog] which is pre-installed by the stemcell.\n\nOnly Linux stemcells are supported at the moment.\n\n## Introduction\n\nUsually, platform logs are sent to ELK stacks which store and index events on-the-fly.\nFinally, users can build fancy Kibana dashboards by extracting metrics from elasticsearch queries.\n\nWith the development of micro-services architectures, the number of emitted logs recently exploded,\nmaking these ELKs very hardware and, therefore, money consuming.\nEven more, these stacks are often built with heavy redundancy and high availability even when most of the emitted events are not critical.\n\nThe idea here is having a much more lightweight architecture, providing only the most essential\nfeatures of log processing:\n- midterm storage for debug and production incident analysis\n- hardware-efficient generation of metrics\n- redundancy and availability matching the actual criticality of the logs\n\nThis is achieved by using both good old technologies such as [RSyslog] and modern\ntools like [Prometheus].\nThe bridge between logs and metrics is provided by a brilliant tool [grok_exporter].\n\n## Components\n\n### Concentrator\n\nThe job `loghost_concentrator` configures local `rsyslogd` to store received logs to persistent disk.\n\n#### Format\n\nOnly syslog events received in [RFC5424] format with `instance@47450` in\n[Structured Data ID][structured-data] are handled. The `47450` private enterprise number is the one\ngenerated by the [syslog-release] generally used to forward VM log events to a given endpoint.\n\n#### Generated files\n\nReceived logs are stored on persistent disk in root directory `/var/vcap/store/loghost` where\n`{path}` depends on parsed [Structured Data ID][structured-data] fields of the event.\n\nAssuming logs are forwarded by [syslog-release], the parsed fields are:\n- `$.director`: the configured name of bosh director\n- `$.deployment`: the name of deployment from which event was sent\n- `$.group`: the name of the instance from which event was sent\n\nFinally, logs are stored under `/var/vcap/store/loghost/{$.director}/{$.deployment}/{$.group}.log`\n\n#### Rotation\n\nThe job also configures local `logrotate` in order to rotate and compress logs every hour.\nRotated logs are stored in the same directories with the `-%Y%m%d%H.gz` suffix.\n\nThe number of kept rotations can be configured `loghost_concentrator.logrotate.max-hours` property\nwith a default value of `360` (i.e.: 15 days).\n\n#### Forwarding and clustering\n\nThe job also provides the possibility to re-forward received syslog event under specified\nconditions; this can be useful for:\n- Clusterize multiple concentrators in order to create a kind of backup across independent BOSH directors\n- Forward business or security critical events to an external log handling platform\n\n![clustering]\n\nForwarding is configured from the `loghost_concentrator.syslog.forward` property by defined\ntarget objects as follows:\n\n```yml\n\u003ctarget-name\u003e:\n  conditions:\n  - \u003ccondition\u003e\n  - ...\n  targets:\n    - address: hostname\n      port: port\n      transport: tcp|udp|relp\n    - ...\n```\n\n**Where:**\n\n- `\u003ccondition\u003e` are valid [rainerscript] expressions with parenthesis. Multiple conditions can be\n  given, all must be true to trigger the forward\n- `\u003ctargets\u003e`: is a list of syslog endpoints where matching events are forwarded. When multiple\n  targets are defined, matching events will be forwarded to all endpoints\n\n**Example:**\n\n```yml\njobs:\n  - name: loghost_concentrator\n    release: loghost\n    properties:\n      loghost_concentrator:\n        syslog:\n          forward:\n            my-forward-target:\n              conditions:\n              - ($.director   isequal \"local-director-name\")\n              - ($.deployment isequal \"cf\")\n              targets:\n                - address: target1.hostname.example.com\n                  port: 514\n                  transport: tcp\n                - address: target2.hostname.example.com\n                  port: 514\n                  transport: tcp\n```\n\n### DNS\n\nAssuming that your deployment uses [bosh-dns], the job `loghost_dns` can be used to define new\naliases.\n\nDNS aliases are configured from the `loghost_dns.aliases` key with the same syntax as the\n`aliases` key of [bosh-dns job][bosh-dns-job].\n\n**Example:**\n\n```yaml\njobs:\n  - name: loghost_dns\n    release: loghost\n    properties:\n      loghost_dns:\n        aliases:\n          my.alias.internal:\n          - 127.0.0.1\n          my.other.alias.internal:\n          - '*.collector-z1.default.logsearch.bosh'\n          - '*.collector-z2.default.logsearch.bosh'\n```\n\n### Exporter\n\nThe `loghost_exporter` job installs and configures the [grok_exporter]. This brilliant program\nprocesses log files and computes [Prometheus] metrics according to parse rules given in\n[grok] format.\n\nParsing rules are defined by the `loghost_exporter.metrics` key with the exact same syntax defined\nby the [grok_exporter-metrics].\n\nIn addition, `loghost_exporter.directors` and `loghost_exporter.deployments` keys must be configured\nto give the list of logs files that the exported should watch.\n\n\u003e **Note**: A limitation in the grok_exporter implementation forces watched directories to pre-exist\n\u003e at exporter startup. Because rsyslog files are created on the fly when events are received, the\n\u003e job creates required directories in its `pre-start` script.\n\nIn addition to user-defined metrics, the exporter provides\n[builtin metrics][grok-builtin-metrics].\n\nOps-files provided in the release also provide metrics, as described in the [usage section](#usage).\n\n### Alerts\n\nThe job `loghost_alerts` defines the following alerts for your [prometheus-boshrelease] deployment:\n- `LoghostNoLogReceived`: triggers if exporter reports no processed logs in the last 15 minutes\n- `LoghostDroppedMessages`: triggers when there is an increase of \"failed to write to target.example.net:6067\" in the logs\n\nWhen `loghost_alerts.security.enabled` key is set to `true` (default `false`), the job also defines\nthe following alerts:\n- `SecurityTooManySystemAuthFailures`: triggers when `audispd` reports too many auth failures.\n  `audispd` logs are generated by all virtual machines deployed by bosh\n- `SecurityTooManyUaaClientFailures`:  triggers when `uaa` component reports too many\n   client authentication failures\n- `SecurityTooManyUaaUserFailures`: triggers when `uaa` component reports too many\n   user authentication failures\n- `SecurityTooManyDiegoSshFailures`: triggers when `ssh_proxy` component running on (`scheduler`\n   instance) reports too many SSH authentication failures to containers\n- `SecurityTooManyDiegoSshSuccess`: triggers when `ssh_proxy` component running on (`scheduler`\n   instance) reports too many SSH authentications to containers\n\nAlert thresholds and evaluation time can be configured from job's spec.\n\n### Dashboards\n\nThe job `loghost_dashboards` adds [Grafana] dashboards for your [prometheus-boshrelease]\ndeployment.\n\n- a global overview giving the system status, number of processed logs per rules, deployments and\n  instances\n\n- a security dashboard overview giving information on authentications when\n  `loghost_dashboards.security.enabled` key is enabled.\n\n\n## Usage\n\n### Step 1: Deploy loghost\n\nFirst, you must add `loghost` instance to the deployment of your choice. You can use the following\nops-files:\n- `manifests/operations/loghost-concentrator-enable.yml`\n- `manifests/operations/loghost-exporter-enable.yml`\n- `manifests/operations/loghost-exporter-enable-security.yml`\n\nIt will add the instance `loghost` with basic features enabled:\n- received log written to `/var/vcap/store/loghost`\n- [grok_exporter] reading and generating metrics from received logs\n\n### Step 2: Forward all logs to loghost instance\n\nThe simplest way to forward all logs at once is to create a `runtime-config.yml` using the [syslog-release].\n\n\nWith file `runtime-syslog-forward.yml`:\n\n```yaml\naddons:\n- exclude:\n    instance_groups:\n    - loghost\n  jobs:\n  - name: syslog_forwarder\n    properties:\n      syslog:\n        address: q-s0.loghost.default.((deployment)).bosh\n        director: ((director_name))\n        transport: udp\n    release: syslog\n  name: syslog_forwarder\nreleases:\n- name: syslog\n  sha1: 658fe5d6f049ec50383c09c0b227261251bfd4eb\n  url: https://artifactory/cloudfoundry/syslog/syslog-11.6.1-ubuntu-xenial-621.tgz\n  version: 11.6.1\n```\n\nUpload to bosh director: `bosh update-runtime-config --name syslog-forward runtime-syslog-forward.yml`\n\n### Step 3: Add alerts and dashboard to prometheus\n\nAdd the following ops-files to your prometheus deployment:\n\n- `manifests/operations/prometheus/loghost-enable.yml`\n- `manifests/operations/prometheus/loghost-enable-security.yml`\n\nIt will:\n\n- define scrape config based on `bosh_exporter` discovery\n- define new alerts\n- add dashboards to Grafana\n\n## Reference\n\n### Ops-files\n\n| name                                   | description                                                                           |\n|----------------------------------------|---------------------------------------------------------------------------------------|\n| loghost-concentrator-enable.yml        | add  instance with `loghost_concentrator` job listening on udp                        |\n| loghost-concentrator-enable-tcp.yml    | configure `loghost_concentrator` to listen on tcp addition to udp                     |\n| loghost-dns-enable.yml                 | add `loshost_dns` job with empty aliases list                                         |\n| loghost-exporter-enable.yml            | add `loghost_exporter` job which spawns `grok_exporter` with a default set of metrics |\n| loghost-exporter-enable-security.yml   | add security metrics to `loghost_exporter` job, grok rules for `uaa` and `audispd`    |\n| prometheus/loghost-enable.yml          | add discovery scraping of `grok_exporter`, default alerts and dashboards              |\n| prometheus/loghost-enable-security.yml | add security alerts and dashboards                                                    |\n\n### Metrics\n\nIn addition to [grok_exporter] [grok-builtin-metrics], the release defines:\n\n| name                                     | dimensions                                        | type      | description                                                  |\n|------------------------------------------|---------------------------------------------------|-----------|--------------------------------------------------------------|\n| loghost_total                            | director, deployment, group                       | (Counter) | log processed                                                |\n| loghost_error_total                      | director, deployment, group                       | (Counter) | log detected as level *error*                                |\n| loghost_auth_failures                    | director, deployment, group, source, ip           | (Counter) | system authentication failures                               |\n| loghost_auth_failures_last_5m            | director, deployment, group, source, ip           | (Gauge)   | system authentication failures in the last 5 minutes (*)     |\n| loghost_auth_success                     | director, deployment, group, source, ip, username | (Counter) | system authentication success                                |\n| loghost_auth_success_last_5m             | director, deployment, group, source, ip, username | (Gauge)   | system authentication success in the last 5 minutes (*)      |\n| loghost_uaa_client_login_success         | director, deployment, group, ip, clientid         | (Counter) | UAA client authentication success                            |\n| loghost_uaa_client_login_success_last_5m | director, deployment, group, ip, clientid         | (Gauge)   | UAA client authentication success in the last 5 minutes (*)  |\n| loghost_uaa_client_login_failure         | director, deployment, group, ip, clientid         | (Counter) | UAA client authentication failures                           |\n| loghost_uaa_client_login_failure_last_5m | director, deployment, group, ip, clientid         | (Gauge)   | UAA client authentication failures in the last 5 minutes (*) |\n| loghost_uaa_user_login_success           | director, deployment, group, ip, username         | (Counter) | UAA user authentication success                              |\n| loghost_uaa_user_login_success_last_5m   | director, deployment, group, ip, username         | (Gauge)   | UAA user authentication success in the last 5 minutes (*)    |\n| loghost_uaa_user_login_failure           | director, deployment, group, ip, username         | (Counter) | UAA user  authentication failures                            |\n| loghost_uaa_user_login_failure_last_5m   | director, deployment, group, ip, username         | (Gauge)   | UAA user failures in the last 5 minutes (*)                  |\n\nWith dimension values:\n- `director`, `deployment`, `group`: BOSH director name, deployment name and instance group name\n   from where the log was originally emitted\n- `source`: the `exe` field of type=`USER.*` message of `audispd`\n- `ip`: the remote address from which the authentication was attempted\n- `clientid`: the `clientid` used to authenticate a client on `UAA`\n- `username`: the `username` used to authenticate a user on `UAA`\n\n\u003e **(*) Tech note**: Because metrics dimensions values are created over time depending on encountered\n\u003e logs, we cannot rely on `rate` or `increase` prometheus function to compute the number of failures\n\u003e on a period of time. As a bypass, we manually compute this metric with a hackish record rule\n\u003e defined as:\n\u003e ```\n\u003e   sum(\u003cmetric\u003e or \u003cmetric\u003e{} * 0) by (\u003cdimensions...\u003e)\n\u003e   -\n\u003e   sum(\u003cmetric\u003e offset 5m or \u003cmetric\u003e{} * 0) by (\u003cdimensions...\u003e)\n\u003e ```\n\n\u003c!-- Local Variables: --\u003e\n\u003c!-- ispell-local-dictionary: \"american\" --\u003e\n\u003c!-- End: --\u003e\n\n[RSyslog]: http://www.rsyslog.com/\n[RFC5424]: https://tools.ietf.org/html/rfc5424\n[syslog-release]: https://github.com/cloudfoundry/syslog-release#format\n[Grafana]: https://grafana.com/\n[prometheus-boshrelease]: https://github.com/bosh-prometheus/prometheus-boshrelease\n[grok-builtin-metrics]: https://github.com/fstab/grok_exporter/blob/master/BUILTIN.md\n[structured-data]: https://tools.ietf.org/html/rfc5424#section-6.3.2\n[clustering]: ./doc/clustering.png\n[rainerscript]: https://www.rsyslog.com/doc/v8-stable/rainerscript/index.html\n[bosh-dns]: https://github.com/cloudfoundry/bosh-dns-release\n[bosh-dns-job]: https://github.com/cloudfoundry/bosh-dns-release/blob/master/jobs/bosh-dns/spec\n[grok_exporter]: https://github.com/fstab/grok_exporter\n[Prometheus]: https://prometheus.io/\n[grok]: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html\n[grok_exporter-metrics]: https://github.com/fstab/grok_exporter/blob/master/CONFIG.md#metrics-section\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Floghost-boshrelease","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forange-cloudfoundry%2Floghost-boshrelease","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forange-cloudfoundry%2Floghost-boshrelease/lists"}