{"id":20801861,"url":"https://github.com/philips-software/fluent-bit-out-hsdp","last_synced_at":"2025-10-26T11:47:16.921Z","repository":{"id":38089486,"uuid":"249044832","full_name":"philips-software/fluent-bit-out-hsdp","owner":"philips-software","description":"fluent bit output plugin to HSDP logging","archived":false,"fork":false,"pushed_at":"2025-04-24T12:28:36.000Z","size":591,"stargazers_count":7,"open_issues_count":0,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-04T02:16:55.743Z","etag":null,"topics":["fluent-bit","forwarder","golang","hsdp","json-logging","kubernetes","log","logging"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philips-software.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-21T19:20:37.000Z","updated_at":"2025-04-24T12:28:39.000Z","dependencies_parsed_at":"2024-06-21T20:31:00.387Z","dependency_job_id":"ea93aad3-26f1-4258-aea0-cc7e4a19ce91","html_url":"https://github.com/philips-software/fluent-bit-out-hsdp","commit_stats":null,"previous_names":[],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips-software%2Ffluent-bit-out-hsdp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips-software%2Ffluent-bit-out-hsdp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips-software%2Ffluent-bit-out-hsdp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philips-software%2Ffluent-bit-out-hsdp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philips-software","download_url":"https://codeload.github.com/philips-software/fluent-bit-out-hsdp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252793546,"owners_count":21805053,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluent-bit","forwarder","golang","hsdp","json-logging","kubernetes","log","logging"],"created_at":"2024-11-17T18:24:49.782Z","updated_at":"2025-10-26T11:47:11.901Z","avatar_url":"https://github.com/philips-software.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent bit HSDP logging output plugin\n\nThis plugin outputs your logs to the HSDP Host Logging service. This is useful when your workloads are not running on Cloud foundry, but you still want to utilize the central logging facilities of HSDP. \n\n  ![flow](resources/fluent-bit.svg)\n\nFluent bit supports parser and filter plugin which can convert unstructured data gathered from the log Input interface into a structured one and to alter existing structured data before ingestion.\n\n[More on fluent-bit](https://fluentbit.io/documentation/0.14/getting_started/)\n\n## Cloud and On-Premise\n\nThe plugin supports deployment to both Cloud and On-Premise environments. Depending on \nthe deployment type you can either specify the Cloud `Region` and `Environment` or the On-Premise\n`IamUrl` and `IdmUrl` values.\n\n## API Signing, Service Identities or Logdrainer\n\nThe plugin supports both the API Signing authorization mechanism or the use\nof a IAM Service Identity with the `LOG.CREATE` scope. It also supports using a CF Logdrain endpoint \nalthough that limits the log format output (e.g. no support for the `custom` field)\n\n## Configuration options\nYour `fluent-bit.conf` file should include an entry like below to enable the plugin:\n\n```\n[output]\n    Name hsdp\n    Match *\n```\n\nConfiguring the authorization mechanism and HSDP Logging endpoints should ideally\nbe done by setting the right Environment variables:\n\n## Settings\n\n### General\n\nThese keys are relevant when using either SigningKey or Service identities\n\n| Key                | Description                                          | Environment variable         | Required |\n|--------------------|------------------------------------------------------|------------------------------|----------|\n| ProductKey         | The Product key of your proposition                  | HSDP\\_PRODUCT\\_KEY           | Required |\n| Region             | The HSP Region (Cloud)                               | HSDP\\_REGION                 | Required |\n| Environment        | THE HSP Environment (Cloud)                          | HSDP\\_ENVIRONMENT            | Required |\n| IamUrl             | The IAM URL (On-Premise)                             | HSDP\\_IAM\\_URL               | Optional |\n| IdmUrl             | The IDM URL (On-Premise)                             | HSDP\\_IDM\\_URL               | Optional |\n| IngestorHost       | The HSDP ingestor host                               | HSDP\\_INGESTOR\\_HOST         | Optional |\n| Debug              | Shows request details when set to true               | HSDP\\_DEBUG                  | Optional |\n| CustomField        | Adds the field hash to custom field when set to true | HSDP\\_CUSTOM\\_FIELD          | Optional |\n| InsecureSkipVerify | Skip checking HSDP ingestor TLS cert. Insecure!      | HSDP\\_INSECURE\\_SKIP\\_VERIFY | Optional |\n| SynchronousFlush   | Flushes log messages synchronously without batching. By default this is set to *false*  |   | Optional |\n| RetryOnError       | Returns retry to FLB if flush fails. Applicable only when *SynchronousFlush* option is set. By default this is set to *false* | | Optional | \n| Proxy              | HTTP Proxy URL in case there is a proxy redirection required             |  | Optional |\n\n### Signing keys\n\n| Key       | Description                         | Environment variable | Required |\n|-----------|-------------------------------------|----------------------|----------|\n| SharedKey | The Shared key for signing requests | HSDP\\_SHARED\\_KEY    | Optional |\n| SecretKey | The Secret key for signing requests | HSDP\\_SECRET\\_KEY    | Optional |\n\n### Service identities\n\n| Key               | Description                              | Environment variable        | Required |\n|-------------------|------------------------------------------|-----------------------------|----------|\n| ServiceId         | The Service ID to use for authentication | HSDP\\_SERVICE\\_ID           | Optional |\n| ServicePrivateKey | The Service private key                  | HSDP\\_SERVICE\\_PRIVATE\\_KEY | Optional |\n\n### Logdrain\n\nYou can reuse an existing Cloud foundry logdrainer endpoint to ship your logs to HSDP logging. The advantage is\nthat you only need the Logdrain URL itself, no other configuration. This URL is considered\na credential so care should be taken to protect it though.\n\n| Key                     | Description                 | Environment variable           | Required |\n|-------------------------|-----------------------------|--------------------------------|----------|\n| LogdrainUrl             | The HSP Logdrain URL        | HSDP_LOGDRAIN_URL              | Optional |\n| LogdrainApplicationName | The Application name to use | HSDP_LOGDRAIN_APPLICATION_NAME | Optional |\n| LogdrainServerName      | The Server name to use      | HSDP_LOGDRAIN_SERVER_NAME      | Optional |\n\n\u003e Environment variable values take precedence over those in configuration files.\n\n## Record field mapping to HSDP logging resource\n\nThe plugin supports full pass-through of the native `LogEvent` JSON message type, example:\n\n```json\n{\n  \"resourceType\": \"LogEvent\",\n  \"id\": \"7f4c85a8-e472-479f-b772-2916353d02a4\",\n  \"applicationName\": \"OPS\",\n  \"eventId\": \"110114\",\n  \"category\": \"TRACELOG\",\n  \"component\": \"TEST\",\n  \"transactionId\": \"2abd7355-cbdd-43e1-b32a-43ec19cd98f0\",\n  \"serviceName\": \"OPS\",\n  \"applicationInstance\": \"INST‐00002\",\n  \"applicationVersion\": \"1.0.0\",\n  \"originatingUser\": \"SomeUsr\",\n  \"serverName\": \"ops-dev.cloud.pcftest.com\",\n  \"logTime\": \"2017-01-31T08:00:00Z\",\n  \"severity\": \"INFO\",\n  \"logData\": {\n    \"message\": \"VGVzdCBNZXNzYWdl\"\n  },\n  \"custom\": {\n    \"foo\": \"bar\"\n  }\n}\n```\n\n\u003e **NOTE**: the `logData.message` field must be [base64 encoded](https://www.hsdp.io/documentation/logging/api-documents/log-event-api/log-event-api)\n\n### field mapping\n\nAlternatively, the plugin maps certain record fields to defined HSDP logging resource fields. The below\ntable shows the mapping, and the default value.\n\n| Record field       | HSDP logging field  | Default value | Details |\n|--------------------|---------------------|---------------|-----------------------|\n| server\\_name       | serverName          | fluent-bit    ||\n| app\\_name          | applicationName     | fluent-bit    ||\n| app\\_instance      | applicationInstance | fluent-bit    ||\n| app\\_version       | applicationVersion  | 1.0           ||\n| category           | category            | TraceLog      ||\n| severity           | severity            | informational ||\n| service\\_name      | service\\_name       | fluent-bit    ||\n| originating\\_user  | originating\\_user   | fluent-bit    ||\n| event\\_id          | event\\_id           | 1             ||\n| transaction\\_id    | transaction\\_id     | random UUID   |if original input is not a valid UUID a new one will be generated|\n| trace\\_id          | trace\\_id           |               ||\n| span\\_id           | span\\_id            |               ||\n| logdata\\_message   | logData.Message     | field hash    |will replace the default field hash dump went present|\n\n\u003e Fields mapped to a HSDP logging resource field will be removed from the log message dump\n\nThe below filter definition shows an example of assigning fields\n\n```yaml\n[filter]\n    Name record_modifier\n    Match *\n    Record server_name ${HOSTNAME}\n    Record service_name Awesome_Tool\n```\n\n```yaml\n[filter]\n    Name modify\n    Match *\n    Rename container_name app_name\n    Rename container_name service_name\n    Rename component_name component\n    Rename container_id app_instance\n```\n\n\u003e Remaining fields will be rendered to a JSON hash and assigned to `logData.Message`\n\n## Building\n\n```shell\ndocker build -t fluent-bit-out-hsdp .\n```\n\n## Testing with Docker\n\n```shell\ndocker run --rm \\\n    -p 127.0.0.1:24224:24224 \\\n    -e HSDP_PRODUCT_KEY=product-key-here \\\n    -e HSDP_REGION=us-east \\\n    -e HSDP_ENVIRONMENT=client-test \\\n    -e HSDP_SERVICE_ID=my.service.id@app.prop.philips-healthsuite.com \\\n    -e HSDP_SERVICE_PRIVATE_KEY=\"$(cat service_private_key.pem)\" \\\n    -it ghcr.io/philips-software/fluent-bit-out-hsdp:latest\n```\n\nOnce the above is running you can start other Docker containers and \nuse fluentd log driver to start logging to HSDP logging:\n\n```shell\ndocker run --rm -it --log-driver fluentd alpine echo \"hello world\"\n```\n\n## Helm Chart\n\nYou can deploy fluent-bit and the HSDP plugin using a Helm chart\n\n### Secret\n\nThe chart will attempt to read credentials from an `hsdp-logging` Kubernetes secret which should reside\nin the namespace. An example `hsdp-logging-secret.yaml` is included below. Make sure you replace the values accordingly.\n\n\u003e **_NOTE:_**  All the values should be in the base64 encoded.\n\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: hsdp-logging\ntype: Opaque\ndata:\n  shared_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz\n  secret_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz\n  product_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz\n  ingestor_host: aHR0cHM6Ly9sb2dpbmdlc3RvcjItY2xpZW50LXRlc3QuZXUtd2VzdC5waGlsaXBzLWhlYWx0aHN1aXRlLmNvbQ==\n```\n\nApply the secret to the right namepace (for Fiesta this is `kube-system`):\n\n```shell\nkubectl apply -f hsdp-logging-secret.yaml -n logging\n```\n\nAdd helm repo:\n\n```shell\nhelm repo add philips-software https://philips-software.github.io/helm-charts/\n```\n\nDownload the helm chart\n```shell\nhelm pull philips-software/fluent-bit-out-hsdp\n```\n\nNavigate to the download Helm chart and unzip.\n\nAfter unzip, open `fluent-bit-out-hsdp\\values.yml` and add the enviroment details like:\n```yaml\nfluent-bit:\n  env:\n   - name: HSDP_REGION\n     valueFrom:\n       secretKeyRef:\n         key: region\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_ENVIRONMENT\n     valueFrom:\n       secretKeyRef:\n         key: environment\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_PRODUCT_KEY\n     valueFrom:\n       secretKeyRef:\n         key: product_key\n         name: hsdp-logging\n         optional: false\n   - name: HSDP_INGESTOR_HOST\n     valueFrom:\n       secretKeyRef:\n         key: ingestor_host\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_SHARED_KEY\n     valueFrom:\n       secretKeyRef:\n         key: shared_key\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_SECRET_KEY\n     valueFrom:\n       secretKeyRef:\n         key: secret_key\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_SERVICE_ID\n     valueFrom:\n       secretKeyRef:\n         key: service_id\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_SERVICE_PRIVATE_KEY\n     valueFrom:\n       secretKeyRef:\n         key: service_private_key\n         name: hsdp-logging\n         optional: true\n```\n\nYou can skip the unwanted fields from above yaml. For e.g, if you are going to use secret_key and shared_key based authentication, then only fields required are: \n```yaml\nfluent-bit:\n  env:\n   - name: HSDP_PRODUCT_KEY\n     valueFrom:\n       secretKeyRef:\n         key: product_key\n         name: hsdp-logging\n         optional: false\n   - name: HSDP_INGESTOR_HOST\n     valueFrom:\n       secretKeyRef:\n         key: ingestor_host\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_SHARED_KEY\n     valueFrom:\n       secretKeyRef:\n         key: shared_key\n         name: hsdp-logging\n         optional: true\n   - name: HSDP_SECRET_KEY\n     valueFrom:\n       secretKeyRef:\n         key: secret_key\n         name: hsdp-logging\n         optional: true\n```\n\nInstall the Helm chart\n\n```shell\nhelm install my-fluent-bit-out-hsdp fluent-bit-out-hsdp -n logging\n```\n\nIf the credentials are correct you should now see your Kubernetes cluster logs in the HSDP Logging system.\n\n## Contact / Getting help\n\nAndy Lo-A-Foe \u003candy.lo-a-foe@philips.com\u003e\n\n## License\n\nLicense is MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilips-software%2Ffluent-bit-out-hsdp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilips-software%2Ffluent-bit-out-hsdp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilips-software%2Ffluent-bit-out-hsdp/lists"}