{"id":22268202,"url":"https://github.com/curityio/kubernetes-logaggregation-elasticsearch","last_synced_at":"2026-04-20T03:31:42.036Z","repository":{"id":74437545,"uuid":"424603582","full_name":"curityio/kubernetes-logaggregation-elasticsearch","owner":"curityio","description":"Demonstrates aggregation of Curity Identity Server logs to Elasticsearch","archived":false,"fork":false,"pushed_at":"2025-03-18T10:54:12.000Z","size":754,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-01-14T05:59:39.298Z","etag":null,"topics":["devops","elasticsearch","kubernetes","logging"],"latest_commit_sha":null,"homepage":"https://curity.io/resources/learn/log-to-elasticsearch/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/curityio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-04T13:21:25.000Z","updated_at":"2025-03-18T10:54:16.000Z","dependencies_parsed_at":"2025-03-04T18:31:30.362Z","dependency_job_id":"e81102e4-5344-40f3-a181-6cc9412390e0","html_url":"https://github.com/curityio/kubernetes-logaggregation-elasticsearch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/curityio/kubernetes-logaggregation-elasticsearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/curityio%2Fkubernetes-logaggregation-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/curityio%2Fkubernetes-logaggregation-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/curityio%2Fkubernetes-logaggregation-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/curityio%2Fkubernetes-logaggregation-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/curityio","download_url":"https://codeload.github.com/curityio/kubernetes-logaggregation-elasticsearch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/curityio%2Fkubernetes-logaggregation-elasticsearch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32031539,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T00:18:06.643Z","status":"online","status_checked_at":"2026-04-20T02:00:06.527Z","response_time":94,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devops","elasticsearch","kubernetes","logging"],"created_at":"2024-12-03T11:11:54.030Z","updated_at":"2026-04-20T03:31:42.021Z","avatar_url":"https://github.com/curityio.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Curity Identity Server Log Aggregation to Elastic Search\n\nAggregates the following logs to Elasticsearch, where logs include OpenTelemetry trace and span IDs.\n\n- System logs\n- Request logs\n- Audit logs\n\n## Prerequisites\n\nStart with base deployments such as the following examples from the Kubernetes Training repository.\n\n- The [Curity Identity Server](https://github.com/curityio/kubernetes-training?tab=readme-ov-file#3---curity-identity-server-example) example deployment.\n- The [Curity Token Handler](https://github.com/curityio/kubernetes-training?tab=readme-ov-file#4---curity-token-handler-example) example deployment.\n\n## 1. Configure Outgoing Logging from the Curity Identity Server\n\nBefore deploying the Curity product, edit its [log4j2.xml](https://github.com/curityio/kubernetes-training/blob/main/resources/curity/idsvr-final/log4j2.xml) file.\\\nReplace default layouts with JSON layouts for the system, request and audit logs.\n\n```xml\n\u003cAppenders\u003e\n    \u003cConsole name=\"stdout\" target=\"SYSTEM_OUT\"\u003e\n        \u003cJSONLayout compact=\"true\" eventEol=\"true\" properties=\"true\" includeTimeMillis=\"true\"\u003e\n                \u003cKeyValuePair key=\"hostname\" value=\"${env:HOSTNAME}\" /\u003e\n        \u003c/JSONLayout\u003e\n        ...\n    \u003c/Console\u003e\n\u003cAppenders\u003e\n```\n\nUse sidecar containers to tail request and audit log files to write them to Kubernetes nodes, ready for log shipping.\\\nDo so by updating the Helm chart [values.yaml](https://github.com/curityio/kubernetes-training/blob/main/resources/curity/idsvr-final/values.yaml) file.\n\n```yaml\ncurity:\n  runtime:\n    logging:\n      level: INFO\n      image: 'busybox:latest'\n      stdout: true\n      logs:\n      - request\n      - audit\n...\n```\n\n## 2. Configure Incoming Logging into Elastic Search\n\n- An [index template](ingestion/indextemplate.json) helps to ensure the type safety storage of fields in logging events.\n- An [ingest pipeline](ingestion/README.md) enables Elasticsearch to transform recived log data to the final JSON format.\n- A Kubernetes job runs a [script](ingestion/initdata.sh) to create the index template and the ingest pipeline.\n\nElasticsearch creates indexes when Filebeat first sends a particular type of log data for a new day.\\\nEach document in the results has an Elasticsearch index such as `curity-request-2025.03.05`.\\\nUse Elasticsearch commands to view the index template and ensure that it gets matched to indexes.\n\n```text\nGET  /_index_template/curity\nPOST /_index_template/_simulate_index/curity-request-2025.03.05\n```\n\n## 3. Configure Log Shipping\n\nThe Filebeat log shipper reads log files from the `/var/log/containers` folder on Kubernetes nodes.\\\nThe log shipper uploads logging events to an Elasticsearch index calculated from the file path and date.\\\nThe following partial configuration shows the approach.\n\n```yaml\nfilebeat.inputs:\n- type: container\n  paths:\n    - /var/log/containers/curity-idsvr-runtime*audit*.log\n    - /var/log/containers/tokenhandler-runtime*-audit*.log\n  json:\n    keys_under_root: true\n    add_error_key: false\n  fields:\n    logtype: 'audit'\n\noutput.elasticsearch:\n  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']\n  username: ${ELASTICSEARCH_USERNAME}\n  password: ${ELASTICSEARCH_PASSWORD}\n  index: \"curity-%{[fields.logtype]}-%{+yyyy.MM.dd}\"\n  pipelines:\n  - pipeline: curity\n```\n\n## 4. Deploy Elastic Stack Components\n\nIf you use the example deployment, run the following script to deploy log aggregation components.\\\nAlternatively, adapt the scripting to match your own deployments.\n\n```bash\n./deploy-elastic-stack.sh\n```\n\nThe script runs a demo deployment of Elasticsearch, Kibana and Filebeat.\\\nThe Kibana frontend uses an external URL of `https://logs.testcluster.example`.\\\nTo make the URL resolvable, get the API gateway's external IP address.\n\n```bash\nkubectl get svc -n apigateway\n```\n\nThen add the Kibana hostname to any other entries for that IP address in the local computer's `/etc/hosts` file.\n\n```text\n172.20.0.5 logs.testcluster.example\n```\n\n## 5. Use Kibana for Live Log Analysis\n\nSign in to Kibana with the following details and access log data from Dev Tools.\n\n- URL: `https://logs.testcluster.example/app/dev_tools#/console`\n- User: elastic\n- Password: Password1\n\nFor example, run Lucene or SQL queries on these indexes to operate on JSON log data.\\\nYou can quickly filter logging events using index fields like an OpenTelemetry trace ID.\n\n```text\nGET curity-system*/_search\n{ \n  \"query\":\n  {\n    \"match\":\n    {\n      \"contextMap.TraceId\": \"ce41b85c6f00f167baa53fd814d23c30\"\n    }\n  }\n}\n```\n\n## Documentation\n\n- See the [Logging Best Practices](https://curity.io/resources/learn/logging-best-practices) article to learn more about Curity Identity Server logs.\n- See the [Elasticsearch Tutorial](https://curity.io/resources/learn/log-to-elasticsearch) for a summary of the Elasticsearch integration.\n\n## More Information\n\nPlease visit [curity.io](https://curity.io/) for more information about the Curity Identity Server.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcurityio%2Fkubernetes-logaggregation-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcurityio%2Fkubernetes-logaggregation-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcurityio%2Fkubernetes-logaggregation-elasticsearch/lists"}