{"id":15291063,"url":"https://github.com/azure/logstash-output-kusto","last_synced_at":"2025-04-06T01:09:53.697Z","repository":{"id":34030994,"uuid":"145303365","full_name":"Azure/logstash-output-kusto","owner":"Azure","description":"Logstash output for Kusto","archived":false,"fork":false,"pushed_at":"2025-03-21T03:26:14.000Z","size":16119,"stargazers_count":14,"open_issues_count":10,"forks_count":15,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-04-06T01:09:48.843Z","etag":null,"topics":["kusto","logstash-plugin","ruby"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Azure.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-19T12:58:54.000Z","updated_at":"2025-03-20T11:51:04.000Z","dependencies_parsed_at":"2024-08-09T06:43:44.908Z","dependency_job_id":"20c49273-f334-46f4-9771-2dea4edae2e8","html_url":"https://github.com/Azure/logstash-output-kusto","commit_stats":{"total_commits":29,"total_committers":12,"mean_commits":"2.4166666666666665","dds":0.7241379310344828,"last_synced_commit":"f903842f9f286985569d1c8fb1a840a93d8df5da"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Flogstash-output-kusto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Flogstash-output-kusto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Flogstash-output-kusto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Flogstash-output-kusto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Azure","download_url":"https://codeload.github.com/Azure/logstash-output-kusto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247419861,"owners_count":20936012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kusto","logstash-plugin","ruby"],"created_at":"2024-09-30T16:10:47.825Z","updated_at":"2025-04-06T01:09:53.670Z","avatar_url":"https://github.com/Azure.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Logstash Output Plugin for Azure Data Explorer (Kusto)\n\n![build](https://github.com/Azure/logstash-output-kusto/workflows/build/badge.svg)\n![build](https://github.com/Azure/logstash-output-kusto/workflows/build/badge.svg?branch=master)\n[![Gem](https://img.shields.io/gem/v/logstash-output-kusto.svg)](https://rubygems.org/gems/logstash-output-kusto)\n[![Gem](https://img.shields.io/gem/dt/logstash-output-kusto.svg)](https://rubygems.org/gems/logstash-output-kusto)\n\nThis is a plugin for [Logstash](https://github.com/elastic/logstash).\n\nIt is fully free and open source. The license is Apache 2.0.\n\nThis Azure Data Explorer (ADX) Logstash plugin enables you to process events from Logstash into an **Azure Data Explorer** database for later analysis. \n\nThis connector forwards data to\n[Azure Data Explorer](https://docs.microsoft.com/en-us/azure/data-explorer),\n[Azure Synapse Data Explorer](https://docs.microsoft.com/en-us/azure/synapse-analytics/data-explorer/data-explorer-overview) and\n[Real time analytics in Fabric](https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview)\n\n## Requirements\n\n- Logstash version 6+. [Installation instructions](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html) \n- Azure Data Explorer cluster with a database. Read [Create a cluster and database](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal) for more information.\n- AAD Application credentials with permission to ingest data into Azure Data Explorer. Read [Creating an AAD Application](https://docs.microsoft.com/en-us/azure/kusto/management/access-control/how-to-provision-aad-app) for more information.\n\n## Installation\n\nTo make the Azure Data Explorer plugin available in your Logstash environment, run the following command:\n```sh\nbin/logstash-plugin install logstash-output-kusto\n```\n\n## Configuration\n\nPerform configuration before sending events from Logstash to Azure Data Explorer. The following example shows the minimum you need to provide. It should be enough for most use-cases:\n\n```ruby\noutput {\n\tkusto {\n            path =\u003e \"/tmp/kusto/%{+YYYY-MM-dd-HH-mm}.txt\"\n            ingest_url =\u003e \"https://ingest-\u003ccluster-name\u003e.kusto.windows.net/\"\n            app_id =\u003e \"\u003capplication id\u003e\"\n            app_key =\u003e \"\u003capplication key/secret\u003e\"\n            app_tenant =\u003e \"\u003ctenant id\u003e\"\n            database =\u003e \"\u003cdatabase name\u003e\"\n            table =\u003e \"\u003ctarget table\u003e\"\n            json_mapping =\u003e \"\u003cmapping name\u003e\"\n            proxy_host =\u003e \"\u003cproxy host\u003e\"\n            proxy_port =\u003e \u003cproxy port\u003e\n            proxy_protocol =\u003e \u003c\"http\"|\"https\"\u003e              \n\t}\n}\n```\nMore information about configuring Logstash can be found in the [logstash configuration guide](https://www.elastic.co/guide/en/logstash/current/configuration.html)\n\n### Available Configuration Keys\n\n| Parameter Name | Description | Notes |\n| --- | --- | --- |\n| **path** | The plugin writes events to temporary files before sending them to ADX. This parameter includes a path where files should be written and a time expression for file rotation to trigger an upload to the ADX service. The example above shows how to rotate the files every minute and check the Logstash docs for more information on time expressions. | Required\n| **ingest_url** | The Kusto endpoint for ingestion-related communication. See it on the Azure Portal.| Required|\n| **app_id, app_key, app_tenant**| Credentials required to connect to the ADX service. Be sure to use an application with 'ingest' privileges. | Optional|\n| **managed_identity**| Managed Identity to authenticate. For user-based managed ID, use the Client ID GUID. For system-based, use the value `system`. The ID needs to have 'ingest' privileges on the cluster. | Optional|\n| **database**| Database name to place events | Required |\n| **table** | Target table name to place events | Required\n| **json_mapping** | Maps each attribute from incoming event JSON strings to the appropriate column in the table. Note that this must be in JSON format, as this is the interface between Logstash and Kusto | Optional |\n| **recovery** | If set to true (default), plugin will attempt to resend pre-existing temp files found in the path upon startup | |\n| **delete_temp_files** | Determines if temp files will be deleted after a successful upload (true is default; set false for debug purposes only)| |\n| **flush_interval** | The time (in seconds) for flushing writes to temporary files. Default is 2 seconds, 0 will flush on every event. Increase this value to reduce IO calls but keep in mind that events in the buffer will be lost in case of abrupt failure.| |\n| **proxy_host** | The proxy hostname for redirecting traffic to Kusto.| |\n| **proxy_port** | The proxy port for the proxy. Defaults to 80.| |\n| **proxy_protocol** | The proxy server protocol , is one of http or https.| |\n\n\u003e Note : LS_JAVA_OPTS can be used to set proxy parameters as well (using export or SET options)\n\n```bash\nexport  LS_JAVA_OPTS=\"-Dhttp.proxyHost=1.2.34 -Dhttp.proxyPort=8989 -Dhttps.proxyHost=1.2.3.4 -Dhttps.proxyPort=8989\"\n```\n\n\n### Release Notes and versions\n\n| Version | Release Date | Notes |\n| --- | --- | --- |\n| 2.0.8 | 2024-10-23 | - Fix library deprecations, fix issues in the Azure Identity library  |\n| 2.0.7 | 2024-01-01 | - Update Kusto JAVA SDK  |\n| 2.0.3 | 2023-12-12 | - Make JSON mapping field optional. If not provided logstash output JSON attribute names will be used for column resolution  |\n| 2.0.2 | 2023-11-28 | - Bugfix for the scenario where the plugin uses managed identity. Instead of providing the managed identity name as empty in the config,it can completely be skipped  |\n| 2.0.0 | 2023-09-19 | - Upgrade to the latest Java SDK version [5.0.2](https://github.com/Azure/azure-kusto-java/releases/tag/v5.0.2). Tests have been performed on **__Logstash 8.5__**  and up (Does not work with 6.x or 7.x versions of Logstash - For these versions use 1.x.x versions of logstash-output-kusto gem) - Fixes CVE's in common-text \u0026 outdated Jackson libraries  |\n| 1.0.6 | 2022-11-29 | - Upgrade to the latest Java SDK [3.2.1](https://github.com/Azure/azure-kusto-java/releases/tag/v3.2.1) version. Tests have been performed on Logstash 6.x and up.|\n\n\n## Development Requirements\n\n- Openjdk **8 64bit** (https://www.openlogic.com/openjdk-downloads)\n- JRuby 9.2 or higher, defined with openjdk 8 64bit\n- Logstash, defined with openjdk 8 64bit\n\n*It is reccomened to use the bundled jdk and jruby with logstash to avoid compatibility issues.*\n\nTo fully build the gem, run: \n\n```shell\nbundle install\nlock_jars\ngem build\n```\n\n## Contributing\n\nAll contributions are welcome: ideas, patches, documentation, bug reports, and complaints.\nProgramming is not a required skill. It is more important to the community that you are able to contribute.\nFor more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazure%2Flogstash-output-kusto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazure%2Flogstash-output-kusto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazure%2Flogstash-output-kusto/lists"}