{"id":15287983,"url":"https://github.com/rfoltyns/log4j2-elasticsearch","last_synced_at":"2025-04-07T17:07:21.854Z","repository":{"id":23024609,"uuid":"89024696","full_name":"rfoltyns/log4j2-elasticsearch","owner":"rfoltyns","description":"Log4j2 Elasticsearch Appender plugins","archived":false,"fork":false,"pushed_at":"2024-09-21T13:03:56.000Z","size":2992,"stargazers_count":175,"open_issues_count":17,"forks_count":44,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-31T16:13:50.034Z","etag":null,"topics":["elasticsearch","jackson","log4j2","log4j2-appender","x-pack"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rfoltyns.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-21T21:23:12.000Z","updated_at":"2025-03-26T09:02:25.000Z","dependencies_parsed_at":"2024-12-06T23:15:33.514Z","dependency_job_id":null,"html_url":"https://github.com/rfoltyns/log4j2-elasticsearch","commit_stats":{"total_commits":585,"total_committers":9,"mean_commits":65.0,"dds":0.01538461538461533,"last_synced_commit":"0194cd215000672c78cc4c9ba761b5ac641921b3"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rfoltyns%2Flog4j2-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rfoltyns%2Flog4j2-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rfoltyns%2Flog4j2-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rfoltyns%2Flog4j2-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rfoltyns","download_url":"https://codeload.github.com/rfoltyns/log4j2-elasticsearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247694875,"owners_count":20980733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","jackson","log4j2","log4j2-appender","x-pack"],"created_at":"2024-09-30T15:43:40.295Z","updated_at":"2025-04-07T17:07:21.799Z","avatar_url":"https://github.com/rfoltyns.png","language":"Java","funding_links":[],"categories":["日志库"],"sub_categories":[],"readme":"# log4j2-elasticsearch overview\n\n[![Build Status](https://app.travis-ci.com/rfoltyns/log4j2-elasticsearch.svg?branch=master)](https://travis-ci.com/github/rfoltyns/log4j2-elasticsearch)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.appenders.log4j/parent/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.appenders.log4j/parent)\n[![codecov](https://codecov.io/gh/rfoltyns/log4j2-elasticsearch/branch/master/graph/badge.svg)](https://codecov.io/gh/rfoltyns/log4j2-elasticsearch)\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/rfoltyns/log4j2-elasticsearch)\n\nThis is a parent project for log4j2 appender plugins capable of pushing logs in batches to Elasticsearch clusters.\n\nLatest released code (1.6.x) is available [here](https://github.com/rfoltyns/log4j2-elasticsearch/tree/1.6).\n\nProject consists of:\n* `log4j2-elasticsearch-core` - skeleton provider for conrete implementations\n* `log4j2-elasticsearch-hc` - optimized Apache Async HTTP client compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters\n* (Since 1.6.0) `log4j2-elasticsearch-ahc` - [AsyncHttpClient](https://github.com/AsyncHttpClient/async-http-client) compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters\n* `log4j2-elasticsearch-jest` - [Jest HTTP Client](https://github.com/searchbox-io/Jest) compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters\n* `log4j2-elasticsearch2-bulkprocessor` - [TCP client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.4/java-docs-bulk-processor.html) compatible with 2.x clusters\n* `log4j2-elasticsearch5-bulkprocessor` - [TCP client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/java-docs-bulk-processor.html) compatible with 5.x and 6.x clusters\n* `log4j2-elasticsearch6-bulkprocessor` - [TCP client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/6.2/java-docs-bulk-processor.html) compatible with 6.x clusters\n\n## Features\n\n* Asynchronous log delivery\n* Batch size and flush interval configuration\n* Failover (redirect failed batch to alternative target)\n* JSON message format ([user-provided](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-jest/src/test/java/org/appenders/log4j2/elasticsearch/jest/smoke/CustomMessageFactoryTest.java) or [JacksonJsonLayout](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#jacksonjsonlayout) by default since 1.3 or Log4j2 JsonLayout)\n* (Since 1.1) Index rollover (hourly, daily, etc.)\n* Index template configuration\n* (1.2) Basic Authentication (XPack Security and Shield support)\n* HTTPS support (XPack Security and Shield - visit submodules for compatibility matrix)\n* (1.3) [Pooled buffers](log4j2-elasticsearch-core#object-pooling) (lower memory footprint)\n* Configurable JSON output using [JacksonJsonLayout](log4j2-elasticsearch-core#jacksonjsonlayout)\n* (1.4) Failover with persistence and retry\n* Log overflow prevention with backoff policies\n* [`log4j2-elasticsearch-hc`](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-hc) module - optimized HTTP client\n* Custom JSON output properties support using [VirtualProperty](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#virtual-properties) and (since 1.4.3) [filters](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#virtual-property-filters)\n* Pluggable [internal logging](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/core/logging/InternalLogging.java) (since 1.4.3)\n* (1.5) [ILM policy](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#index-lifecycle-management) configuration\n* Configurable [Jackson modules](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#log4j2-elasticsearch-core#jackson-modules) support\n* [Component templates](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#component-templates) configuration\n* [Composable index templates](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#composable-index-template) configuration\n* [Service Discovery](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc#service-discovery) for HC module\n* (1.6) Elasticsearch 8.x support (`null` mapping type)\n* [Data Streams](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#data-streams) support\n* [Metrics](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#metrics)\n* [`log4j2-elasticsearch-ahc`](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-ahc) module - HTTP client with GZIP support\n\n### Roadmap [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/rfoltyns/log4j2-elasticsearch)\n\n* Send [Metrics](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-core#metrics) to Elasticsearch\n* Prep for \"batch-core\" (or whatever the final name is) module based on common AHC and HC classes\n* More Elasticsearch API integrations\n* TRY to maintain compatibility with OpenSearch\n\nFeature Requests welcome!\n\n## Usage\n\n1. Add this snippet to your `pom.xml` file:\n    ```xml\n    \u003cdependency\u003e\n        \u003cgroupId\u003eorg.appenders.log4j\u003c/groupId\u003e\n        \u003cartifactId\u003elog4j2-elasticsearch-jest\u003c/artifactId\u003e\n        \u003cversion\u003e1.6.1\u003c/version\u003e\n    \u003c/dependency\u003e\n    ```\n\n    Ensure that Log4j2 and Jackson FasterXML jars are added as well - see `Dependencies` section below\n\n\n2. Use simple `log4j2.xml` configuration:\n    ```xml\n    \u003cAppenders\u003e\n        \u003cElasticsearch name=\"elasticsearchAsyncBatch\"\u003e\n            \u003cIndexName indexName=\"log4j2\" /\u003e\n            \u003cJacksonJsonLayout /\u003e\n            \u003cAsyncBatchDelivery\u003e\n                \u003cIndexTemplate name=\"log4j2\" path=\"classpath:indexTemplate.json\" /\u003e\n                \u003cJestHttp serverUris=\"http://localhost:9200\" /\u003e\n            \u003c/AsyncBatchDelivery\u003e\n        \u003c/Elasticsearch\u003e\n    \u003c/Appenders\u003e\n    ```\n\n    or use [optimized Apache HC based HTTP client](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-hc)\n\n    or new [AsyncHttpClient (Netty) based HTTP client](https://github.com/rfoltyns/log4j2-elasticsearch/tree/master/log4j2-elasticsearch-ahc)\n\n    or [log4j2.properties](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/test/resources/log4j2.properties)\n\n    or [configure programmatically](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/test/java/org/appenders/log4j2/elasticsearch/hc/load/SmokeTest.java)\n\n    NOTE: `indexTemplate.json` file is not a part of main jars. You have to create it on your own (because only YOU know which mapping you'd like to use). You can find a few basic ones in tests jars and [log4j2-elasticsearch-examples](https://github.com/rfoltyns/log4j2-elasticsearch-examples).\n\n\n3. Start logging directly to Elasticsearch!\n    ```java\n    Logger log = LogManager.getLogger(\"Logger that references elasticsearchAsyncBatch\");\n    log.info(\"Hello, World!\");\n    ```\n\n    Logs not arriving? Visit [examples](https://github.com/rfoltyns/log4j2-elasticsearch-examples) and verify your config.\n\n## General recommendations\n\n* Start simple with `jest` module. Suitable for smaller loads, up to few thousands of logs per second\n* Use `hc` up to 100-200kps (depends on log size and network bandwidth)\n* Use `ahc` and GZIP for all of the above and 200kps+ (depends on log size and network bandwidth)\n\n## Dependencies\n\nBe aware that Jackson FasterXML, Log4j2, Apache HC, AsyncHttpClient, Netty, Chronicle or JCTools jars (depends on the module you choose) may need to be provided for this library to work. By design, you can choose which jars you'd like to have on your classpath.\nPlease visit [mvnrepository](https://mvnrepository.com/artifact/org.appenders.log4j) for an overview of provided and compile dependencies\n\nIn order to fix [#56](https://github.com/rfoltyns/log4j2-elasticsearch/issues/56), two new modules were extracted from `log4j2-elasticsearch-core`:\n* (1.5+) [appenders-logging](https://github.com/appenders/appenders-logging) (`compile`) available [here](https://mvnrepository.com/artifact/org.appenders.logging/appenders-logging)\n* (1.5+) [appenders-jackson-st](https://github.com/appenders/appenders-jackson-st) (`compile`) available [here](https://mvnrepository.com/artifact/org.appenders.st/appenders-jackson-st)\n\nThis will not cause any issues if you're using packaging tools with transitive dependencies support (Maven, Gradle, etc.). However, in some cases e.g. if you're managing your jars explicitly, classloaders will complain. Sorry for the inconvenience.\n\n## Released to [Sonatype OSS repos](https://oss.sonatype.org/content/repositories/releases/org/appenders/log4j/)\nVisit submodules' documentation or [mvnrepository](https://mvnrepository.com/artifact/org.appenders.log4j) for XML snippets.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frfoltyns%2Flog4j2-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frfoltyns%2Flog4j2-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frfoltyns%2Flog4j2-elasticsearch/lists"}