{"id":15152455,"url":"https://github.com/ivanocortesini/log4j-2-elastic","last_synced_at":"2025-10-24T08:31:12.783Z","repository":{"id":45421724,"uuid":"169988734","full_name":"ivanocortesini/log4j-2-elastic","owner":"ivanocortesini","description":"Appender for Elasticsearch with support for all Log4j 2 features","archived":false,"fork":false,"pushed_at":"2022-11-16T06:59:53.000Z","size":42,"stargazers_count":4,"open_issues_count":7,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-31T00:33:17.924Z","etag":null,"topics":["appender","beats","elastic","elasticsearch","log","log4j","log4j2","logger","logging","logs","logstash"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivanocortesini.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-10T14:47:27.000Z","updated_at":"2021-04-07T02:09:00.000Z","dependencies_parsed_at":"2022-08-25T15:03:05.203Z","dependency_job_id":null,"html_url":"https://github.com/ivanocortesini/log4j-2-elastic","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivanocortesini%2Flog4j-2-elastic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivanocortesini%2Flog4j-2-elastic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivanocortesini%2Flog4j-2-elastic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivanocortesini%2Flog4j-2-elastic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivanocortesini","download_url":"https://codeload.github.com/ivanocortesini/log4j-2-elastic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237937810,"owners_count":19390543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appender","beats","elastic","elasticsearch","log","log4j","log4j2","logger","logging","logs","logstash"],"created_at":"2024-09-26T16:01:44.521Z","updated_at":"2025-10-24T08:31:12.283Z","avatar_url":"https://github.com/ivanocortesini.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# log4j-2-elastic\n[![build_status](https://travis-ci.org/ivanocortesini/log4j-2-elastic.svg?branch=master)](https://travis-ci.org/ivanocortesini/log4j-2-elastic)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.ivanocortesini/log4j-2-elastic/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.github.ivanocortesini/log4j-2-elastic)\n[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](CONTRIBUTING.md)\n\nI've created this project to share and enhance a [Log4j2](https://logging.apache.org/log4j/2.x/) appender that logs messages directly into an [Elasticsearch](https://www.elastic.co/products/elasticsearch) cluster.\nThis approach to log aggregation into Elasticsearch can be a good alternative to Elastic Beats in some specific scenario.\n\n## Features\nThis product includes some standard and some particular features:\n* Take advantage of Log4j2 asynchronous logging with bulk store requests. Batch mode is recognized automatically. A timeout based buffer flush can be configured.\n* Store @Logged annotated log message parameter objects as embedded JSON fields into the log document stored in Elasticsearch.\n* Mapped Diagnostic Context (MDC) support.\n* Connection to multiple cluster nodes.\n* Basic authentication.\n* Log4j \"ignoreExceptions\" option support.\n* Log4j \"includeLocation\" option support.\n* Log4j layout and filters support.\n\n## Getting Started\n\nIf you want to create your own local development environment you only need to import this code in your preferred IDE as a standard Maven project.\n\n### Prerequisites\n\n* git\n* Java 8+ JDK\n* Maven 3+\n* Elasticsearch 6.x\n* Your preferred IDE \n\n## Running the tests\n\nFor this project I've included a small set of JUnit 5 unit tests to verify the working status of main use cases.\nTo run those tests you need to update \"log4j2.xml\" and \"elasticsearch.properties\" configuration files to connect to an up and running Elasticsearch node or cluster.\n\n## Deployment\n\nThe simplest way to use this appender in a Java application is:\n* include this Maven dependency in your pom.xml file:\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.ivanocortesini\u003c/groupId\u003e\n    \u003cartifactId\u003elog4j-2-elastic\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n* include an \"Elastic\" type appender into your log4j2 configuration like in the following example:\n```\n\u003cElastic\n        name=\"example-elastic-appender\"\n        cluster=\"http://host-1:9200 http://host-2:9200\"\n        index=\"example-log-index\"\n        flushTimeOut=\"60\"\n        ignoreExceptions=\"true\"\n        username=\"usr\",\n        password=\"psw\"\n        \u003e\n    \u003cPatternLayout pattern=\"%m%n\"/\u003e\n\u003c/Elastic\u003e\n```\nAll needed dependencies are included (to Elasticsearch client, Log4J2 and Jackson). If you have to use another version of those dependencies (e.g. another Elasticsearch version) you can simply include an \"exclusions\" element for this artifact dependency declaration. \n\n### Configuration\nIn the table below are summarized available attribute parameters (all optional except \"name\"):\n\nParameter | Type | Default | Description\n---|---|---|---\nname | String | | Appender name\ncluster | String | http://localhost:9200 | Elasticsearch cluster nodes URL list (blank or comma separated) \nindex | String | Appender name | Elasticsearch destination index \nflushTimeOut | Integer | 0 | Timeout (in seconds) to flush the current log messages bulk transmission. Timeout refers to bulk messages creation time. The timer is active if value is greater than 0.  \nignoreExceptions | Boolean | true | If value is \"false\" the appender throw internal errors to the caller method. If value is \"true\" the appender try to use default logger to log internal error message.\nusername | String | | Username for basic authentication (if required)\npassword | String | | Password for basic authentication (if required)\n\nFor performance purposes I really recommend to use this appender with asynchronous loggers like in the example below:\n```\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\n\u003cConfiguration status=\"WARN\" name=\"example-configuration\" packages=\"\"\u003e\n    \u003cAppenders\u003e\n\n        \u003cElastic\n                name=\"example-elastic-appender\"\n                cluster=\"http://localhost:9200\"\n                index=\"example-log-index\"\n                flushTimeOut=\"60\"\n                ignoreExceptions=\"true\"\n                username=\"usr\"\n                password=\"psw\" \u003e\n            \u003cPatternLayout pattern=\"%m%n\"/\u003e\n        \u003c/Elastic\u003e\n\n        \u003cConsole name=\"STDOUT\" target=\"SYSTEM_OUT\"\u003e\n            \u003cPatternLayout pattern=\"%m%n\"/\u003e\n        \u003c/Console\u003e\n\n    \u003c/Appenders\u003e\n\n    \u003cLoggers\u003e\n        \u003cAsyncLogger name=\"example-logger\" level=\"info\" includeLocation=\"true\" additivity=\"false\"\u003e\n            \u003cAppenderRef ref=\"example-elastic-appender\"/\u003e\n        \u003c/AsyncLogger\u003e\n        \u003cRoot level=\"info\" includeLocation=\"true\"\u003e\n            \u003cAppenderRef ref=\"STDOUT\"/\u003e\n        \u003c/Root\u003e\n    \u003c/Loggers\u003e\n\u003c/Configuration\u003e\n```\n\n### @Logged \u0026 MDC\n@Logged annotation gives the possibility to add multiple objects data into a log message in the form of embedded fields inside the document stored in Elasticsearch. Below there's an example POJO (I've used [Lombok](https://github.com/rzwitserloot/lombok) here) annotated with @Logged.\n\n```\n@Data @AllArgsConstructor\n@Logged(\"data-example\")\npublic class DataExample {\n\n    private String title;\n    private String address;\n    private Date creationTime;\n    private List\u003cString\u003e phone;\n\n}\n```\nThis annotation requires to specify the name of the destination field for wich you should **avoid to use reserved field names**:\nmessage, level, logger, timestamp, thread, class, method, line, errorMessage, errorType, stack.\n\nActually dates are serialized in standard Elasticsearch [date_time](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html) format but you must include a document schema to save and manage them as real dates into Elasticsearch storage.\n\nMDC is another Log4J standard way to dynamically add fields to the log message and this appender support this feature. Fields value must be of Sring type in this case:\n```\n...\nThreadContext.put(\"property_1\", \"String value\");\nThreadContext.put(\"property_2\", new SimpleDateFormat(\"yyyy-MM-dd\").format(new Date()));\nlogger.info(\"Messegae with 2 additional properties...\");\n...\n```\nDestination log message document will contain 2 additional fields: 'property_1' and 'property_2'. MDC properties will be cleared automatically after log message process.\n \n### Usage details and considerations\nWhen you produce a log message a correspondent JSON document will be generated and stored into Elasticsearch. So the code below (Log4j2 support [lambda expressions](https://logging.apache.org/log4j/2.x/manual/api.html)):\n\n```\nlogger.info(\"Single message: {} {}\", \n        ()-\u003e\"param value\",\n        ()-\u003e2,\n        ()-\u003enew DataExample(\"title\", \"address\", LocalDateTime.now(), Arrays.asList(\"0039061111111\", \"0039062222222\"))\n);\n```\n\nwill produce a document like:\n\n```\n{\n    \"_index\":\"example-log-index\",\n    \"_type\":\"doc\",\n    \"_id\":\"nSQS3GgBLia8gni1U7cV\",\n    \"_score\":1.0,\n    \"_source\":{\n       \"message\":\"Single message: param value 2\",\n       \"level\":\"INFO\",\n       \"logger\":\"example-logger\",\n       \"timestamp\":\"2019-02-11T10:20:17.793Z\",\n       \"thread\":\"main\",\n       \"class\":\"tech.elasticlog.appender.ElasticAppenderTest\",\n       \"method\":\"appendSingleWithLambda\",\n       \"line\":75,\n       \"data-example\":{\n          \"title\":\"title\",\n          \"address\":\"address\",\n          \"creationTime\":\"2019-02-11T11:20:17.577+0100\",\n          \"phone\":[\n             \"0039061111111\",\n             \"0039062222222\"\n          ]\n       }\n    }\n}\n```\nIf you log an exception, other fields will be added:\n\n```\n...\n\"errorMessage\" : \"\u003cexception localized error message\u003e\",\n\"errorType\" : \"\u003cexception class full name\u003e\",\n\"stack\" : [{\n               \"class\":\"\u003cstacktrace element class name\u003e\",\n               \"method\":\"\u003cstacktrace element method name\u003e\",\n               \"line\": \u003cstacktrace element line number\u003e\n            }],\n...\n```\nStack, class, method and line fields will be present only when logger \"includeLocation\" attribute is set to true. Take into account that location inclusion consumes time and resources.\n\nDepending on Log4j configuration, generated documents will be stored immediately (with synchronous loggers) or in bulk (with asynchronous loggers). This appender autodetect if the logger require the usage of single message or bulk transmission. It's important to **avoid to use the same \"Elastic\" appender instance for mixed asynchronous and synchronous loggers**.\n\nIf you want to be 100% sure to avoid log messages loss you can use synchronous logger and set \"ignoreExceptions\" to false.\nHowever this approach can bring performance issues. Even using asynchronous loggers, if an Elasticsearch cluster of 3+ nodes is available you should have enough guarantees and in any case you can add some other appender to provide an information backup.\n\nThe Elasticsearch destination **index will be created automatically** (if not present) during appender startup phase. Any change you wanto to make in this sense requires to update Log4j configuration and restart your application.\n\nThe bulk documents buffer flush timeout is checked every 5 seconds. So if you set a timeout of 30 seconds you can expect that the document buffer will be flushed in less than 35 seconds by the logger (that can close the log batch) or by the timeout manager (after 30 seconds from batch creation).\nLog4j2 has an algorithm to dynamically define the batch size based on log activities frequency and usually it close the batch in few seconds. However you can specify buffer size by setting speficic [system parameters](). \"flushTimeOut\" parameter have no effect if the appender is included in a synchronous logger.    \n\n## Built With\n\n[Maven](https://maven.apache.org/) - Dependency Management\n\n## Contributing\n\nThis is a tiny product with a simple and clean design and minimum external dependencies. The main components are:\n* A standard Log4j2 appender implementation.\n* An Elasticsearch client component based on Elasticsearch high level Java client.\n* A log message content translator enhanced with parameters JSON encoding with Jackson. \n  \nAny kind of contribution will be appreciated. You can contact me by email or open issues or pull requests for bugs, enhancements and suggestions about any aspect of this product (e.g. architectural considerations, code design, performance, etc...).  \n\nI've included a standard [CONTRIBUTING](CONTRIBUTING.md) file for details on code of conduct, and process for submitting pull requests.\n\n## Versioning\n\nI use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/ivanocortesini/log4j-2-elastic/tags). \n\n## Authors\n\n* **IvanoCortesini** ([Github](https://github.com/ivanocortesini) - [LinkedIn](https://www.linkedin.com/in/ivanocortesini/))\n\nSee also the list of [contributors](https://github.com/ivanocortesini/log4j-2-elastic/contributors) who participated in this project.\n\n## License\n\nThis project is licensed under the Apache License 2.0 License - see the [LICENSE](LICENSE) file for details\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivanocortesini%2Flog4j-2-elastic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivanocortesini%2Flog4j-2-elastic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivanocortesini%2Flog4j-2-elastic/lists"}