{"id":23118547,"url":"https://github.com/urielha/log4stash","last_synced_at":"2025-04-09T11:07:00.989Z","repository":{"id":7061134,"uuid":"56092822","full_name":"urielha/log4stash","owner":"urielha","description":"Module to Log log4net Messages to ElasticSearch","archived":false,"fork":false,"pushed_at":"2024-11-07T08:24:58.000Z","size":24887,"stargazers_count":60,"open_issues_count":25,"forks_count":44,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-02T10:08:44.952Z","etag":null,"topics":["appender","elasticsearch","filters","grok","log","log4net","logging","logstash","regex"],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/urielha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-04-12T19:46:04.000Z","updated_at":"2024-11-07T08:24:55.000Z","dependencies_parsed_at":"2025-01-28T13:14:25.868Z","dependency_job_id":"7c16d955-1718-4ff7-b9b2-0bf09f666951","html_url":"https://github.com/urielha/log4stash","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urielha%2Flog4stash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urielha%2Flog4stash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urielha%2Flog4stash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urielha%2Flog4stash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/urielha","download_url":"https://codeload.github.com/urielha/log4stash/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027407,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appender","elasticsearch","filters","grok","log","log4net","logging","logstash","regex"],"created_at":"2024-12-17T05:18:15.190Z","updated_at":"2025-04-09T11:07:00.963Z","avatar_url":"https://github.com/urielha.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"log4stash\n=====================\n\n\u003e **NOTE:** This package is no longer maintained, I will be happy to get help with maintaing it.\n\n[![Build status][AzureDevOpsImg]][azure-devops]\n\nlog4stash is a [log4net](http://logging.apache.org/log4net/) appender to log messages to the [ElasticSearch](http://www.elasticsearch.org) document database. ElasticSearch offers robust full-text search engine and analyzation so that errors and messages can be indexed quickly and searched easily.\n\nlog4stash provides a few logging filters similar to the filters on [logstash](http://logstash.net).\n\nThe origin of log4stash is [@jptoto](https://github.com/jptoto)'s [log4net.ElasticSearch](https://github.com/jptoto/log4net.ElasticSearch) repository.\n\nlog4stash is based on RestSharp and Newtonsoft.Json but uses ILRepack to avoid nuget dependencies.\n\n**log4stash is fully open source, MIT licensed.**\n\n### Features:\n* Supports .NET 4.5.2+ and .NET Core 2.0+\n* Easy installation and setup via [Nuget](https://nuget.org/packages/log4stash/)\n* Ability to analyze the log event before sending it to ElasticSearch using built-in filters and custom filters similar to [logstash](http://logstash.net/docs/1.4.2/).\n\n### Breaking Changes:\nNavigate to breaking changes page [here](https://github.com/urielha/log4stash/blob/master/docs/breaking.md). See also [Version notes](https://github.com/urielha/log4stash/blob/master/docs/version_notes.md) page.\n\n### Filters:\n* [**Add**][docs-filters-add] - add new key and value to the event.\n* [**Remove**][docs-filters-remove] - remove key from the event.\n* [**Rename**][docs-filters-rename] - rename key to another name.\n* **Kv** - analyze value (default is to analyze the 'Message' value) and export key-value pairs using regex (similar to logstash's kv filter).\n* **Grok** - analyze value (default is 'Message') using custom regex and saved patterns (similar to logstash's grok filter).\n* **ConvertToArray** - split raw string to an array by given seperators. \n* **Json** - convert json string to an object (so it will be parsed as object in elasticsearch).\n* **Convert** - Available convertors: `ToString`, `ToLower`, `ToUpper`, `ToInt` and `ToArray`. See [config example][config-example] for more information. \n* **Xml** - Parse xml into an object.\n\n#### Custom filter:\nTo add your own filters you just need to implement the interface IElasticAppenderFilter on your assembly and configure it on the log4net configuration file.\n\n\u003c!-- ### Usage:\nPlease see the [DOCUMENTATION](https://github.com/urielha/log4stash/wiki/0-Documentation) Wiki page to begin logging errors to ElasticSearch! --\u003e\n\n### Issues:\nI do my best to reply to issues or questions ASAP. Please use the [ISSUES](https://github.com/urielha/log4stash/issues) page to submit questions or errors.\n\n### Configuration Examples:\n\nAlmost all the parameters are optional, to see the default values check the [c'tor](https://github.com/urielha/log4stash/blob/master/src/log4stash/ElasticSearchAppender.cs#L86) of the appender and the c'tor of every filter. \nYou can also set any public property in the appender/filter which didn't appear in the example.\n\n##### Simple configuration:\n```xml\n\u003cappender name=\"ElasticSearchAppender\" type=\"log4stash.ElasticSearchAppender, log4stash\"\u003e\n    \u003cServer\u003elocalhost\u003c/Server\u003e\n    \u003cPort\u003e9200\u003c/Port\u003e\n    \u003cElasticFilters\u003e\n      \u003c!-- example of using filter with default parameters --\u003e\n      \u003ckv /\u003e \n    \u003c/ElasticFilters\u003e\n\u003c/appender\u003e\n```\n\n##### (Almost) Full configuration:\n```xml\n\u003cappender name=\"ElasticSearchAppender\" type=\"log4stash.ElasticSearchAppender, log4stash\"\u003e\n\t\u003cServer\u003elocalhost\u003c/Server\u003e\n\t\u003cPort\u003e9200\u003c/Port\u003e\n\t\u003c!-- optional: in case elasticsearch is located behind a reverse proxy the URL is like http://Server:Port/Path, default = empty string --\u003e\n\t\u003cPath\u003e/es5\u003c/Path\u003e\n\t\u003c!-- The time zone for the formatter is based on the character before the index. '+' = local time, '~' = utc time --\u003e\n\t\u003cIndexName\u003elog_test_%{+yyyy-MM-dd}\u003c/IndexName\u003e\n\t\u003c!-- type support was removed in ElasticSearch 7, so if not defined in configuration there won't be a type in the request --\u003e\n\t\u003cIndexType\u003eLogEvent\u003c/IndexType\u003e\n\t\u003cBulkSize\u003e2000\u003c/BulkSize\u003e\n\t\u003cBulkIdleTimeout\u003e10000\u003c/BulkIdleTimeout\u003e\n\t\u003cIndexAsync\u003eFalse\u003c/IndexAsync\u003e\n\t\u003cDropEventsOverBulkLimit\u003eFalse\u003c/DropEventsOverBulkLimit\u003e\n\n\t\u003c!-- Serialize log object as json (default is true).\n      -- This in case you log the object this way: `logger.Debug(obj);` and not: `logger.Debug(\"string\");` --\u003e\n\t\u003cSerializeObjects\u003eTrue\u003c/SerializeObjects\u003e \n\n\t\u003c!-- optional: elasticsearch timeout for the request, default = 10000 --\u003e\n\t\u003cElasticSearchTimeout\u003e10000\u003c/ElasticSearchTimeout\u003e\n\n\t\u003c!-- optional: ssl connection --\u003e\n\t\u003cSsl\u003eFalse\u003c/Ssl\u003e\n\t\u003cAllowSelfSignedServerCert\u003eFalse\u003c/AllowSelfSignedServerCert\u003e\n\n\t\u003c!--You can add parameters to the request to control the parameters sent to ElasticSearch.\n    for example, as you can see here, you can add a custom id source to the appender.\n    The Key is the key to be added to the request, and the value is the parameter's name in the log event properties.--\u003e\n\t\u003cIndexOperationParams\u003e\n\t\t\u003cParameter\u003e\n\t\t\t\u003cKey\u003e_id\u003c/Key\u003e\n\t\t\t\u003cValue\u003e%{IdSource}\u003c/Value\u003e\n\t\t\u003c/Parameter\u003e\n\t\t\u003cParameter\u003e\n\t\t\t\u003cKey\u003ekey\u003c/Key\u003e\n\t\t\t\u003cValue\u003evalue\u003c/Value\u003e\n\t\t\u003c/Parameter\u003e\n\t\u003c/IndexOperationParams\u003e\n\n\t\u003c!-- for more information read about log4net.Core.FixFlags --\u003e\n\t\u003cFixedFields\u003ePartial\u003c/FixedFields\u003e\n\n\t\u003cTemplate\u003e\n\t\t\u003cName\u003etemplateName\u003c/Name\u003e\n\t\t\u003cFileName\u003epath2template.json\u003c/FileName\u003e\n\t\u003c/Template\u003e\n\n\t\u003c!--Only one credential type can be used at once--\u003e\n\t\u003c!--Here we list all possible types--\u003e\n\t\u003cAuthenticationMethod\u003e\n\t\t\u003c!--For basic authentication purposes--\u003e\n\t\t\u003cBasic\u003e\n\t\t\t\u003cUsername\u003eUsername\u003c/Username\u003e\n\t\t\t\u003cPassword\u003ePassword\u003c/Password\u003e\n\t\t\u003c/Basic\u003e\n\t\t\u003c!--For AWS ElasticSearch service--\u003e\n\t\t\u003cAws\u003e\n\t\t\t\u003cAws4SignerSecretKey\u003eSecret\u003c/Aws4SignerSecretKey\u003e\n\t\t\t\u003cAws4SignerAccessKey\u003eAccessKey\u003c/Aws4SignerAccessKey\u003e\n\t\t\t\u003cAws4SignerRegion\u003eRegion\u003c/Aws4SignerRegion\u003e\n\t\t\u003c/Aws\u003e\n\t\t\u003c!-- For Api Key (X-Pack) authentication --\u003e\n\t\t\u003cApiKey\u003e\n\t\t\t\u003c!-- ApiKeyBase64 takes precedence over Id/ApiKey  --\u003e\n\t\t\t\u003cApiKeyBase64\u003eaWQ6YXBpa2V5\u003c/ApiKey\u003e\n\t\t\t\u003c!-- Or --\u003e\n\t\t\t\u003cId\u003eid\u003c/Id\u003e\n\t\t\t\u003cApiKey\u003eapikey\u003c/ApiKey\u003e\n\t\t\u003c/ApiKey\u003e\n\t\u003c/AuthenticationMethod\u003e\n\n\t\u003c!-- all filters goes in ElasticFilters tag --\u003e\n\t\u003cElasticFilters\u003e\n\t\t\u003cAdd\u003e\n\t\t\t\u003cKey\u003e@type\u003c/Key\u003e\n\t\t\t\u003cValue\u003eSpecial\u003c/Value\u003e\n\t\t\u003c/Add\u003e\n\n\t\t\u003c!-- using the @type value from the previous filter --\u003e\n\t\t\u003cAdd\u003e\n\t\t\t\u003cKey\u003eSmartValue\u003c/Key\u003e\n\t\t\t\u003cValue\u003ethe type is %{@type}\u003c/Value\u003e\n\t\t\u003c/Add\u003e\n\n\t\t\u003cRemove\u003e\n\t\t\t\u003cKey\u003e@type\u003c/Key\u003e\n\t\t\u003c/Remove\u003e\n\n\t\t\u003c!-- you can load custom filters like I do here --\u003e\n\t\t\u003cFilter type=\"log4stash.Filters.RenameKeyFilter, log4stash\"\u003e\n\t\t\t\u003cKey\u003eSmartValue\u003c/Key\u003e\n\t\t\t\u003cRenameTo\u003eSmartValue2\u003c/RenameTo\u003e\n\t\t\u003c/Filter\u003e\n\n\t\t\u003c!-- converts a json object to fields in the document --\u003e\n\t\t\u003cJson\u003e\n\t\t\t\u003cSourceKey\u003eJsonRaw\u003c/SourceKey\u003e\n\t\t\t\u003cFlattenJson\u003efalse\u003c/FlattenJson\u003e\n\t\t\t\u003c!-- the separator property is only relevant when setting the FlattenJson property to 'true' --\u003e\n\t\t\t\u003cSeparator\u003e_\u003c/Separator\u003e \n\t\t\u003c/Json\u003e\n\n\t\t\u003c!-- converts an xml object to fields in the document --\u003e\n\t\t\u003cXml\u003e\n\t\t\t\u003cSourceKey\u003eXmlRaw\u003c/SourceKey\u003e\n\t\t\t\u003cFlattenXml\u003efalse\u003c/FlattenXml\u003e\n\t\t\u003c/Xml\u003e\n\n\t\t\u003c!-- kv and grok filters similar to logstash's filters --\u003e\n\t\t\u003cKv\u003e\n\t\t\t\u003cSourceKey\u003eMessage\u003c/SourceKey\u003e\n\t\t\t\u003cValueSplit\u003e:=\u003c/ValueSplit\u003e\n\t\t\t\u003cFieldSplit\u003e ,\u003c/FieldSplit\u003e\n\t\t\u003c/kv\u003e\n\n\t\t\u003cGrok\u003e\n\t\t\t\u003cSourceKey\u003eMessage\u003c/SourceKey\u003e\n\t\t\t\u003cPattern\u003ethe message is %{WORD:Message} and guid %{UUID:the_guid}\u003c/Pattern\u003e\n\t\t\t\u003cOverwrite\u003etrue\u003c/Overwrite\u003e\n\t\t\u003c/Grok\u003e\n\n\t\t\u003c!-- Convert string like: \"1,2, 45 9\" into array of numbers [1,2,45,9] --\u003e\n\t\t\u003cConvertToArray\u003e\n\t\t\t\u003cSourceKey\u003esomeIds\u003c/SourceKey\u003e\n\t\t\t\u003c!-- The separators (space and comma) --\u003e\n\t\t\t\u003cSeperators\u003e, \u003c/Seperators\u003e \n\t\t\u003c/ConvertToArray\u003e\n\n\t\t\u003cConvert\u003e\n\t\t\t\u003c!-- convert given key to string --\u003e\n\t\t\t\u003cToString\u003eshouldBeString\u003c/ToString\u003e\n\n\t\t\t\u003c!-- same as ConvertToArray. Just for convenience --\u003e\n\t\t\t\u003cToArray\u003e\n\t\t\t\t\u003cSourceKey\u003eanotherIds\u003c/SourceKey\u003e\n\t\t\t\u003c/ToArray\u003e\n\t\t\u003c/Convert\u003e\n\t\u003c/ElasticFilters\u003e\n\u003c/appender\u003e\n```\n\nNote that the filters got called by the order they appeared in the config (as shown in the example).\n\n### Templates:\nTo get to know the [ElasticSearch templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) follow the link.\n\nSample template could be found in: [log-index-spec.json](https://github.com/urielha/log4stash/blob/master/scripts/log-index-spec.json). And more complex template with dynamic mappings can be found in the tests template: [template.json](https://github.com/urielha/log4stash/blob/master/src/log4stash.Tests/template.json)\n\nYou can follow the link to read more about [dynamic mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html).\n\n### License:\n[MIT License](https://github.com/urielha/log4stash/blob/master/LICENSE)\n\n### Thanks:\n\nThanks to [@eran-gil](https://github.com/eran-gil) for helping me updating this package to support newer versions of ES, creating continous deployment and fixing issues.\n\nThanks to [@jptoto](https://github.com/jptoto) for the idea and the first working ElasticAppender.\nMany thanks to [@mpdreamz](https://github.com/Mpdreamz) and the team for their great work on the NEST library!\nThe inspiration to the filters and style had taken from [elasticsearch/logstash](https://github.com/elasticsearch/logstash) project.\n\n### Build status:\n\nThe CI is running on Azure DevOps and tested against ElasticSearch 5,6,7 every time.\nSupport for lower ElasticSearch versions is no longer maintained.\n\n[AzureDevOpsImg]:https://erangil.visualstudio.com/log4stash/_apis/build/status/log4stash%20CI?branchName=master\n\n[azure-devops]:https://azure.microsoft.com/en-us/services/devops/\n\n[config-example]:https://github.com/urielha/log4stash#almost-full-configuration\n[filters-section]:https://github.com/urielha/log4stash#filters\n\n[docs-filters-add]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Add.md\n[docs-filters-remove]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Remove.md\n[docs-filters-rename]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Rename.md\n\n[erangil2]:https://github.com/erangil2\n[ignasv]:https://github.com/ignasv\n[mfpalladino]:https://github.com/mfpalladino\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furielha%2Flog4stash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furielha%2Flog4stash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furielha%2Flog4stash/lists"}