Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/urielha/log4stash
Module to Log log4net Messages to ElasticSearch
https://github.com/urielha/log4stash
appender elasticsearch filters grok log log4net logging logstash regex
Last synced: 4 days ago
JSON representation
Module to Log log4net Messages to ElasticSearch
- Host: GitHub
- URL: https://github.com/urielha/log4stash
- Owner: urielha
- License: mit
- Created: 2016-04-12T19:46:04.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2024-11-07T08:24:58.000Z (3 months ago)
- Last Synced: 2025-01-28T13:15:39.572Z (11 days ago)
- Topics: appender, elasticsearch, filters, grok, log, log4net, logging, logstash, regex
- Language: C#
- Size: 23.7 MB
- Stars: 60
- Watchers: 8
- Forks: 44
- Open Issues: 25
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
log4stash
=====================
> **NOTE:** This package is no longer maintained, I will be happy to get help with maintaing it.
[![Build status][AzureDevOpsImg]][azure-devops]
log4stash is a [log4net](http://logging.apache.org/log4net/) appender to log messages to the [ElasticSearch](http://www.elasticsearch.org) document database. ElasticSearch offers robust full-text search engine and analyzation so that errors and messages can be indexed quickly and searched easily.
log4stash provides a few logging filters similar to the filters on [logstash](http://logstash.net).
The origin of log4stash is [@jptoto](https://github.com/jptoto)'s [log4net.ElasticSearch](https://github.com/jptoto/log4net.ElasticSearch) repository.
log4stash is based on RestSharp and Newtonsoft.Json but uses ILRepack to avoid nuget dependencies.
**log4stash is fully open source, MIT licensed.**
### Features:
* Supports .NET 4.5.2+ and .NET Core 2.0+
* Easy installation and setup via [Nuget](https://nuget.org/packages/log4stash/)
* Ability to analyze the log event before sending it to ElasticSearch using built-in filters and custom filters similar to [logstash](http://logstash.net/docs/1.4.2/).
### Breaking Changes:
Navigate to breaking changes page [here](https://github.com/urielha/log4stash/blob/master/docs/breaking.md). See also [Version notes](https://github.com/urielha/log4stash/blob/master/docs/version_notes.md) page.
### Filters:
* [**Add**][docs-filters-add] - add new key and value to the event.
* [**Remove**][docs-filters-remove] - remove key from the event.
* [**Rename**][docs-filters-rename] - rename key to another name.
* **Kv** - analyze value (default is to analyze the 'Message' value) and export key-value pairs using regex (similar to logstash's kv filter).
* **Grok** - analyze value (default is 'Message') using custom regex and saved patterns (similar to logstash's grok filter).
* **ConvertToArray** - split raw string to an array by given seperators.
* **Json** - convert json string to an object (so it will be parsed as object in elasticsearch).
* **Convert** - Available convertors: `ToString`, `ToLower`, `ToUpper`, `ToInt` and `ToArray`. See [config example][config-example] for more information.
* **Xml** - Parse xml into an object.
#### Custom filter:
To add your own filters you just need to implement the interface IElasticAppenderFilter on your assembly and configure it on the log4net configuration file.
### Issues:
I do my best to reply to issues or questions ASAP. Please use the [ISSUES](https://github.com/urielha/log4stash/issues) page to submit questions or errors.
### Configuration Examples:
Almost all the parameters are optional, to see the default values check the [c'tor](https://github.com/urielha/log4stash/blob/master/src/log4stash/ElasticSearchAppender.cs#L86) of the appender and the c'tor of every filter.
You can also set any public property in the appender/filter which didn't appear in the example.
##### Simple configuration:
```xml
localhost
9200
```
##### (Almost) Full configuration:
```xml
localhost
9200
/es5
log_test_%{+yyyy-MM-dd}
LogEvent
2000
10000
False
False
True
10000
False
False
_id
%{IdSource}
key
value
Partial
templateName
path2template.json
Username
Password
Secret
AccessKey
Region
aWQ6YXBpa2V5
id
apikey
@type
Special
SmartValue
the type is %{@type}
@type
SmartValue
SmartValue2
JsonRaw
false
_
XmlRaw
false
Message
:=
,
Message
the message is %{WORD:Message} and guid %{UUID:the_guid}
true
someIds
,
shouldBeString
anotherIds
```
Note that the filters got called by the order they appeared in the config (as shown in the example).
### Templates:
To get to know the [ElasticSearch templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) follow the link.
Sample template could be found in: [log-index-spec.json](https://github.com/urielha/log4stash/blob/master/scripts/log-index-spec.json). And more complex template with dynamic mappings can be found in the tests template: [template.json](https://github.com/urielha/log4stash/blob/master/src/log4stash.Tests/template.json)
You can follow the link to read more about [dynamic mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html).
### License:
[MIT License](https://github.com/urielha/log4stash/blob/master/LICENSE)
### Thanks:
Thanks to [@eran-gil](https://github.com/eran-gil) for helping me updating this package to support newer versions of ES, creating continous deployment and fixing issues.
Thanks to [@jptoto](https://github.com/jptoto) for the idea and the first working ElasticAppender.
Many thanks to [@mpdreamz](https://github.com/Mpdreamz) and the team for their great work on the NEST library!
The inspiration to the filters and style had taken from [elasticsearch/logstash](https://github.com/elasticsearch/logstash) project.
### Build status:
The CI is running on Azure DevOps and tested against ElasticSearch 5,6,7 every time.
Support for lower ElasticSearch versions is no longer maintained.
[AzureDevOpsImg]:https://erangil.visualstudio.com/log4stash/_apis/build/status/log4stash%20CI?branchName=master
[azure-devops]:https://azure.microsoft.com/en-us/services/devops/
[config-example]:https://github.com/urielha/log4stash#almost-full-configuration
[filters-section]:https://github.com/urielha/log4stash#filters
[docs-filters-add]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Add.md
[docs-filters-remove]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Remove.md
[docs-filters-rename]:https://github.com/urielha/log4stash/blob/master/docs/Filters/Rename.md
[erangil2]:https://github.com/erangil2
[ignasv]:https://github.com/ignasv
[mfpalladino]:https://github.com/mfpalladino