Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/serilog-contrib/serilog-sinks-elasticsearch
A Serilog sink that writes events to Elasticsearch
https://github.com/serilog-contrib/serilog-sinks-elasticsearch
elasticsearch looking-for-maintainer serilog serilog-sink
Last synced: about 1 month ago
JSON representation
A Serilog sink that writes events to Elasticsearch
- Host: GitHub
- URL: https://github.com/serilog-contrib/serilog-sinks-elasticsearch
- Owner: serilog-contrib
- License: apache-2.0
- Archived: true
- Created: 2015-02-22T22:47:26.000Z (almost 10 years ago)
- Default Branch: dev
- Last Pushed: 2024-06-01T10:15:59.000Z (7 months ago)
- Last Synced: 2024-11-08T05:46:55.488Z (about 1 month ago)
- Topics: elasticsearch, looking-for-maintainer, serilog, serilog-sink
- Language: C#
- Homepage:
- Size: 644 KB
- Stars: 434
- Watchers: 20
- Forks: 196
- Open Issues: 96
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.md
- License: LICENSE
Awesome Lists containing this project
README
> TLDR: This Sink is archived, you can use the [Elastic.Serilog.Sink](https://www.nuget.org/packages/Elastic.Serilog.Sinks) instead. This one is supported by Elastic themselves.
Longer version: This sink is archived as it became too hard to keep it maintained and up to date with the new version of Elastic. I have been [searching](https://github.com/serilog-contrib/serilog-sinks-elasticsearch/issues/486) for a maintainer for some time, but unfortunately, there was nobody who could take over the responsibilities and give it more attention than I could give it.
Luckily, Elastic has a very good .net team and they not only provided PRs and support to this sink, they also worked on their own implementation.
Combined with the current state of this sink (there is some legacy code, and some improvements to do as well) and the fact there is a great and supported alternative, I decided to [archive](https://github.com/serilog-contrib/serilog-sinks-elasticsearch/issues/583) this one.
Feel free to fork and improve, or use the existing versions, but there won't be any updates.
I want to thank all the contributors and supporters for all the work during all those years. It has served the community well since it was first created in 2014, and has over 74M downloads on NuGet, so I'm glad to see that it was successful and helpful to others.
---
# Serilog.Sinks.Elasticsearch [](https://github.com/serilog-contrib/serilog-sinks-elasticsearch/actions/workflows/cicd.yaml) [](https://www.nuget.org/packages/Serilog.Sinks.Elasticsearch)
This repository contains two nuget packages: `Serilog.Sinks.Elasticsearch` and `Serilog.Formatting.Elasticsearch`.
> __Just a heads up that the .NET team @elastic have created their own new Serilog Sink called Elastic.Serilog.Sinks (Package: https://www.nuget.org/packages/Elastic.Serilog.Sinks#readme-body-tab and documentation: https://www.elastic.co/guide/en/ecs-logging/dotnet/current/serilog-data-shipper.html). Although this current sink will still work, I advise you to have a look first at the official Elastic implementation as it is better supported and more up to date.__
## Table of contents
* [What is this sink](#what-is-this-sink)
* [Features](#features)
* [Quick start](#quick-start)
* [Elasticsearch sinks](#elasticsearch-sinks)
* [Elasticsearch formatters](#elasticsearch-formatters)
* [More information](#more-information)
* [A note about fields inside Elasticsearch](#a-note-about-fields-inside-elasticsearch)
* [A note about Kibana](#a-note-about-kibana)
* [JSON `appsettings.json` configuration](#json-appsettingsjson-configuration)
* [Handling errors](#handling-errors)
* [Breaking changes](#breaking-changes)
## What is this sink
The Serilog Elasticsearch sink project is a sink (basically a writer) for the Serilog logging framework. Structured log events are written to sinks and each sink is responsible for writing it to its own backend, database, store etc. This sink delivers the data to Elasticsearch, a NoSQL search engine. It does this in a similar structure as Logstash and makes it easy to use Kibana for visualizing your logs.
## Features
* Simple configuration to get log events published to Elasticsearch. Only server address is needed.
* All properties are stored inside fields in ES. This allows you to query on all the relevant data but also run analytics over this data.
* Be able to customize the store; specify the index name being used, the serializer or the connections to the server (load balanced).
* Durable mode; store the logevents first on disk before delivering them to ES making sure you never miss events if you have trouble connecting to your ES cluster.
* Automatically create the right mappings for the best usage of the log events in ES or automatically upload your own custom mapping.
* Starting from version 3, compatible with Elasticsearch 2.
* Version 6.x supports the new Elasticsearch.net version 6.x library.
* From version 8.x there is support for Elasticsearch.net version 7.
* From version 9.x there is support for Elasticsearch.net version 8. Version detection is enabled by default, in which case `TypeName` is handled automatically across major versions 6, 7 and 8. Versions 2 and 5 of Elasticsearch are no longer supported. Version 9.0.0 of the sink targets netstandard2.0 and therefore can be run on any .NET Framework that supports it (both .NET Core and .NET Framework), however, we are focused on testing it with .NET 6.0 to make the maintenance simpler.
## Quick start
### Elasticsearch sinks
```powershell
Install-Package serilog.sinks.elasticsearch
```
Simplest way to register this sink is to use default configuration:
```csharp
var loggerConfig = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200")));
```
Or, if using .NET Core and `Serilog.Settings.Configuration` Nuget package and `appsettings.json`, default configuration would look like this:
```json
{
"Serilog": {
"Using": [ "Serilog.Sinks.Elasticsearch" ],
"MinimumLevel": "Warning",
"WriteTo": [
{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9200"
}
}
]
}
}
```
More elaborate configuration, using additional Nuget packages (e.g. `Serilog.Enrichers.Environment`) would look like:
```json
{
"Serilog": {
"Using": [ "Serilog.Sinks.Elasticsearch" ],
"MinimumLevel": "Warning",
"WriteTo": [
{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9200"
}
}
],
"Enrich": [ "FromLogContext", "WithMachineName" ],
"Properties": {
"Application": "My app"
}
}
}
```
This way the sink will detect version of Elasticsearch server (`DetectElasticsearchVersion` is set to `true` by default) and handle `TypeName` behavior correctly, based on the server version (6.x, 7.x or 8.x).
### Disable detection of Elasticsearch server version
Alternatively, `DetectElasticsearchVersion` can be set to `false` and certain option can be configured manually. In that case, the sink will assume version 7 of Elasticsearch, but options will be ignored due to a potential version incompatibility.
For example, you can configure the sink to force registeration of v6 index template. Be aware that the AutoRegisterTemplate option will not overwrite an existing template.
```csharp
var loggerConfig = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200") ){
DetectElasticsearchVersion = false,
AutoRegisterTemplate = true,
AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6
});
```
### Configurable properties
Besides a registration of the sink in the code, it is possible to register it using appSettings reader (from v2.0.42+) reader (from v2.0.42+) as shown below.
This example shows the options that are currently available when using the appSettings reader.
```xml
```
With the appSettings configuration the `nodeUris` property is required. Multiple nodes can be specified using `,` or `;` to separate them. All other properties are optional. Also required is the `` setting to include this sink. All other properties are optional. If you do not explicitly specify an indexFormat-setting, a generic index such as 'logstash-[current_date]' will be used automatically.
And start writing your events using Serilog.
### Elasticsearch formatters
```powershell
Install-Package serilog.formatting.elasticsearch
```
The `Serilog.Formatting.Elasticsearch` nuget package consists of a several formatters:
* `ElasticsearchJsonFormatter` - custom json formatter that respects the configured property name handling and forces `Timestamp` to @timestamp.
* `ExceptionAsObjectJsonFormatter` - a json formatter which serializes any exception into an exception object.
Override default formatter if it's possible with selected sink
```csharp
var loggerConfig = new LoggerConfiguration()
.WriteTo.Console(new ElasticsearchJsonFormatter());
```
## More information
* [Basic information](https://github.com/serilog/serilog-sinks-elasticsearch/wiki/basic-setup) on how to configure and use this sink.
* [Configuration options](https://github.com/serilog/serilog-sinks-elasticsearch/wiki/Configure-the-sink) which you can use.
* How to use the [durability](https://github.com/serilog/serilog-sinks-elasticsearch/wiki/durability) mode.
* Get the [NuGet package](http://www.nuget.org/packages/Serilog.Sinks.Elasticsearch).
* Report issues to the [issue tracker](https://github.com/serilog/serilog-sinks-elasticsearch/issues). PR welcome, but please do this against the dev branch.
* For an overview of recent changes, have a look at the [change log](https://github.com/serilog/serilog-sinks-elasticsearch/blob/master/CHANGES.md).
### A note about fields inside Elasticsearch
Be aware that there is an explicit and implicit mapping of types inside an Elasticsearch index. A value called `X` as a string will be indexed as being a string. Sending the same `X` as an integer in a next log message will not work. ES will raise a mapping exception, however it is not that evident that your log item was not stored due to the bulk actions performed.
So be careful about defining and using your fields (and type of fields). It is easy to miss that you first send a {User} as a simple username (string) and next as a User object. The first mapping dynamically created in the index wins. See also issue [#184](https://github.com/serilog/serilog-sinks-elasticsearch/issues/184) for details and a possible solution. There are also limits in ES on the number of dynamic fields you can actually throw inside an index.
### A note about Kibana
In order to avoid a potentially deeply nested JSON structure for exceptions with inner exceptions,
by default the logged exception and it's inner exception is logged as an array of exceptions in the field `exceptions`. Use the 'Depth' field to traverse the inner exceptions flow.
However, not all features in Kibana work just as well with JSON arrays - for instance, including
exception fields on dashboards and visualizations. Therefore, we provide an alternative formatter, `ExceptionAsObjectJsonFormatter`, which will serialize the exception into the `exception` field as an object with nested `InnerException` properties. This was also the default behavior of the sink before version 2.
To use it, simply specify it as the `CustomFormatter` when creating the sink:
```csharp
new ElasticsearchSink(new ElasticsearchSinkOptions(url)
{
CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage:true)
});
```
### JSON `appsettings.json` configuration
To use the Elasticsearch sink with _Microsoft.Extensions.Configuration_, for example with ASP.NET Core or .NET Core, use the [Serilog.Settings.Configuration](https://github.com/serilog/serilog-settings-configuration) package. First install that package if you have not already done so:
```powershell
Install-Package Serilog.Settings.Configuration
```
Instead of configuring the sink directly in code, call `ReadFrom.Configuration()`:
```csharp
var configuration = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json")
.Build();
var logger = new LoggerConfiguration()
.ReadFrom.Configuration(configuration)
.CreateLogger();
```
In your `appsettings.json` file, under the `Serilog` node, :
```json
{
"Serilog": {
"WriteTo": [{
"Name": "Elasticsearch",
"Args": {
"nodeUris": "http://localhost:9200;http://remotehost:9200/",
"indexFormat": "custom-index-{0:yyyy.MM}",
"templateName": "myCustomTemplate",
"typeName": "myCustomLogEventType",
"pipelineName": "myCustomPipelineName",
"batchPostingLimit": 50,
"batchAction": "Create",
"period": 2,
"inlineFields": true,
"restrictedToMinimumLevel": "Warning",
"bufferBaseFilename": "C:/Temp/docker-elk-serilog-web-buffer",
"bufferFileSizeLimitBytes": 5242880,
"bufferLogShippingInterval": 5000,
"bufferRetainedInvalidPayloadsLimitBytes": 5000,
"bufferFileCountLimit": 31,
"connectionGlobalHeaders" :"Authorization=Bearer SOME-TOKEN;OtherHeader=OTHER-HEADER-VALUE",
"connectionTimeout": 5,
"emitEventFailure": "WriteToSelfLog",
"queueSizeLimit": "100000",
"autoRegisterTemplate": true,
"autoRegisterTemplateVersion": "ESv2",
"overwriteTemplate": false,
"registerTemplateFailure": "IndexAnyway",
"deadLetterIndexName": "deadletter-{0:yyyy.MM}",
"numberOfShards": 20,
"numberOfReplicas": 10,
"templateCustomSettings": [{ "index.mapping.total_fields.limit": "10000000" } ],
"formatProvider": "My.Namespace.MyFormatProvider, My.Assembly.Name",
"connection": "My.Namespace.MyConnection, My.Assembly.Name",
"serializer": "My.Namespace.MySerializer, My.Assembly.Name",
"connectionPool": "My.Namespace.MyConnectionPool, My.Assembly.Name",
"customFormatter": "My.Namespace.MyCustomFormatter, My.Assembly.Name",
"customDurableFormatter": "My.Namespace.MyCustomDurableFormatter, My.Assembly.Name",
"failureSink": "My.Namespace.MyFailureSink, My.Assembly.Name"
}
}]
}
}
```
See the XML `` example above for a discussion of available `Args` options.
### Handling errors
From version 5.5 you have the option to specify how to handle issues with Elasticsearch. Since the sink delivers in a batch, it might be possible that one or more events could actually not be stored in the Elasticsearch store.
Can be a mapping issue for example. It is hard to find out what happened here. There is a new option called *EmitEventFailure* which is an enum (flagged) with the following options:
* WriteToSelfLog, the default option in which the errors are written to the SelfLog.
* WriteToFailureSink, the failed events are send to another sink. Make sure to configure this one by setting the FailureSink option.
* ThrowException, in which an exception is raised.
* RaiseCallback, the failure callback function will be called when the event cannot be submitted to Elasticsearch. Make sure to set the FailureCallback option to handle the event.
An example:
```csharp
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
{
FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
EmitEventFailureHandling.WriteToFailureSink |
EmitEventFailureHandling.RaiseCallback,
FailureSink = new FileSink("./failures.txt", new JsonFormatter(), null)
})
```
With the AutoRegisterTemplate option the sink will write a default template to Elasticsearch. When this template is not there, you might not want to index as it can influence the data quality.
Since version 5.5 you can use the RegisterTemplateFailure option. Set it to one of the following options:
* IndexAnyway; the default option, the events will be send to the server
* IndexToDeadletterIndex; using the deadletterindex format, it will write the events to the deadletter queue. When you fix your template mapping, you can copy your data into the right index.
* FailSink; this will simply fail the sink by raising an exception.
Since version 7 you can specify an action to do when log row was denied by the elasticsearch because of the data (payload) if durable file is specied.
i.e.
```csharp
BufferCleanPayload = (failingEvent, statuscode, exception) =>
{
dynamic e = JObject.Parse(failingEvent);
return JsonConvert.SerializeObject(new Dictionary()
{
{ "@timestamp",e["@timestamp"]},
{ "level","Error"},
{ "message","Error: "+e.message},
{ "messageTemplate",e.messageTemplate},
{ "failingStatusCode", statuscode},
{ "failingException", exception}
});
},
```
The IndexDecider didnt worked well when durable file was specified so an option to specify BufferIndexDecider is added.
Datatype of logEvent is string
i.e.
```csharp
BufferIndexDecider = (logEvent, offset) => "log-serilog-" + (new Random().Next(0, 2)),
```
Option BufferFileCountLimit is added. The maximum number of log files that will be retained. including the current log file. For unlimited retention, pass null. The default is 31.
Option BufferFileSizeLimitBytes is added The maximum size, in bytes, to which the buffer log file for a specific date will be allowed to grow. By default `100L * 1024 * 1024` will be applied.
### Breaking changes
#### Version 9
* Dropped support for 456 and sticking now with NETSTANDARD
* Dropped support for Opensearch - This package supported writing to Opensearch (without guarantees) up untill the last version, Updated ES packages dropped support for Opensearch
#### Version 7
* Nuget Serilog.Sinks.File is now used instead of deprecated Serilog.Sinks.RollingFile
* SingleEventSizePostingLimit option is changed from int to long? with default value null, Don't use value 0 nothing will be logged then!!!!!
#### Version 6
Starting from version 6, the sink has been upgraded to work with Elasticsearch 6.0 and has support for the new templates used by ES 6.
> If you use the `AutoRegisterTemplate` option, you need to set the `AutoRegisterTemplateVersion` option to `ESv6` in order to generate default templates that are compatible with the breaking changes in ES 6.
#### Version 4
Starting from version 4, the sink has been upgraded to work with Serilog 2.0 and has .NET Core support.
#### Version 3
Starting from version 3, the sink supports the Elasticsearch.Net 2 package and Elasticsearch version 2. If you need Elasticsearch 1.x support, then stick with version 2 of the sink.
The function
```csharp
protected virtual ElasticsearchResponse EmitBatchChecked(IEnumerable events)
```
now uses a generic type. This allows you to map to either DynamicResponse when using Elasticsearch.NET or to BulkResponse if you want to use NEST.
We also dropped support for .NET 4 since the Elasticsearch.NET client also does not support this version of the framework anymore. If you need to use .net 4, then you need to stick with the 2.x version of the sink.
#### Version 2
Be aware that version 2 introduces some breaking changes.
* The overloads have been reduced to a single Elasticsearch function in which you can pass an options object.
* The namespace and function names are now Elasticsearch instead of ElasticSearch everywhere
* The Exceptions recorded by Serilog are customer serialized into the Exceptions property which is an array instead of an object.
* Inner exceptions are recorded in the same array but have an increasing depth parameter. So instead of nesting objects you need to look at this parameter to find the depth of the exception.
* Do no longer use the mapping once provided in the Gist. The Sink can automatically create the right mapping for you, but this feature is disabled by default. We advice you to use it.
* Since version 2.0.42 the ability to register this sink using the AppSettings reader is restored. You can pass in a node (or collection of nodes) and optionally an indexname and template.