Skip to content
This repository was archived by the owner on Jun 1, 2024. It is now read-only.

Commit 8fbb35a

Browse files
mivanomivano
authored
[WIP] Added more extensive error handling (#127)
* Missing {"@ in body #102. Enable reading from UTF-8 files with or without BOM * bulkresponse deserialisation and logging of failed operations * Bugfix - message id has te be unique. * Removed NEST dependency - parse elasticsearch bulkresponse using dynamics. * Added Kibana to docker-compose. Added exception handling in Sink when no connection is for example available. * Implemented further error handling * Added additional validations * +semver:minor Added documentation
1 parent 1633c3a commit 8fbb35a

22 files changed

+1057
-99
lines changed

‎.env

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
TAG=5.6.2
2+
ELASTIC_VERSION=5.6.2
3+
ELASTIC_PASSWORD=changeme

‎.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ TestResult.xml
3434
[Rr]eleasePS/
3535
dlldata.c
3636

37+
buffer*.json
38+
*.bookmark
39+
failures.txt
40+
3741
*_i.c
3842
*_p.c
3943
*_i.h

‎.idea/modules.xml

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/vcs.xml

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/workspace.xml

Lines changed: 267 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.vscode/launch.json

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
// Use IntelliSense to find out which attributes exist for C# debugging
3+
// Use hover for the description of the existing attributes
4+
// For further information visit https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md
5+
"version": "0.2.0",
6+
"configurations": [
7+
{
8+
"name": ".NET Core Launch (console)",
9+
"type": "coreclr",
10+
"request": "launch",
11+
"preLaunchTask": "build",
12+
// If you have changed target frameworks, make sure to update the program path.
13+
"program": "${workspaceRoot}/sample/Serilog.Sinks.Elasticsearch.Sample/bin/Debug/netcoreapp1.1/Serilog.Sinks.Elasticsearch.Sample.dll",
14+
"args": [],
15+
"cwd": "${workspaceRoot}/sample/Serilog.Sinks.Elasticsearch.Sample",
16+
// For more information about the 'console' field, see https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md#console-terminal-window
17+
"console": "internalConsole",
18+
"stopAtEntry": false,
19+
"internalConsoleOptions": "openOnSessionStart"
20+
},
21+
{
22+
"name": ".NET Core Attach",
23+
"type": "coreclr",
24+
"request": "attach",
25+
"processId": "${command:pickProcess}"
26+
}
27+
]
28+
}

‎.vscode/tasks.json

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"version": "0.1.0",
3+
"command": "dotnet",
4+
"isShellCommand": true,
5+
"args": [],
6+
"tasks": [
7+
{
8+
"taskName": "build",
9+
"args": [
10+
"${workspaceRoot}/sample/Serilog.Sinks.Elasticsearch.Sample/Serilog.Sinks.Elasticsearch.Sample.csproj"
11+
],
12+
"isBuildCommand": true,
13+
"problemMatcher": "$msCompile"
14+
}
15+
]
16+
}

‎CHANGES.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
11
== Changelog
22

3+
5.5
4+
* Errors from Elasticsearch can now be handled. Either by looking into the selflog, sending the failty events to another sink, handle the logevent yourself by using a callback or let the sink throw an exception.
5+
* BOM fix for buffered option.
6+
* The creation of the template might fail. You can now specify what kind of action should be taken if this happens.
7+
* Added a sample application.
8+
* Added a docker-compose file that allows you to start a local elasticsearch and kibana instance.
9+
* ConnectionTimeout is now set to be 5 seconds instead of 1 minute.
10+
* You can now set the queueSizeLimit, which limits the amount of events stored in the PeriodicBatching buffer. Does not impact the durable buffer.
11+
312
5.4
413
* Added support for pipelines in Elasticsearch. Pipelines allows you to change the ingress data by running it through Processors (https://www.elastic.co/blog/new-way-to-ingest-part-1).
514

‎README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,36 @@ In your `appsettings.json` file, under the `Serilog` node, :
131131

132132
See the XML `<appSettings>` example above for a discussion of available `Args` options.
133133

134+
### Handling errors
135+
136+
From version 5.5 you have the option to specify how to handle issues with Elasticsearch. Since the sink delivers in a batch, it might be possible that one or more events could actually nog be stored in the Elasticseach store.
137+
Can be a mapping issue for example. It is hard to find out what happened here. There is a new option called *EmitEventFailure* which is an enum (flagged) with the following options:
138+
139+
- WriteToSelfLog, the default option in which the errors are written to the SelfLog.
140+
- WriteToFailureSink, the failed events are send to another sink. Make sure to configure this one by setting the FailureSink option.
141+
- ThrowException, in which an exception is raised.
142+
- RaiseCallback, the failure callback function will be called when the event cannot be submitted to Elasticsearch. Make sure to set the FailureCallback option to handle the event.
143+
144+
An example:
145+
146+
```csharp
147+
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
148+
{
149+
FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
150+
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
151+
EmitEventFailureHandling.WriteToFailureSink |
152+
EmitEventFailureHandling.RaiseCallback,
153+
FailureSink = new FileSink("./failures.txt", new JsonFormatter(), null)
154+
})
155+
```
156+
157+
With the AutoRegisterTemplate option the sink will write a default template to Elasticsearch. When this template is not there, you might not want to index as it can influence the data quality.
158+
Since version 5.5 you can use the RegisterTemplateFailure option. Set it to one of the following options:
159+
160+
- IndexAnyway; the default option, the events will be send to the server
161+
- IndexToDeadletterIndex; using the deadletterindex format, it will write the events to the deadletter queue. When you fix your template mapping, you can copy your data into the right index.
162+
- FailSink; this will simply fail the sink by raising an exception.
163+
134164
### Breaking changes for version 4
135165

136166
Starting from version 4, the sink has been upgraded to work with Serilog 2.0 and has .NET Core support.

‎docker-compose.readme.txt

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
Elastic (http://localhost:9200)= elastic:changeme
2+
Elastic Head: http://localhost:9100/?auth_user=elastic&auth_password=changeme
3+
Kibana: http://localhost:5601
4+
5+
To start the components:
6+
docker-compose up
7+
8+
add the -d option to run in the background
9+
10+
To scale the nodes:
11+
12+
docker-compose up --scale elasticsearch=3
13+
14+
To shutdown:
15+
16+
docker-compose down

‎docker-compose.yml

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
version: '2'
2+
3+
services:
4+
5+
elasticsearch:
6+
image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
7+
volumes:
8+
- esdata2:/usr/share/elasticsearch/data
9+
ports:
10+
- "9200:9200"
11+
- "9300"
12+
ulimits:
13+
memlock:
14+
soft: -1
15+
hard: -1
16+
mem_limit: 1g
17+
environment:
18+
- cluster.name=docker-cluster
19+
- bootstrap.memory_lock=true
20+
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
21+
- discovery.type=zen
22+
- discovery.zen.ping.unicast.hosts=elasticsearch
23+
- http.cors.enabled=true
24+
- http.cors.allow-origin=*
25+
- http.cors.allow-headers=Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With
26+
- http.cors.allow-credentials=true
27+
networks:
28+
- elk
29+
30+
kibana:
31+
image: docker.elastic.co/kibana/kibana:${TAG}
32+
environment:
33+
- "ELASTICSEARCH_URL=http://elasticsearch:9200"
34+
- server.name=kibana
35+
- server.host="0"
36+
- xpack.security.enabled=true
37+
- xpack.monitoring.enabled=true
38+
- xpack.ml.enabled=false
39+
- xpack.graph.enabled=false
40+
- xpack.reporting.enabled=false
41+
- xpack.grokdebugger.enabled=false
42+
ports:
43+
- "5601:5601"
44+
networks:
45+
- esnet
46+
depends_on:
47+
- elasticsearch
48+
49+
head:
50+
image: mobz/elasticsearch-head:5
51+
ulimits:
52+
memlock:
53+
soft: -1
54+
hard: -1
55+
mem_limit: 1g
56+
ports:
57+
- "9100:9100"
58+
networks:
59+
- elk
60+
61+
kibana:
62+
image: docker.elastic.co/kibana/kibana:${TAG}
63+
environment:
64+
- "ELASTICSEARCH_URL=http://elasticsearch:9200"
65+
ports:
66+
- "5601:5601"
67+
networks:
68+
- elk
69+
depends_on:
70+
- elasticsearch
71+
72+
volumes:
73+
esdata1:
74+
driver: local
75+
esdata2:
76+
driver: local
77+
#esconfig1:
78+
# driver: local
79+
#esconfig2:
80+
# driver: local
81+
82+
networks:
83+
elk:
84+
driver: bridge

‎sample/Serilog.Sinks.Elasticsearch.Sample/Program.cs

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
using System;
2+
using Serilog;
3+
using Serilog.Debugging;
4+
using Serilog.Formatting.Json;
5+
using Serilog.Sinks.File;
6+
using Serilog.Sinks.SystemConsole.Themes;
7+
8+
namespace Serilog.Sinks.Elasticsearch.Sample
9+
{
10+
class Program
11+
{
12+
static void Main(string[] args)
13+
{
14+
Log.Logger = new LoggerConfiguration()
15+
.MinimumLevel.Debug()
16+
.WriteTo.Console(theme: SystemConsoleTheme.Literate)
17+
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://elastic:changeme@localhost:9200")) // for the docker-compose implementation
18+
{
19+
AutoRegisterTemplate = true,
20+
//BufferBaseFilename = "./buffer",
21+
RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
22+
FailureCallback = e => Console.WriteLine("Unable to submit event " + e.MessageTemplate),
23+
EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
24+
EmitEventFailureHandling.WriteToFailureSink |
25+
EmitEventFailureHandling.RaiseCallback,
26+
FailureSink = new FileSink("./failures.txt", new JsonFormatter(), null)
27+
})
28+
.CreateLogger();
29+
30+
// Enable the selflog output
31+
SelfLog.Enable(Console.Error);
32+
33+
Log.Information("Hello, world!");
34+
35+
int a = 10, b = 0;
36+
try
37+
{
38+
Log.Debug("Dividing {A} by {B}", a, b);
39+
Console.WriteLine(a / b);
40+
}
41+
catch (Exception ex)
42+
{
43+
Log.Error(ex, "Something went wrong");
44+
}
45+
46+
// Introduce a failure by storing a field as a different type
47+
Log.Debug("Reusing {A} by {B}", "string", true);
48+
49+
Log.CloseAndFlush();
50+
Console.Read();
51+
}
52+
}
53+
}

‎sample/Serilog.Sinks.Elasticsearch.Sample/Serilog.Sinks.Elasticsearch.Sample.csproj

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
3+
<PropertyGroup>
4+
<OutputType>Exe</OutputType>
5+
<TargetFramework>netcoreapp1.1</TargetFramework>
6+
</PropertyGroup>
7+
8+
<ItemGroup>
9+
<PackageReference Include="Serilog" Version="2.5.0" />
10+
<PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
11+
</ItemGroup>
12+
13+
<ItemGroup>
14+
<ProjectReference Include="..\..\src\Serilog.Sinks.Elasticsearch\Serilog.Sinks.Elasticsearch.csproj" />
15+
</ItemGroup>
16+
17+
</Project>

‎serilog-sinks-elasticsearch.sln

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
Microsoft Visual Studio Solution File, Format Version 12.00
22
# Visual Studio 15
3-
VisualStudioVersion = 15.0.26730.16
3+
VisualStudioVersion = 15.0.26730.3
44
MinimumVisualStudioVersion = 10.0.40219.1
55
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Files", "Files", "{148431F6-5BA9-4987-80CF-DF9F23F54947}"
66
ProjectSection(SolutionItems) = preProject
@@ -19,6 +19,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Serilog.Sinks.Elasticsearch
1919
EndProject
2020
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Serilog.Sinks.Elasticsearch.Tests", "test\Serilog.Sinks.Elasticsearch.Tests\Serilog.Sinks.Elasticsearch.Tests.csproj", "{CA358673-C4B6-49D0-8EC5-D6CB50A11CC0}"
2121
EndProject
22+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Serilog.Sinks.Elasticsearch.Sample", "sample\Serilog.Sinks.Elasticsearch.Sample\Serilog.Sinks.Elasticsearch.Sample.csproj", "{253B37AB-D82E-4A5F-BA16-F1BE398818C8}"
23+
EndProject
2224
Global
2325
GlobalSection(SolutionConfigurationPlatforms) = preSolution
2426
Debug|Any CPU = Debug|Any CPU
@@ -33,8 +35,15 @@ Global
3335
{CA358673-C4B6-49D0-8EC5-D6CB50A11CC0}.Debug|Any CPU.Build.0 = Debug|Any CPU
3436
{CA358673-C4B6-49D0-8EC5-D6CB50A11CC0}.Release|Any CPU.ActiveCfg = Release|Any CPU
3537
{CA358673-C4B6-49D0-8EC5-D6CB50A11CC0}.Release|Any CPU.Build.0 = Release|Any CPU
38+
{253B37AB-D82E-4A5F-BA16-F1BE398818C8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
39+
{253B37AB-D82E-4A5F-BA16-F1BE398818C8}.Debug|Any CPU.Build.0 = Debug|Any CPU
40+
{253B37AB-D82E-4A5F-BA16-F1BE398818C8}.Release|Any CPU.ActiveCfg = Release|Any CPU
41+
{253B37AB-D82E-4A5F-BA16-F1BE398818C8}.Release|Any CPU.Build.0 = Release|Any CPU
3642
EndGlobalSection
3743
GlobalSection(SolutionProperties) = preSolution
3844
HideSolutionNode = FALSE
3945
EndGlobalSection
46+
GlobalSection(ExtensibilityGlobals) = postSolution
47+
SolutionGuid = {2A076FF0-B90E-451A-B858-3144CE509516}
48+
EndGlobalSection
4049
EndGlobal

‎serilog-sinks-elasticsearch.sln.iml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<module type="RIDER_MODULE" version="4">
3+
<component name="NewModuleRootManager">
4+
<content url="file://$MODULE_DIR$">
5+
<sourceFolder url="file://$MODULE_DIR$" isTestSource="false" />
6+
</content>
7+
<orderEntry type="sourceFolder" forTests="false" />
8+
</component>
9+
</module>

‎src/Serilog.Sinks.Elasticsearch/Serilog.Sinks.Elasticsearch.csproj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@
2929
<ItemGroup>
3030
<PackageReference Include="Elasticsearch.Net" Version="5.5.0" />
3131
<PackageReference Include="Serilog" Version="2.5.0" />
32-
<PackageReference Include="Serilog.Sinks.File" Version="3.2.0" />
33-
<PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="2.1.0" />
32+
<PackageReference Include="Serilog.Sinks.File" Version="4.0.0" />
33+
<PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="2.1.1" />
3434
<PackageReference Include="Serilog.Sinks.RollingFile" Version="3.3.0" />
3535
</ItemGroup>
3636

0 commit comments

Comments
 (0)