Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/marusyk/grok.net
.NET implementation of the grok 📝
https://github.com/marusyk/grok.net
c-sharp-library csharp dotnet dotnet-standard grok grok-parser grok-patterns grokking hacktoberfest nuget nuget-package
Last synced: about 1 month ago
JSON representation
.NET implementation of the grok 📝
- Host: GitHub
- URL: https://github.com/marusyk/grok.net
- Owner: Marusyk
- License: mit
- Created: 2019-07-10T21:30:01.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-08-16T07:25:20.000Z (3 months ago)
- Last Synced: 2024-09-25T18:42:38.769Z (about 2 months ago)
- Topics: c-sharp-library, csharp, dotnet, dotnet-standard, grok, grok-parser, grok-patterns, grokking, hacktoberfest, nuget, nuget-package
- Language: C#
- Homepage:
- Size: 145 KB
- Stars: 287
- Watchers: 9
- Forks: 55
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
[](https://stand-with-ukraine.pp.ua)
Cross-platform .NET grok implementation as a NuGet package
[](https://github.com/Marusyk/grok.net/actions/workflows/builds.yml)
[](https://github.com/Marusyk/grok.net/releases)
[](https://github.com/Marusyk/grok.net/blob/main/LICENSE)
[](https://github.com/Marusyk/grok.net/blob/main/CONTRIBUTING.md)
[](https://www.nuget.org/packages/grok.net)
[](https://www.nuget.org/packages/Grok.Net)
[](https://www.powershellgallery.com/packages/Grok)
[](https://www.powershellgallery.com/packages/Grok)
# Code Coverage
[](https://coveralls.io/github/Marusyk/grok.net?branch=main)
# How to Install
Install as a library from [Nuget](http://nuget.org):
**[Grok.Net](https://www.nuget.org/packages/Grok.Net)**
PM> Install-Package Grok.Net
Install as a PowerShell module from [PowershellGallery](https://www.powershellgallery.com):
**[Grok](https://www.powershellgallery.com/packages/Grok)**
```powershell
Install-Module -Name Grok
```
# Dependency
Since [v.2.0.0](https://github.com/Marusyk/grok.net/releases/tag/v2.0.0), the grok uses the [PCRE.NET](https://github.com/ltrzesniewski/pcre-net) library for regex.
# What is grok
Grok is a great way to parse unstructured log data into something structured and queryable. It sits on top of Regular Expression (regex) and uses text patterns to match lines in log files.
A great way to get started with building your grok filters is this grok debug tool: https://grokdebugger.com
What can I use Grok for?
- reporting errors and other patterns from logs and processes
- parsing complex text output and converting it to JSON for external processing
- apply 'write-once use-everywhere' to regular expressions
- automatically providing patterns for unknown text inputs (logs you want patterns generated for future matching)
The syntax for a grok pattern is `%{SYNTAX:SEMANTIC}`
The `SYNTAX` is the name of [the pattern](https://github.com/logstash-plugins/logstash-patterns-core/blob/main/patterns/ecs-v1/grok-patterns) that will match your text. `SEMANTIC` is the key.
For example, `3.44` will be matched by the `NUMBER` pattern, and `55.3.244.1` will be matched by the `IP` pattern. `3.44` could be the duration of an event, so you could call it simply `duration`. Further, a string `55.3.244.1` might identify the `client` making a request.
For the above example, your grok filter would look something like this:
```text
%{NUMBER:duration} %{IP:client}
```
Examples: With that idea of syntax and semantics, we can pull out useful fields from a sample log like this fictional HTTP request log:
```text
55.3.244.1 GET /index.html 15824 0.043
```
The pattern for this could be:
```text
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
```
More about [grok](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)
# How to use
Create a new instance with grok pattern:
```csharp
Grok grok = new Grok("%{MONTHDAY:month}-%{MONTHDAY:day}-%{MONTHDAY:year} %{TIME:timestamp};%{WORD:id};%{LOGLEVEL:loglevel};%{WORD:func};%{GREEDYDATA:msg}");
```
then prepare some logs to parse
```csharp
string logs = @"06-21-19 21:00:13:589241;15;INFO;main;DECODED: 775233900043 DECODED BY: 18500738 DISTANCE: 1.5165
06-22-19 22:00:13:589265;156;WARN;main;DECODED: 775233900043 EMPTY DISTANCE: --------";
```
You are ready to parse and print the result
```csharp
var grokResult = grok.Parse(logs);
foreach (var item in grokResult)
{
Console.WriteLine($"{item.Key} : {item.Value}");
}
```
output:
```text
month : 06
day : 21
year : 19
timestamp : 21:00:13:589241
id : 15
loglevel : INFO
func : main
msg : DECODED: 775233900043 DECODED BY: 18500738 DISTANCE: 1.5165
month : 06
day : 22
year : 19
timestamp : 22:00:13:589265
id : 156
loglevel : WARN
func : main
msg : DECODED: 775233900043 EMPTY DISTANCE: --------
```
or use `ToDictionary()` on `grokResult` to get the result as `IReadOnlyDictionary>`
# Custom grok patterns
There is the possibility to add your own patterns.
## using file
Create a file and write the pattern you need as the pattern name, space, and then the regexp for that pattern.
For example, Patterns\grok-custom-patterns:
```text
ZIPCODE [1-9]{1}[0-9]{2}\s{0,1}[0-9]{3}
```
then load the file and pass the stream to Grok:
```csharp
FileStream customPatterns = System.IO.File.OpenRead(@"Patterns\grok-custom-patterns");
Grok grok = new Grok("%{ZIPCODE:zipcode}:%{EMAILADDRESS:email}", customPatterns);
var grokResult = grok.Parse($"122001:[email protected]");
```
## using in-memory
Define a collection of patterns
```csharp
var custom = new Dictionary
{
{"BASE64", "(?=(.{4})*$)[A-Za-z0-9+/]*={0,2}$"}
};
```
and use it as follows
```csharp
var grok = new Grok("Basic %{BASE64:credentials}", custom);
GrokResult grokResult = grok.Parse("Basic YWRtaW46cGEkJHdvcmQ=");
```
# PowerShell Module
Install and use the Grok as a PowerShell module
```powershell
grok -i "06-21-19 21:00:13:589241;15;INFO;main;DECODED: 775233900043 DECODED BY: 18500738 DISTANCE: 1.5165" -g "%{MONTHDAY:month}-%{MONTHDAY:day}-%{MONTHDAY:year} %{TIME:timestamp};%{WORD:id};%{LOGLEVEL:loglevel};%{WORD:func};%{GREEDYDATA:msg}"
```
To get help use `help grok` command
## Build
On Windows:
```powershell
build.ps1
```
On Linux/Mac:
```bash
build.sh
```
## Contributing
Would you like to help make grok.net even better? We keep a list of issues that are approachable for newcomers under the [good-first-issue](https://github.com/Marusyk/grok.net/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) label.
Also. please read [CONTRIBUTING.md](https://github.com/Marusyk/grok.net/blob/main/CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
## License
This project is licensed under the MIT License - see the [LICENSE.md](https://github.com/Marusyk/grok.net/blob/main/LICENSE) file for details
Thanks to [@martinjt](https://github.com/martinjt). The project is based on [martinjt/grokdotnet](https://github.com/martinjt/grokdotnet).