https://github.com/symonk/log-analyse
:gear: Production log file monitoring at scale with actionability
https://github.com/symonk/log-analyse
Last synced: about 2 months ago
JSON representation
:gear: Production log file monitoring at scale with actionability
- Host: GitHub
- URL: https://github.com/symonk/log-analyse
- Owner: symonk
- License: apache-2.0
- Created: 2024-07-20T16:13:49.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-07-28T15:58:28.000Z (10 months ago)
- Last Synced: 2025-02-08T20:47:31.663Z (3 months ago)
- Language: Go
- Homepage:
- Size: 138 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
README
[](https://pkg.go.dev/github.com/symonk/log-analyse)
[](https://github.com/symonk/log-analyse/actions/workflows/go_test.yml)
[](https://codecov.io/gh/symonk/log-analyse)
[](https://goreportcard.com/report/github.com/symonk/log-analyse)
[](https://github.com/symonk/log-analyse/blob/master/LICENSE)
> [!CAUTION]
> log-analyse is currently in early phase development and not fit for use
# Log Analyse
`log-analyse` is a tool for asynchronously monitoring log files for pre defined pattern
matches and causing a trigger when matches are found based on arbitrary options. It can
easily monitoring thousands of individual files for `Write` events.
`log-analyse` can be leveraged as a tool for basic visibility and alerting, aswell as a
security utility.
> [!IMPORTANT]
> log-analyse will only ever need read permissions on the files it is monitoring
-----
## Planned Features
`log-analyse` aims to support the following:
* tail mode - live monitoring of log files with rotation support etc.
* trigger system for dispatching actions
* highly performant (and configurable) scanning of log files.
-----
## Triggers
for now `log-analyse` allows the following (basic) triggers:
* `trigger:slack`: Dispatch a notification to slack.
* `trigger:teams`: Dispatch a notification to teams.
* `trigger:cloud_watch`: Publish a metric to cloudwatch.
* `trigger:shell (experimental)`: Invoke a shell script with context args.
* `trigger:print`: Print violations to stdout.
-----
## Quick start
`log-analyse` by default will look for a configuration file in `~/.loganalyse/loganalyse.yaml`, however you can provide
an explicit absolute path to a yaml file via the `--config` file.
An example of the current configuration (changing rapidly):
```yaml
---
files:
- glob: ~/logs/*.log
options:
active: false
hits: 5
period: 30s
trigger: email
patterns:
- .*FATAL.*
- .*payment failed.*
- glob: ~/logs/foo.log
options:
active: true
hits: 1
period: 1h10s
trigger: slack
patterns:
- .*critical error.*
```
-----
## Running Log-analyse
Running log analyse on your system is as easy as:
```bash
# ensure to use the minimum permissions necessary for the below:
go install github.com/symonk/log-analyse
mkdir ~/.loganalyse/loganalyse.yaml
# populate loganalyse.yaml with your configuration
log-analyse
```
----
## Configuring log-analyse
Log analyse can be configured on a per `glob` basis. It is possible with overlapping globs
that the same file on disk may be traversed, this behaviour is controlled by the `strict`
flag at the top level and duplicate files can cause an exit during the collection phase.
The config is composed of an array of objects, each of which currently supports the following:
* `glob`: A glob pattern for file collection.
* `options`: An object of object for all files matching the glob.
* `active`: If the glob is enabled and should be monitored.
* `hits`: How many matches before alerting.
* `period`: Over what period should hits be considered before alerting.
* `patterns`: Per line regex patterns for lines of interest.
* `trigger`: Which notification mechanism to fire for detections.
-----