https://github.com/d0ng1ee/logdeep
log anomaly detection toolkit including DeepLog
https://github.com/d0ng1ee/logdeep
aiops anomaly-detection deeplog failure-detection log-analysis log-anomaly pytorch sequence-prediction
Last synced: 5 months ago
JSON representation
log anomaly detection toolkit including DeepLog
- Host: GitHub
- URL: https://github.com/d0ng1ee/logdeep
- Owner: d0ng1ee
- License: mit
- Created: 2020-03-11T12:48:32.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-04-23T14:58:57.000Z (about 6 years ago)
- Last Synced: 2024-12-27T23:42:56.547Z (over 1 year ago)
- Topics: aiops, anomaly-detection, deeplog, failure-detection, log-analysis, log-anomaly, pytorch, sequence-prediction
- Language: Python
- Homepage:
- Size: 6 MB
- Stars: 409
- Watchers: 4
- Forks: 117
- Open Issues: 31
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# logdeep
## Introduction
LogDeep is an open source deeplearning-based log analysis toolkit for automated anomaly detection.
*Note: This repo does not include log parsing，if you need to use it, please check [logparser](https://github.com/logpai/logparser)*
## Major features
- Modular Design
- Support multi log event features out of box
- State of the art(Including resluts from deeplog,loganomaly,robustlog...)
## Models
| Model | Paper reference |
| :--- | :--- |
|DeepLog| [**CCS'17**] [DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning](https://www.cs.utah.edu/~lifeifei/papers/deeplog.pdf)|
|LogAnomaly| [**IJCAI'19**] [LogAnomaly: UnsupervisedDetectionof SequentialandQuantitativeAnomaliesinUnstructuredLogs](https://www.ijcai.org/Proceedings/2019/658)|
|RobustLog| [**FSE'19**] [RobustLog-BasedAnomalyDetectiononUnstableLogData](https://dl.acm.org/doi/10.1145/3338906.3338931)
## Requirement
- python>=3.6
- pytorch >= 1.1.0
## Quick start
```
git clone https://github.com/donglee-afar/logdeep.git
cd logdeep
```
Example of building your own log dataset
[SAMPLING_EXAMPLE.md](data/sampling_example/README.md)
Train & Test DeepLog example
```
cd demo
# Train
python deeplog.py train
# Test
python deeplog.py test
```
The output results, key parameters and train logs will be saved under `result/` path
## DIY your own pipeline
Here is an example of the key parameters of the loganomaly model which in `demo/loganomaly.py`
Try to modify these parameters to build a new model!
```
# Smaple
options['sample'] = "sliding_window"
options['window_size'] = 10
# Features
options['sequentials'] = True
options['quantitatives'] = True
options['semantics'] = False
Model = loganomaly(input_size=options['input_size'],
hidden_size=options['hidden_size'],
num_layers=options['num_layers'],
num_keys=options['num_classes'])
```
## Benchmark results
| | | HDFS | | |
| :----:|:----:|:----:|:----:|:----:|
| **Model** | **feature** | **Precision** | **Recall** | **F1** |
| DeepLog(unsupervised)| seq |0.9583 | 0.9330 | 0.9454 |
| LogAnomaly(unsupervised) | seq+quan|0.9690 |0.9825 |0.9757 |
| RobustLog(supervised)| semantic |0.9216 |0.9586 |0.9397 |