https://github.com/HelenGuohx/logbert

log anomaly detection via BERT
https://github.com/HelenGuohx/logbert

Last synced: 7 days ago
JSON representation

log anomaly detection via BERT

• Host: GitHub
• URL: https://github.com/HelenGuohx/logbert
• Owner: HelenGuohx
• License: mit
• Created: 2020-12-05T00:13:28.000Z (almost 4 years ago)
• Default Branch: main
• Last Pushed: 2024-06-21T21:45:11.000Z (5 months ago)
• Last Synced: 2024-08-02T15:32:34.394Z (3 months ago)
• Language: Python
• Size: 179 KB
• Stars: 221
• Watchers: 8
• Forks: 97
• Open Issues: 19
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# LogBERT: Log Anomaly Detection via BERT

### [ARXIV](https://arxiv.org/abs/2103.04475) 

This repository provides the implementation of Logbert for log anomaly detection. 

The process includes downloading raw data online, parsing logs into structured data, 

creating log sequences and finally modeling. 

![alt](img/log_preprocess.png)

## Configuration

- Ubuntu 20.04

- NVIDIA driver 460.73.01 

- CUDA 11.2

- Python 3.8

- PyTorch 1.9.0

## Installation

This code requires the packages listed in requirements.txt.

An virtual environment is recommended to run this code

On macOS and Linux:  

```

python3 -m pip install --user virtualenv

python3 -m venv env

source env/bin/activate

pip install -r ./environment/requirements.txt

deactivate

```

Reference: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

An alternative is to create a conda environment:

```

    conda create -f ./environment/environment.yml

    conda activate logbert

```

Reference: https://docs.conda.io/en/latest/miniconda.html

## Experiment

Logbert and other baseline models are implemented on [HDFS](https://github.com/logpai/loghub/tree/master/HDFS), [BGL](https://github.com/logpai/loghub/tree/master/BGL), and [thunderbird]() datasets

### HDFS example

```shell script

cd HDFS

sh init.sh

# process data

python data_process.py

#run logbert

python logbert.py vocab

python logbert.py train

python logbert.py predict

#run deeplog

python deeplog.py vocab

# set options["vocab_size"] =  above

python deeplog.py train

python deeplog.py predict 

#run loganomaly

python loganomaly.py vocab

# set options["vocab_size"] =  above

python loganomaly.py train

python loganomaly.py predict

#run baselines

baselines.ipynb

```

### Folders created during execution

```shell script 

~/.dataset //Stores original datasets after downloading

project/output //Stores intermediate files and final results during execution

```