Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/h21lab/anomaly-detection
Scripts to help to detect anomalies in pcap file. Anomaly Detection using tensorflow and tshark.
https://github.com/h21lab/anomaly-detection
detect-anomalies json machine-learning monitoring neural-network pcap tensorflow tshark wireshark
Last synced: 3 months ago
JSON representation
Scripts to help to detect anomalies in pcap file. Anomaly Detection using tensorflow and tshark.
- Host: GitHub
- URL: https://github.com/h21lab/anomaly-detection
- Owner: H21lab
- License: apache-2.0
- Created: 2017-03-08T06:41:03.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2020-06-10T11:23:37.000Z (over 4 years ago)
- Last Synced: 2024-08-02T15:54:11.345Z (3 months ago)
- Topics: detect-anomalies, json, machine-learning, monitoring, neural-network, pcap, tensorflow, tshark, wireshark
- Language: Jupyter Notebook
- Homepage: https://www.h21lab.com/tools/anomaly-detection
- Size: 1.79 MB
- Stars: 79
- Watchers: 8
- Forks: 22
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-network-stuff - **37**星
README
# Unsupervised Anomaly Detection using tensorflow and tshark
Unsupervised learning by using autoencoder neural network by using tensorflow.
See the [ad_tf_autoencoder.ipynb](https://github.com/H21lab/Anomaly-Detection/blob/master/ad_tf_autoencoder.ipynb)
# Supervised Anomaly Detection using tensorflow and tshark
```shell-session
Script to help to detect anomalies in pcap file.
Using tensorflow neural network classifier and tshark -T ek -x input.
Input is tshark ek json generate by:
./tshark -T ek -x -r trace.pcap > input.json
Run script:
cat input.pcap.json | python ad_tf.py -i normal.pcap.json \
-a anomaly.pcap.json -f field_1 field_2 .... field_n
For fields the name of the fields from json ek should be used, e.g.:
tshark -T ek -x -r ./res/input.pcap.gz | python ad_tf.py \
-i res/normal.json -a res/anomaly.json -f tcp_tcp_flags_raw \
tcp_tcp_dstport_raw
Output pcap
ad_test.pcap
The script uses the tshark ek jsons including the raw hex data generated
from pcaps by command as described above. The fields arguments are used for
anomaly detection. The fields are used as columns, hashed and used as input
to tensorflow neural classifier network.
The neural classifier network is first trained with normal.pcap.json input
with label 0 and with anomaly.pcap.json input with label 1. After training
then from stdin is read the input.pcap.json and evaluated. The neural
network predicts the label.
The output pcap contains then the frames predicted by neural network as
anomalies with label 1.
```
# Simple Anomaly Detection using tshark
```shell-session
Simple script to help to detect anomalies in pcap file.
Input is tshark ek json generate by:
./tshark -T ek -x -r trace.pcap > input.json
Run script:
cat input.json | python ad_simple.py field_1 field_2 .... field_n
For fields the name of the fields from json ek should be used, e.g.:
cat input.json | python ad_simple.py ip_ip_src ip_ip_dst
Output pcap
ad_test.pcap
The script read the tshark ek json including the raw hex data. The input is
generated from pcap using tshark. The fields arguments are used for simple
anomaly detection. The behavior is similar like SQL GROUP BY command. The
fields are hashed together and the output pcap contains the frames
beginning with most unique combination of selected fields and descending to
most frequent frames containing the selected fields.
The following example
cat input.json | python ad_simple.py ip_ip_src ip_ip_dst
will generate pcap starting with less frequent combinations of source and
dest IP pairs and descending to frames with common
combinations.
```
## Limitations
Program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY.
## Attribution
This code was created by Martin Kacer, H21 lab, Copyright 2020.
https://www.h21lab.com