Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/slgobinath/pcap-processor
Read and process pcap files using this nifty tool
https://github.com/slgobinath/pcap-processor
csv grpc http kafka pcap pcap-processor sink stream wisdom
Last synced: about 1 month ago
JSON representation
Read and process pcap files using this nifty tool
- Host: GitHub
- URL: https://github.com/slgobinath/pcap-processor
- Owner: slgobinath
- License: gpl-3.0
- Created: 2018-05-08T00:07:22.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2018-12-21T13:41:29.000Z (almost 6 years ago)
- Last Synced: 2024-08-03T17:11:19.312Z (4 months ago)
- Topics: csv, grpc, http, kafka, pcap, pcap-processor, sink, stream, wisdom
- Language: Python
- Size: 39.1 KB
- Stars: 40
- Watchers: 4
- Forks: 21
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-network-stuff - **20**星
README
# Pcap Processor
Read and process pcap files using this nifty tool.
This tool can read pcap files, process them internally and write them to one or more sinks.
Currently there are mappers written for pcap length conversion and protocol normalization.
I also have written sinks to write the pcap file to console, csv file or http endpoint.
```text
usage: pcap-processor [-h] [--map {length,protocol}]
[--sink {console,kafka,http,csv,grpc}] [--version]
file [file ...]
Read and process pcap files using this nifty tool.
positional arguments:
file pcap file to read
optional arguments:
-h, --help show this help message and exit
--map {length,protocol}
enable a mapper with the given name. You can use this
option multiple times to enable more than one mappers
--sink {console,kafka,http,csv,grpc}
enable a sink with the given name. You can use this
option multiple times to enable more than one sinks
--version show program's version number and exit
```
## Note
Currently the `pcap-processor` reads only packets with IP or IPV6 layers. Feel free to modify the code based on your requirement. If you find any enhancement, please send a Pull Request.
## Requirements
pcap-reader relies on external command line tool: `tshark` and some Python modules.
Install `tshark` using the following command in Ubuntu and its derivatives:
```bash
sudo apt install tshark
```
Install Python dependencies using the following command:
```bash
pip3 install -r requirements.txt
```
## Research Work
This tool is developed as part of my research project. If you are using this tool in your research,
please cite the following paper:
**Citation:**
```text
Loganathan, G., Samarabandu, J., & Wang, X. (2018). Sequence to Sequence Pattern Learning Algorithm for Real-time Anomaly Detection in Network Traffic. In 2018 IEEE Canadian Conference on Electrical & Computer Engineering (CCECE) (CCECE 2018). Quebec City, Canada.
```
**BibTex**
```bibtex
@INPROCEEDINGS{Loga1805:Sequence,
AUTHOR="Gobinath Loganathan and Jagath Samarabandu and Xianbin Wang",
TITLE="Sequence to Sequence Pattern Learning Algorithm for Real-time Anomaly
Detection in Network Traffic",
BOOKTITLE="2018 IEEE Canadian Conference on Electrical \& Computer Engineering (CCECE)
(CCECE 2018)",
ADDRESS="Quebec City, Canada",
DAYS=13,
MONTH=may,
YEAR=2018,
KEYWORDS="Seq2Seq; Anomaly Detection",
ABSTRACT="Network intrusions can be modeled as anomalies in network traffic in which
the expected order of packets and their attributes deviate from regular
traffic. Algorithms that predict the next sequence of events based on
previous sequences are a promising avenue for detecting such anomalies. In
this paper, we present a novel multi-attribute model for predicting a
network packet sequence based on previous packets using a
sequence-to-sequence (Seq2Seq) encoder-decoder model. This model is trained
on an attack-free dataset to learn the normal sequence of packets in TCP
connections and then it is used to detect anomalous packets in TCP traffic.
We show that in DARPA 1999 dataset, the proposed multi-attribute Seq2Seq
model detects anomalous raw TCP packets which are part of intrusions with
97\% accuracy. Also, it can detect selected intrusions in real-time with
100\% accuracy and outperforms existing algorithms based on recurrent
neural network models such as LSTM."
}
```
## Use Cases
Read a pcap file and send all packets to Apache Kafka:
```bash
python3 -m pcap_processor --sink kafka samples/cicids_2017.pcap
```
Read a pcap file, map protocols and write them to a CSV file:
```bash
python3 -m pcap_processor --map protocol --sink csv samples/cicids_2017.pcap
```
Mappers and sinks have their own properties. Please modify them in the relevant `plugins/.py`.
For example, to change the output CSV file location, modify the `self.path = "packets.csv"` in `pcap_processor/plugins/csv_sink.py`.