{"id":13583943,"url":"https://github.com/H21lab/Anomaly-Detection","last_synced_at":"2025-04-06T21:33:30.499Z","repository":{"id":53822525,"uuid":"84287590","full_name":"H21lab/Anomaly-Detection","owner":"H21lab","description":"Scripts to help to detect anomalies in pcap file. Anomaly Detection using tensorflow and tshark.","archived":false,"fork":false,"pushed_at":"2020-06-10T11:23:37.000Z","size":1873,"stargazers_count":79,"open_issues_count":0,"forks_count":21,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-17T10:39:39.545Z","etag":null,"topics":["detect-anomalies","json","machine-learning","monitoring","neural-network","pcap","tensorflow","tshark","wireshark"],"latest_commit_sha":null,"homepage":"https://www.h21lab.com/tools/anomaly-detection","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/H21lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-08T06:41:03.000Z","updated_at":"2024-10-10T19:50:31.000Z","dependencies_parsed_at":"2022-08-21T18:40:15.391Z","dependency_job_id":null,"html_url":"https://github.com/H21lab/Anomaly-Detection","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FAnomaly-Detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FAnomaly-Detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FAnomaly-Detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/H21lab%2FAnomaly-Detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/H21lab","download_url":"https://codeload.github.com/H21lab/Anomaly-Detection/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247556844,"owners_count":20958034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["detect-anomalies","json","machine-learning","monitoring","neural-network","pcap","tensorflow","tshark","wireshark"],"created_at":"2024-08-01T15:03:54.899Z","updated_at":"2025-04-06T21:33:29.584Z","avatar_url":"https://github.com/H21lab.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# Unsupervised Anomaly Detection using tensorflow and tshark\nUnsupervised learning by using autoencoder neural network by using tensorflow.\n\nSee the [ad_tf_autoencoder.ipynb](https://github.com/H21lab/Anomaly-Detection/blob/master/ad_tf_autoencoder.ipynb)\n\n\n# Supervised Anomaly Detection using tensorflow and tshark\n```shell-session\nScript to help to detect anomalies in pcap file.\nUsing tensorflow neural network classifier and tshark -T ek -x input.\n\nInput is tshark ek json generate by:\n./tshark -T ek -x -r trace.pcap \u003e input.json\n\nRun script:\ncat input.pcap.json | python ad_tf.py -i normal.pcap.json \\\n -a anomaly.pcap.json -f field_1 field_2 .... field_n\n\nFor fields the name of the fields from json ek should be used, e.g.:\ntshark -T ek -x -r ./res/input.pcap.gz | python ad_tf.py \\\n   -i res/normal.json -a res/anomaly.json -f tcp_tcp_flags_raw \\\n   tcp_tcp_dstport_raw\n\nOutput pcap\nad_test.pcap\n\nThe script  uses the tshark ek  jsons including the raw  hex data generated\nfrom pcaps by command as described above. The fields arguments are used for\nanomaly detection. The fields are used as columns, hashed and used as input\nto tensorflow neural classifier network.\n\nThe neural classifier network is  first trained with normal.pcap.json input\nwith label 0 and with anomaly.pcap.json  input with label 1. After training\nthen  from stdin  is read  the  input.pcap.json and  evaluated. The  neural\nnetwork predicts the label.\n\nThe output  pcap contains then  the frames  predicted by neural  network as\nanomalies with label 1.\n```\n\n# Simple Anomaly Detection using tshark\n```shell-session\nSimple script to help to detect anomalies in pcap file.\n\nInput is tshark ek json generate by:\n./tshark -T ek -x -r trace.pcap \u003e input.json\n\nRun script:\ncat input.json | python ad_simple.py field_1 field_2 .... field_n\n\nFor fields the name of the fields from json ek should be used, e.g.:\ncat input.json | python ad_simple.py ip_ip_src ip_ip_dst\n\nOutput pcap\nad_test.pcap\n\nThe script read the tshark ek json including the raw hex data. The input is\ngenerated from pcap using tshark. The  fields arguments are used for simple\nanomaly detection. The  behavior is similar like SQL GROUP  BY command. The\nfields  are  hashed  together  and  the output  pcap  contains  the  frames\nbeginning with most unique combination of selected fields and descending to\nmost frequent frames containing the selected fields.\n\nThe following example\n    cat input.json | python ad_simple.py ip_ip_src ip_ip_dst\nwill  generate pcap starting with less  frequent combinations of source and\ndest IP pairs and descending to frames with common\ncombinations.\n```\n\n## Limitations\n\nProgram is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY.\n\n## Attribution\n\nThis code was created by Martin Kacer, H21 lab, Copyright 2020.\nhttps://www.h21lab.com\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FH21lab%2FAnomaly-Detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FH21lab%2FAnomaly-Detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FH21lab%2FAnomaly-Detection/lists"}