{"id":17130903,"url":"https://github.com/herrfeder/ai_cybersecurity_ids_poc","last_synced_at":"2025-04-13T07:33:16.698Z","repository":{"id":54221491,"uuid":"314457392","full_name":"herrfeder/AI_Cybersecurity_IDS_PoC","owner":"herrfeder","description":"Winning Contribution of Michael Schwabe and David Lassig to BWI Data Analytics Hackathon 2020 in the Category Cyber Security. Proof of Concept Intrusion Detection using Zeek with selfmade MachineLearning in a nice WebApp.","archived":false,"fork":false,"pushed_at":"2021-03-03T06:39:46.000Z","size":110967,"stargazers_count":9,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-05T14:12:09.071Z","etag":null,"topics":["circleci","cloudformation","cyber-security","dash","docker-container","intrusion-detection","keras","kubernetes","machine-learning","plotly","python","scikit-learn","tensorflow","zeek"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/herrfeder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-20T05:39:37.000Z","updated_at":"2024-11-30T14:15:59.000Z","dependencies_parsed_at":"2022-08-13T09:30:51.533Z","dependency_job_id":null,"html_url":"https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herrfeder%2FAI_Cybersecurity_IDS_PoC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herrfeder%2FAI_Cybersecurity_IDS_PoC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herrfeder%2FAI_Cybersecurity_IDS_PoC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herrfeder%2FAI_Cybersecurity_IDS_PoC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/herrfeder","download_url":"https://codeload.github.com/herrfeder/AI_Cybersecurity_IDS_PoC/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240045016,"owners_count":19739185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["circleci","cloudformation","cyber-security","dash","docker-container","intrusion-detection","keras","kubernetes","machine-learning","plotly","python","scikit-learn","tensorflow","zeek"],"created_at":"2024-10-14T19:13:31.506Z","updated_at":"2025-02-23T02:30:31.202Z","avatar_url":"https://github.com/herrfeder.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/herrfeder/AI_Cybersecurity_IDS_PoC.svg?style=svg)](https://app.circleci.com/pipelines/github/herrfeder/AI_Cybersecurity_IDS_PoC/86/workflows/1c2fe7ae-4c19-412a-80f4-9c6b9cf2139a)\n\n# AI_Cybersecurity_IDS_PoC \nand \n# Davids Udacity CloudDevOps Nanodegree Capstone Project\n\n  * Winning Solution of BWI Data Analytics Hackathon 2020\n  * CloudDevOps Pipeline with Green-Blue-Deployment for Davids Udacity CloudDevOps Nanodegree Capstone Project\n\n![bwi_hackathon_badge](https://abload.de/img/bwi_dataanalyticshack7ujy4.png)\n\n\n## App Screenshots\n\n  * (as App is running on privately-owned real Internet-connected Infrastructure IPs are blurred)\n\n| Monitoring Dashboard | Model Performance | Anomaly Training | Application of Models |\n|--------------------------------------|--------------------------------------|--------------------------------------|--------------------------------------|\n| ![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/screenshots/analysis_dashboard.png) | ![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/screenshots/model_performance.png) | ![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/screenshots/train_anomaly.png) | ![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/screenshots/apply_model.png) |\n\n\n\n## Concept\n\n  * unfortunately only in german :/\n\n![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/concept/pitch_final.png)\n\n\n## Features\n\n  * Live-updating Webapp with DataPipeline from live running Zeek-Logs\n    * extensive and easily extentable Monitoring Dashboard\n  * Application of Neural Net and Random Forest models trained on pretrained labelled data against live Zeek logs\n  * Training of Anomaly Detection using IsolationForest can be triggered during Runtime\n\n## Content\n\n  * [analysis](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/tree/main/analysis) contains all stuff Michael did for \n    * exploring the used labelled data from [UNSW-NB15 Datasets](https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-NB15-Datasets/)\n    * checking out the performance of different models (mainly Random Forest and Neural Nets)\n    * train and optimize the best model approaches using [keras-tuner](https://github.com/keras-team/keras-tuner)\t\n\n  * [app](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/tree/main/app) contains all stuff David did for\n    * creating the live-updating Datapipeline using [zeek](https://github.com/zeek) logs\n      * parsing them with an tinkered version of [ParseZeekLogs](https://github.com/dgunter/ParseZeekLogs) for enabling continuously feeding the logs into the pipeline\n      * and [pygtail](https://github.com/bgreenlee/pygtail) for also continuously feeding the logs into the pipeline\n    * creating Webapp using [plotly](https://github.com/plotly) and [Dash](https://github.com/plotly/dash)\n    * Implementing live trained Anomaly Detection using Isolation Forest from [scikit-learn](https://github.com/scikit-learn/scikit-learn)  \n\n\n## Installation/Deployment (CloudDevOps Nanodegree Part)\n\n| CircleCI Branch CI/CD Pipeline | CircleCI Main CI/CD Pipeline |\n|--------------------------------------|--------------------------------------|\n| ![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/screenshots/capstone_broai_branch_pipeline.png) | ![](https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC/raw/main/screenshots/capstone_broai_main_pipeline.png) |\n\n\n\n### Local Docker-Compose Deployment\n\n\n1. Clone the repository:\n    ```bash\n    git clone https://github.com/herrfeder/AI_Cybersecurity_IDS_PoC.git\n    ```\n\n2. Go into Deploy Folder and `run_compose.sh` to run `file`-based or `kafka`-based Stack:\n    ```\n    deploy/run_compose.sh kafka\n    # OR\n    deploy/run_compose.sh file\n    ```\n\n  * first run will take very long because Docker Containers will be build locally and the zeek compilation and Kafka Plugin Install will take a while \n\n3. Go to http://127.0.0.1:8050/\n\n\n### Local Kubernetes Deployment\n\n1. You need to build the previous Compose-based stack at least once and upload the resulting Docker Container using the `upload-docker.sh` script or you relying on my public-built Container:\n  * zeek_kafka https://hub.docker.com/repository/docker/herrfeder/zeek_kafka (already in k8s Configs)\n  * broai https://hub.docker.com/repository/docker/herrfeder/broai (already in k8s Configs)    \n    \n2. You have to prepare and start minikube and run `run_kube_local.sh`:    \n    ```bash\n    cd deploy\n    ./run_kube_local.sh file\n    # OR (you can run booth as well)\n    ./run_kube_local.sh file \n    ```\n\n3. Now add local Ingress Rule to reach the broai endpoint:\n    ```bash\n    kubectl apply -f broai_kubernetes/ingress-local-service.yaml\n    # Check now these ingress service with\n    kubectl get svc\n    ```\n\n4. Now add `green.broai` and `blue.broai` with your minikube IP to your `/etc/hosts` and visit this domains. \n\n\n### AWS Kubernetes Deployment\n\n1. You need to build the previous Compose-based stack at least once and upload the resulting Docker Container using the `upload-docker.sh` script or you relying on my public-built Container:\n  * zeek_kafka https://hub.docker.com/repository/docker/herrfeder/zeek_kafka (already in k8s Configs)\n  * broai https://hub.docker.com/repository/docker/herrfeder/broai (already in k8s Configs)    \n\n2. Install `aws-cli` and deploy the Network and Cluster Requirements with the provided AWS Cloudformation Scripts:\n    ```bash\n    cd .circleci\n\n    scripts/push_cloudformation_stack.sh broainetwork cloudformation/network.yaml \u003cyour individual id\u003e\n    scripts/push_cloudformation_stack.sh broaicluster cloudformation/cluster.yaml \u003cyour individual id\u003e\n    ```\n \n3. Get Access Token to acess your AWS EKS Cluster with kubectl:\n    ```bash\n    cd deploy\n\n    mkdir .kube\n    aws eks --region us-west-2 update-kubeconfig --kubeconfig .kube/config-aws --name AWSK8SCluster\n    ``` \n\n4. Deploy Kubernetes Manifests:\n    ```bash\n    ./run_kube_aws.sh\n    ```\n    \n4. Go to http://127.0.0.1:8050/\n\n\n5. Wait for finishing and check with `kubectl --kubeconfig .kube/config-aws get svc` the resulting Loadbalancer Hostnames and access them. :)\n\n\n## TODO\n\n  * replacing filebased Datapipeline by Apache Kafka feed (DONE in scope of Davids Udacity CloudDevOps Nanodegree Capstone Project)\n    * faster feeding into webapp\n    * more elegant data management\n  * also enabling Random Forest and Neural Net training during runtime\n  * feeding predicted live-data into analysis workflow for automatic re-evaluation and re-training\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fherrfeder%2Fai_cybersecurity_ids_poc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fherrfeder%2Fai_cybersecurity_ids_poc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fherrfeder%2Fai_cybersecurity_ids_poc/lists"}