{"id":18123274,"url":"https://github.com/jay-johnson/antinex-core","last_synced_at":"2025-04-14T23:03:10.207Z","repository":{"id":57410639,"uuid":"124046877","full_name":"jay-johnson/antinex-core","owner":"jay-johnson","description":"Network exploit detection using highly accurate pre-trained deep neural networks with Celery + Keras + Tensorflow + Redis","archived":false,"fork":false,"pushed_at":"2018-12-07T20:06:21.000Z","size":389,"stargazers_count":21,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-28T11:05:02.711Z","etag":null,"topics":["ai-security","anti-nex","artificial-intelligence","celery","docker","jupyter","keras","redis","tensorflow"],"latest_commit_sha":null,"homepage":"http://antinex.readthedocs.io/en/latest/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jay-johnson.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-06T08:35:27.000Z","updated_at":"2024-11-09T08:43:56.000Z","dependencies_parsed_at":"2022-08-27T22:01:53.903Z","dependency_job_id":null,"html_url":"https://github.com/jay-johnson/antinex-core","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fantinex-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fantinex-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fantinex-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fantinex-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jay-johnson","download_url":"https://codeload.github.com/jay-johnson/antinex-core/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246301963,"owners_count":20755512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-security","anti-nex","artificial-intelligence","celery","docker","jupyter","keras","redis","tensorflow"],"created_at":"2024-11-01T07:08:55.994Z","updated_at":"2025-03-30T09:32:42.545Z","avatar_url":"https://github.com/jay-johnson.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"AntiNex Core\n============\n\nAutomating network exploit detection using highly accurate pre-trained deep neural networks.\n\nAs of 2018-03-12, the core can repeatedly predict attacks on Django, Flask, React + Redux, Vue, and Spring application servers by training using the pre-recorded `AntiNex datasets`_ with cross validation scores above **~99.8%** with automated scaler normalization.\n\n.. image:: https://travis-ci.org/jay-johnson/antinex-core.svg?branch=master\n    :target: https://travis-ci.org/jay-johnson/antinex-core\n\nAccuracy + Training + Cross Validation in a Jupyter Notebook\n------------------------------------------------------------\n\nhttps://github.com/jay-johnson/antinex-core/blob/master/docker/notebooks/AntiNex-Protecting-Django.ipynb\n\nUsing a Pre-Trained Deep Neural Network in a Jupyter Notebook\n-------------------------------------------------------------\n\nhttps://github.com/jay-johnson/antinex-core/blob/master/docker/notebooks/AntiNex-Using-Pre-Trained-Deep-Neural-Networks-For-Defense.ipynb\n\n.. _AntiNex datasets: https://github.com/jay-johnson/antinex-datasets\n\nOverview\n--------\n\nThe core is a Celery worker pool for processing training and prediction requests for deep neural networks to detect network exploits (Nex) using Keras and Tensorflow in near real-time. Internally each worker manages a buffer of pre-trained models identified by the ``label`` from the initial training request. Once trained, a model can be used for rapid prediction testing provided the same ``label`` name is used on the prediction request. Models can also be re-trained by using the training api with the same ``label``. While the initial focus is on network exploits, the repository also includes mock stock data for demonstrating running a worker pool to quickly predict regression data (like stock prices) with many, pre-trained deep neural networks.\n\nThis repository is a standalone training and prediction worker pool that is decoupled from the AntiNex REST API:\n\nhttps://github.com/jay-johnson/train-ai-with-django-swagger-jwt\n\nAntiNex Stack Status\n--------------------\n\nAntiNex Core Worker is part of the AntiNex stack:\n\n.. list-table::\n   :header-rows: 1\n\n   * - Component\n     - Build\n     - Docs Link\n     - Docs Build\n   * - `REST API \u003chttps://github.com/jay-johnson/train-ai-with-django-swagger-jwt\u003e`__\n     - .. image:: https://travis-ci.org/jay-johnson/train-ai-with-django-swagger-jwt.svg?branch=master\n           :alt: Travis Tests\n           :target: https://travis-ci.org/jay-johnson/train-ai-with-django-swagger-jwt.svg\n     - `Docs \u003chttp://antinex.readthedocs.io/en/latest/\u003e`__\n     - .. image:: https://readthedocs.org/projects/antinex/badge/?version=latest\n           :alt: Read the Docs REST API Tests\n           :target: https://readthedocs.org/projects/antinex/badge/?version=latest\n   * - `Core Worker \u003chttps://github.com/jay-johnson/antinex-core\u003e`__\n     - .. image:: https://travis-ci.org/jay-johnson/antinex-core.svg?branch=master\n           :alt: Travis AntiNex Core Tests\n           :target: https://travis-ci.org/jay-johnson/antinex-core.svg\n     - `Docs \u003chttp://antinex-core-worker.readthedocs.io/en/latest/\u003e`__\n     - .. image:: https://readthedocs.org/projects/antinex-core-worker/badge/?version=latest\n           :alt: Read the Docs AntiNex Core Tests\n           :target: http://antinex-core-worker.readthedocs.io/en/latest/?badge=latest\n   * - `Network Pipeline \u003chttps://github.com/jay-johnson/network-pipeline\u003e`__\n     - .. image:: https://travis-ci.org/jay-johnson/network-pipeline.svg?branch=master\n           :alt: Travis AntiNex Network Pipeline Tests\n           :target: https://travis-ci.org/jay-johnson/network-pipeline.svg\n     - `Docs \u003chttp://antinex-network-pipeline.readthedocs.io/en/latest/\u003e`__\n     - .. image:: https://readthedocs.org/projects/antinex-network-pipeline/badge/?version=latest\n           :alt: Read the Docs AntiNex Network Pipeline Tests\n           :target: https://readthedocs.org/projects/antinex-network-pipeline/badge/?version=latest\n   * - `AI Utils \u003chttps://github.com/jay-johnson/antinex-utils\u003e`__\n     - .. image:: https://travis-ci.org/jay-johnson/antinex-utils.svg?branch=master\n           :alt: Travis AntiNex AI Utils Tests\n           :target: https://travis-ci.org/jay-johnson/antinex-utils.svg\n     - `Docs \u003chttp://antinex-ai-utilities.readthedocs.io/en/latest/\u003e`__\n     - .. image:: https://readthedocs.org/projects/antinex-ai-utilities/badge/?version=latest\n           :alt: Read the Docs AntiNex AI Utils Tests\n           :target: http://antinex-ai-utilities.readthedocs.io/en/latest/?badge=latest\n   * - `Client \u003chttps://github.com/jay-johnson/antinex-client\u003e`__\n     - .. image:: https://travis-ci.org/jay-johnson/antinex-client.svg?branch=master\n           :alt: Travis AntiNex Client Tests\n           :target: https://travis-ci.org/jay-johnson/antinex-client.svg\n     - `Docs \u003chttp://antinex-client.readthedocs.io/en/latest/\u003e`__\n     - .. image:: https://readthedocs.org/projects/antinex-client/badge/?version=latest\n           :alt: Read the Docs AntiNex Client Tests\n           :target: https://readthedocs.org/projects/antinex-client/badge/?version=latest\n\nInstall\n-------\n\npip install antinex-core\n\nOptional for Generating Images\n------------------------------\n\nIf you want to generate images please install ``python3-tk`` on Ubuntu.\n\n::\n\n    sudo apt-get install python3-tk\n\nDocker\n------\n\nStart the container for browsing with Jupyter:\n\n::\n\n    # if you do not have docker compose installed, you can try installing it with:\n    # pip install docker-compose\n    cd docker\n    ./start-stack.sh\n\nOpen Jupyter Notebook with Django Deep Neural Network Analysis\n--------------------------------------------------------------\n\nDefault password is: ``admin``\n\nhttp://localhost:8888/notebooks/AntiNex-Protecting-Django.ipynb\n\nView Notebook Presentation Slides\n---------------------------------\n\n#.  Use ``Alt + r`` inside the notebook\n\n#.  Use the non-vertical scolling url: http://localhost:8889/Slides-AntiNex-Protecting-Django.slides.html\n\n#.  Use the non-vertical scolling url: http://localhost:8890/Slides-AntiNex-Using-Pre-Trained-Deep-Neural-Networks-For-Defense.slides.html\n\nRun\n---\n\nPlease make sure redis is running and accessible before starting the core:\n\n::\n\n    redis-cli \n    127.0.0.1:6379\u003e\n\nWith redis running and the antinex-core pip installed in the python 3 runtime, use this command to start the core:\n\n::\n\n    ./run-antinex-core.sh\n\nOr with celery:\n\n::\n\n    celery worker -A antinex_core.antinex_worker -l DEBUG\n\nPublish a Predict Request\n-------------------------\n\nTo train and predict with the new automated scaler-normalized dataset with a 99.8% prediction accuracy for detecting attacks using a wide, two-layer deep neural network with the `AntiNex datasets`_ run the following steps.\n\n.. _AntiNex datasets: https://github.com/jay-johnson/antinex-datasets\n\nClone\n-----\n\nPlease make sure to clone the dataset repo to the pre-configured location:\n\n::\n\n    mkdir -p -m 777 /opt/antinex\n    git clone https://github.com/jay-johnson/antinex-datasets.git /opt/antinex/antinex-datasets\n\nDjango - Train and Predict\n--------------------------\n\n::\n\n    ./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-django-antinex-simple.json\n\nFlask - Train and Predict\n-------------------------\n\n::\n\n    ./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-flask-antinex-simple.json\n\nReact and Redux - Train and Predict\n-----------------------------------\n\n::\n\n    ./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-react-redux-antinex-simple.json\n\nVue - Train and Predict\n-----------------------\n\n::\n\n    ./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-vue-antinex-simple.json\n\nSpring - Train and Predict\n--------------------------\n\n::\n\n    ./antinex_core/scripts/publish_predict_request.py -f training/scaler-full-spring-antinex-simple.json\n\nAccuracy and Prediction Report\n------------------------------\n\nAfter a few minutes the final report will be printed out like:\n\n::\n\n    2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30178 - label_value=1.0 predicted=1 label=attack\n    2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30179 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30180 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30181 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,944 - antinex-prc - INFO - sample=30182 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30183 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30184 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30185 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30186 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30187 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30188 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30189 - label_value=1.0 predicted=1 label=attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30190 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,945 - antinex-prc - INFO - sample=30191 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30192 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30193 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30194 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30195 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30196 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30197 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30198 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,946 - antinex-prc - INFO - sample=30199 - label_value=-1.0 predicted=-1 label=not_attack\n    2018-03-11 23:35:00,947 - antinex-prc - INFO - Full-Django-AntiNex-Simple-Scaler-DNN made predictions=30200 found=30200 accuracy=99.84685430463577\n    2018-03-11 23:35:00,947 - antinex-prc - INFO - Full-Django-AntiNex-Simple-Scaler-DNN - saving model=full-django-antinex-simple-scaler-dnn\n\n\nIf you do not have the datasets cloned locally, you can use the included minimized dataset from the repo:\n\n::\n\n    ./antinex_core/scripts/publish_predict_request.py -f training/scaler-django-antinex-simple.json\n\nPublish a Train Request\n-----------------------\n\n::\n\n    ./antinex_core/scripts/publish_train_request.py\n\nPublish a Regression Prediction Request\n---------------------------------------\n\n::\n\n    ./antinex_core/scripts/publish_regression_predict.py\n\nJSON API\n--------\n\nThe AntiNex core manages a pool of workers that are subscribed to process tasks found in two queues (``webapp.train.requests`` and ``webapp.predict.requests``). Tasks are defined as JSON dictionaries and must have the following structure:\n\n::\n\n    {\n        \"label\": \"Django-AntiNex-Simple-Scaler-DNN\",\n        \"dataset\": \"./tests/datasets/classification/cleaned_attack_scans.csv\",\n        \"apply_scaler\": true,\n        \"ml_type\": \"classification\",\n        \"predict_feature\": \"label_value\",\n        \"features_to_process\": [\n            \"eth_type\",\n            \"idx\",\n            \"ip_ihl\",\n            \"ip_len\",\n            \"ip_tos\",\n            \"ip_version\",\n            \"tcp_dport\",\n            \"tcp_fields_options.MSS\",\n            \"tcp_fields_options.Timestamp\",\n            \"tcp_fields_options.WScale\",\n            \"tcp_seq\",\n            \"tcp_sport\"\n        ],\n        \"ignore_features\": [\n        ],\n        \"sort_values\": [\n        ],\n        \"seed\": 42,\n        \"test_size\": 0.2,\n        \"batch_size\": 32,\n        \"epochs\": 10,\n        \"num_splits\": 2,\n        \"loss\": \"binary_crossentropy\",\n        \"optimizer\": \"adam\",\n        \"metrics\": [\n            \"accuracy\"\n        ],\n        \"histories\": [\n            \"val_loss\",\n            \"val_acc\",\n            \"loss\",\n            \"acc\"\n        ],\n        \"model_desc\": {\n            \"layers\": [\n                {\n                    \"num_neurons\": 250,\n                    \"init\": \"uniform\",\n                    \"activation\": \"relu\"\n                },\n                {\n                    \"num_neurons\": 1,\n                    \"init\": \"uniform\",\n                    \"activation\": \"sigmoid\"\n                }\n            ]\n        },\n        \"label_rules\": {\n            \"labels\": [\n                \"not_attack\",\n                \"not_attack\",\n                \"attack\"\n            ],\n            \"label_values\": [\n                -1,\n                0,\n                1\n            ]\n        },\n        \"version\": 1\n    }\n\nRegression prediction tasks are also supported, and here is an example from an included dataset with mock stock prices:\n\n::\n\n    {\n        \"label\": \"Scaler-Close-Regression\",\n        \"dataset\": \"./tests/datasets/regression/stock.csv\",\n        \"apply_scaler\": true,\n        \"ml_type\": \"regression\",\n        \"predict_feature\": \"close\",\n        \"features_to_process\": [\n            \"high\",\n            \"low\",\n            \"open\",\n            \"volume\"\n        ],\n        \"ignore_features\": [\n        ],\n        \"sort_values\": [\n        ],\n        \"seed\": 7,\n        \"test_size\": 0.2,\n        \"batch_size\": 32,\n        \"epochs\": 50,\n        \"num_splits\": 2,\n        \"loss\": \"mse\",\n        \"optimizer\": \"adam\",\n        \"metrics\": [\n            \"accuracy\"\n        ],\n        \"model_desc\": {\n            \"layers\": [\n                {\n                    \"activation\": \"relu\",\n                    \"init\": \"uniform\",\n                    \"num_neurons\": 200\n                },\n                {\n                    \"activation\": null,\n                    \"init\": \"uniform\",\n                    \"num_neurons\": 1\n                }\n            ]\n        }\n    }\n\nSplunk Environment Variables\n----------------------------\n\nThis repository uses the `Spylunking \u003chttps://github.com/jay-johnson/spylunking\u003e`__ logger that supports publishing logs to Splunk over the authenticated HEC REST API. You can set these environment variables to publish to Splunk:\n\n::\n\n    export SPLUNK_ADDRESS=\"\u003csplunk address host:port\u003e\"\n    export SPLUNK_API_ADDRESS=\"\u003csplunk api address host:port\u003e\"\n    export SPLUNK_USER=\"\u003csplunk username for login\u003e\"\n    export SPLUNK_PASSWORD=\"\u003csplunk password for login\u003e\"\n    export SPLUNK_TOKEN=\"\u003cOptional - username and password will login or you can use a pre-existing splunk token\u003e\"\n    export SPLUNK_INDEX=\"\u003csplunk index\u003e\"\n    export SPLUNK_QUEUE_SIZE=\"\u003cnum msgs allowed in queue - 0=infinite\u003e\"\n    export SPLUNK_RETRY_COUNT=\"\u003cattempts per log to retry publishing\u003e\"\n    export SPLUNK_RETRY_BACKOFF=\"\u003ccooldown in seconds per failed POST\u003e\"\n    export SPLUNK_SLEEP_INTERVAL=\"\u003csleep in seconds per batch\u003e\"\n    export SPLUNK_SOURCE=\"\u003csplunk source\u003e\"\n    export SPLUNK_SOURCETYPE=\"\u003csplunk sourcetype\u003e\"\n    export SPLUNK_TIMEOUT=\"\u003ctimeout in seconds\u003e\"\n    export SPLUNK_DEBUG=\"\u003c1 enable debug|0 off - very verbose logging in the Splunk Publishers\u003e\"\n\nDevelopment\n-----------\n::\n\n    virtualenv -p python3 ~/.venvs/antinexcore \u0026\u0026 source ~/.venvs/antinexcore/bin/activate \u0026\u0026 pip install -e .\n\nTesting\n-------\n\nRun all\n\n::\n\n    python setup.py test\n\nRun a test case\n\n::\n\n    python -m unittest tests.test_train.TestTrain.test_train_antinex_simple_success_retrain\n\nLinting\n-------\n\nflake8 .\n\npycodestyle .\n\nLicense\n-------\n\nApache 2.0 - Please refer to the LICENSE_ for more details\n\n.. _License: https://github.com/jay-johnson/antinex-core/blob/master/LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fantinex-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjay-johnson%2Fantinex-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fantinex-core/lists"}