{"id":19174621,"url":"https://github.com/equalitie/learn2ban","last_synced_at":"2025-05-07T18:20:55.940Z","repository":{"id":14407205,"uuid":"17117969","full_name":"equalitie/learn2ban","owner":"equalitie","description":"Open source machine learning DDOS detection tool","archived":false,"fork":false,"pushed_at":"2015-11-23T21:13:09.000Z","size":95,"stargazers_count":137,"open_issues_count":0,"forks_count":34,"subscribers_count":27,"default_branch":"master","last_synced_at":"2025-04-20T01:33:01.511Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/equalitie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-02-23T21:49:29.000Z","updated_at":"2025-04-11T00:42:24.000Z","dependencies_parsed_at":"2022-08-19T01:00:59.579Z","dependency_job_id":null,"html_url":"https://github.com/equalitie/learn2ban","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Flearn2ban","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Flearn2ban/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Flearn2ban/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Flearn2ban/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/equalitie","download_url":"https://codeload.github.com/equalitie/learn2ban/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931812,"owners_count":21827170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T10:18:32.798Z","updated_at":"2025-05-07T18:20:55.917Z","avatar_url":"https://github.com/equalitie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Learn2ban\n=========\nOpen source machine learning DDOS detection tool\n\nCopyright 2013 eQualit.ie\n\nLearn2ban is free software: you can redistribute it and/or modify\nit under the terms of the GNU Affero General Public License as\npublished by the Free Software Foundation, either version 3 of the\nLicense, or (at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU Affero General Public License for more details.\n\nYou should have received a copy of the GNU Affero General Public License\nalong with this program.  If not, see `\u003chttp://www.gnu.org/licenses/\u003e`.\n\nInstallation\n============\n\nThe following libraries should be installed\n\n    [sudo] apt-get install libmysqlclient-dev\n    [sudo] apt-get install build-essential python-dev python-numpy python-setuptools python-scipy libatlas-dev\n    [sudo] apt-get install python-matplotlib\n    easy_install pip\n\nInstall required packages\n\n    pip install -r requirements.txt\n\nInitialize Learn2ban training database\n\n    python src/initialise_db.py\n    \nTesting:\n--------\nRun python unit tests in the Learn2ban/src/test/ directory to ensure functionality.\n\nConfiguration\n=============\n\nAccess to mysql server\n----------------------\nUser needs to enter the access detail for a mysql server in config/train2ban.cfg. For example:\n\n    db_user = root\n    db_password = thisisapassword\n    db_host = mydb.myserver.com\n    db_name = learn2ban\n    config_profile = myconfig\n\nThen user need to run initialise_db.py\n\n    python initialise_db.py\n\nTo create the database. User then is required to make a record in config table with profile name equal to config_profile (myconfig in this example) and enter the relevant directories in the table.\n\nRegex filters\n-------------\n\nIn order to annotate input logs, Learn2ban uses the fail2ban regex filtering system to mark IP addresses as malicious or legitimate. The regex rules to apply can be added to regex_filter table in learn2ban database\n\nTraining data\n-------------\nThe data from which the Learn2ban SVM model will be constructed should be placed in the directory defined in the profile or entered in absolute path in experiment table if the profile asks for absolute path.\n\nRunning Learn2ban Experiments\n=============================\n\nLearn2ban is currently designed to run in an experimental mode to allow users to create multiple variations of models, based on their training data, and to easily analyze the efficacy and accuracy of these models.\n\nUser needs to enter the log file names in logs table, assign the regexes which identifies the bots in the log in regex_assignment table. User then design an experiment in experiments table, and assign the log to it in experiment_logs. \n\nTo run a configured learn2ban experiment, enable the experiment in experiments table and execute\n\n    python src/analysis/experimentor.py\n\nLearn2ban model feature set\n===========================\nIn order to classify requesting IP addresses as legitimate or malicious the Learn2ban SVM model takes into account the following set of features derived from HTTP log data.\n\nThese features are implemented at Learn2ban/src/features.\n\n* average_request_interval - this feature considers the behaviour of the requester in terms of the average number of request made within a given interval. This is essentially the frequency with which a requester attempts to access a given host. It takes into account the requests as whole not merely in terms of a single page.\n* cycling_user_agent - a common attack for DDOS Botnets is to change user agent repeatedly during an attack. This strategy can be quite effective against even the most generalised regex rules. If the IP never repeats its user agent then rules put in place to block requesters using obscure user agents will still be subverted. In the context of a real human user, or even a spider bot, user agent rotation is highly aberrant.\n* html_to_image_ratio - This feature considers the type of content that is being requested. It considers if a requester is only retrieving HTML content but no ancillary data such as images, css or javascript files.\n* variance_request_interval - While many DDOS attacks use a very simplistic brute force approach, some have incorporated a slightly more sophisticated approach by making burst requests in order to avoid being blocked by simple rules which allow only a certain number of requests within a time frame.\n* payload_size_average - this feature looks at the size of the content that a requester is retrieving.\n* HTTP_response_code_rate - Considers http response rate, primarily looking for error codes that may signal a cache busting attack.\n* request_depth - Normal site users with commonly browse beyond the home page of a given site. Human users interaction with a website will resemble browsing more than that of a botnet.\n* request_depth_std - As an adjunct to request depth, this feature considers the standard deviation of a bot's request.\n* session_length - This feature also elucidates general behaviour considering the requester's interaction with a given sight in terms of session time.\n* percentage_consecutive_requests - To further elucidate the requester's interaction with a given site we additionally consider how many of the requests made were consecutive as another window onto frequency.\n\nAdding new features\n===================\n\nIt is possible to easily extend Learn2ban's feature set by inheriting from the prototype feature at Lear2ban/src/features/learn2ban_feature.py.\n\nThe new feature needs to register the log data index of the feature under consideration and implement the compute() method which will return the feature value.\n\nThis project forms part of the [Deflect](https://deflect.ca) project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalitie%2Flearn2ban","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fequalitie%2Flearn2ban","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalitie%2Flearn2ban/lists"}