{"id":23080630,"url":"https://github.com/chen0040/keras-anomaly-detection","last_synced_at":"2025-04-06T11:10:25.043Z","repository":{"id":85884574,"uuid":"115834868","full_name":"chen0040/keras-anomaly-detection","owner":"chen0040","description":"Anomaly detection implemented in Keras","archived":false,"fork":false,"pushed_at":"2018-04-01T04:37:46.000Z","size":72107,"stargazers_count":375,"open_issues_count":5,"forks_count":155,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-03-30T10:07:33.307Z","etag":null,"topics":["anomaly-detection","bidirectonal-lstm","convolutional-neural-networks","keras","lstm","recurrent-neural-networks"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chen0040.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-31T01:14:26.000Z","updated_at":"2025-02-26T08:23:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"f0959e1c-bbee-4fa1-b3de-f3f1381c6772","html_url":"https://github.com/chen0040/keras-anomaly-detection","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fkeras-anomaly-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fkeras-anomaly-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fkeras-anomaly-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chen0040%2Fkeras-anomaly-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chen0040","download_url":"https://codeload.github.com/chen0040/keras-anomaly-detection/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247471521,"owners_count":20944158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anomaly-detection","bidirectonal-lstm","convolutional-neural-networks","keras","lstm","recurrent-neural-networks"],"created_at":"2024-12-16T13:15:53.787Z","updated_at":"2025-04-06T11:10:25.024Z","avatar_url":"https://github.com/chen0040.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# keras-anomaly-detection\n\nAnomaly detection implemented in Keras\n\nThe source codes of the recurrent, convolutional and feedforward networks auto-encoders for anomaly detection can be found in\n[keras_anomaly_detection/library/convolutional.py](keras_anomaly_detection/library/convolutional.py) and\n[keras_anomaly_detection/library/recurrent.py](keras_anomaly_detection/library/recurrent.py) and\n[keras_anomaly_detection/library/feedforward.py](keras_anomaly_detection/library/feedforward.py)\n\nThe the anomaly detection is implemented using auto-encoder with convolutional, feedforward, and recurrent networks and can be applied\nto:\n\n* timeseries data to detect timeseries time windows that have anomaly pattern\n    * LstmAutoEncoder in [keras_anomaly_detection/library/recurrent.py](keras_anomaly_detection/library/recurrent.py)\n    * Conv1DAutoEncoder in [keras_anomaly_detection/library/convolutional.py](keras_anomaly_detection/library/convolutional.py)\n    * CnnLstmAutoEncoder in [keras_anomaly_detection/library/recurrent.py](keras_anomaly_detection/library/recurrent.py)\n    * BidirectionalLstmAutoEncoder in [keras_anomaly_detection/library/recurrent.py](keras_anomaly_detection/library/recurrent.py)\n* structured data (i.e., tabular data) to detect anomaly in data records\n    * Conv1DAutoEncoder in [keras_anomaly_detection/library/convolutional.py](keras_anomaly_detection/library/convolutional.py)\n    * FeedforwardAutoEncoder in [keras_anomaly_detection/library/feedforward.py](keras_anomaly_detection/library/feedforward.py)\n\n# Usage\n\n### Detect Anomaly within the ECG Data\n\nThe sample codes can be found in the [demo/ecg_demo](demo/ecg_demo).\n\nThe following sample codes show how to fit and detect anomaly using Conv1DAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom keras_anomaly_detection.library.plot_utils import visualize_reconstruction_error\nfrom keras_anomaly_detection.library.convolutional import Conv1DAutoEncoder\n\n\ndef main():\n    data_dir_path = './data'\n    model_dir_path = './models'\n\n    # ecg data in which each row is a temporal sequence data of continuous values\n    ecg_data = pd.read_csv(data_dir_path + '/ecg_discord_test.csv', header=None)\n    print(ecg_data.head())\n    ecg_np_data = ecg_data.as_matrix()\n    scaler = MinMaxScaler()\n    ecg_np_data = scaler.fit_transform(ecg_np_data)\n\n    print(ecg_np_data.shape)\n\n    ae = Conv1DAutoEncoder()\n\n    # fit the data and save model into model_dir_path\n    ae.fit(ecg_np_data[:23, :], model_dir_path=model_dir_path, estimated_negative_sample_ratio=0.9)\n\n    # load back the model saved in model_dir_path detect anomaly\n    ae.load_model(model_dir_path)\n    anomaly_information = ae.anomaly(ecg_np_data[:23, :])\n    reconstruction_error = []\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        print('# ' + str(idx) + ' is ' + ('abnormal' if is_anomaly else 'normal') + ' (dist: ' + str(dist) + ')')\n        reconstruction_error.append(dist)\n\n    visualize_reconstruction_error(reconstruction_error, ae.threshold)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nThe following sample codes show how to fit and detect anomaly using LstmAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom keras_anomaly_detection.library.plot_utils import visualize_reconstruction_error\nfrom keras_anomaly_detection.library.recurrent import LstmAutoEncoder\n\n\ndef main():\n    data_dir_path = './data'\n    model_dir_path = './models'\n    ecg_data = pd.read_csv(data_dir_path + '/ecg_discord_test.csv', header=None)\n    print(ecg_data.head())\n    ecg_np_data = ecg_data.as_matrix()\n    scaler = MinMaxScaler()\n    ecg_np_data = scaler.fit_transform(ecg_np_data)\n    print(ecg_np_data.shape)\n\n    ae = LstmAutoEncoder()\n\n    # fit the data and save model into model_dir_path\n    ae.fit(ecg_np_data[:23, :], model_dir_path=model_dir_path, estimated_negative_sample_ratio=0.9)\n\n    # load back the model saved in model_dir_path detect anomaly\n    ae.load_model(model_dir_path)\n    anomaly_information = ae.anomaly(ecg_np_data[:23, :])\n    reconstruction_error = []\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        print('# ' + str(idx) + ' is ' + ('abnormal' if is_anomaly else 'normal') + ' (dist: ' + str(dist) + ')')\n        reconstruction_error.append(dist)\n\n    visualize_reconstruction_error(reconstruction_error, ae.threshold)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nThe following sample codes show how to fit and detect anomaly using CnnLstmAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom keras_anomaly_detection.library.plot_utils import visualize_reconstruction_error\nfrom keras_anomaly_detection.library.recurrent import CnnLstmAutoEncoder\n\n\ndef main():\n    data_dir_path = './data'\n    model_dir_path = './models'\n    ecg_data = pd.read_csv(data_dir_path + '/ecg_discord_test.csv', header=None)\n    print(ecg_data.head())\n    ecg_np_data = ecg_data.as_matrix()\n    scaler = MinMaxScaler()\n    ecg_np_data = scaler.fit_transform(ecg_np_data)\n    print(ecg_np_data.shape)\n\n    ae = CnnLstmAutoEncoder()\n\n    # fit the data and save model into model_dir_path\n    ae.fit(ecg_np_data[:23, :], model_dir_path=model_dir_path, estimated_negative_sample_ratio=0.9)\n\n    # load back the model saved in model_dir_path detect anomaly\n    ae.load_model(model_dir_path)\n    anomaly_information = ae.anomaly(ecg_np_data[:23, :])\n    reconstruction_error = []\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        print('# ' + str(idx) + ' is ' + ('abnormal' if is_anomaly else 'normal') + ' (dist: ' + str(dist) + ')')\n        reconstruction_error.append(dist)\n\n    visualize_reconstruction_error(reconstruction_error, ae.threshold)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nThe following sample codes show how to fit and detect anomaly using BidirectionalLstmAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom keras_anomaly_detection.library.plot_utils import visualize_reconstruction_error\nfrom keras_anomaly_detection.library.recurrent import BidirectionalLstmAutoEncoder\n\n\ndef main():\n    data_dir_path = './data'\n    model_dir_path = './models'\n    ecg_data = pd.read_csv(data_dir_path + '/ecg_discord_test.csv', header=None)\n    print(ecg_data.head())\n    ecg_np_data = ecg_data.as_matrix()\n    scaler = MinMaxScaler()\n    ecg_np_data = scaler.fit_transform(ecg_np_data)\n    print(ecg_np_data.shape)\n\n    ae = BidirectionalLstmAutoEncoder()\n\n    # fit the data and save model into model_dir_path\n    ae.fit(ecg_np_data[:23, :], model_dir_path=model_dir_path, estimated_negative_sample_ratio=0.9)\n\n    # load back the model saved in model_dir_path detect anomaly\n    ae.load_model(model_dir_path)\n    anomaly_information = ae.anomaly(ecg_np_data[:23, :])\n    reconstruction_error = []\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        print('# ' + str(idx) + ' is ' + ('abnormal' if is_anomaly else 'normal') + ' (dist: ' + str(dist) + ')')\n        reconstruction_error.append(dist)\n\n    visualize_reconstruction_error(reconstruction_error, ae.threshold)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nThe following sample codes show how to fit and detect anomaly using FeedForwardAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.preprocessing import MinMaxScaler\nfrom keras_anomaly_detection.library.plot_utils import visualize_reconstruction_error\nfrom keras_anomaly_detection.library.feedforward import FeedForwardAutoEncoder\n\n\ndef main():\n    data_dir_path = './data'\n    model_dir_path = './models'\n\n    # ecg data in which each row is a temporal sequence data of continuous values\n    ecg_data = pd.read_csv(data_dir_path + '/ecg_discord_test.csv', header=None)\n    print(ecg_data.head())\n    ecg_np_data = ecg_data.as_matrix()\n    scaler = MinMaxScaler()\n    ecg_np_data = scaler.fit_transform(ecg_np_data)\n\n    print(ecg_np_data.shape)\n\n    ae = FeedForwardAutoEncoder()\n\n    # fit the data and save model into model_dir_path\n    ae.fit(ecg_np_data[:23, :], model_dir_path=model_dir_path, estimated_negative_sample_ratio=0.9)\n\n    # load back the model saved in model_dir_path detect anomaly\n    ae.load_model(model_dir_path)\n    anomaly_information = ae.anomaly(ecg_np_data[:23, :])\n    reconstruction_error = []\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        print('# ' + str(idx) + ' is ' + ('abnormal' if is_anomaly else 'normal') + ' (dist: ' + str(dist) + ')')\n        reconstruction_error.append(dist)\n\n    visualize_reconstruction_error(reconstruction_error, ae.threshold)\n\n\nif __name__ == '__main__':\n    main()\n```\n\n# Detect Fraud in Credit Card Transaction\n\nThe sample codes can be found in the [demo/credit_card_demo](demo/credit_card_demo).\n\nThe credit card sample data is from [this repo](https://github.com/curiousily/Credit-Card-Fraud-Detection-using-Autoencoders-in-Keras/blob/master/fraud_detection.ipynb)\n\nBelow is the sample code using FeedforwardAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\nfrom keras_anomaly_detection.library.feedforward import FeedForwardAutoEncoder\nfrom keras_anomaly_detection.demo.credit_card_demo.unzip_utils import unzip\nfrom keras_anomaly_detection.library.plot_utils import plot_confusion_matrix, plot_training_history, visualize_anomaly\nfrom keras_anomaly_detection.library.evaluation_utils import report_evaluation_metrics\nimport numpy as np\n\nDO_TRAINING = False\n\n\ndef preprocess_data(csv_data):\n    credit_card_data = csv_data.drop(labels=['Class', 'Time'], axis=1)\n    credit_card_data['Amount'] = StandardScaler().fit_transform(credit_card_data['Amount'].values.reshape(-1, 1))\n    # print(credit_card_data.head())\n    credit_card_np_data = credit_card_data.as_matrix()\n    y_true = csv_data['Class'].as_matrix()\n    return credit_card_np_data, y_true\n\n\ndef main():\n    seed = 42\n    np.random.seed(seed)\n\n    data_dir_path = './data'\n    model_dir_path = './models'\n\n    unzip(data_dir_path + '/creditcardfraud.zip', data_dir_path)\n    csv_data = pd.read_csv(data_dir_path + '/creditcard.csv')\n    estimated_negative_sample_ratio = 1 - csv_data['Class'].sum() / csv_data['Class'].count()\n    print(estimated_negative_sample_ratio)\n    X, Y = preprocess_data(csv_data)\n    print(X.shape)\n\n    ae = FeedForwardAutoEncoder()\n\n    training_history_file_path = model_dir_path + '/' + FeedForwardAutoEncoder.model_name + '-history.npy'\n    # fit the data and save model into model_dir_path\n    epochs = 100\n    history = None\n    if DO_TRAINING:\n        history = ae.fit(X, model_dir_path=model_dir_path,\n                         estimated_negative_sample_ratio=estimated_negative_sample_ratio,\n                         nb_epoch=epochs,\n                         random_state=seed)\n        np.save(training_history_file_path, history)\n    else:\n        history = np.load(training_history_file_path).item()\n\n    # load back the model saved in model_dir_path\n    ae.load_model(model_dir_path)\n    # detect anomaly for the test data\n    Ypred = []\n    _, Xtest, _, Ytest = train_test_split(X, Y, test_size=0.2, random_state=seed)\n    reconstruction_error = []\n    adjusted_threshold = 14\n    anomaly_information = ae.anomaly(Xtest, adjusted_threshold)\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        predicted_label = 1 if is_anomaly else 0\n        Ypred.append(predicted_label)\n        reconstruction_error.append(dist)\n\n    report_evaluation_metrics(Ytest, Ypred)\n    plot_training_history(history)\n    visualize_anomaly(Ytest, reconstruction_error, adjusted_threshold)\n    plot_confusion_matrix(Ytest, Ypred)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nThe sample code below uses Conv1DAutoEncoder:\n\n```python\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\nfrom keras_anomaly_detection.library.convolutional import Conv1DAutoEncoder\nfrom keras_anomaly_detection.demo.credit_card_demo.unzip_utils import unzip\nfrom keras_anomaly_detection.library.plot_utils import plot_confusion_matrix, plot_training_history, visualize_anomaly\nfrom keras_anomaly_detection.library.evaluation_utils import report_evaluation_metrics\nimport numpy as np\nimport os\n\nDO_TRAINING = False\n\n\ndef preprocess_data(csv_data):\n    credit_card_data = csv_data.drop(labels=['Class', 'Time'], axis=1)\n    credit_card_data['Amount'] = StandardScaler().fit_transform(credit_card_data['Amount'].values.reshape(-1, 1))\n    # print(credit_card_data.head())\n    credit_card_np_data = credit_card_data.as_matrix()\n    y_true = csv_data['Class'].as_matrix()\n    return credit_card_np_data, y_true\n\n\ndef main():\n    seed = 42\n    np.random.seed(seed)\n\n    data_dir_path = './data'\n    model_dir_path = './models'\n\n    unzip(data_dir_path + '/creditcardfraud.zip', data_dir_path)\n    csv_data = pd.read_csv(data_dir_path + '/creditcard.csv')\n    estimated_negative_sample_ratio = 1 - csv_data['Class'].sum() / csv_data['Class'].count()\n    print(estimated_negative_sample_ratio)\n    X, Y = preprocess_data(csv_data)\n    print(X.shape)\n\n    ae = Conv1DAutoEncoder()\n\n    training_history_file_path = model_dir_path + '/' + Conv1DAutoEncoder.model_name + '-history.npy'\n    # fit the data and save model into model_dir_path\n    epochs = 10\n    history = None\n    if DO_TRAINING:\n        history = ae.fit(X, model_dir_path=model_dir_path,\n                         estimated_negative_sample_ratio=estimated_negative_sample_ratio,\n                         epochs=epochs)\n        np.save(training_history_file_path, history)\n    elif os.path.exists(training_history_file_path):\n        history = np.load(training_history_file_path).item()\n\n    # load back the model saved in model_dir_path\n    ae.load_model(model_dir_path)\n    # detect anomaly for the test data\n    Ypred = []\n    _, Xtest, _, Ytest = train_test_split(X, Y, test_size=0.2, random_state=seed)\n    reconstruction_error = []\n    adjusted_threshold = 10\n    anomaly_information = ae.anomaly(Xtest, adjusted_threshold)\n    for idx, (is_anomaly, dist) in enumerate(anomaly_information):\n        predicted_label = 1 if is_anomaly else 0\n        Ypred.append(predicted_label)\n        reconstruction_error.append(dist)\n\n    report_evaluation_metrics(Ytest, Ypred)\n    plot_training_history(history)\n    visualize_anomaly(Ytest, reconstruction_error, adjusted_threshold)\n    plot_confusion_matrix(Ytest, Ypred)\n\n\nif __name__ == '__main__':\n    main()\n\n```\n\n\n# Note\n\nThere is also an autoencoder from H2O for timeseries anomaly detection in \n[demo/h2o_ecg_pulse_detection.py](demo/ecg_demo/h2o_ecg_pulse_detection.py)\n\n### Configure to run on GPU on Windows\n\n* Step 1: Change tensorflow to tensorflow-gpu in requirements.txt and install tensorflow-gpu\n* Step 2: Download and install the [CUDA® Toolkit 9.0](https://developer.nvidia.com/cuda-90-download-archive) (Please note that\ncurrently CUDA® Toolkit 9.1 is not yet supported by tensorflow, therefore you should download CUDA® Toolkit 9.0)\n* Step 3: Download and unzip the [cuDNN 7.0.4 for CUDA@ Toolkit 9.0](https://developer.nvidia.com/cudnn) and add the\nbin folder of the unzipped directory to the $PATH of your Windows environment \n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchen0040%2Fkeras-anomaly-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchen0040%2Fkeras-anomaly-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchen0040%2Fkeras-anomaly-detection/lists"}