{"id":22039189,"url":"https://github.com/shivanmathur/anomaly-detection-using-autoencoder","last_synced_at":"2025-03-23T13:13:37.872Z","repository":{"id":253059742,"uuid":"842293993","full_name":"ShivanMathur/Anomaly-Detection-using-AutoEncoder","owner":"ShivanMathur","description":null,"archived":false,"fork":false,"pushed_at":"2024-08-14T05:32:21.000Z","size":1036,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-28T19:17:00.431Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShivanMathur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-14T04:05:44.000Z","updated_at":"2024-08-14T05:33:22.000Z","dependencies_parsed_at":"2024-08-16T23:31:37.246Z","dependency_job_id":null,"html_url":"https://github.com/ShivanMathur/Anomaly-Detection-using-AutoEncoder","commit_stats":null,"previous_names":["shivanmathur/anomaly-detection-using-autoencoder"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShivanMathur%2FAnomaly-Detection-using-AutoEncoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShivanMathur%2FAnomaly-Detection-using-AutoEncoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShivanMathur%2FAnomaly-Detection-using-AutoEncoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShivanMathur%2FAnomaly-Detection-using-AutoEncoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShivanMathur","download_url":"https://codeload.github.com/ShivanMathur/Anomaly-Detection-using-AutoEncoder/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245104530,"owners_count":20561380,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-30T11:09:41.918Z","updated_at":"2025-03-23T13:13:37.823Z","avatar_url":"https://github.com/ShivanMathur.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Anomaly-Detection-using-AutoEncoder\n\nThe project's objective is to utilize the AutoEncoder architecture to detect anomalies in the real production traffic data so that the organization can take appropriate action whenever they encounter such scenarios. \n\n## Dataset Description:\n\n- The dataset has been taken from Yahoo Webscope program. It consists of time-series data with labeled anomalies. The data represents real production traffic of some Yahoo properties. The dataset comprises of three columns, namely:\n  - _timestamp_: in the dataset, it is replaced by integers incremented by 1, which represents 1 hour worth of data;\n  - _value_: value recorded at the corresponding timestamp; and\n  - _is_anomaly_: Boolean indicating if the current value at the given timestamp is an anomaly or not\n- Dataset Link: \u003ca href = \"https://webscope.sandbox.yahoo.com/catalog.php?datatype=s\u0026did=70\"\u003e Yahoo Webscope S5 Dataset\u003c/a\u003e\n\n## Implementation:\n\n- **Data Preprocessing:**\n  - Imported and combined multiple CSV files containing time series data from the A1Benchmark dataset. The dataset was merged into a single DataFrame for easier manipulation and analysis.\n  - Verified the dataset for any missing values to ensure data quality and consistency before proceeding with further steps.\n  - Performed normalization on the dataset using the StandardScaler from Scikit-learn. This scaling ensures that the features have a mean of 0 and a standard deviation of 1. \n\n- **Model Architecture:**\n\n  - Model 1: Dense AutoEncoder\n\n    - This model is a straightforward autoencoder built with fully connected (dense) layers. It includes three encoding layers and three corresponding decoding layers. The model is designed to compress the input data into a smaller latent representation and then reconstruct the original data.\n\n  - Model 2: LSTM-based AutoEncoder\n\n    -  This model utilizes LSTM layers for both the encoder and decoder. It is designed to capture temporal dependencies in sequential data, making it well-suited for time series anomaly detection.\n\n  - Model 3 \u0026 4: Enhanced AutoEncoder with Modified Hyperparameters\n    - An extension of Model 1, this version replaces ReLU with Tanh activation functions and increases the hidden units. The adjustments in activation functions and hidden layer dimensions aim to improve the model's learning capabilities. \n   \n   - Building on Model 3, this architecture replaces Tanh with LeakyReLU activation functions and introduces Dropout layers for regularization. These changes help prevent overfitting and enhance the model's robustness. \n\n## Model Performance:\n- Model 2 with LSTM architecture achieves the highest accuracy (93.93%), followed by Model 3 (93.63%), Model 1 (93.438%) and then Model 4 (93.05%).\n- Model 2 has the highest precision and recall, which indicates that it can correctly identify and capture anomalies. It also shows the highest number of true positives and lowest number of false negatives.\n\n\n### Code:\n\nThe complete project code is available in the \u003ca href = \"https://github.com/ShivanMathur/Anomaly-Detection-using-AutoEncoder/blob/main/Anomaly_Detection_using_AutoEncoder.ipynb\"\u003e Anomaly_Detection_using_AutoEncoder.ipynb \u003c/a\u003e.\n\n### Report:\n\nFor a detailed project report, \u003ca href = \"https://github.com/ShivanMathur/Anomaly-Detection-using-AutoEncoder/blob/main/report.pdf\" \u003e click here\u003c/a\u003e.\n\n### Contact:\nFor questions or feedback, please reach out to me at [shivanmthr18@gmail.com](mailto:shivanmthr18@gmail.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivanmathur%2Fanomaly-detection-using-autoencoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshivanmathur%2Fanomaly-detection-using-autoencoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshivanmathur%2Fanomaly-detection-using-autoencoder/lists"}