{"id":26005433,"url":"https://github.com/wb-az/timeseries-sensor-anomaly-detection","last_synced_at":"2025-10-10T13:04:39.120Z","repository":{"id":169710292,"uuid":"643629413","full_name":"Wb-az/timeseries-sensor-anomaly-detection","owner":"Wb-az","description":"Unsupervised anomaly detection in vibration signal using PyCaret vs BiLSTM","archived":false,"fork":false,"pushed_at":"2025-06-21T17:37:05.000Z","size":45032,"stargazers_count":12,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-21T18:33:04.140Z","etag":null,"topics":["bilstm-model","clustering-algorithm","conover","deep-learning","experimental-design","friedman-test","histogram-based-outlier-score","hypothesis-testing","iforest-model","knn-model","minimum-covariance-determinant","nonparametric-statistics","pairwise-comparison","pycaret-library","pytorch","svm-model","time-series","unsupervides-deep-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Wb-az.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-21T19:06:37.000Z","updated_at":"2025-06-21T17:37:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"806573af-da36-4204-adb4-9ab77e2712b0","html_url":"https://github.com/Wb-az/timeseries-sensor-anomaly-detection","commit_stats":null,"previous_names":["wb-az/sequence-anomaly-detection","wb-az/timeseries-sensor-anomaly-detection"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Wb-az/timeseries-sensor-anomaly-detection","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Ftimeseries-sensor-anomaly-detection","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Ftimeseries-sensor-anomaly-detection/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Ftimeseries-sensor-anomaly-detection/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Ftimeseries-sensor-anomaly-detection/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Wb-az","download_url":"https://codeload.github.com/Wb-az/timeseries-sensor-anomaly-detection/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Ftimeseries-sensor-anomaly-detection/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279004005,"owners_count":26083661,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bilstm-model","clustering-algorithm","conover","deep-learning","experimental-design","friedman-test","histogram-based-outlier-score","hypothesis-testing","iforest-model","knn-model","minimum-covariance-determinant","nonparametric-statistics","pairwise-comparison","pycaret-library","pytorch","svm-model","time-series","unsupervides-deep-learning"],"created_at":"2025-03-05T20:56:41.470Z","updated_at":"2025-10-10T13:04:39.062Z","avatar_url":"https://github.com/Wb-az.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003ch1 align=\"center\"\u003eAnomaly Sensor Detection - IMS Bearing NASA Acoustics and Vibration Database\u003c/h1\u003e\n\n\u003ch2 align=\"center\"\u003ePyCaret vs Custom BiLSTM (Bidirectional LSTM)\u003c/h2\u003e\n\n## Introduction\n\nThis project will use four unsupervised anomaly detection models from Pycaret to detect anomalies in sensor-bearing vibration signals. These models are Decision Tree and Support Vector Machine. In addition, a customed LSTM model will be built using the PyTorch Framework to autoencode and decode the signal input. The data comprises three vibrational sensor readings from the NASA Acoustics and Vibration Database. The datasets contained text files with 1-second vibration signal snapshots (20,480 data points) recorded at 5 and 10 minutes intervals at a sampling rate of 20 kHz. \n\n•\tPycaret is a high-performance, open-source, low-code library for ML modelling. It provides highly-efficient CPU/GPU implementations of ML models for cluster analysis, classification, time series and anomaly detection. Snap ML accelerates ML algorithms through system awareness and offers novel ML algorithms with best-in-class accuracy. Pycaret also automates ML workflows and aims to democratise ML. For more information, please visit [Pycaret](https://pycaret.org/)[1]. \n•\tPyTorch is a machine learning framework used for computer vision and natural language processing applications, initially developed by Meta AI and now part of the Linux Foundation umbrella. For more information, please visit [PyTorch](https://pytorch.org/).\n\nChoi et al. conducted a study of the impact of deep Learning reported on anomaly detection in time-series data [7]. Several works on these datasets have been reported to prevent early anomalies successfully using deep learning architectures [2][3][4][5]. These works followed a semi-supervised approach to detecting failure. However, there is no strong rationale for selecting a specific train data subset. In addition, the models are not challenged against unseen data (another dataset from the three groups. Other studies have produced successful results but only used a tiny fraction of the data [5].\n\n\nIn this work, we will use deep learning architectures, a BiLSTM, to predict anomalies in an unsupervised fashion. The Autoencoder decoder will be trained in all datasets without separating the anomalies portion and evaluated on test dataset 3. In addition, Cluster, Histogram, iForest, KNN, MCD and SVM anomaly detection models will be trained and assessed on the same datasets. Further, the performance of the unsupervised models will be compared [6] using the nonparametric statistics Friedman test and the posthoc test Friedman-Conover.\n\nThe aim of this study is to determine whether PyCaret offers a similar or better performance than Artificial Neural Networks but at low code, therefore, optimising resources.\n\n\n\n## Data\nThe data were sourced from [kaggle](https://www.kaggle.com/datasets/vinayak123tyagi/bearing-dataset) and comprises three datasets of vibrational sensor readings from the NASA Acoustics and Vibration Database. The datasets contained text files with 1-second vibration signal snapshots (20,480 data points) recorded at 5 and 10-minute intervals at a sampling rate of 20 kHz.\n\n## Methods\n\n### Experimental setup PyCaret Models\n\n__Table 1__ Anomaly Models - PyCaret\n\n|ID|Name|Reference|\n|---|---|---|\n|cluster|Clustering-Based Local Outlier|pyod\\.models\\.cblof\\.CBLOF|\n|iforest|Isolation Forest|pyod\\.models\\.iforest\\.IForest|\n|histogram|Histogram-based Outlier Detection|pyod\\.models\\.hbos\\.HBOS|\n|knn|K-Nearest Neighbors Detector|pyod\\.models\\.knn\\.KNN|\n|svm|One-class SVM detector|pyod\\.models\\.ocsvm\\.OCSVM|\n|mcd|Minimum Covariance Determinant|pyod\\.models\\.mcd\\.MCD|\n\n### Experimental Setup for BiLSTM\n\nAll experiments were run for 50 epochs, learning rate of 2 e-4 and a batch size of 32. The architecture used was a configurable Bidirectional-LSTM. For this work, only one layer with 32 hidden units and a dropout of 0.1 was used. The encoder-decoder can be customised to multiple BiLSTM layers.\n\n\n__Table 2__ BILSTM Experiomental Setup Models\n|Exp|Model|Loss|Optim|\n|---|---|---|---|\n|1|bilstm|mae\\_loss|adam|\n|2|bilstm|huber\\_loss|adam|\n|3|bilstm|mae\\_loss|adamw|\n|4|bilstm|huber\\_loss|adamw|\n\nThe results from each model will be further analysed using cluster metrics three metrics, the silhouette coefficient, Calinski-Harabaz and Davies-Boulding indexes.\n\n1. **Silhouette Score** is the mean Silhouette Coefficient for all clusters, which is calculated using the mean intra-cluster distance and the mean nearest-cluster distance. This score is between -1 and 1; the higher the score, the more well-defined and distinct the clusters are.\n\n1. **Calinski-Harabaz Index** is calculated using the between-cluster and within-cluster dispersion to measure the distinctiveness between groups. Like the Silhouette Score, the higher the score, the more well-defined the clusters are. This score has no bound, meaning no ‘acceptable’ or ‘good’ value exists.\n\n3. **Davies-Bouldin Index** is the average similarity of each cluster with its most similar cluster. Unlike the previous two metrics, this score measures the similarity of the clusters, meaning that the lower the score, the better the separation between the clusters. Zero is the lowest possible score. Values closer to zero indicate a better partition. The usage of centroid distance limits the distance metric to Euclidean space.\n\nThe disadvantage of the three metrics is that they generally score higher for convex clusters than other concepts of clusters. Therefore further compare their performance with **non-parametric statistics**.\n\n\n## Results\n\nTables 3 and Figures 2-5 show the anomalies detected for each of the selected PyCaret and BiLST models for the training dataset and the independent test dataset.\n\n__Table 3__: Anomalies detected by model for the training and test datasets.\n\n| Model     | Anomalies - \u003c/br\u003etraining dataset | Anomalies- \u003c/br\u003eTest dataset |\n|-----------|:---------------------------------:|:----------------------------:|\n| Cluster   |                50                 |              -               |\n| Histogram |                50                 |              -               |\n| iforest   |                50                 |             187              |\n| KNN       |                50                 |              -               |\n| MCD       |                50                 |              78              |\n| SVM       |                50                 |              95              |\n| Exp-01    |                99                 |             190              |\n| Exp-02    |                98                 |             191              |\n| Exp-03    |                99                 |             190              |\n| Exp-04    |                104                |             191              |\n\n\n\n### Training\n\n\n*Cluster Metrics*\n\nThe models that obtained the highest Calinski-Harabasz and Davies-Bouldin indexes were SVM, MCV and Histogram. These models, in addition to IForest, showed the highest Silhouette scores (Table 4).\n\n*Non-parametric Comparison*\n\n* From the nonparametric statistical Conover-Friedman test, we found a significant difference in detecting anomalies among the models with 95 \\% certainty.\n\n* For the training dataset, there is no significant difference in the performance of each Pycaret model. However, all BiLSTM experiments differed significantly from all PyCaret models (Table 6, Figure 6). \n* All BiLSTM experiments were not significantly different from each other, as shown in the Friedman-Conover and critical difference diagrams. \n* Exp-04 ranked the highest scored but was not significantly different to __Exp-02__ to __Exp-04__. The anomaly could have been detected with 22:00 lead time with Exp-02 and Exp-04, while for the PyCaret models, the Clustering-Based Local outlier **cluster** could have detected the anomalies 14:40 hr in advance.\n\n\n### Test\n\n*Cluster Metrics*\n\nMCD and SVM obtained the highest Silhouette scores and Calinski-Harabasz and Davies-Bouldin indexes (Table 5).\n\n\n* From the nonparametric test, we can reject the null hypothesis that the performance of all models at detecting anomalies is not significantly different with 95% certainty\n\n* The ranking of the models shows that the best model. The Conover-Friedman test and critical difference diagram showed no statistical difference amongst experiments Exp-01 to Exp-04 and iForest with 95% certainty (Table 6, Figure 7).\n* Similarly, MCD and SVM showed no significant difference in detecting anomalies but significantly differed from all other models.\n\n* Experiments Exp-02 and Exp-04, which minimised the Huber-Loss, ranked the highest of all models.\n* The Conover-Friedman test and critical difference diagram showed no statistical difference amongst experiments Exp-01 to Exp-04 and iForest.  \n* Any of the BiLSTM models could have detected the anomaly within a 27:00 hr lead time, specifically Exp-03 within 27:20:00, while Iforest detected the anomalies within a lead time of 25:30:00.\n\n\n### PyCaret\n\u003cp align=\"center\"\u003e\n  Train dataset - Dataset 2 (avg_df2)\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg height=\"1200\" src=\"plots/scatter_anomalies.png\" width=\"600\" alt=\"Anomaly distribution on train dataset\"/\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\nFigure 2. Anomalies detected by the PyCaret models on the training dataset.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\nTest dataset - Dataset 3 (avg_df3)\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Fig\" height=\"500\" src=\"plots/merged_test_anomaly_prediction.png\" width=\"600\"/\u003e \n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  Figure 3. Anomalies detected by the PyCaret models on the test dataset.\n\u003c/p\u003e\n  \n\u003cbr\u003e\n\u003c/br\u003e\n\n\n### BiLSTM - PyTorch\n\n\u003cp align=\"center\"\u003e  \n  Train dataset - Dataset 2 (avg_df2)\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"plots/merged_bilstm_val_pred_anom_exp1.png\" width=\"400\" height=\"150\" alt=\"train_anomaly exp1\"/\u003eA \u003cimg height=\"150\" src=\"plots/merged_bilstm_val_pred_anom_exp2.png\" title=\"train anomaly exp2\" \nwidth=\"400\"/\u003eB\n\u003cimg alt=\"train anomalies exp3\" height=\"150\" src=\"plots/merged_bilstm_val_pred_anom_exp3.png\" width=\"400\"/\u003eC \u003cimg height=\"150\" src=\"plots/merged_bilstm_val_pred_anom_exp4.png\" width=\"400\" title=\"train anomalies exp4\"/\u003eB\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  Figure 4. Anomalies distribution detected on the training dataset. The experimental setup is outlined in Table 2. A. Exp-01, B.Exp-02, C. Exp-03, D.Exp-04\n\u003c/p\u003e\n\n\n\u003cbr\u003e\n\u003c/br\u003e\n\n\u003cp align=\"center\"\u003e\n  Test dataset - Dataset 3 (avg_df3)\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"plots/merged_test_bilstm_anom_exp1.png\" width=\"400\" height=\"150\" alt=\"test_anomaly exp1\"/\u003eA \u003cimg src=\"plots/merged_test_bilstm_anom_exp2.png\" width=\"400\" height=\"150\" alt=\"test_anomaly exp2\"/\u003eB\n\u003cimg src=\"plots/merged_test_bilstm_anom_exp3.png\" width=\"400\" height=\"150\" alt=\"test_anomaly exp3\"/\u003eC \u003cimg src=\"plots/merged_test_bilstm_anom_exp4.png\" width=\"400\" height=\"150\" alt=\"test_anomaly exp4\"/\u003eD\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\nFigure 5. Anomalies distribution detected on the test dataset. The experimental setup is outlined in Table 2. A. Exp-01, B.Exp-02, C. Exp-03, D.Exp-04.\n\u003c/p\u003e\n\n## Cluster Metrics\n\n__Table 4__: Cluster metrics on the training dataset.\n|index|silhoutte|calinski\\_harabasz|davies\\_bouldin|\n|---|---|---|---|\n|cluster|0\\.7762|646\\.9312|0\\.8175|\n|histogram|0\\.8124|1001\\.4351|0\\.6754|\n|iforest|0\\.8124|992\\.9829|0\\.6791|\n|knn|0\\.8017|897\\.689|0\\.7167|\n|mcd|0\\.8124|1004\\.5345|0\\.6739|\n|svm|0\\.814|1011\\.3661|0\\.6723|\n|exp1|0\\.7442|879\\.5458|0\\.7615|\n|exp2|0\\.747|892\\.5695|0\\.7562|\n|exp3|0\\.7442|879\\.5458|0\\.7615|\n|exp4|0\\.7386|875\\.7697|0\\.764|\n\n\u003cbr\u003e\n\u003c/br\n\n__Table 5__: Cluster metrics on the test dataset.\n|index|silhoutte|calinski\\_harabasz|davies\\_bouldin|\n|---|---|---|---|\n|iforest|0\\.923|7629\\.4839|0\\.7218|\n|mcd|0\\.9557|15873\\.0144|0\\.3554|\n|svm|0\\.9533|16564\\.1182|0\\.3939|\n|exp1|0\\.9263|8549\\.185|0\\.6857|\n|exp2|0\\.9256|8389\\.7269|0\\.692|\n|exp3|0\\.9267|8656\\.1093|0\\.6818|\n|exp4|0\\.9256|8433\\.0381|0\\.6902|\n\n\n## Nonparametric Statistical Models Comparison - Friedman-Conover\n\n__Table 6__: Models performance ranking for the training and test datasets.\n\n| Model     | Training Ranks | Test Ranks |\n|-----------|:--------------:|:----------:|\n| Exp-01    |    0.5647      |  0.5738    |\n| Exp-02    |    0.5642      |  0.5739    |\n| Exp-03    |    0.5647      |  0.5738    |\n| Exp-04    |    0.5673      |  0.5739    |\n| Cluster   |    0.5398      |     -      |\n| Histogram |    0.5398      |     -      |\n| iForest   |    0.5398      |  0.5735    |\n| KNN       |    0.5398      |     -      |\n| MCD       |    0.5398      |  0.5663    |\n| SVM       |    0.5398     |   0.5649    |\n\n## Training results\n\n* Friedman-Chisquare nonparametric statistical test - p-value = 8.85e-75. Therefore the H\u003csub\u003e0\u003c/sub\u003e is rejected.\n* Posthoc- Friedman-Conover pairwise comparison\n*  Critical Difference Diagram\n\u003cp align=\"center\"\u003e\n\u003cimg height=\"300\" src=\"plots/posthoc_train_models.png\" width=\"400\" title=\"train_post\"/\u003e\n\n\u003cimg height=\"100\" src=\"plots/critical_dif_train_models.png\" width=\"400\" title=\"train_post\"/\u003e\n\u003cp align=\"center\"\u003e\nFigure 6.  Figure 6. Conover-Friedman posthoc comparison training results and critical distance on the unseen test data by model.\n\u003c/p\u003e\n\u003cbr\u003e\n\u003c/br\u003e\n\n\n## Test results\n\n* Non-parametri test - Friedman-chisquare pvalue = 6.51e-19. Therefore the H\u003csub\u003e0\u003c/sub\u003e is rejected.\n* Posthoc- Friedman-Conover pairwise comparison\n* Critical Difference Diagram\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg height=\"250\" src=\"plots/posthoc_test_models.png\" width=\"350\" title=\"train_post\"/\u003e\n\u003cimg height=\"100\" src=\"plots/critical_dif_test_models.png\" width=\"400\" title=\"train_post\"/\u003e\n  \n\nFigure 6. Conover-Friedman posthoc comparison test results and critical distance on the unseen test data by model.\n\u003c/p\u003e\n\n\n# Conclusion\nIn summary, Exp-04 consistently obtained the best performance in both datasets. The test dataset was almost six times larger than the train set. It also presented spikes at the beginning and middle of the test. We can observe that the data's size and quality impact the model's choice to detect failures. Models Histogram, Cluster, and KNN were excluded in the test comparison since they accounted for more than 50% of the test dataset which is inaccurate as per the data visualisations. These models showed to be less robust for unseen data.\n\nConversely, the __IForest__ model took the lead in detecting anomalies together with Exp-01 and Exp-04 with no significant difference in their performance for the unseen test dataset. It showed to be __more robust__ to changes independently of dataset size and unknown contamination. \n\nWe can conclude that the PyCaret Anomalies models selected in this work and the Bilstm (Bidirectional LSTM) Artificial Neural Networks can detect failures on the bearing sensors' signals at the same performance level. Whether these models can detect failures days or weeks in advance in other unseen data requires further testing and optimisation.\n\n## References\n\n[1] [PyCaret](https://pycaret.gitbook.io/docs/)\n\u003cbr\u003e\n\u003c/br\u003e\n[2] https://towardsdatascience.com/lstm-autoencoder-for-anomaly-detection-e1f4f2ee7ccf\n\u003cbr\u003e\n\u003c/br\u003e\n[3] https://towardsdatascience.com/machine-learning-for-anomaly-detection-and-condition-monitoring-d4614e7de770\n\u003cbr\u003e\n\u003c/br\u003e\n[4] https://sabtainahmadml.medium.com/condition-monitoring-through-diagnosis-of-anomalies-lstm-based-unsupervised-ml-approach-5f0565735dff\n\u003cbr\u003e\n\u003c/br\u003e\n[5] Zhang, R.; Peng, Z.; Wu, L.; Yao, B.; Guan, Y. Fault Diagnosis from Raw Sensor Data Using Deep Neural Networks Considering Temporal Coherence. Sensors 2017, 17, 549. https://doi.org/10.3390/s1703054\n\u003cbr\u003e\n\u003c/br\u003e\n[6] Goldstein, M. and Uchida, S. (2016) ‘A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data’, PLOS ONE, 11(4). doi:10.1371/journal.pone.0152173. \n\u003cbr\u003e\n\u003c/br\u003e\n[7] K. Choi, J. Yi, C. Park and S. Yoon, \"Deep Learning for Anomaly Detection in Time-Series Data: Review, Analysis, and Guidelines,\" in IEEE Access, vol. 9, pp. 120043-120065, 2021, doi: 10.1109/ACCESS.2021.3107975.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwb-az%2Ftimeseries-sensor-anomaly-detection","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwb-az%2Ftimeseries-sensor-anomaly-detection","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwb-az%2Ftimeseries-sensor-anomaly-detection/lists"}