{"id":21328308,"url":"https://github.com/alumik/bagel-tensorflow","last_synced_at":"2025-03-16T00:16:52.151Z","repository":{"id":60135670,"uuid":"306698692","full_name":"alumik/bagel-tensorflow","owner":"alumik","description":"A robust and unsupervised KPI anomaly detection algorithm based on conditional variational autoencoder","archived":false,"fork":false,"pushed_at":"2023-09-21T07:36:14.000Z","size":2875,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T12:48:13.771Z","etag":null,"topics":["anomaly-detection","cvae","tensorflow2","time-series"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alumik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-23T17:06:22.000Z","updated_at":"2023-12-31T07:13:41.000Z","dependencies_parsed_at":"2023-01-18T20:30:32.981Z","dependency_job_id":null,"html_url":"https://github.com/alumik/bagel-tensorflow","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alumik%2Fbagel-tensorflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alumik%2Fbagel-tensorflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alumik%2Fbagel-tensorflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alumik%2Fbagel-tensorflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alumik","download_url":"https://codeload.github.com/alumik/bagel-tensorflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806092,"owners_count":20350775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anomaly-detection","cvae","tensorflow2","time-series"],"created_at":"2024-11-21T21:26:32.953Z","updated_at":"2025-03-16T00:16:52.132Z","avatar_url":"https://github.com/alumik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bagel\n\n![version-2.2.0](https://img.shields.io/badge/version-2.2.0-blue)\n![python-\u003e=3.10](https://img.shields.io/badge/python-\u003e=3.10-blue?logo=python\u0026logoColor=white)\n![TensorFlow 2.13](https://img.shields.io/badge/TensorFlow-2.13-FF6F00?logo=tensorflow\u0026logoColor=white)\n![license-MIT](https://img.shields.io/badge/license-MIT-green)\n\n\u003cimg width=\"140\" alt=\"Bagel Logo\" align=\"right\" src=\"https://www.svgrepo.com/show/275681/bagel.svg\"/\u003e\n\nBagel is a robust and unsupervised KPI anomaly detection algorithm based on conditional variational autoencoder.\n\nThis is an implementation of Bagel in TensorFlow 2. The original PyTorch 0.4 implementation can be found at\n[NetManAIOps/Bagel](https://github.com/NetManAIOps/Bagel).\n\n## Install\n\n`pip` will automatically install required PyPI dependencies when you install this package:\n\n- For development use:\n\n    ```\n    git clone https://github.com/alumik/bagel-tensorflow.git\n    cd bagel-tensorflow\n    pip install -e .\n    ```\n\n- For production use:\n\n    ```\n    pip install git+https://github.com/alumik/bagel-tensorflow.git\n    ```\n\nAn `environment.yml` is also provided if you prefer `conda` to manage dependencies:\n\n```\nconda env create -f environment.yml\n```\n\n## Run\n\n### KPI Format\n\nKPI data must be stored in csv files in the following format:\n\n```\ntimestamp,   value,       label\n1469376000,  0.847300274, 0\n1469376300, -0.036137314, 0\n1469376600,  0.074292384, 0\n1469376900,  0.074292384, 0\n1469377200, -0.036137314, 0\n1469377500,  0.184722083, 0\n1469377800, -0.036137314, 0\n1469378100,  0.184722083, 0\n```\n\n- `timestamp`: timestamps in seconds (10-digit).\n- `label` (optional): `0` for normal points, `1` for anomaly points.\n- Labels are used only for evaluation and are not required in model training and inference. However, if labels are\n  provided, the model can still take labeled data to improve the performance.\n\n### Sample Script\n\nA sample script can be found at `sample/main.py`:\n\n## Usage\n\nTo prepare the data:\n\n```python\nimport bagel\n\nkpi = bagel.data.load_kpi('kpi.csv')\nkpi.complete_timestamp()\ntrain_kpi, valid_kpi, test_kpi = kpi.split((0.49, 0.21, 0.3))\ntrain_kpi, mean, std = train_kpi.standardize()\nvalid_kpi, _, _ = valid_kpi.standardize(mean=mean, std=std)\ntest_kpi, _, _ = test_kpi.standardize(mean=mean, std=std)\ndataset = bagel.data.KPIDataset(\n    train_kpi.use_labels(0.),\n    window_size=window_size,\n    time_feature=time_feature,\n    missing_injection_rate=missing_injection_rate,\n)\nvalid_dataset = bagel.data.KPIDataset(valid_kpi, window_size=window_size, time_feature=time_feature)\ntest_dataset = bagel.data.KPIDataset(test_kpi.no_labels(), window_size=window_size, time_feature=time_feature)\n```\n\nTo build and train a Bagel model:\n\n```python\nmodel = bagel.Bagel(\n    window_size=window_size,\n    hidden_dims=hidden_dims,\n    latent_dim=latent_dim,\n    dropout_rate=dropout_rate,\n)\nlr_scheduler = tf.keras.optimizers.schedules.ExponentialDecay(\n    initial_learning_rate=learning_rate,\n    decay_steps=10 * len(dataset) // batch_size,\n    decay_rate=0.75,\n    staircase=True,\n)\noptimizer = tf.keras.optimizers.Adam(learning_rate=lr_scheduler, clipnorm=clipnorm)\nmodel.compile(optimizer=optimizer, jit_compile=True)\nmodel.fit(\n    x=[dataset.values, dataset.time_code, dataset.normal],\n    batch_size=batch_size,\n    epochs=epochs,\n    validation_data=([valid_dataset.values, valid_dataset.time_code, valid_dataset.normal], None),\n    validation_batch_size=batch_size,\n)\n```\n\nTo use the trained model for prediction:\n\n```python\nanomaly_scores = model.predict(\n    x=[test_dataset.values, test_dataset.time_code, test_dataset.normal],\n    batch_size=batch_size,\n)\n```\n\nUse `tf.keras.Model.save` API to save the model.\n\n## Citation\n\n```bibtex\n@inproceedings{conf/ipccc/LiCP18,\n    author    = {Zeyan Li and\n                 Wenxiao Chen and\n                 Dan Pei},\n    title     = {Robust and Unsupervised {KPI} Anomaly Detection Based on Conditional\n                 Variational Autoencoder},\n    booktitle = {37th {IEEE} International Performance Computing and Communications\n                 Conference, {IPCCC} 2018, Orlando, FL, USA, November 17-19, 2018},\n    pages     = {1--9},\n    publisher = {{IEEE}},\n    year      = {2018},\n    url       = {https://doi.org/10.1109/PCCC.2018.8710885},\n    doi       = {10.1109/PCCC.2018.8710885}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falumik%2Fbagel-tensorflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falumik%2Fbagel-tensorflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falumik%2Fbagel-tensorflow/lists"}