{"id":13415309,"url":"https://github.com/NetManAIOps/donut","last_synced_at":"2025-03-14T22:33:17.474Z","repository":{"id":49356674,"uuid":"115477704","full_name":"NetManAIOps/donut","owner":"NetManAIOps","description":"WWW 2018: Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications","archived":false,"fork":false,"pushed_at":"2019-03-06T03:12:00.000Z","size":12647,"stargazers_count":452,"open_issues_count":25,"forks_count":144,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-04-13T18:54:13.607Z","etag":null,"topics":["code"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NetManAIOps.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-27T03:32:19.000Z","updated_at":"2024-03-25T09:33:44.000Z","dependencies_parsed_at":"2022-09-16T04:31:00.199Z","dependency_job_id":null,"html_url":"https://github.com/NetManAIOps/donut","commit_stats":null,"previous_names":["korepwx/donut"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetManAIOps%2Fdonut","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetManAIOps%2Fdonut/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetManAIOps%2Fdonut/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NetManAIOps%2Fdonut/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NetManAIOps","download_url":"https://codeload.github.com/NetManAIOps/donut/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243658057,"owners_count":20326459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code"],"created_at":"2024-07-30T21:00:46.909Z","updated_at":"2025-03-14T22:33:17.464Z","avatar_url":"https://github.com/NetManAIOps.png","language":"Python","funding_links":[],"categories":["异常检测包","Python","2018"],"sub_categories":[],"readme":"DONUT\n=====\n\n.. image:: https://travis-ci.org/haowen-xu/donut.svg?branch=master\n    :target: https://travis-ci.org/haowen-xu/donut\n.. image:: https://coveralls.io/repos/github/haowen-xu/donut/badge.svg?branch=master\n    :target: https://coveralls.io/github/haowen-xu/donut?branch=master\n\nDonut is an anomaly detection algorithm for seasonal KPIs.\n\nCitation\n--------\n\n.. code-block:: bibtex\n\n    @inproceedings{donut,\n      title={Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications},\n      author={Xu, Haowen and Chen, Wenxiao and Zhao, Nengwen and Li, Zeyan and Bu, Jiahao and Li, Zhihan and Liu, Ying and Zhao, Youjian and Pei, Dan and Feng, Yang and others},\n      booktitle={Proceedings of the 2018 World Wide Web Conference on World Wide Web},\n      pages={187--196},\n      year={2018},\n      organization={International World Wide Web Conferences Steering Committee}\n    }\n\nDependencies\n------------\n\nTensorFlow \u003e= 1.5\n\nInstallation\n------------\n\nCheckout this repository and execute:\n\n.. code-block:: bash\n\n    pip install git+https://github.com/thu-ml/zhusuan.git\n    pip install git+https://github.com/haowen-xu/tfsnippet.git@v0.1.2\n    pip install .\n\nThis will first install `ZhuSuan \u003chttps://github.com/thu-ml/zhusuan\u003e`_ and\n`TFSnippet \u003chttps://github.com/haowen-xu/tfsnippet\u003e`_, the two major dependencies\nof Donut, then install the Donut package itself.\n\nAPI Usage\n---------\n\nTo prepare the data:\n\n.. code-block:: python\n\n    import numpy as np\n    from donut import complete_timestamp, standardize_kpi\n\n    # Read the raw data.\n    timestamp, values, labels = ...\n    # If there is no label, simply use all zeros.\n    labels = np.zeros_like(values, dtype=np.int32)\n\n    # Complete the timestamp, and obtain the missing point indicators.\n    timestamp, missing, (values, labels) = \\\n        complete_timestamp(timestamp, (values, labels))\n\n    # Split the training and testing data.\n    test_portion = 0.3\n    test_n = int(len(values) * test_portion)\n    train_values, test_values = values[:-test_n], values[-test_n:]\n    train_labels, test_labels = labels[:-test_n], labels[-test_n:]\n    train_missing, test_missing = missing[:-test_n], missing[-test_n:]\n\n    # Standardize the training and testing data.\n    train_values, mean, std = standardize_kpi(\n        train_values, excludes=np.logical_or(train_labels, train_missing))\n    test_values, _, _ = standardize_kpi(test_values, mean=mean, std=std)\n\nTo construct a Donut model:\n\n.. code-block:: python\n\n    import tensorflow as tf\n    from donut import Donut\n    from tensorflow import keras as K\n    from tfsnippet.modules import Sequential\n\n    # We build the entire model within the scope of `model_vs`,\n    # it should hold exactly all the variables of `model`, including\n    # the variables created by Keras layers.\n    with tf.variable_scope('model') as model_vs:\n        model = Donut(\n            h_for_p_x=Sequential([\n                K.layers.Dense(100, kernel_regularizer=K.regularizers.l2(0.001),\n                               activation=tf.nn.relu),\n                K.layers.Dense(100, kernel_regularizer=K.regularizers.l2(0.001),\n                               activation=tf.nn.relu),\n            ]),\n            h_for_q_z=Sequential([\n                K.layers.Dense(100, kernel_regularizer=K.regularizers.l2(0.001),\n                               activation=tf.nn.relu),\n                K.layers.Dense(100, kernel_regularizer=K.regularizers.l2(0.001),\n                               activation=tf.nn.relu),\n            ]),\n            x_dims=120,\n            z_dims=5,\n        )\n\nTo train the Donut model, and use a trained model for prediction:\n\n.. code-block:: python\n\n    from donut import DonutTrainer, DonutPredictor\n\n    trainer = DonutTrainer(model=model, model_vs=model_vs)\n    predictor = DonutPredictor(model)\n\n    with tf.Session().as_default():\n        trainer.fit(train_values, train_labels, train_missing, mean, std)\n        test_score = predictor.get_score(test_values, test_missing)\n\nTo save and restore a trained model:\n\n.. code-block:: python\n\n    from tfsnippet.utils import get_variables_as_dict, VariableSaver\n\n    with tf.Session().as_default():\n        # Train the model.\n        ...\n\n        # Remember to get the model variables after the birth of a\n        # `predictor` or a `trainer`.  The :class:`Donut` instances\n        # does not build the graph until :meth:`Donut.get_score` or\n        # :meth:`Donut.get_training_loss` is called, which is\n        # done in the `predictor` or the `trainer`.\n        var_dict = get_variables_as_dict(model_vs)\n\n        # save variables to `save_dir`\n        saver = VariableSaver(var_dict, save_dir)\n        saver.save()\n\n    with tf.Session().as_default():\n        # Restore variables from `save_dir`.\n        saver = VariableSaver(get_variables_as_dict(model_vs), save_dir)\n        saver.restore()\n\n\nIf you need more advanced outputs from the model, you may derive the outputs\nby using `model.vae` directly, for example:\n\n.. code-block:: python\n\n    from donut import iterative_masked_reconstruct\n\n    # Obtain the reconstructed `x`, with MCMC missing data imputation.\n    # See also:\n    #   :meth:`donut.Donut.get_score`\n    #   :func:`donut.iterative_masked_reconstruct`\n    #   :meth:`tfsnippet.modules.VAE.reconstruct`\n    input_x = ...  # 2-D `float32` :class:`tf.Tensor`, input `x` windows\n    input_y = ...  # 2-D `int32` :class:`tf.Tensor`, missing point indicators\n                   # for the `x` windows\n    x = model.vae.reconstruct(\n        iterative_masked_reconstruct(\n            reconstruct=model.vae.reconstruct,\n            x=input_x,\n            mask=input_y,\n            iter_count=mcmc_iteration,\n            back_prop=False\n        )\n    )\n    # `x` is a :class:`tfsnippet.stochastic.StochasticTensor`, from which\n    # you may derive many useful outputs, for example:\n    x.tensor  # the `x` samples\n    x.log_prob(group_ndims=0)  # element-wise log p(x|z) of sampled x\n    x.distribution.log_prob(input_x)  # the reconstruction probability\n    x.distribution.mean, x.distribution.std  # mean and std of p(x|z)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNetManAIOps%2Fdonut","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNetManAIOps%2Fdonut","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNetManAIOps%2Fdonut/lists"}