{"id":19361963,"url":"https://github.com/numaproj/numalogic","last_synced_at":"2025-05-15T23:03:27.166Z","repository":{"id":45084472,"uuid":"513323345","full_name":"numaproj/numalogic","owner":"numaproj","description":"Collection of operational time series ML models and tools","archived":false,"fork":false,"pushed_at":"2024-10-15T22:18:19.000Z","size":48502,"stargazers_count":169,"open_issues_count":52,"forks_count":31,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-15T23:02:50.178Z","etag":null,"topics":["autoencoders","deep-learning","hacktoberfest","machine-learning","outlier-detection","python","time-series","unsupervised-learning","variational-autoencoder"],"latest_commit_sha":null,"homepage":"https://numalogic.numaproj.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/numaproj.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-12T23:48:54.000Z","updated_at":"2025-05-15T20:33:23.000Z","dependencies_parsed_at":"2023-10-05T03:51:48.281Z","dependency_job_id":"562ce729-2199-42a3-8730-f76069874526","html_url":"https://github.com/numaproj/numalogic","commit_stats":null,"previous_names":[],"tags_count":92,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumalogic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumalogic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumalogic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/numaproj%2Fnumalogic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/numaproj","download_url":"https://codeload.github.com/numaproj/numalogic/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoencoders","deep-learning","hacktoberfest","machine-learning","outlier-detection","python","time-series","unsupervised-learning","variational-autoencoder"],"created_at":"2024-11-10T07:26:02.266Z","updated_at":"2025-05-15T23:03:26.885Z","avatar_url":"https://github.com/numaproj.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# numalogic\n\n[![Build](https://github.com/numaproj/numalogic/actions/workflows/ci.yml/badge.svg)](https://github.com/numaproj/numalogic/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/numaproj/numalogic/branch/main/graph/badge.svg?token=020HF97A32)](https://codecov.io/gh/numaproj/numalogic)\n[![black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![slack](https://img.shields.io/badge/slack-numaproj-brightgreen.svg?logo=slack)](https://join.slack.com/t/numaproj/shared_invite/zt-19svuv47m-YKHhsQ~~KK9mBv1E7pNzfg)\n[![Release Version](https://img.shields.io/github/v/release/numaproj/numalogic?label=numalogic)](https://github.com/numaproj/numalogic/releases/latest)\n\n\n## Background\nNumalogic is a collection of ML models and algorithms for operation data analytics and AIOps.\nAt Intuit, we use Numalogic at scale for continuous real-time data enrichment including\nanomaly scoring. We assign an anomaly score (ML inference) to any time-series\ndatum/event/message we receive on our streaming platform (say, Kafka). 95% of our\ndata sets are time-series, and we have a complex flowchart to execute ML inference on\nour high throughput sources. We run multiple models on the same datum, say a model that is\nsensitive towards +ve sentiments, another more tuned towards -ve sentiments, and another\noptimized for neutral sentiments. We also have a couple of ML models trained for the same\ndata source to provide more accurate scores based on the data density in our model store.\nAn ensemble of models is required because some composite keys in the data tend to be less\ndense than others, e.g., forgot-password interaction is less frequent than a status check\ninteraction. At runtime, for each datum that arrives, models are picked based on a conditional\nforwarding filter set on the data density. ML engineers need to worry about only their\ninference container; they do not have to worry about data movement and quality assurance.\n\n## Numalogic realtime training\nFor an always-on ML platform, the key requirement is the ability to train or retrain models\nautomatically based on the incoming messages. The composite key built at per message runtime\nlooks for a matching model, and if the model turns out to be stale or missing, an automatic\nretriggering is applied. The conditional forwarding feature of the platform improves the\ndevelopment velocity of the ML developer when they have to make a decision whether to forward\nthe result further or drop it after a trigger request.\n\n\n## Key Features\n\n1. Ease of use: simple and efficient tools for predictive data analytics\n2. Reusability: all the functionalities can be re-used in various contexts\n3. Model selection: easy to compare, validate, fine-tune and choose the model that works best with each data set\n4. Data processing: readily available feature extraction, scaling, transforming and normalization tools\n5. Extensibility: adding your own functions or extending over the existing capabilities\n6. Model Storage: out-of-the-box support for MLFlow and support for other model ML lifecycle management tools\n\n## Use Cases\n1. Deployment failure detection\n2. System failure detection for node failures or crashes\n3. Fraud detection\n4. Network intrusion detection\n5. Forecasting on time series data\n\n## Getting Started\n\nFor set-up information and running your first pipeline using numalogic, please see our [getting started guide](./quick-start.md).\n\n\n## Installation\n\nNumalogic requires Python 3.8 or higher.\n\n### Prerequisites\nNumalogic needs [PyTorch](https://pytorch.org/) and\n[PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/) to work.\nBut since these packages are platform dependendent,\nthey are not included in the numalogic package itself. Kindly install them first.\n\nNumalogic supports pytorch versions `2.0.0` and above.\n\nnumalogic can be installed using pip.\n```shell\npip install numalogic\n```\n\nIf using mlflow for model registry, install using:\n```shell\npip install numalogic[mlflow]\n```\n\n### Build locally\n\n1. Install [Poetry](https://python-poetry.org/docs/):\n    ```\n    curl -sSL https://install.python-poetry.org | python3 -\n    ```\n2. To activate virtual env:\n    ```\n    poetry shell\n    ```\n3. To install dependencies:\n    ```\n    poetry install --with dev,torch\n    ```\n   If extra dependencies are needed:\n    ```\n    poetry install --all-extras\n    ```\n4. To run unit tests:\n    ```\n    make test\n    ```\n5. To format code style using black and ruff:\n    ```\n    make lint\n    ```\n6. Setup [pre-commit](https://pre-commit.com/) hooks:\n   ```bash\n   pre-commit install\n   ```\n\n## Contributing\nWe would love contributions in the numalogic project in one of the following (but not limited to) areas:\n\n- Adding new time series anomaly detection models\n- Making it easier to add user's custom models\n- Support for additional model registry frameworks\n\nFor contribution guildelines please refer [here](https://github.com/numaproj/numaproj/blob/main/CONTRIBUTING.md).\n\n\n## Resources\n- [QUICK_START](docs/quick-start.md)\n- [EXAMPLES](examples)\n- [CONTRIBUTING](https://github.com/numaproj/numaproj/blob/main/CONTRIBUTING.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumaproj%2Fnumalogic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnumaproj%2Fnumalogic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnumaproj%2Fnumalogic/lists"}