{"id":13473078,"url":"https://github.com/salesforce/logai","last_synced_at":"2025-05-14T21:10:41.426Z","repository":{"id":65613380,"uuid":"558512915","full_name":"salesforce/logai","owner":"salesforce","description":"LogAI - An open-source library for log analytics and intelligence","archived":false,"fork":false,"pushed_at":"2024-11-14T15:16:16.000Z","size":31383,"stargazers_count":540,"open_issues_count":38,"forks_count":80,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-06T17:03:07.775Z","etag":null,"topics":["ai","aiops","anomaly-detection","benchmarking","log-analysis","log-intelligence","machine-learning","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salesforce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-27T17:42:30.000Z","updated_at":"2025-04-04T19:05:39.000Z","dependencies_parsed_at":"2024-11-22T15:12:46.060Z","dependency_job_id":null,"html_url":"https://github.com/salesforce/logai","commit_stats":{"total_commits":71,"total_committers":6,"mean_commits":"11.833333333333334","dds":"0.45070422535211263","last_synced_commit":"1aba01e9de1285d512c239f9cfdac9713b6b11d7"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Flogai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Flogai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Flogai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Flogai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salesforce","download_url":"https://codeload.github.com/salesforce/logai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248766687,"owners_count":21158302,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","aiops","anomaly-detection","benchmarking","log-analysis","log-intelligence","machine-learning","python"],"created_at":"2024-07-31T16:01:00.568Z","updated_at":"2025-04-13T19:21:04.491Z","avatar_url":"https://github.com/salesforce.png","language":"Python","readme":"\u003c!--\nCopyright (c) 2023 Salesforce.com, inc.\nAll rights reserved.\nSPDX-License-Identifier: BSD-3-Clause\nFor full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause\n\n--\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"./img/logai_logo.jpg\" width=\"400\"/\u003e\n    \u003c/br\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/salesforce/logai/actions/workflows/tests.yml\"\u003e\n    \u003cimg alt=\"Latest Release\" src=\"https://github.com/salesforce/logai/actions/workflows/tests.yml/badge.svg?branch=main\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/salesforce/logai/actions/workflows/pages/pages-build-deployment\"\u003e\n    \u003cimg alt=\"pages-build-deployment\" src=\"https://github.com/salesforce/logai/actions/workflows/pages/pages-build-deployment/badge.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/salesforce/logai/releases\"\u003e\n    \u003cimg alt=\"Latest Release\" src=\"https://img.shields.io/github/release/salesforce/logai.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://badge.fury.io/py/logai\"\u003e\n    \u003cimg src=\"https://badge.fury.io/py/logai.svg\" alt=\"PyPI version\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/BSD-3-Clause\"\u003e\n    \u003cimg alt=\"license\" src=\"https://img.shields.io/badge/License-BSD_3--Clause-blue.svg\"/\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n# LogAI: A Library for Log Analytics and Intelligence\n\n## Table of Contents\n* [Introduction](#introduction)\n* [Installation](#installation)\n* [Getting Started](#getting-started)\n  * [Explore LogAI GUI Portal](#explore-logai-gui-portal)\n  * [Run Simple Time-series Anomaly Detection Application](#run-simple-time-series-anomaly-detection-application)\n  * [Build Customized LogAI Applications](#build-customized-logai-applications)\n  * [Deep-learning Anomaly Detection Benchmarking](#deep-learning-anomaly-detection-benchmarking)\n* [Documentation](#documentation)\n* [Technical Report and Citing LogAI](#technical-report-and-citing-logai)\n* [Contact](#contact)\n* [License](#license)\n\n## Introduction\nLogAI is a one-stop open source library for log analytics and intelligence. LogAI supports various log analytics and log intelligence tasks such as log summarization, log clustering, log anomaly detection and more. It adopts the OpenTelemetry data model, to enable compatibility with different log management platforms. LogAI provides a unified model interface and integrates popular time-series, statistical learning and deep \nlearning models. Alongside this, LogAI also provides an out-of-the-box GUI toolkit for users to conduct interactive log \nanalysis. With LogAI, we can also easily benchmark popular ML and deep-learning algorithms for log anomaly detection \nwithout putting in redundant effort to process the logs. We have opensourced LogAI to facilitate a wide range of \napplications benefiting both academic research and industrial prototyping. \n \nThe following table compares LogAI with several existing log analysis Tools, include both commercial log management platforms\nlike NewRelic and DataDog, or popular log analysis open source tools on Github. \n\n| Coverage  | LogAI | NewRelic Log Monitoring | DataDog Log Explorer | logparser | loglizer | deep-loglizer | log3C | \n| ------------- | ------------- | ------------- |  ------------- | ------------- | ------------- | ------------- | ------------- |\n| OpenTelemetry log data model  | :white_check_mark:  | :white_check_mark:  | :white_check_mark:  | | | | | | \n| Unified data loader and preprocessing | :white_check_mark:  | :white_check_mark:  | :white_check_mark:  | :white_check_mark:  | :white_check_mark:  | :white_check_mark: | |\n| Auto log parsing  | :white_check_mark:  | :white_check_mark: | :white_check_mark: | | | \n| Log clustering | :white_check_mark: | :white_check_mark:  | :white_check_mark:  | | | | :white_check_mark: | \n| Log anomaly detection - time-series | :white_check_mark: | :white_check_mark:  | :white_check_mark:  | | | | | | \n| Log anomaly detection - traditional ML | :white_check_mark: |  |  | | :white_check_mark: |  |  |  \n| Log anomaly detection - deep Learning | :white_check_mark: |  | | | :white_check_mark: | :white_check_mark: |  |  \n| Huggingface integration | :white_check_mark: | | | | | |  |\n| GUI for result visualization | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | | |\n\n## Installation\n\n### Quick Install\nYou can install LogAI core library using `pip install`:\n\n```shell\n\n# Check out LogAI code repo from Github\ngit clone https://github.com/salesforce/logai.git\ncd logai\n\n# [Optional] Create virtual environment\npython3 -m venv venv\nsource venv/bin/activate\n\n# Install LogAI\npip install logai\n\n```\n\n### Install Optional Dependencies\nLogAI core library is light-weight with limited dependent packages installed. Users can install optional dependencies\nto enable extended functionalities of LogAI.\n\n**Deep Learning Log Analysis**. To conduct deep learning model related tasks and run benchmarking,\nplease install extra requirements by `pip install \"logai[deep-learning]\"`.\n\n**Enable LogAI GUI Portal***. To use LogAI GUI portal, \nplease install extra requirements by `pip install \"logai[gui]\"`.\n\n**LogAI Development**. To contribute to LogAI development, build and test code changes, \nplease install extra requirements by `pip install \"logai[dev]\"`.\n\n**Complete Installation**. you can install the full list of dependencies by `pip install \"logai[all]\"`.\n\n### Known Issues\n\n\u003e :warning: You may see `Resource punkt not found` while using LogAI. You can download `punkt`\n\u003e package from NLTK to solve the problem.\n\u003e ```shell\n\u003e python -m nltk.downloader punkt\n\u003e ```\n\n## Getting Started\n\nBelow we briefly introduce several ways to explore and use LogAI, including exploring LogAI GUI\nportal, benchmarking deep-learning based log anomaly detection using LogAI, and building your \nown log analysis application with LogAI.\n\n### Explore LogAI GUI Portal \n\nYou can also start a local LogAI service and use the GUI portal to explore LogAI.\n\n```shell\n\n# Check out LogAI code repo from Github\ngit clone https://github.com/salesforce/logai.git\ncd logai\n\n# [Optional] Create virtual environment\npython3 -m venv venv # create virtual environment\nsource venv/bin/activate # activate virtual env\n\n# install LogAI and GUI dependencies\npip install \".[dev]\"\npip install \".[gui]\"\n\n# Start LogAI service\nexport PYTHONPATH='.'  # make sure to add current root to PYTHONPATH\npython3 gui/application.py # Run local plotly dash server.\n```\n\nThen open the LogAI portal via http://localhost:8050/ or http://127.0.0.1:8050/ in your browser:\n\n![portal](img/logai_gui_landing.png)\n\nThe control panel is on the left side of the page. There are three applications you can choose from:\nLog Summarization, Log Clustering and Anomaly Detection.\n\n#### Control Panel\n\n**File Setting**. You can select the log type and log files to be processed. Now LogAI supports three \npublic datasets: HDFS, BGL and HealthApp. For each log type we included several sample log data. After\nlog file selected, you can choose the attributes you want to be involved in log processing. The selected \nattributes will be treated as structured log attributes.\n\n**Algorithm Setting**. For different applications, the algorithms options may be different.\nFor example, auto-log parsing algorithms are utilized for log summarization, but log clustering uses auto-parsing algorithms,\n vectorization algorithms, categorical encoding and clustering algorithms. You can select an\n algorithm and change the parameters in each algorithm section. After algorithm configuration, simply click \"run\" to\nrun the application.\n\n#### Log Summarization\n\nLog summarization App summarize and group the raw logs by log patterns and attributes. You can click on \neach log pattern and see what the pattern looks like and the dynamic values in each position. You can also \nsee the chart of occurrance trend of this pattern on the right side.\n\n![log summarization](img/logai_summarization_res.png)\n\n#### Log Clustering\n\nLog clustering App groups raw logs into clusters by calculating the semantic representation of each logline. \nThen using clustering algorithms to generate log clusters. In this example, we choose k-mean where `k==8` to\ngenerate 8 clusters. The result is shown as a pie chart and you can click each portion of the pie chart to check\nthe raw logs in this cluster.\n\n![log clustering](img/logai_clustering_res.png)\n\n#### Anomaly Detection\n\nLog anomaly detection App conduct log anomaly detection tasks. Similar to log clustering, log anomaly detection\nalso needs to extract information from raw logs and generate representation of loglines. Depend on the type of anomaly detection,\nThe representation can be different. \n**Time-series anomaly detection**. If we use time-series algorithm like ETS, the raw logs will be converted\ninto log counter vectors by given time interval. Then ETS is performed on the generated log counter vectors and detect\nanomalous timestamps on the counter vector time-series. \n\n**Semantic anomaly detection**. If we use unsupervised outlier detection algorithms such as One-class SVM, the raw logs will \nbe converted into semantic vectors and feed the One-class SVM model. Then the model will detect anomalous loglines.\n\n![log anomaly detection](img/logai_anomaly_detection.png)\n\nLogAI GUI portal is just an example to demo LogAI capabilities. We know this may not be the best way to visualize the \nresults and there might be bugs in how the results are displayed. We will keep working with the open source community\nto improve usability of the portal. Any feedbacks and contributions are welcome :blush:. \n\n### Run Simple Time-series Anomaly Detection Application\n\nYou can also use LogAI in more programtic ways. LogAI supports configuration files in `.json` or `.yaml`. \nBelow is a sample `log_anomaly_detection_config.json` configuration for anomaly detection application. \nMake sure to set `filepath` to the target log dataset file path.\n\n```json\n{\n      \"open_set_data_loader_config\": {\n        \"dataset_name\": \"HDFS\",\n        \"filepath\": \"\"\n      },\n      \"preprocessor_config\": {\n          \"custom_delimiters_regex\":[]\n      },\n      \"log_parser_config\": {\n        \"parsing_algorithm\": \"drain\",\n        \"parsing_algo_params\": {\n          \"sim_th\": 0.5,\n          \"depth\": 5\n        }\n      },\n      \"feature_extractor_config\": {\n          \"group_by_category\": [\"Level\"],\n          \"group_by_time\": \"1s\"\n      },\n      \"log_vectorizer_config\": {\n          \"algo_name\": \"word2vec\"\n      },\n      \"categorical_encoder_config\": {\n          \"name\": \"label_encoder\"\n      },\n      \"anomaly_detection_config\": {\n          \"algo_name\": \"one_class_svm\"\n      }\n    }\n```\n\nThen to run log anomaly detection. You can simply create below python script:\n\n```python\nimport json\n\nfrom logai.applications.application_interfaces import WorkFlowConfig\nfrom logai.applications.log_anomaly_detection import LogAnomalyDetection\n\n# path to json configuration file\njson_config = \"./log_anomaly_detection_config.json\"\n\n# Create log anomaly detection application workflow configuration\nconfig = json.loads(json_config)\nworkflow_config = WorkFlowConfig.from_dict(config) \n\n# Create LogAnomalyDetection Application for given workflow_config\napp = LogAnomalyDetection(workflow_config)\n\n# Execute App\napp.execute()\n\n```\n\nThen you can check anomaly detection results by calling `app.anomaly_results`. \n\nFor full context of this example please check\n[Tutorial: Use Log Anomaly Detection Application](./examples/jupyter_notebook/log_anomaly_detection_example.ipynb).\n\n### Build Customized LogAI Applications\nYou can build your own customized log analysis applications using LogAI. Here we show two examples:\n\n* [Tutorial: Log Clustering Using LogAI](./examples/jupyter_notebook/tutorial_log_clustering.ipynb)\n* [Tutorial: Log Anomaly Detection Using LogAI](./examples/jupyter_notebook/tutorial_log_anomaly_detection.ipynb)\n\n### Deep-learning Anomaly Detection Benchmarking\n\nLogAI can be used to benchmark deep-learning anomaly detection results. \nA [tutorial](examples/jupyter_notebook/tutorial_deep_ad.md) is provided for \nAnomaly Detection Benchmarking using LSTM anomaly detector for HDFS Dataset. More examples of deep-learning anomaly \ndetection benchmarking on different datasets and algorithms can be found in \n[Deep Anomaly Detection Benchmarking Examples](examples/jupyter_notebook/nn_ad_benchmarking).\n\n## Documentation\n\nFor more detail about LogAI library and advanced use cases, please visit \n[LogAI Documentation](https://opensource.salesforce.com/logai).\n\n## Technical Report and Citing LogAI\n\nYou can find more details about LogAI in the [technical report](https://arxiv.org/abs/2301.13415). \nIf you're using LogAI in your research or applications, please cite using this BibTeX:\n\n```\n@misc{https://doi.org/10.48550/arxiv.2301.13415,\n    title = {LogAI: A Library for Log Analytics and Intelligence},\n    author = {Cheng, Qian and Saha, Amrita and Yang, Wenzhuo and Liu, Chenghao and Sahoo, Doyen and Hoi, Steven},\n    publisher = {arXiv},\n    year = {2023},\n    doi = {10.48550/ARXIV.2301.13415},\n    url = {https://arxiv.org/abs/2301.13415},\n    copyright = {arXiv.org perpetual, non-exclusive license}\n}\n\n```\n\n## Contact\nIf you have any questions, comments or suggestions, \nplease do not hesitate to contact us at [logai@salesforce.com](logai@salesforce.com). \n\n## License\n[BSD 3-Clause License](LICENSE.txt)\n\n","funding_links":[],"categories":["Python","Observability \u0026 Monitoring"],"sub_categories":["Log Analysis \u0026 Intelligence"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Flogai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalesforce%2Flogai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Flogai/lists"}