{"id":21158188,"url":"https://github.com/attackiq/pysigma-backend-kusto","last_synced_at":"2025-04-06T07:09:58.202Z","repository":{"id":153385465,"uuid":"617115260","full_name":"AttackIQ/pySigma-backend-kusto","owner":"AttackIQ","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-13T20:36:54.000Z","size":1247,"stargazers_count":36,"open_issues_count":1,"forks_count":10,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-30T06:03:51.867Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AttackIQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-21T18:20:48.000Z","updated_at":"2025-03-24T17:14:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"a92ef95e-a25c-4e35-86c9-113a87bc1c2f","html_url":"https://github.com/AttackIQ/pySigma-backend-kusto","commit_stats":null,"previous_names":["attackiq/pysigma-backend-kusto","attackiq/pysigma-backend-microsoft365defender"],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AttackIQ%2FpySigma-backend-kusto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AttackIQ%2FpySigma-backend-kusto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AttackIQ%2FpySigma-backend-kusto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AttackIQ%2FpySigma-backend-kusto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AttackIQ","download_url":"https://codeload.github.com/AttackIQ/pySigma-backend-kusto/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247445668,"owners_count":20939958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T12:17:53.842Z","updated_at":"2025-04-06T07:09:58.178Z","avatar_url":"https://github.com/AttackIQ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pySigma Kusto Query Language (KQL) Backend\n\n![Tests](https://github.com/AttackIQ/pySigma-backend-microsoft365defender/actions/workflows/test.yml/badge.svg)\n![Coverage Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/slincoln-aiq/9c0879725c7f94387801390bbb0ac8d6/raw/slincoln-aiq-pySigma-backend-microsoft365defender.json)\n![Status](https://img.shields.io/badge/Status-pre--release-orange)\n![PyPI version](https://badge.fury.io/py/pysigma-backend-kusto.svg)\n![Python versions](https://img.shields.io/pypi/pyversions/pysigma-backend-kusto.svg)\n![pySigma version](https://img.shields.io/badge/pySigma-%3E%3D0.10.0-blue)\n![License](https://img.shields.io/github/license/AttackIQ/pySigma-backend-microsoft365defender.svg)\n\n## Contents\n\n- [pySigma Kusto Query Language (KQL) Backend](#pysigma-kusto-query-language-kql-backend)\n  - [📖 Overview](#-overview)\n    - [🔑 Key Features](#-key-features)\n    - [🧑‍💻 Maintainer](#-maintainer)\n  - [🚀 Quick Start](#-quick-start)\n  - [📘 Usage](#-usage)\n    - [🖥️ sigma-cli](#️-sigma-cli)\n    - [🐍 Python Script](#-python-script)\n  - [🛠️ Advanced Features](#️-advanced-features)\n    - [🔄 Pipeline \\\u0026 Backend Args (New in 0.2.0)](#-pipeline--backend-args-new-in-020)\n    - [🗃️ Custom Table Names (New in 0.3.0) (Beta)](#️-custom-table-names-new-in-030-beta)\n  - [🔄 Processing Pipelines](#-processing-pipelines)\n    - [📊 Rule Support](#-rule-support)\n    - [🖥️ Commonly Supported Categories](#️-commonly-supported-categories)\n  - [🧪 Custom Transformations](#-custom-transformations)\n    - [📊 Custom Postprocessing Item](#-custom-postprocessing-item)\n  - [❓Frequently Asked Questions](#frequently-asked-questions)\n    - [How do I set the table name for a rule?](#how-do-i-set-the-table-name-for-a-rule)\n    - [How do I set the table name for a rule in YAML?](#how-do-i-set-the-table-name-for-a-rule-in-yaml)\n    - [How is the table name determined for a rule?](#how-is-the-table-name-determined-for-a-rule)\n    - [How are field mappings determined for a rule?](#how-are-field-mappings-determined-for-a-rule)\n    - [What tables are supported for each pipeline?](#what-tables-are-supported-for-each-pipeline)\n    - [I am receiving an `Invalid SigmaDetectionItem field name encountered` error. What does this mean?](#i-am-receiving-an-invalid-sigmadetectionitem-field-name-encountered-error-what-does-this-mean)\n    - [My query\\_table or custom field mapping isn't working](#my-query_table-or-custom-field-mapping-isnt-working)\n  - [🤝 Contributing](#-contributing)\n  - [📄 License](#-license)\n\n## 📖 Overview\n\nThe **pySigma Kusto Backend** transforms Sigma Rules into queries using [Kusto Query Language (KQL)](https://learn.microsoft.com/en-us/kusto/query/?view=microsoft-fabric). This backend supports multiple Microsoft products, including:\n\n- [Microsoft XDR Advanced Hunting Queries](https://learn.microsoft.com/en-us/defender-xdr/advanced-hunting-overview) (Formally Microsoft 365 Defender Advanced Hunting Queries)\n- [Azure Sentinel Advanced Security Information Model (ASIM) Queries](https://learn.microsoft.com/en-us/azure/sentinel/normalization)\n- [Azure Monitor Queries](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/get-started-queries)\n\n\u003e **Note:** This backend was previously named **pySigma Microsoft 365 Defender Backend**.\n\n### 🔑 Key Features\n\n- **Backend**: `sigma.backends.kusto` with `KustoBackend` class\n- **Pipelines**: Provides `microsoft_xdr_pipeline`, `sentinelasim_pipeline`, and `azure_monitor_pipeline` for query tables and field renames\n- **Output**: Query strings in Kusto Query Language (KQL)\n\n### 🧑‍💻 Maintainer\n\n- [Stephen Lincoln](https://github.com/slincoln-aiq) via [AttackIQ](https://github.com/AttackIQ)\n\n## 🚀 Quick Start\n\n1. Install the package:\n\n   ```bash\n   pip install pysigma-backend-kusto\n   ```\n\n   \u003e **Note:** This package requires `pySigma` version 0.10.0 or higher.\n\n2. Convert a Sigma rule to MIcrosoft XDR KQL query using sigma-cli:\n\n   ```bash\n   sigma convert -t kusto -p microsoft_xdr path/to/your/rule.yml\n   ```\n\n3. Or use in a Python script:\n\n   ```python\n   from sigma.rule import SigmaRule\n\n   from sigma.backends.kusto import KustoBackend\n   from sigma.pipelines.microsoftxdr import microsoft_xdr_pipeline\n\n   # Load your Sigma rule\n   rule = SigmaRule.from_yaml(\n      \"\"\"\n      title: Mimikatz CommandLine\n      status: test\n      logsource:\n            category: process_creation\n            product: windows\n      detection:\n            sel:\n               CommandLine|contains: mimikatz.exe\n            condition: sel\n      \"\"\"\n   )\n\n   # Convert the rule\n   xdr_pipeline = microsoft_xdr_pipeline()\n   backend = KustoBackend(processing_pipeline=xdr_pipeline)\n   print(backend.convert_rule(rule)[0])\n\n   ```\n\n## 📘 Usage\n\n### 🖥️ sigma-cli\n\nUse with `sigma-cli` per [typical sigma-cli usage](https://github.com/SigmaHQ/sigma-cli#usage):\n\n```bash\nsigma convert -t kusto -p microsoft_xdr -f default -s ~/sigma/rules\n```\n\n### 🐍 Python Script\n\nUse the backend and pipeline in a standalone Python script. Note, the backend automatically applies the pipeline, but\nyou can manually add it if you would like.\n\n```python\nfrom sigma.rule import SigmaRule\nfrom sigma.backends.kusto import KustoBackend\nfrom sigma.pipelines.microsoftxdr import microsoft_xdr_pipeline\n\n# Define an example rule as a YAML str\nsigma_rule = SigmaRule.from_yaml(\"\"\"\n  title: Mimikatz CommandLine\n  status: test\n  logsource:\n      category: process_creation\n      product: windows\n  detection:\n      sel:\n          CommandLine|contains: mimikatz.exe\n      condition: sel\n\"\"\")\n# Create backend, which automatically adds the pipeline\nkusto_backend = KustoBackend()\n\n# Or apply the pipeline manually\npipeline = microsoft_xdr_pipeline()\npipeline.apply(sigma_rule)\n\n# Convert the rule\nprint(sigma_rule.title + \" KQL Query: \\n\")\nprint(kusto_backend.convert_rule(sigma_rule)[0])\n```\n\nOutput:\n\n```text\nMimikatz CommandLine KQL Query: \n\nDeviceProcessEvents\n| where ProcessCommandLine contains \"mimikatz.exe\"\n```\n\n## 🛠️ Advanced Features\n\n### 🔄 Pipeline \u0026 Backend Args (New in 0.2.0)\n\nFor the `microsoft_xdr_pipeline`:\n\n- `transform_parent_image`: Controls ParentImage field mapping behavior\n  - When set to `True` (default), maps ParentImage to InitiatingProcessParentFileName\n  - When set to `False`, maps ParentImage to InitiatingProcessFileName\n  - Useful for adjusting field mappings based on specific rule requirements\n  - Example usage:\n\n```python\nfrom sigma.pipelines.microsoftxdr import microsoft_xdr_pipeline\npipeline = microsoft_xdr_pipeline(transform_parent_image=False)\n```\n\nThis argument allows fine-tuning of the ParentImage field mapping, which can be crucial for accurate rule conversion in certain scenarios. By default, it follows the behavior of mapping ParentImage to the parent process name, but setting it to `False` allows for mapping to the initiating process name instead.\n\n### 🗃️ Custom Table Names (New in 0.3.0) (Beta)\n\nThe `query_table` argument allows users to override table mappings and set custom table names.  This is useful for converting Sigma rules where the rule category does not easily map to the default table names.\n\n#### YAML Pipelines\n\nTo set a custom table name, ensure your pipeline has a priority of 9 or lower, as sigma-cli merges pipelines based on priority (default is 10). Field mappings in `mappings.py` will apply according to your specified table name, along with any custom field mapping transformations.\n\n```YAML\n# test_table_name_pipeline.yml\nname: Custom Query Table Pipeline\npriority: 1\ntransformations:\n- id: test_name_name\n  type: set_state\n  key: \"query_table\"\n  val: [\"DeviceProcessEvents\"]\n```\n\n```bash\nsigma convert -t kusto -p microsoft_xdr -p test_table_name_pipeline.yml test_rule.yml\n```\n\n#### Python Pipelines\n\nYou can also set the table name in the pipeline via Python by passing the `query_table` parameter to the pipeline.\n\n```python\nfrom sigma.pipelines.microsoftxdr import microsoft_xdr_pipeline\nmy_pipeline = microsoft_xdr_pipeline(query_table=\"DeviceProcessEvents\")\n```\n\n## 🔄 Processing Pipelines\n\nThis project includes three main processing pipelines, each designed for a specific Microsoft product:\n\n1. **Microsoft XDR Pipeline** (formerly Microsoft 365 Defender)\n   - Status: Production-ready\n   - Supports a wide range of Sigma rule categories\n   - All tables supported, but additional field mapping contributions welcome\n\n2. **Sentinel ASIM Pipeline**\n   - Status: Beta\n   - Transforms rules for Microsoft Sentinel Advanced Security Information Model (ASIM)\n   - All tables supported, but field mappings are limited\n\n3. **Azure Monitor Pipeline**\n   - Status: Alpha\n   - Currently supports field mappings for `SecurityEvent` and `SigninLogs` tables only\n   - All tables supported, but requires custom field mappings for other tables\n\nEach pipeline includes a `query_table` parameter for setting custom table names.\n\n### 📊 Rule Support\n\nRules are supported if either:\n\n- A valid table name is supplied via the `query_table` parameter or YAML pipeline\n- The rule's logsource category is supported and mapped in the pipeline's `mappings.py` file\n- The rule has an `EventID` or `EventCode` field in the `detection` section, and the eventid is present in the pipeline's `eventid_to_table_mappings` dictionary\n\n### 🖥️ Commonly Supported Categories\n\n- process_creation\n- image_load\n- network_connection\n- file_access, file_change, file_delete, file_event, file_rename\n- registry_add, registry_delete, registry_event, registry_set\n\nSpecific pipelines may support additional categories. Check each pipeline's `mappings.py` file for details.\n\n## 🧪 Custom Transformations\n\nThis package includes several custom `ProcessingPipeline` `Transformation` classes:\n\n1. **DynamicFieldMappingTransformation**\n   - Determines field mappings based on the `query_table` state parameter\n\n2. **GenericFieldMappingTransformation**\n   - Applies common field mappings across all tables in a pipeline\n\n3. **BaseHashesValuesTransformation**\n   - Transforms the Hashes field, removing hash algorithm prefixes\n\n4. **ParentImageValueTransformation**\n   - Extracts parent process name from Sysmon ParentImage field\n\n5. **SplitDomainUserTransformation**\n   - Splits User field into separate domain and username fields\n\n6. **RegistryActionTypeValueTransformation**\n   - Adjusts registry ActionType values for compatibility\n\n7. **InvalidFieldTransformation**\n   - Identifies unsupported or invalid fields in rules\n\n8. **SetQueryTableStateTransformation**\n   - Manages the `query_table` state based on rule category or custom settings\n\n### 📊 Custom Postprocessing Item\n\n1. **PrependQueryTablePostprocessingItem**\n\n- Adds table name as prefix to each query in a SigmaCollection, or single query in a SigmaRule\n\n## ❓Frequently Asked Questions\n\n### How do I set the table name for a rule?\n\nYou can set the table name for a rule by adding the `query_table` parameter to the pipeline and setting it to the table name you want to use.\n\n```python\nfrom sigma.pipelines.microsoftxdr import microsoft_xdr_pipeline\npipeline = microsoft_xdr_pipeline(query_table=\"DeviceProcessEvents\")\n```\n\n### How do I set the table name for a rule in YAML?\n\nYou can set the table name for a rule in YAML by adding the `query_table` parameter to the pipeline and setting it to the table name you want to use.\n\n```YAML\n# test_table_name_pipeline.yml\nname: \npriority: 1\ntransformations:\n- id: test_name_name\n  type: set_state\n  key: \"query_table\"\n  val: [\"DeviceProcessEvents\"]\n```\n\n```bash\nsigma convert -t kusto -p microsoft_xdr -p test_table_name_pipeline.yml test_rule.yml\n```\n\n### How is the table name determined for a rule?\n\nThe table name is set by the `SetQueryTableStateTransformation` transformation, which is the first transformation in each pipeline. The `query_table` is set to the pipeline's `state` parameter with the following priority:\n1. The `query_table` parameter passed to the pipeline, if using a Python script/code.\n2. The `query_table` parameter passed to the pipeline in a custom YAML pipeline, if using sigma-cli.\n3. The `logsource.category` field in the rule, if the category is present in the pipeline's `category_to_table_mappings` dictionary.\n4. The `EventID` or `EventCode` field, if present in the rule's `detection` section, and if the eventid is present in the pipeline's `eventid_to_table_mappings` dictionary.\n5. If none of the above are present, an error is raised.\n\n### How are field mappings determined for a rule?\n\nThe field mappings are determined by the `DynamicFieldMappingTransformation` transformation. It will use the table name from the pipeline state's `query_table` key.  The field mapping logic is defined in each pipeline's `mappings.py` file for each table.  If a field is not found in the table, the `GenericFieldMappingTransformation` will apply generic field mappings.  If a field is not found in the generic field mappings, the field will be kept the same.\n\n### What tables are supported for each pipeline?\n\nThe tables that are supported for each pipeline are defined in each pipeline's `tables.py` file. This file is automatically generated by the scripts in the `utils` folder. These scripts pull documentation from Microsoft to get all documented tables and their fields and schema.\n\n### I am receiving an `Invalid SigmaDetectionItem field name encountered` error. What does this mean?\n\nThis error means that the field name(s) provided in the error are not found in the tables fields defined in `tables.py` for the pipeline you are using. This probably means that a Sigma rule's field was not found in the field mappings for the table.  To fix this error, you can supply your own custom field mappings to convert the unsupported field into a supported one. For example, in using YAML:\n\n```YAML\n# custom_field_mapping_pipeline.yml\nname: Custom Field Mapping\npriority: 1\ntransformations:\n- id: field_mapping\n    type: field_name_mapping\n    mapping:\n        MyNotSupportedField: a_supported_field\n    rule_conditions:\n    - type: logsource\n        service: sysmon\n```\n\n```bash\nsigma convert -t kusto -p custom_field_mapping_pipeline.yml -p microsoft_xdr test_rule.yml\n```\n\nIf you find the field mapping useful, please consider submitting a PR to add it to the pipeline's field mappings :)\n\n### My query_table or custom field mapping isn't working\n\nEach pipeline in the project has a priority of 10. If you are trying to set the table name or custom field mappings, your pipeline needs to have a priority of 9 or less.  You can set the priority in the YAML pipeline like so:\n\n```YAML\n# test_table_name_pipeline.yml\nname: \npriority: 9\ntransformations:\n- id: test_name_name\n  type: set_state\n  key: \"query_table\"\n  val: [\"DeviceProcessEvents\"]\n```\n\n## 🤝 Contributing\n\nContributions are welcome, especially for table and field mappings! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\nPlease make sure to update tests as appropriate.\n\n## 📄 License\n\nThis project is licensed under the GNU Lesser General Public License v3.0 - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fattackiq%2Fpysigma-backend-kusto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fattackiq%2Fpysigma-backend-kusto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fattackiq%2Fpysigma-backend-kusto/lists"}