{"id":14971828,"url":"https://github.com/grafana/pysigma-backend-loki","last_synced_at":"2025-04-09T06:13:31.024Z","repository":{"id":64781929,"uuid":"551338163","full_name":"grafana/pySigma-backend-loki","owner":"grafana","description":"pySigma backend for generating Grafana Loki/LogQL rules","archived":false,"fork":false,"pushed_at":"2025-03-31T05:04:57.000Z","size":795,"stargazers_count":41,"open_issues_count":7,"forks_count":2,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-01T14:50:12.103Z","etag":null,"topics":["loki","siem","sigma"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grafana.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-14T08:04:28.000Z","updated_at":"2025-03-28T19:57:57.000Z","dependencies_parsed_at":"2023-10-20T11:39:00.482Z","dependency_job_id":"0e0fa32f-ca1b-4031-b07f-215e1da4d106","html_url":"https://github.com/grafana/pySigma-backend-loki","commit_stats":{"total_commits":254,"total_committers":5,"mean_commits":50.8,"dds":"0.32677165354330706","last_synced_commit":"29e88d1f702d80f8d82fcb0e007f9479aae8c168"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2FpySigma-backend-loki","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2FpySigma-backend-loki/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2FpySigma-backend-loki/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grafana%2FpySigma-backend-loki/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grafana","download_url":"https://codeload.github.com/grafana/pySigma-backend-loki/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247987285,"owners_count":21028895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["loki","siem","sigma"],"created_at":"2024-09-24T13:45:53.767Z","updated_at":"2025-04-09T06:13:31.006Z","avatar_url":"https://github.com/grafana.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![PyPI](https://img.shields.io/pypi/v/pysigma-backend-loki)\r\n![Tests](https://github.com/grafana/pySigma-backend-loki/actions/workflows/test.yml/badge.svg)\r\n[![Coverage Status](https://coveralls.io/repos/github/grafana/pySigma-backend-loki/badge.svg?branch=main\u0026t=lvM1Ns)](https://coveralls.io/github/grafana/pySigma-backend-loki?branch=main)\r\n\r\n# pySigma Loki Backend\r\n\r\nThis is the Loki backend for pySigma. It provides the package `sigma.backends.loki` with the `LogQLBackend` class.\r\n\r\nIt supports the following output formats for Sigma rules:\r\n\r\n* `default`: plain Loki LogQL queries\r\n* `ruler`: creates Loki LogQL queries in the ruler (YAML) format for generating alerts\r\n\r\nIt also supports the following query formats for and categories of [Sigma Correlation rules](https://github.com/SigmaHQ/sigma-specification/blob/version_2/Sigma_meta_rules.md):\r\n* `default` format using [LogQL metric queries](https://grafana.com/docs/loki/latest/query/metric_queries/):\r\n  * `event_count`\r\n  * `value_count`\r\n\r\nIt includes the following pipeline transformations in `sigma.pipelines.loki`:\r\n\r\n* `SetCustomAttributeTransformation`: adds a specified custom attribute to a rule, which can be used to introduce a [stream selector](https://grafana.com/docs/loki/latest/logql/log_queries/#log-stream-selector) or [parser expression](https://grafana.com/docs/loki/latest/logql/log_queries/#parser-expression) into the generated query\r\n  * The `LokiCustomAttributes` enum contains the relevant custom attribute names used by the backend\r\n\r\nFurther, it contains the processing pipelines in `sigma.pipelines.loki`:\r\n\r\n* `loki_log_parser`: converts field names to logfmt labels used by Grafana\r\n* `loki_promtail_sysmon`: parse and adjust field names for Windows sysmon data produced by promtail\r\n  * Note: most rules lack the `sysmon` service tag, and hence this pipeline should be used in combination with the [generic sysmon pipeline](https://github.com/SigmaHQ/pySigma-pipeline-sysmon)\r\n* `loki_okta_system_log`: parse the Okta System Log event json, adjusting field-names appropriately\r\n\r\nWhen converting rules into queries, the backend has the following optional arguments:\r\n\r\n* `add_line_filters` (boolean, default: `False`): if `True`, attempts to infer and add new line filters to queries without line filters, to [improve Loki query performance](https://grafana.com/docs/loki/latest/logql/log_queries/#line-filter-expression)\r\n* `case_sensitive` (boolean, default: `False`): if `True`, defaults to generating case-sensitive query filters, instead of case-insensitive filters that [the Sigma specification expects](https://github.com/SigmaHQ/sigma-specification/blob/main/Sigma_specification.md#general), trading between Loki query performance and potentially missing data with unexpected casing\r\n  * Note: if the generated query will be executed on Loki v2.8.2 or older, this argument **should** be set to `False`, as these versions of Loki may contain issues with case-insensitive filters, which cause such queries to fail to match desired data\r\n\r\nThis backend is currently maintained by:\r\n\r\n* [Nick Moore](https://github.com/kelnage)\r\n* [Mostafa Moradian](https://github.com/mostafa)\r\n\r\n## Installation\r\n\r\nTo get started developing/testing pySigma-backend-loki, these steps may help you get started:\r\n\r\n1. [Install poetry](https://python-poetry.org/docs/#installation)\r\n2. Clone this repository and open a terminal/shell in the top-level directory\r\n3. Run `poetry install` to install the Python dependencies\r\n4. Run `poetry shell` to activate the poetry environment\r\n5. Check it all works by running `poetry run pytest`\r\n6. (Optional) If you wish to validate the generated rules using sigma\\_backend\\_tester.py, install\r\n   [LogCLI](https://grafana.com/docs/loki/latest/tools/logcli/)\r\n7. (Optional, but recommended) To enable the Git hooks, run the following command from the root directory of the repository:\r\n```sh\r\ngit config --local core.hooksPath .githooks/\r\n```\r\n\r\n## Releasing\r\n\r\nTo release new versions of pySigma-backend-loki, we use GitHub actions to update PyPI. When the main branch is in state that is ready to release, the process is as follows:\r\n\r\n1. Determine the correct version number using the [Semantic Versioning](https://semver.org/) methodology. All version numbers should be in the format `\\d+\\.\\d+\\.\\d+(-[0-9A-Za-z-]+)?`\r\n2. Update [pyproject.toml](https://github.com/grafana/pySigma-backend-loki/blob/main/pyproject.toml) with the new version number\r\n3. Commit and push the change to GitHub, validate that the GitHub actions tests pass, and merge the PR into main\r\n4. Checkout main and create a signed tag for the release, named the version number prefixed with a v, e.g., `git tag --sign --message=\"Release vX.X.X\" vX.X.X`\r\n5. Push the tag to GitHub, e.g., `git push --tags`, and validate that the release to the test instance of PyPI is successful\r\n6. Run `poetry build` to produce distributable versions in `dist/`\r\n7. Create a release in GitHub against the appropriate tag. If the version number starts with `v0`, or ends with `-alpha/beta` etc., mark it as a pre-release, and attach the distributable files to the release\r\n8. Validate that the release to PyPI GitHub action is successful\r\n9. If this release supports a new minor or major version of `pySigma`, do a pull request on the [pySigma-plugin-directory](https://github.com/SigmaHQ/pySigma-plugin-directory) to reflect that\r\n\r\n## Work in progress\r\n\r\nThese features are currently either WIP or are planned to be implemented in the near future.\r\n\r\n* Various processing pipelines for other applications and log sources\r\n* Generating more accurate log stream selectors based on logsource\r\n* Translate field names in Sigma signatures into relevant labels for Loki using pipelines\r\n\r\n## Won't implement (probably)\r\n\r\nThese features are not easily supported by the backend, and hence are unlikely to be implemented.\r\n\r\n* More complex keyword/line filter searches than ANDs of ORs\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrafana%2Fpysigma-backend-loki","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrafana%2Fpysigma-backend-loki","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrafana%2Fpysigma-backend-loki/lists"}