Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/runreveal/pql
Pipelined Query Language
https://github.com/runreveal/pql
clickhouse detection-engineering go golang query-language siem sql
Last synced: about 1 month ago
JSON representation
Pipelined Query Language
- Host: GitHub
- URL: https://github.com/runreveal/pql
- Owner: runreveal
- License: apache-2.0
- Created: 2024-01-26T23:15:38.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-06-15T03:33:07.000Z (6 months ago)
- Last Synced: 2024-08-14T09:47:13.218Z (4 months ago)
- Topics: clickhouse, detection-engineering, go, golang, query-language, siem, sql
- Language: Go
- Homepage: https://pql.dev
- Size: 183 KB
- Stars: 628
- Watchers: 5
- Forks: 24
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-clickhouse - runreveal/pql - PQL is a Go library that compiles a pipelined-based query language into SQL, primarily tested with Clickhouse. (Language bindings / Golang)
README
# pipelined query language
[](https://pql.dev)
[](https://pql.dev)
[](https://discord.gg/PbeXzrWP)
This Go library compiles a pipelined-based query language
(inspired by the [Kusto Query Language][])
into SQL.
It has been specifically tested to work with the [Clickhouse SQL dialect][],
but the generated SQL is intentionally database agnostic. This repository
contains a Go library, and a CLI to invoke the library.
For example, the following expression:
```plain
StormEvents
| where DamageProperty > 5000 and EventType == "Thunderstorm Wind"
| top 3 by DamageProperty
```
will be compiled to SQL that is similar to:
```sql
SELECT *
FROM StormEvents
WHERE DamageProperty > 5000 AND EventType = 'Thunderstorm Wind'
ORDER BY DamageProperty DESC
LIMIT 3;
```
[Kusto Query Language]: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/
[Clickhouse SQL dialect]: https://clickhouse.com/docs/en/sql-reference
## Getting Started
If you'd like to see a demo along with some examples, check out https://pql.dev.
To use pql in your go code, a minimal example might look like this
```
package main
import (
"github.com/runreveal/pql"
)
func main() {
sql, err := pql.Compile("users | project id, email | limit 5")
if err != nil {
panic(err)
}
println(sql)
}
```
Running this program should give you the following output
```
$ go run test.go
WITH "__subquery0" AS (SELECT "id" AS "id", "email" AS "email" FROM "users")
SELECT * FROM "__subquery0" LIMIT 5;
```
## Documentation
The following tabular operators are supported and the Microsoft KQL
documentation is representative of the current pql api.
- [`as`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/as-operator)
- [`count`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/count-operator)
- [`join`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/join-operator)
- [`let` statements](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/let-statement),
but only scalar expressions are supported.
- [`project`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/project-operator)
- [`extend`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/extend-operator)
- [`sort`/`order`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/sort-operator)
- [`summarize`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/summarize-operator)
- [`take`/`limit`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/take-operator)
- [`top`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/top-operator)
- [`where`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/where-operator)
The following scalar functions are implemented within pql. Functions not in this
list will be passed through to the underlying SQL engine. This allows the usage
of the full APIs implemented by the underlying engine.
- [`not`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/not-function)
- [`now`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/now-function)
- [`isnull`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/isnull-function)
and [`isnotnull`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/isnotnull-function)
- [`strcat`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/strcat-function)
- [`iff`/`iif`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/iff-function)
- [`count`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/count-aggregation-function)
- [`countif`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/countif-aggregation-function)
- [`tolower`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/tolower-function)
- [`toupper`](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/toupper-function)
Column names with special characters can be escaped with backticks.
## Get involved
- Join our [discord](https://discord.gg/NZS9QtCJXt)
- Contribute a [scalar function](./CONTRIBUTING.md)
## License
[Apache 2.0](LICENSE)