{"id":25487942,"url":"https://github.com/anonymity94/spl2dsl","last_synced_at":"2025-04-09T22:15:45.716Z","repository":{"id":126761874,"uuid":"350177015","full_name":"Anonymity94/spl2dsl","owner":"Anonymity94","description":"Convert Splunk SPL to Elasticsearch DSL with pegjs","archived":false,"fork":false,"pushed_at":"2022-04-25T02:05:20.000Z","size":127,"stargazers_count":13,"open_issues_count":4,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-09T22:15:40.879Z","etag":null,"topics":["dsl","elasticsearch","pegjs","splunk"],"latest_commit_sha":null,"homepage":"https://anonymity94.github.io/spl2dsl/examples/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Anonymity94.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-22T02:10:25.000Z","updated_at":"2024-10-23T09:14:57.000Z","dependencies_parsed_at":"2023-06-17T23:33:39.936Z","dependency_job_id":null,"html_url":"https://github.com/Anonymity94/spl2dsl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anonymity94%2Fspl2dsl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anonymity94%2Fspl2dsl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anonymity94%2Fspl2dsl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anonymity94%2Fspl2dsl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Anonymity94","download_url":"https://codeload.github.com/Anonymity94/spl2dsl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119289,"owners_count":21050755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dsl","elasticsearch","pegjs","splunk"],"created_at":"2025-02-18T20:38:07.400Z","updated_at":"2025-04-09T22:15:45.706Z","avatar_url":"https://github.com/Anonymity94.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Splunk-SPL-to-ElasticSearch-DSL\n\n基于 `Splunk` 的 `SPL` 查询语言转换成 `ElasticSearch` 的 `DSL`。\n\n~~转换结果和 [SQL access » SQL Translate API](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/sql-translate.html) 对齐。~~\n\n可以配置 [Wrapper query](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-wrapper-query.html) 进行表达式搜索。\n\n## Usage\n\n```js\nconst converter = require(\"./lib/converter\")\n\ntry {\n  const { target, dev } = converter.parse(`| search a=1 and b=2`);\n  // 完整的es搜索语句\n  console.log(target)\n  // 一些查询字段值，目前只存放了时间范围\n  // 以后可以在基础上拓展，返回所有的查询字段，方便自定义校验字段名和字段值\n  console.log(dev)\n} catch (error) {\n  console.log(error.message);\n}\n\n```\n\n```html\n\u003cscript src=\"../lib/converter.min.js\"\u003e\u003c/script\u003e\n\n\u003cscript\u003e\n  try {\n    var result = splToDslConverter.parse(value, {\n      json: true,\n    });\n  } catch (error) {\n    console.log(error);\n  }\n\u003c/script\u003e\n```\n\n\n\n## 开发\n\n```sh\nyarn\n\n# build\nyarn build\n\n# test\nyarn test\n```\n\n## 一个完整的搜索\n\n```\n# `ip_initiator` 为 `'10.0.0.1'`\n# 并且 `ip_protocol` 的值为 `TCP` 或 `UDP`\n# 并且 `port_initiator` 大于 `80`\n# 并且 `port_initiator` 小于 `100`\n# 并且 `start_time` 的值在7天前到现在之间\n# 以 `start_time` 倒序排序\n# 返回30条数据\n\nip_initiator = '10.0.0.1' AND ip_protocol in ('TCP', 'UDP') AND port_initiator \u003e 80 AND port_initiator \u003c 100\n| gentimes start_time start=now-7d end=now\n| sort -start_time\n| head 30\n```\n\n## 语法说明\n\n```\n# 搜索表名，可以省略\n[source \u003ctableName\u003e]\n# 搜索字段\n[[| search] \u003cfield-name\u003e \u003coperate\u003e \u003cfield-value\u003e] [\u003clogical-connector\u003e \u003cfield-name\u003e \u003coperate\u003e \u003cfield-value\u003e]]\n\n# 限制时间\n[| gentimes \u003ctime-field\u003e start \u003ctime-value\u003e [end \u003ctime-value\u003e]]\n\n# 排序,+为正序，-为倒序\n[| sort \u003csort-operate\u003e \u003csort-field\u003e [, \u003csort-operate\u003e \u003csort-field\u003e]]\n\n# 返回前多少条\n[| head \u003cint\u003e]\n\n```\n\n\n\n## 参数说明\n\n|         参数          |    名称    | 描述                                                         |\n| :-------------------: | :--------: | ------------------------------------------------------------ |\n|    `\u003cfield-name\u003e`     |   字段名   | 允许输入大小字母、数字、下划线[`_`]、英文的点[`.`]。例如：`start_time`、`cup.usage`。\u003cbr /\u003e支持前缀符号 `@` 或 `_`。例如 `@timestamp` `_timestamp` |\n|      `\u003coperate\u003e`      |   操作符   | `=`、`!=`、`\u003e`、`\u003e=`、`\u003c`、`\u003c=`                              |\n|    `\u003cfield-value\u003e`    |   字段值   | 允许输入大小字母、数字、下划线[`_`]、英文的点[`.`]、冒号[`:`]、正斜杠[`/`]、通配符[`*`]、通配符[`?`]。\u003cbr /\u003e允许内容被单引号[`''`]或双引号[`\"\"`]包裹。含有通配符时，将会使用ES中的[Wildcard query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html)\u003cbr /\u003e例如：`12`、`\"1.2\"`、`\"中国\"`、`\"a_b\"` |\n| `\u003clogical-connector\u003e` | 逻辑关系符 | `and`、`AND`、`or`、`OR`、`\u0026\u0026`、`||`                         |\n|    `\u003ctime-field\u003e`     | 时间字段名 | 同`\u003cfield-name\u003e`                                             |\n|    `\u003ctime-value\u003e`     | 时间内容值 | [时间范围](#时间范围)                                        |\n|    `\u003csort-field\u003e`     | 排序字段名 | 同`\u003cfield-name\u003e`                                             |\n|   `\u003csort-operate\u003e`    |  排序符号  | `+` 正序\u003cbr /\u003e`-` 倒序                                       |\n\n\n## Demo\n\n### 时间条件\n\n```\n| gentimes start_time start=2020-07-13T00:00:00+08 end=2020-07-13T23:59:59+08\n\n// end时间可以省略，下面2个查询条件是等价的\n| gentimes start_time start=now-2d\n| gentimes start_time start=now-2d end=now\n\n| gentimes start_time start=1594569600000 end=1594624363506\n```\n\n### 字段条件\n\n⚠️ 开头的 `| search` 可省略\n\n#### 查询一个字段\n\n```\n| search a=1\n等价于\n a=1\n```\n\n#### 使用逻辑关系表达式查询多个字段\n\n```\n | search a=1 and b\u003e4\n a=1 \u0026\u0026 (b=1 AND (c=\"2\" OR c='3')) OR d!='2'\n | search a=1 and b in ('2','3','4')\n | search a=1 or b in ('2','3','4')\n```\n\n#### 模糊查询\n\n⚠️ 为了保证搜索性能，请避免使用 * 或开头模式 ?\n\n支持两个通配符运算符： \n\n- `?`，它与任何单个字符匹配\n- `*`，可以匹配零个或多个字符，包括一个空字符\n\n\n\n例1，匹配 `kiy`、` kity` 或  `kimchy`\n\n```\n| search a=\"ki*y\"\n```\n\n\n\n例2，匹配 `C1K0-KD345`、` C2K5-DFG65`、 `C4K8-UI365`\n\n```\n# 搜索以C开头，第一个字符必须为C，第二字符随意，第三个字符必须是K\n| search a=\"C?K*\"\n```\n\n#### 查询范围\n\n```\n| search a\u003e1 and a\u003c10\n| search a\u003e1 and a\u003c=10\n| search a\u003e=1 and a\u003c=10\n```\n\n\n\n#### 字段命中多个值\n\n```| search a in (2,5,6)\n等价于\n| search a=2 and a=5 and a=6\n```\n\n#### 字段排除多个值\n\n```\n| search a NOT IN (2,5,6)\n等价于\n| search a!=2 and a!=5 and a!=6\n```\n\n#### 操作符 `EXISTS`\n\n\u003e [query-dsl-exists-query](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/query-dsl-exists-query.html)\n\u003e Returns documents that contain an indexed value for a field.\\n\n\u003e An indexed value may not exist for a document’s field due to a variety of reasons:\n\u003e - The field in the source JSON is null or []\n\u003e - The field has \"index\" : false set in the mapping\n\u003e - The length of the field value exceeded an ignore_above setting in the mapping\n\u003e - The field value was malformed and ignore_malformed was defined in the mapping\n\n`ES` 中只会排除 `NULL` 或 `[]`这 2 类值，我给做出了拓展，新增了空字符串 `''`，这 3 类值以外的其他的都会被命中。\n\n```json\n# name 字段不为空\nname EXISTS\n```\n\n#### 操作符 `NOT_EXISTS`\n\n搜索不存在值的字段，字段值为 `''` 或 `NULL` 或 `[]` 时会被命中。\n\n```json\n# name 不存在值\nname NOT_EXISTS\n```\n\n### 限制返回条数\n\n```\n# 返回前100条数据\n| head 100\n```\n\n###  排序\n\n```\n# create_time倒序，state正序\n| sort -create_time, +state\n```\n\n\n\n## 时间范围\n\n针对时间格式做处理一些调整，这里的时间格式和`Splunk`中标准的时间格式不同。\n\n#### splunk标准格式\n\n`Splunk` 中的时间格式为：`| gentimes start=\u003ctimestamp\u003e [end=\u003ctimestamp\u003e] [increment=\u003cincrement\u003e]` [Gentimes文档](https://docs.splunk.com/Documentation/Splunk/8.0.5/SearchReference/Gentimes)\n\n其中 `timestamp` 的格式为：`MM/DD/YYYY[:HH:MM:SS] | \u003cint\u003e`\n\n\n\n---\n\n#### 修改后的时间内容值\n\n`| gentimes \u003ctime-field\u003e start=\u003ctime-value\u003e [end=\u003ctime-value\u003e]`\n\n时间的内容值可以分为**相对时间**和**绝对时间**：\n\n- 相对时间\n\n  - `now` 当前时间\n\n  - `now-\u003cint\u003e(y | M | w | d | H | h | m | s)`\n\n    | 单位       | 说明      |\n    | ---------- | --------- |\n    | `y`        | `Year`    |\n    | `M`        | `Months`  |\n    | `w`        | `Weeks`   |\n    | `d`        | `Days`    |\n    | `h` or `H` | `Hours`   |\n    | `m`        | `Minutes` |\n    | `s`        | `Seconds` |\n\n    例如：`now-7d`，7天前\n\n- 绝对时间\n\n  - `2017-04-01T12:34:56+08`\n  - `2017-04-01T12:34:56+0800`\n  - `2017-04-01T12:34:56+08:00`\n  - 时间戳（毫秒）\n\n#### 使用Demo\n\n- `| gentimes time-field start=2020-07-13T00:00:00+08 end=2020-07-13T23:59:59+08`\n- `| gentimes start=now-7d end=now`\n- `| gentimes start=1594569600000 end=1594624363506`\n\n\n\n## Links\n\n- [Splunk Search Reference](https://docs.splunk.com/Documentation/Splunk/8.0.4/SearchReference/Abstract)\n- [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/query-dsl.html)\n- [查询Elasticsearch中的数据 (基于DSL的查询, 包括validate、match、bool)](https://www.cnblogs.com/shoufeng/p/11096521.html)\n- [SQL access » SQL Translate API](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/sql-translate.html)\n- [PEG.js Online Version](https://pegjs.org/online)\n\n\n## FAQ\n\n### 🤔`terms` or `match`?\n\n- [Term Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html)精确查询，对查询的值不分词,直接进倒排索引去匹配。\n- [Match Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html) ：模糊查询，对查询的值分词，对分词的结果一一进入倒排索引去匹配\n\n\n\n### 🤔 `GET` 查询中加不加`.keyword`?\n\n--\n\n### 🤔 `filter` 和 `query` 查询的不同?\n\n[Elasticsearch DSL中Query与Filter的区别](https://blog.csdn.net/xifeijian/article/details/50823110)\n[Elasticsearch filter和query的不同](https://blog.csdn.net/wojiushiwo987/article/details/80468757)\n\n### 🤔 前缀匹配查询？通配符查询？\n\n[Prefix Query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-prefix-query.html)\n[Wildcard query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html)\n\n\n## 参考\n- [Wrapper query](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-wrapper-query.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanonymity94%2Fspl2dsl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanonymity94%2Fspl2dsl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanonymity94%2Fspl2dsl/lists"}