{"id":22275598,"url":"https://github.com/wyl/es-libr","last_synced_at":"2026-04-13T23:31:51.524Z","repository":{"id":248991740,"uuid":"828879597","full_name":"wyl/es-libr","owner":"wyl","description":"Reduce reported data and optimize Elasticsearch response speed","archived":false,"fork":false,"pushed_at":"2024-12-30T03:59:10.000Z","size":2235,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-30T15:19:25.820Z","etag":null,"topics":["elasticsearch","koa","koa-body","mongo","ndjson"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wyl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-15T10:04:25.000Z","updated_at":"2024-12-30T03:59:13.000Z","dependencies_parsed_at":"2024-07-21T07:47:17.754Z","dependency_job_id":"1e0913bb-c36d-4d11-9af9-344359db7a1a","html_url":"https://github.com/wyl/es-libr","commit_stats":null,"previous_names":["wyl/es-libr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyl%2Fes-libr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyl%2Fes-libr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyl%2Fes-libr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wyl%2Fes-libr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wyl","download_url":"https://codeload.github.com/wyl/es-libr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245509066,"owners_count":20626901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","koa","koa-body","mongo","ndjson"],"created_at":"2024-12-03T14:10:08.495Z","updated_at":"2026-04-13T23:31:46.498Z","avatar_url":"https://github.com/wyl.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\n\n通过此程序减少发送到 ES 的数据量，之前的任何使用方式都不需要做更改。\n通过优化数据量，来优化响应速度和减少成本\n\n![Request Follow](image/Request-Follow.png)\n\n# ES 使用姿势：\n\n- 仅满足搜索条件的数据结构\n- 除搜索需求之外，报表属性\n- 除搜索需求之外，报表属性，业务展示需求的属性\n- 原始数据结构\n\n# Key Feature\n\n## 定义数据 Mapping\n\n[Index Mapping Demo](index-mappings/blog-post.ts)\n定义好 Es 索引结构，上报的数据只会是定义的数据结构。\n\n\u003e 服务使用 NodeJs Koa 框架开发，使用 Middleware 来拦截修改请求数据，在 Request 和 Response 中修改数据逻辑，请求路径和返回值都是 Elasticsearch 的数据。\n\n## 装载索引\n\n[Index](index-mappings/index.ts)\n\n```\nconst indexMappingList = [mapping, blogPostIndexMapping]\n```\n\n\u003e 完成以上步骤，像使用 ES 一样去使用此服务。\n\n# Example\n\n```\n{\n  \"title\": \"Nest eggs\",\n  \"body\":  \"Making your money work...\",\n  \"tags\":  [ \"cash\", \"shares\" ],\n  \"comments\": [\n    {\n      \"name\":    \"John Smith\",\n      \"comment\": \"Great article\",\n      \"age\":     28,\n      \"stars\":   4,\n      \"date\":    \"2014-09-01\"\n    },\n    {\n      \"name\":    \"Alice White\",\n      \"comment\": \"More like this please\",\n      \"age\":     31,\n      \"stars\":   5,\n      \"date\":    \"2014-10-22\"\n    }\n  ]\n}\n```\n\n如上，我们有一个原始文档。搜索需求需要根据 `title`、`tags` 和 `comments` 中的 `age`、`stars` 做检索。\n\n## 定义索引\n\n```\nconst blogPostIndexMapping: T.IndicesPutMappingRequest =\n  {\n    index: 'my_index',\n    properties: {\n      title: {type: 'text'},\n      // body: {type: 'text'},\n      tags: {type: 'keyword'},\n      comments: {\n        type: 'nested',\n        properties: {\n          // name: {type: 'keyword'},\n          age: {type: 'integer'},\n          stars: {type: 'integer'},\n          // date: {type: 'date'},\n        }\n      }\n      // address: {type: 'keyword'},\n    }\n}\n\n```\n\n## 装载索引\n\n## 发起请求\n\n发起 document 插入或更新请求。\n\n```\n  curl --request POST \\\n  --url http://127.0.0.1:3001/my_index/_update/1 \\\n  --header 'Authorization: ApiKey \u003cES API KEY\u003e' \\\n  --header 'Content-Type: application/json' \\\n  --header 'User-Agent: insomnia/9.3.2' \\\n  --data '{\n\t\"doc_as_upsert\": true,\n\t\"doc\": {\n\t\t\"title\": \"Nest eggs\",\n\t\t\"body\": \"Making your money work...\",\n\t\t\"tags\": [\n\t\t\t\"cash\",\n\t\t\t\"shares\"\n\t\t],\n\t\t\"comments\": [\n\t\t\t{\n\t\t\t\t\"name\": \"John Smith\",\n\t\t\t\t\"comment\": \"Great article\",\n\t\t\t\t\"age\": 28,\n\t\t\t\t\"stars\": 4,\n\t\t\t\t\"date\": \"2014-09-01\"\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"name\": \"Alice White\",\n\t\t\t\t\"comment\": \"More like this please\",\n\t\t\t\t\"age\": 31,\n\t\t\t\t\"stars\": 5,\n\t\t\t\t\"date\": \"2014-10-22\"\n\t\t\t}\n\t\t]\n\t}\n}'\n```\n\n### log\n\n![Update Log](image/Update-log.png)\n\n### 实际上报的 Body\n\n```\n{\n  \"doc\": {\n    \"title\": \"Nest eggs\",\n    \"tags\": [\n      \"cash\",\n      \"shares\"\n    ],\n    \"comments\": [\n      {\n        \"age\": 28,\n        \"stars\": 4\n      },\n      {\n        \"age\": 31,\n        \"stars\": 5\n      }\n    ]\n  },\n  \"doc_as_upsert\": true\n}\n\n```\n\n可以看到上报的数据，就是索引定义的数据结构。\n\n将更多的注意力放在业务上\n\n### 在以下方法减少上报的 body\n\n#### [Index API/Create API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)\n\n- PUT /:index/\\_doc/:\\_id\n- POST /:index/\\_doc/\n- PUT /:index/\\_create/:\\_id\n- POST /:index/\\_create/:\\_id\n\n#### [Update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html)\n\n- POST /:index/\\_update{/:\\_id}\n\n#### [Bulk API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)\n\n- POST /\\_bulk\n- POST /:target/\\_bulk\n\n[Elasticsearch Distributed Search](https://github.com/wyl/es-libr/blob/main/elasticsearch%20distributed%20search.md)\n\n### Q/A\n\n\u003cdetails\u003e\n\u003csummary\u003eQ: 定义过索引的数据提交会更改提交的数据结构，未定义索引的数据会怎样处理？\u003c/summary\u003e\n未定义会直接转发原始Body，此时这个服务本身就是ES\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eQ: 认证是怎样处理的？\u003c/summary\u003e\n认证未做任何更改，认证及返回的状态都是ES 本身的真实相应\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eQ:响应速度可能被优化到多少？\u003c/summary\u003e\n是通过优化提交ES 的数据结构，减少上报的数据结构来优化时长。响应速度可能被优化的不多，优化的是ES 的使用成本。\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwyl%2Fes-libr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwyl%2Fes-libr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwyl%2Fes-libr/lists"}