{"id":21026752,"url":"https://github.com/chaokunyang/jkes","last_synced_at":"2026-02-27T01:32:59.680Z","repository":{"id":113392772,"uuid":"99287959","full_name":"chaokunyang/jkes","owner":"chaokunyang","description":"A search framework and multi-tenant search platform based on java, kafka, kafka connect, elasticsearch","archived":false,"fork":false,"pushed_at":"2024-04-01T07:32:55.000Z","size":1696,"stargazers_count":171,"open_issues_count":0,"forks_count":85,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-05-20T11:06:16.110Z","etag":null,"topics":["elasticsearch","java","kafka","kafka-connector","odm","rest-api","search"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chaokunyang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-04T01:00:13.000Z","updated_at":"2025-03-07T08:34:14.000Z","dependencies_parsed_at":"2024-11-19T11:56:34.713Z","dependency_job_id":"d2e3aa56-22ca-476f-b50d-521917373659","html_url":"https://github.com/chaokunyang/jkes","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/chaokunyang/jkes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaokunyang%2Fjkes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaokunyang%2Fjkes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaokunyang%2Fjkes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaokunyang%2Fjkes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chaokunyang","download_url":"https://codeload.github.com/chaokunyang/jkes/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaokunyang%2Fjkes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29880786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T23:51:21.483Z","status":"ssl_error","status_checked_at":"2026-02-26T23:50:46.793Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","java","kafka","kafka-connector","odm","rest-api","search"],"created_at":"2024-11-19T11:46:05.523Z","updated_at":"2026-02-27T01:32:59.647Z","avatar_url":"https://github.com/chaokunyang.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Jkes\nJkes是一个基于Java、Kafka、ElasticSearch的搜索框架。Jkes提供了注解驱动的JPA风格的对象/文档映射，使用rest api用于文档搜索。\n\n## 安装\n可以参考[`jkes-integration-test`](https://github.com/chaokunyang/jkes/tree/master/jkes-integration-test)项目快速掌握jkes框架的使用方法。[`jkes-integration-test`](https://github.com/chaokunyang/jkes/tree/master/jkes-integration-test)是我们用来测试功能完整性的一个Spring Boot Application。\n\n- 安装`jkes-index-connector`和`jkes-delete-connector`到Kafka Connect类路径\n- 安装 Smart Chinese Analysis Plugin\n```shell\nsudo bin/elasticsearch-plugin install analysis-smartcn\n```\n\n## 配置\n- 引入jkes-spring-data-jpa依赖\n- 添加配置\n```java\n@EnableAspectJAutoProxy\n@EnableJkes\n@Configuration\npublic class JkesConfig {\n\n  @Bean\n  public PlatformTransactionManager transactionManager(EntityManagerFactory factory, EventSupport eventSupport) {\n\n    return new SearchPlatformTransactionManager(new JpaTransactionManager(factory), eventSupport);\n  }\n}\n```\n- 提供JkesProperties Bean\n```java\n@Component\n@Configuration\npublic class JkesConf extends DefaultJkesPropertiesImpl {\n\n    @PostConstruct\n    public void setUp() {\n        Config.setJkesProperties(this);\n    }\n\n    @Override\n    public String getKafkaBootstrapServers() {\n        return \"k1-test.com:9292,k2-test.com:9292,k3-test.com:9292\";\n    }\n\n    @Override\n    public String getKafkaConnectServers() {\n        return \"http://k1-test.com:8084,http://k2-test.com:8084,http://k3-test.com:8084\";\n    }\n\n    @Override\n    public String getEsBootstrapServers() {\n        return \"http://es1-test.com:9200,http://es2-test.com:9200,http://es3-test.com:9200\";\n    }\n\n    @Override\n    public String getDocumentBasePackage() {\n        return \"com.timeyang.jkes.integration_test.domain\";\n    }\n\n    @Override\n    public String getClientId() {\n        return \"integration_test\";\n    }\n\n}\n```\n这里可以很灵活，如果使用Spring Boot，可以使用`@ConfigurationProperties`提供配置\n\n- 增加索引管理端点\n因为我们不知道客户端使用的哪种web技术，所以索引端点需要在客户端添加。比如在`Spring MVC`中，可以按照如下方式添加索引端点\n```java\n@RestController\n@RequestMapping(\"/api/search\")\npublic class SearchEndpoint {\n\n    private Indexer indexer;\n\n    @Autowired\n    public SearchEndpoint(Indexer indexer) {\n        this.indexer = indexer;\n    }\n\n    @RequestMapping(value = \"/start_all\", method = RequestMethod.POST)\n    public void startAll() {\n        indexer.startAll();\n    }\n\n    @RequestMapping(value = \"/start/{entityClassName:.+}\", method = RequestMethod.POST)\n    public void start(@PathVariable(\"entityClassName\") String entityClassName) {\n        indexer.start(entityClassName);\n    }\n\n    @RequestMapping(value = \"/stop_all\", method = RequestMethod.PUT)\n    public Map\u003cString, Boolean\u003e stopAll() {\n        return indexer.stopAll();\n    }\n\n    @RequestMapping(value = \"/stop/{entityClassName:.+}\", method = RequestMethod.PUT)\n    public Boolean stop(@PathVariable(\"entityClassName\") String entityClassName) {\n        return indexer.stop(entityClassName);\n    }\n\n    @RequestMapping(value = \"/progress\", method = RequestMethod.GET)\n    public Map\u003cString, IndexProgress\u003e getProgress() {\n        return indexer.getProgress();\n    }\n\n}\n```\n## 快速开始\n### 索引API\n使用`com.timeyang.jkes.core.annotation`包下相关注解标记实体\n```java\n@lombok.Data\n@Entity\n@Document\npublic class Person extends AuditedEntity {\n\n    // @Id will be identified automatically\n    // @Field(type = FieldType.Long)\n    @Id\n    @GeneratedValue(strategy = GenerationType.IDENTITY)\n    private Long id;\n\n    @MultiFields(\n            mainField = @Field(type = FieldType.Text),\n            otherFields = {\n                    @InnerField(suffix = \"raw\", type = FieldType.Keyword),\n                    @InnerField(suffix = \"english\", type = FieldType.Text, analyzer = \"english\")\n            }\n    )\n    private String name;\n\n    @Field(type = FieldType.Keyword)\n    private String gender;\n\n    @Field(type = FieldType.Integer)\n    private Integer age;\n\n    // don't add @Field to test whether ignored\n    // @Field(type = FieldType.Text)\n    private String description;\n\n    @Field(type = FieldType.Object)\n    @ManyToOne(fetch = FetchType.EAGER)\n    @JoinColumn(name = \"group_id\")\n    private PersonGroup personGroup;\n\n}\n```\n```java\n@lombok.Data\n@Entity\n@Document(type = \"person_group\", alias = \"person_group_alias\")\npublic class PersonGroup extends AuditedEntity {\n    @Id\n    @GeneratedValue(strategy = GenerationType.IDENTITY)\n    private Long id;\n    private String name;\n    private String interests;\n    @OneToMany(fetch = FetchType.EAGER, cascade = CascadeType.ALL, mappedBy = \"personGroup\", orphanRemoval = true)\n    private List\u003cPerson\u003e persons;\n    private String description;\n\n    @DocumentId\n    @Field(type = FieldType.Long)\n    public Long getId() {\n        return id;\n    }\n\n    @MultiFields(\n            mainField = @Field(type = FieldType.Text),\n            otherFields = {\n                    @InnerField(suffix = \"raw\", type = FieldType.Keyword),\n                    @InnerField(suffix = \"english\", type = FieldType.Text, analyzer = \"english\")\n            }\n    )\n    public String getName() {\n        return name;\n    }\n\n    @Field(type = FieldType.Text)\n    public String getInterests() {\n        return interests;\n    }\n\n    @Field(type = FieldType.Nested)\n    public List\u003cPerson\u003e getPersons() {\n        return persons;\n    }\n\n    /**\n     * 不加Field注解，测试序列化时是否忽略\n     */\n    public String getDescription() {\n        return description;\n    }\n}\n```\n当更新实体时，文档会被自动索引到ElasticSearch；删除实体时，文档会自动从ElasticSearch删除。\n\n### 搜索API\n启动搜索服务[jkes-search-service](https://github.com/chaokunyang/jkes/tree/master/jkes-services/jkes-search-service)，搜索服务是一个Spring Boot Application，提供rest搜索api，默认运行在9000端口。\n- URI query\n```\ncurl -XPOST localhost:9000/api/v1/integration_test_person_group/person_group/_search?from=3\u0026size=10\n```\n\n- Nested query\n```\nintegration_test_person_group/person_group/_search?from=0\u0026size=10\n{\n  \"query\": {\n    \"nested\": {\n      \"path\": \"persons\",\n      \"score_mode\": \"avg\",\n      \"query\": {\n        \"bool\": {\n          \"must\": [\n            {\n              \"range\": {\n                \"persons.age\": {\n                  \"gt\": 5\n                }\n              }\n            }\n          ]\n        }\n      }\n    }\n  }\n}\n```\n- match query\n```\nintegration_test_person_group/person_group/_search?from=0\u0026size=10\n{\n  \"query\": {\n      \"match\": {\n        \"interests\": \"Hadoop\"\n      }\n    }\n}\n```\n- bool query\n```\n{\n  \"query\": {\n    \"bool\" : {\n      \"must\" : {\n        \"match\" : { \"interests\" : \"Hadoop\" }\n      },\n      \"filter\": {\n        \"term\" : { \"name.raw\" : \"name0\" }\n      },\n      \"should\" : [\n        { \"match\" : { \"interests\" : \"Flink\" } },\n        {\n            \"nested\" : {\n                \"path\" : \"persons\",\n                \"score_mode\" : \"avg\",\n\n                \"query\" : {\n                    \"bool\" : {\n                        \"must\" : [\n                        { \"match\" : {\"persons.name\" : \"name40\"} },\n                        { \"match\" : {\"persons.interests\" : \"interests\"} }\n                        ],\n                        \"must_not\" : {\n                            \"range\" : {\n                              \"age\" : { \"gte\" : 50, \"lte\" : 60 }\n                            }\n                          }\n                    }\n                }\n            }\n        }\n\n      ],\n      \"minimum_should_match\" : 1,\n      \"boost\" : 1.0\n    }\n\n  }\n\n}\n```\n- Source filtering\n```\nintegration_test_person_group/person_group/_search\n{\n    \"_source\": false,\n    \"query\" : {\n        \"match\" : { \"name\" : \"name17\" }\n    }\n}\n```\n```\nintegration_test_person_group/person_group/_search\n{\n    \"_source\": {\n            \"includes\": [ \"name\", \"persons.*\" ],\n            \"excludes\": [ \"date*\", \"version\", \"persons.age\" ]\n        },\n    \"query\" : {\n        \"match\" : { \"name\" : \"name17\" }\n    }\n}\n```\n- prefix\n```\nintegration_test_person_group/person_group/_search\n{ \n  \"query\": {\n    \"prefix\" : { \"name\" : \"name\" }\n  }\n}\n```\n- wildcard\n```\nintegration_test_person_group/person_group/_search\n{\n    \"query\": {\n        \"wildcard\" : { \"name\" : \"name*\" }\n    }\n}\n```\n- regexp\n```\nintegration_test_person_group/person_group/_search\n{\n    \"query\": {\n        \"regexp\":{\n            \"name\": \"na.*17\"\n        }\n    }\n}\n```\n\n## Jkes工作原理\n索引工作原理：\n- 应用启动时，Jkes扫描所有标注`@Document`注解的实体，为它们构建元数据。\n- 基于构建的元数据，创建`index`和`mapping`Json格式的配置，然后通过`ElasticSearch Java Rest Client`将创建/更新`index`配置。\n- 为每个文档创建/更新`Kafka ElasticSearch Connector`，用于创建/更新文档\n- 为整个项目启动/更新`Jkes Deleter Connector`，用于删除文档\n- 拦截数据操作方法。将`* save(*)`方法返回的数据包装为`SaveEvent`保存到`EventContainer`；使用`(* delete*(..)`方法的参数，生成一个`DeleteEvent/DeleteAllEvent`保存到`EventContainer`。\n- 拦截事务。在事务提交后使用`JkesKafkaProducer`发送`SaveEvent`中的实体到Kafka，Kafka会使用我们提供的`JkesJsonSerializer`序列化指定的数据，然后发送到Kafka。\n- 与`SaveEvent`不同，`DeleteEvent`会直接被序列化，然后发送到Kafka，而不是只发送一份数据\n- 与`SaveEvent`和`DeleteEvent`不同，`DeleteAllEvent`不会发送数据到Kafka，而是直接通过`ElasticSearch Java Rest Client`删除相应的`index`，然后重建该索引，重启`Kafka ElasticSearch Connector`\n\n查询工作原理：\n- 查询服务通过rest api提供\n- 我们没有直接使用ElasticSearch进行查询，因为我们需要在后续版本使用机器学习进行搜索排序，而直接与ElasticSearch进行耦合，会增加搜索排序API的接入难度\n- 查询服务是一个Spring Boot Application，使用docker打包为镜像\n- 查询服务提供多版本API，用于API进化和兼容\n- 查询服务解析`json`请求，进行一些预处理后，使用`ElasticSearch Java Rest Client`转发到ElasticSearch，将得到的响应进行解析，进一步处理后返回到客户端。\n- 为了便于客户端人员开发，查询服务提供了一个[查询UI界面](http://localhost:9000/api/v1)，开发人员可以在这个页面得到预期结果后再把json请求体复制到程序中。\n\n## 流程图\n![Jkes流程图](https://raw.githubusercontent.com/chaokunyang/jkes/master/docs/images/Jkes%20Architecture.png)\n\n## 模块介绍\n### jkes-core\n`jkes-core`是整个`jkes`的核心部分。主要包括以下功能：\n- `annotation`包提供了jkes的核心注解\n- `elasticsearch`包封装了`elasticsearch`相关的操作，如为所有的文档创建/更新索引，更新mapping\n- `kafka`包提供了Kafka 生产者，Kafka Json Serializer，Kafka Connect Client\n- `metadata`包提供了核心的注解元数据的构建与结构化模型\n- `event`包提供了事件模型与容器\n- `exception`包提供了常见的Jkes异常\n- `http`包基于`Apache Http Client`封装了常见的http json请求\n- `support`包暴露了Jkes核心配置支持\n- `util`包提供了一些工具类，便于开发。如：Asserts, ClassUtils, DocumentUtils, IOUtils, JsonUtils, ReflectionUtils, StringUtils\n\n### jkes-boot\n`jkes-boot`用于与一些第三方开源框架进行集成。\n\n当前，我们通过`jkes-spring-data-jpa`，提供了与`spring data jpa`的集成。通过使用Spring的AOP机制，对`Repository`方法进行拦截，生成`SaveEvent/DeleteEvent/DeleteAllEvent`保存到`EventContainer`。通过使用我们提供的`SearchPlatformTransactionManager`，对常用的事务管理器(如`JpaTransactionManager`)进行包装，提供事务拦截功能。\n\n在后续版本，我们会提供与更多框架的集成。\n\n`jkes-spring-data-jpa`说明：\n- `ContextSupport`类用于从bean工厂获取`Repository Bean`\n- `@EnableJkes`让客户端能够轻松开启Jkes的功能，提供了与Spring一致的配置模型\n- `EventSupport`处理事件的细节，在保存和删除数据时生成相应事件存放到`EventContainer`，在事务提交和回滚时处理相应的事件\n- `SearchPlatformTransactionManager`包装了客户端的事务管理器，在事务提交和回滚时加入了`回调hook`\n- `audit`包提供了一个简单的`AuditedEntity`父类，方便添加审计功能，版本信息可用于结合`ElasticSearch`的版本机制保证不会索引过期文档数据\n- `exception`包封装了常见异常\n- `intercept`包提供了AOP切点和切面\n- `index`包提供了`全量索引`功能。当前，我们提供了基于`线程池`的索引机制和基于`ForkJoin`的索引机制。在后续版本，我们会重构代码，增加基于`阻塞队列`的`生产者-消费者`模式，提供并发性能\n- \n\n### jkes-services\n`jkes-services`主要用来提供一些服务。\n目前，`jkes-services`提供了以下服务：\n- `jkes-delete-connector`\n    \n  - `jkes-delete-connector`是一个`Kafka Connector`，用于从kafka集群获取索引删除事件(`DeleteEvent`)，然后使用`Jest Client`删除ElasticSearch中相应的文档。\n   \n  - 借助于Kafka Connect的rest admin api，我们轻松地实现了多租户平台上的文档删除功能。只要为每个项目启动一个`jkes-delete-connector`，就可以自动处理该项目的文档删除工作。避免了每启动一个新的项目，我们都得手动启动一个Kafka Consumer来处理该项目的文档删除工作。尽管可以通过正则订阅来减少这样的工作，但是还是非常不灵活\n    \n- `jkes-search-service`\n  \n  - `jkes-search-service`是一个restful的搜索服务，提供了多版本的rest query api。查询服务提供多版本API，用于API进化和兼容\n  - `jkes-search-service`目前支持URI风格的搜索和JSON请求体风格的搜索。\n  - 我们没有直接使用ElasticSearch进行查询，因为我们需要在后续版本使用机器学习进行搜索排序，而直接与ElasticSearch进行耦合，会增加搜索排序的接入难度\n  - 查询服务是一个Spring Boot Application，使用docker打包为镜像\n  - 查询服务解析`json`请求，进行一些预处理后，使用`ElasticSearch Java Rest Client`转发到ElasticSearch，将得到的响应进行解析，进一步处理后返回到客户端。\n  - 为了便于客户端人员开发，查询服务提供了一个[查询UI界面](http://localhost:9000/api/v1)，开发人员可以在这个页面得到预期结果后再把json请求体复制到程序中。\n\n后续，我们将会基于`zookeeper`构建索引集群，提供集群索引管理功能\n\n### jkes-integration-test\n`jkes-integration-test`是一个基于Spring Boot集成测试项目，用于进行`功能测试`。同时测量一些常见操作的`吞吐率`\n\n## Development\nTo build a development version you'll need a recent version of Kafka. You can build jkes with Maven using the standard lifecycle phases.\n\n## Contribute\n- Source Code: https://github.com/chaokunyang/jkes\n- Issue Tracker: https://github.com/chaokunyang/jkes/issues\n\n## LICENSE\nThis project is licensed under Apache License 2.0.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaokunyang%2Fjkes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchaokunyang%2Fjkes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaokunyang%2Fjkes/lists"}