{"id":22941907,"url":"https://github.com/dot123/dht-es-search","last_synced_at":"2025-04-01T21:17:49.148Z","repository":{"id":267703112,"uuid":"902097329","full_name":"dot123/DHT-ES-Search","owner":"dot123","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-10T12:19:29.000Z","size":1063,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-07T13:48:23.260Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dot123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-11T22:41:30.000Z","updated_at":"2025-01-10T12:19:33.000Z","dependencies_parsed_at":"2025-02-07T13:54:01.058Z","dependency_job_id":null,"html_url":"https://github.com/dot123/DHT-ES-Search","commit_stats":null,"previous_names":["dot123/dht-es-search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dot123%2FDHT-ES-Search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dot123%2FDHT-ES-Search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dot123%2FDHT-ES-Search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dot123%2FDHT-ES-Search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dot123","download_url":"https://codeload.github.com/dot123/DHT-ES-Search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246709913,"owners_count":20821298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T13:45:30.682Z","updated_at":"2025-04-01T21:17:49.127Z","avatar_url":"https://github.com/dot123.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DHT 搜索引擎\n\n\u003e 本项目基于 mr-mmajoR/Bittorrent-DHT-Tracker 进行改造，主要针对搜索功能进行了优化，旨在实现海量数据的毫秒级响应。我们将原有的 MySQL 全文搜索替换为 Elasticsearch，以提升搜索效率，并对分页实现进行了优化，显著提升了系统的响应速度和用户体验。\n\n---\n\n## 项目概述\n这是一个基于 DHT 网络的 BT 种子搜索引擎，使用 Go 语言开发的分布式爬虫和搜索系统。项目采用双数据库架构，结合 MySQL 和 Elasticsearch 提供高效的数据存储和搜索服务。\n\n---\n\n## 核心技术栈\n\n### 搜索引擎迁移\n- **Elasticsearch 客户端**\n  - go-elasticsearch/v8\n  - 自定义 mapping 配置\n  - search_after 深分页\n\n### 数据同步\n- **Logstash**\n  - JDBC 输入插件\n  - 数据类型转换\n  - 增量同步配置\n\n### 分词系统\n- **IK 分词器**\n  - 中文分词支持\n  - 自定义词典\n  - 智能分词模式\n\n### 搜索优化\n- **精确匹配**\n  - operator: \"and\"\n  - minimum_should_match: \"100%\"\n  - analyzer: \"standard\"\n\n---\n\n## Web 接口实现\n\n### Elasticsearch 搜索实现\n```go\n// 搜索查询构建\nsearchQuery := map[string]interface{}{\n    \"query\": map[string]interface{}{\n        \"match\": map[string]interface{}{\n            \"textindex\": map[string]interface{}{\n                \"query\":                query,\n                \"operator\":             \"and\",\n                \"minimum_should_match\": \"100%\",\n                \"analyzer\":             \"standard\",\n                \"zero_terms_query\":     \"none\",\n            },\n        },\n    },\n    \"sort\": []map[string]interface{}{\n        {\"updated\": map[string]interface{}{\"order\": \"desc\"}},\n        {\"id\": map[string]interface{}{\"order\": \"asc\"}},\n    },\n    \"size\": pageSize,\n}\n```\n\n### 分页处理\n- 使用 search_after 实现深分页\n- 维护前后页的 sort 值\n- 支持双向导航\n```go\n// 分页参数处理\nif searchAfter != nil {\n    searchQuery[\"search_after\"] = searchAfter\n}\n\n// 获取下一页的 sort 值\nsortValues := hit.(map[string]interface{})[\"sort\"].([]interface{})\nlastSort = sortValues\n```\n\n### 排序实现\n```go\n// 根据排序方式添加排序字段\nif order == \"cnt\" {\n    searchQuery[\"sort\"] = []map[string]interface{}{\n        {\"cnt\": map[string]interface{}{\"order\": \"desc\"}},\n        {\"id\": map[string]interface{}{\"order\": \"asc\"}},\n    }\n} else {\n    searchQuery[\"sort\"] = []map[string]interface{}{\n        {\"updated\": map[string]interface{}{\"order\": \"desc\"}},\n        {\"id\": map[string]interface{}{\"order\": \"asc\"}},\n    }\n}\n```\n\n---\n\n## 数据迁移与配置\n\n### Elasticsearch 映射配置\n**mapping.json**\n```json\n{\n    \"mappings\": {\n        \"properties\": {\n            \"infohash\": {\n                \"type\": \"keyword\"\n            },\n            \"name\": {\n                \"type\": \"keyword\"\n            },\n            \"length\": {\n                \"type\": \"long\"\n            },\n            \"files\": {\n                \"type\": \"boolean\"\n            },\n            \"addeded\": {\n                \"type\": \"date\"\n            },\n            \"updated\": {\n                \"type\": \"date\"\n            },\n            \"cnt\": {\n                \"type\": \"integer\"\n            },\n            \"textindex\": {\n                \"type\": \"text\",\n                \"analyzer\": \"ik_max_word\",\n                \"search_analyzer\": \"ik_smart\"\n            }\n        }\n    }\n}\n```\n\n### IK 分词配置\n1. **安装 IK 分词器插件**\n   ```bash\n   bin/elasticsearch-plugin install https://get.infini.cloud/elasticsearch/analysis-ik/8.16.1\n   ```\n   重启 Elasticsearch 服务：\n   ```bash\n   systemctl restart elasticsearch\n   ```\n\n2. **配置 IK 自定义词库**\n   编辑 `config/analysis-ik/` 下的 `custom.txt` 文件，添加需要的自定义分词。\n\n3. **验证 IK 分词器安装**\n   ```bash\n   curl -X POST \"http://localhost:9200/_analyze\" -H 'Content-Type: application/json' -d'{\n       \"analyzer\": \"ik_max_word\",\n       \"text\": \"测试分词效果\"\n   }'\n   ```\n\n### Logstash 配置\n**mysql_to_es.conf**\n```plaintext\ninput {\n  jdbc {\n    jdbc_driver_library =\u003e \"D:/logstash-8.16.1/logstash-core/lib/jars/mysql-connector-java-5.1.49.jar\"\n    jdbc_driver_class =\u003e \"com.mysql.jdbc.Driver\"\n    jdbc_connection_string =\u003e \"jdbc:mysql://localhost:3306/dhtbt\"\n    jdbc_user =\u003e \"root\"\n    jdbc_password =\u003e \"your_password\"\n    # 查询要导入的数据\n    statement =\u003e \"SELECT id, infohash, name, length, files, addeded, updated, cnt, textindex FROM infohash\"\n    jdbc_paging_enabled =\u003e \"true\"\n    jdbc_page_size =\u003e \"1000\"\n  }\n}\n\nfilter {\n  mutate {\n    convert =\u003e { \"length\" =\u003e \"integer\" }\n    convert =\u003e { \"files\" =\u003e \"boolean\" }\n  }\n}\n\noutput {\n  elasticsearch {\n    hosts =\u003e [\"http://localhost:9200\"]\n    index =\u003e \"infohash_index\"\n    document_id =\u003e \"%{id}\" # 使用 MySQL 的 id 字段作为 Elasticsearch 的文档 ID\n    action =\u003e \"index\"\n  }\n}\n```\n\n### 数据迁移步骤\n1. **创建 Elasticsearch 索引**\n   ```bash\n   curl -X PUT \"http://localhost:9200/infohash_index\" \\\n   -H \"Content-Type: application/json\" \\\n   -d @mapping.json\n   ```\n2. **配置 Logstash 并启动服务**\n   ```bash\n   bin/logstash -f mysql_to_es.conf\n   ```\n3. **验证数据同步**\n   ```bash\n   # 检查索引文档数量\n   curl -X GET \"http://localhost:9200/infohash_index/_count\"\n\n   # 检查映射配置是否正确\n   curl -X GET \"http://localhost:9200/infohash_index/_mapping\"\n   ```\n\n---\n\n## 编译与启动\n\n### 环境要求\n- Go 1.16+\n- MySQL 5.7+\n- Elasticsearch 8.x\n- Logstash 8.x\n\n### 配置文件\n**config.json**\n```json\n{\n    \"database\": {\n        \"host\": \"(localhost:3306)\",\n        \"name\": \"dhtbt\",\n        \"user\": \"root\",\n        \"password\": \"your_password\"\n    },\n    \"elasticsearch\": {\n        \"url\": \"http://localhost:9200\"\n    },\n    \"spider\": {\n        \"port\": \"6882\"\n    },\n    \"webinterface\": {\n        \"interface\": \"\",\n        \"port\": \"8080\"\n    }\n}\n```\n\n### 编译程序\n```bash\n# 编译爬虫程序\ngo build -o spider spider.go\n\n# 编译 Web 接口程序\ngo build -o webinterface webinterface.go\n```\n\n### 启动服务\n1. 启动爬虫：\n   ```bash\n   ./spider\n   ```\n2. 启动 Web 接口：\n   ```bash\n   ./webinterface\n   ```\n\n---\n\n## 注意事项与问题处理\n\n### 常见问题\n1. **数据库连接错误**\n   - 检查 MySQL 服务是否运行\n   - 验证用户名密码是否正确\n   - 确认数据库名称是否存在\n\n2. **Elasticsearch 连接问题**\n   - 检查 ES 服务状态\n   - 验证 ES 版本兼容性\n   - 确认 mapping 配置正确\n\n3. **端口占用问题**\n   ```bash\n   lsof -i :6882\n   lsof -i :8080\n   kill -9 \u003cPID\u003e\n   ```\n\n4. **权限问题**\n   ```bash\n   chmod +x spider webinterface\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdot123%2Fdht-es-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdot123%2Fdht-es-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdot123%2Fdht-es-search/lists"}