https://github.com/wangle201210/go-rag
基于eino实现知识库的rag
https://github.com/wangle201210/go-rag
eino es go llm rag
Last synced: 5 months ago
JSON representation
基于eino实现知识库的rag
- Host: GitHub
- URL: https://github.com/wangle201210/go-rag
- Owner: wangle201210
- License: apache-2.0
- Created: 2025-04-27T06:13:17.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2025-04-27T07:47:26.000Z (5 months ago)
- Last Synced: 2025-04-27T08:19:33.489Z (5 months ago)
- Topics: eino, es, go, llm, rag
- Language: Go
- Homepage:
- Size: 212 KB
- Stars: 9
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# go-rag
基于eino实现知识库的rag
## 存储层
- [x] es8存储向量相关数据
## 功能列表
- [x] md、pdf、html 文档解析
- [x] 网页解析
- [x] 文档检索
## 未来计划
- [ ] 使用mysql存储chunk和文档的映射关系，目前放在es的ext字段
## 使用
安装依赖
```bash
go get github.com/wangle201210/go-rag@latest
```
安装es8
```bash
docker run -d --name elasticsearch \
-e "discovery.type=single-node" \
-e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
-e "xpack.security.enabled=false" \
-p 9200:9200 \
-p 9300:9300 \
elasticsearch:8.18.0
```
初始化Rag对象
```go
client, err := elasticsearch.NewClient(elasticsearch.Config{
Addresses: []string{"http://localhost:9200"},
})
if err != nil {
log.Printf("NewClient of es8 failed, err=%v", err)
return
}
ragSvr, err = New(context.Background(), &config.Config{
Client: client,
IndexName: "rag",
APIKey: os.Getenv("OPENAI_API_KEY"),
BaseURL: os.Getenv("OPENAI_BASE_URL"),
Model: "text-embedding-3-large",
})
if err != nil {
log.Printf("New of rag failed, err=%v", err)
return
}
```
加载各种数据源的数据，并将其向量化后存储进向量数据库。
```golang
ids, err := ragSvr.Index("./test_file/readme.md")
if err != nil {
t.Fatal(err)
}
for _, id := range ids {
t.Log(id)
}
ragSvr.Index("./test_file/readme2.md")
ragSvr.Index("./test_file/readme.html")
ragSvr.Index("./test_file/test.pdf")
ragSvr.Index("https://deepchat.thinkinai.xyz/docs/guide/advanced-features/shortcuts.html")
... ...
```
检索
```go
msg, err := ragSvr.Retrieve("这里有很多内容", 1.5, 5)
if err != nil {
t.Fatal(err)
}
for _, m := range msg {
t.Logf("content: %v, score: %v", m.Content, m.Score())
}
msg, err = ragSvr.Retrieve("代码解析", 1.5, 5)
... ...
```
详情可以参照[test文件](./rag_test.go)