{"id":31072472,"url":"https://github.com/liliang-cn/sqvect","last_synced_at":"2026-02-03T14:00:56.122Z","repository":{"id":308653555,"uuid":"1033618518","full_name":"liliang-cn/sqvect","owner":"liliang-cn","description":"SQLite for Vectors - Simple, fast, embeddable vector storage for Go applications.","archived":false,"fork":false,"pushed_at":"2026-01-12T13:09:02.000Z","size":26206,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-12T19:13:33.225Z","etag":null,"topics":["embeddings","llm","rag","sqlite","vector-database","vector-search"],"latest_commit_sha":null,"homepage":"https://liliang-cn.github.io/sqvect/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liliang-cn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-07T05:15:45.000Z","updated_at":"2026-01-12T13:09:06.000Z","dependencies_parsed_at":"2025-08-28T10:16:26.474Z","dependency_job_id":"2d6e532f-8483-4fd1-9f05-094c43092044","html_url":"https://github.com/liliang-cn/sqvect","commit_stats":null,"previous_names":["liliang-cn/sqvect"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/liliang-cn/sqvect","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liliang-cn%2Fsqvect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liliang-cn%2Fsqvect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liliang-cn%2Fsqvect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liliang-cn%2Fsqvect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liliang-cn","download_url":"https://codeload.github.com/liliang-cn/sqvect/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liliang-cn%2Fsqvect/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29047055,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T10:09:22.136Z","status":"ssl_error","status_checked_at":"2026-02-03T10:09:16.814Z","response_time":96,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embeddings","llm","rag","sqlite","vector-database","vector-search"],"created_at":"2025-09-16T00:54:31.007Z","updated_at":"2026-02-03T14:00:56.115Z","avatar_url":"https://github.com/liliang-cn.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sqvect\n\n[![CI/CD](https://github.com/liliang-cn/sqvect/actions/workflows/ci.yml/badge.svg)](https://github.com/liliang-cn/sqvect/actions/workflows/ci.yml)\n[![codecov](https://codecov.io/gh/liliang-cn/sqvect/v2/branch/main/graph/badge.svg)](https://codecov.io/gh/liliang-cn/sqvect/v2)\n[![Go Report Card](https://goreportcard.com/badge/github.com/liliang-cn/sqvect/v2)](https://goreportcard.com/report/github.com/liliang-cn/sqvect/v2)\n[![Go Reference](https://pkg.go.dev/badge/github.com/liliang-cn/sqvect/v2.svg)](https://pkg.go.dev/github.com/liliang-cn/sqvect/v2)\n[![GitHub release](https://img.shields.io/github/release/liliang-cn/sqvect.svg)](https://github.com/liliang-cn/sqvect/releases)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**A lightweight, embeddable vector database LIBRARY for Go AI projects.**\n\nsqvect is a **100% pure Go library** designed to be the storage kernel for your RAG applications. It provides vector storage, keyword search (FTS5), graph relationships, and chat memory management in a single SQLite file.\n\n## ✨ Features\n\n- 🪶 **Lightweight** – Single SQLite file, zero external dependencies.\n- 🚀 **RAG-Ready** – Built-in tables for **Documents**, **Chat Sessions**, and **Messages**.\n- 🔍 **Hybrid Search** – Combine **Vector Search (HNSW)** + **Keyword Search (FTS5)** with RRF fusion.\n- 🧠 **AI Agent Memory** – **Hindsight** system for long-term agent memory (World, Bank, Opinion, Observation).\n- 🛡️ **Secure** – Row-Level Security (RLS) via **ACL** fields and query filtering.\n- 🕸️ **Graph Storage** – Built-in knowledge graph with entity relationships.\n- 📊 **Quantization** – **SQ8 Quantization** reduces RAM usage by 75%.\n- ⚡ **High Performance** – Optimized WAL mode, concurrent access.\n- 🎯 **Zero Config** – Works out of the box.\n\n## 🚀 Quick Start\n\n```bash\ngo get github.com/liliang-cn/sqvect/v2\n```\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"fmt\"\n    \"github.com/liliang-cn/sqvect/v2/pkg/sqvect\"\n)\n\nfunc main() {\n    // 1. Open DB (auto-creates tables for vectors, docs, chat)\n    db, _ := sqvect.Open(sqvect.DefaultConfig(\"rag.db\"))\n    defer db.Close()\n    ctx := context.Background()\n\n    // 2. Add a Document \u0026 Vector\n    // sqvect manages the relationship between docs and chunks\n    db.Vector().CreateDocument(ctx, \u0026core.Document{ID: \"doc1\", Title: \"Go Guide\"})\n    \n    db.Quick().Add(ctx, []float32{0.1, 0.2, 0.9}, \"Go is awesome\")\n\n    // 3. Search\n    results, _ := db.Quick().Search(ctx, []float32{0.1, 0.2, 0.8}, 1)\n    fmt.Printf(\"Found: %s\\n\", results[0].Content)\n}\n```\n\n## 💡 Why sqvect?\n\n### Key Advantages\n\n**🎯 All-in-One RAG Storage**\n- Stop managing separate databases for vectors, documents, and chat history\n- Single SQLite file = easy backup, migration, and version control\n- Perfect for edge deployment and local-first applications\n\n**🚀 Developer Experience**\n- Zero configuration - works out of the box\n- Type-safe Go API with full IntelliSense support\n- Built-in RAG schemas (no ORM/SQL required)\n- Comprehensive examples for common use cases\n\n**⚡ Performance \u0026 Efficiency**\n- SQ8 quantization reduces memory by 75% (1M vectors ~1GB)\n- Multiple index types (HNSW, IVF, LSH) for different workloads\n- WAL mode + connection pooling for concurrent access\n- Efficient distance calculations\n\n**🔒 Security First**\n- Row-Level Security (ACL) built into the core\n- User-scoped queries enforce permission boundaries\n- No data leakage between tenants\n\n**🧪 Production Ready**\n- 93% test coverage on core APIs\n- Battle-tested algorithms (HNSW, RRF, PQ)\n- CI/CD + Codecov + Go Report Card badges\n- MIT license for easy integration\n\n## 🧠 Hindsight: AI Agent Memory System\n\nsqvect includes **Hindsight**, a biomimetic memory system for AI agents that mirrors how human memory works. Inspired by [vectorize-io/hindsight](https://github.com/vectorize-io/hindsight), it enables agents to learn and improve over time.\n\n### Three Core Operations\n\n```go\nimport \"github.com/liliang-cn/sqvect/v2/pkg/hindsight\"\n\nsys, _ := hindsight.New(\u0026hindsight.Config{DBPath: \"agent_memory.db\"})\n\n// RETAIN: Store memories (caller provides embeddings)\nsys.Retain(ctx, \u0026hindsight.Memory{\n    Type:     hindsight.WorldMemory,\n    Content:  \"Alice works at Google as a senior engineer\",\n    Vector:   embedding,\n    Entities: []string{\"Alice\", \"Google\"},\n})\n\n// RECALL: Search using TEMPR strategies (Temporal, Entity, Memory, Priming)\nresults, _ := sys.Recall(ctx, \u0026hindsight.RecallRequest{\n    BankID:      \"agent-1\",\n    QueryVector: queryEmbedding,\n    Strategy:    hindsight.DefaultStrategy(),\n})\n\n// OBSERVE: Reflect on memories to generate new insights\nresp, _ := sys.Observe(ctx, \u0026hindsight.ReflectRequest{\n    BankID:      \"agent-1\",\n    Query:       \"What does Alice prefer?\",\n    QueryVector: queryEmbedding,\n})\n// resp.Observations contains newly generated insights\n```\n\n### Four Memory Types\n\n| Type | Description | Example |\n|:---|:---|:---|\n| **World** | Objective facts about the world | \"Alice works at Google\" |\n| **Bank** | Agent's own experiences | \"I recommended Python to Bob\" |\n| **Opinion** | Beliefs with confidence scores | \"Python is best for ML\" (0.85) |\n| **Observation** | Insights derived from reflection | \"Users prefer concise answers\" |\n\n### TEMPR Retrieval Strategies\n\nHindsight runs four search strategies in parallel and fuses results with RRF:\n\n- **T**emporal – Time-range filtered search\n- **E**ntity – Graph-based entity relationships\n- **M**emory – Semantic vector similarity\n- **P**riming – Keyword/BM25 exact matching\n- **R**ecall – RRF fusion for ranked results\n\n### Memory Banks \u0026 Disposition\n\n```go\n// Create a memory bank with personality traits\nbank := hindsight.NewBank(\"agent-1\", \"Assistant Agent\")\nbank.Skepticism = 3  // 1=Trusting, 5=Skeptical\nbank.Literalism = 3  // 1=Flexible, 5=Literal\nbank.Empathy = 4     // 1=Detached, 5=Empathetic\nsys.CreateBank(ctx, bank)\n```\n\n**Why Hindsight Matters**\n- Agents form **opinions** with confidence scores (not just retrieve facts)\n- **Disposition traits** influence how observations are generated\n- Agents **learn from experience** – observations persist across sessions\n- Pure memory system – no LLM dependency (caller handles embeddings)\n\n## 🏗 Enterprise RAG Capabilities\n\nsqvect goes beyond simple vector storage. It provides the schema and APIs needed for complex RAG apps.\n\n### 1. Hybrid Search (Vector + Keyword)\nCombine semantic understanding with precise keyword matching using Reciprocal Rank Fusion (RRF).\n\n```go\n// Search for \"apple\" (keyword) AND vector similarity\nresults, _ := db.Vector().HybridSearch(ctx, queryVec, \"apple\", core.HybridSearchOptions{\n    TopK: 5,\n    RRFK: 60, // Fusion parameter\n})\n```\n\n### 2. Chat Memory Management\nStore conversation history directly alongside your data.\n\n```go\n// 1. Create a session\ndb.Vector().CreateSession(ctx, \u0026core.Session{ID: \"sess_1\", UserID: \"user_123\"})\n\n// 2. Add messages (User \u0026 Assistant)\ndb.Vector().AddMessage(ctx, \u0026core.Message{\n    SessionID: \"sess_1\",\n    Role:      \"user\",\n    Content:   \"What is sqvect?\",\n})\n\n// 3. Retrieve history for context window\nhistory, _ := db.Vector().GetSessionHistory(ctx, \"sess_1\", 10)\n```\n\n### 3. Row-Level Security (ACL)\nEnforce permissions at the database level.\n\n```go\n// Insert restricted document\ndb.Vector().Upsert(ctx, \u0026core.Embedding{\n    ID: \"secret_doc\", \n    Vector: vec, \n    ACL: []string{\"group:admin\", \"user:alice\"}, // Only admins and Alice\n})\n\n// Search with user context (auto-filters results)\nresults, _ := db.Vector().SearchWithACL(ctx, queryVec, []string{\"user:bob\"}, opts)\n// Returns nothing for Bob!\n```\n\n### 4. Document Management\nTrack source files, versions, and metadata. Deleting a document automatically deletes all its vector chunks (Cascading Delete).\n\n```go\ndb.Vector().CreateDocument(ctx, \u0026core.Document{\n    ID: \"manual_v1\", \n    Title: \"User Manual\",\n    Version: 1,\n})\n// ... add embeddings linked to \"manual_v1\" ...\n\n// Delete document and ALL its embeddings in one call\ndb.Vector().DeleteDocument(ctx, \"manual_v1\")\n```\n\n## 📚 Database Schema\n\nsqvect manages these tables for you:\n\n| Table | Description |\n| :--- | :--- |\n| `embeddings` | Vectors, content, JSON metadata, ACLs. |\n| `graph_nodes` | Graph nodes for entity relationships. |\n| `graph_edges` | Directed edges between nodes (with weights). |\n| `documents` | Parent records for embeddings (Title, URL, Version). |\n| `sessions` | Chat sessions/threads. |\n| `messages` | Chat logs (Role, Content, Timestamp). |\n| `collections` | Logical namespaces (Multi-tenancy). |\n| `chunks_fts` | **FTS5** virtual table for keyword search. |\n\n## 📊 Performance (128-dim)\n\n| Index Type | Insert Speed | Search QPS | Memory (1M vecs) |\n| :--- | :--- | :--- | :--- |\n| **HNSW** | ~580 ops/s | ~720 QPS | ~1.2 GB (SQ8) |\n| **IVF** | ~14,500 ops/s | ~1,230 QPS | ~1.0 GB (SQ8) |\n\n*Tested on Apple M2 Pro.*\n\n## 🎯 Best Use Cases\n\n### Perfect For ✅\n\n| Use Case | Why sqvect? |\n|:---|:---|\n| **Local-First RAG Apps** | Single file, no server, works offline |\n| **AI Agent Memory** | Hindsight system with TEMPR retrieval |\n| **Edge AI Devices** | Low memory (SQ8), no external deps, pure Go |\n| **Personal Knowledge Bases** | Simple backup (copy file), easy to query |\n| **Internal Tools** | Fast setup, no DevOps overhead |\n| **Chat Memory Systems** | Built-in sessions/messages tables |\n| **Multi-Tenant SaaS** | ACL + Collections for isolation |\n| **Document Clustering** | Graph algorithms (PageRank, community detection) |\n| **Hybrid Search Apps** | Vector + FTS5 with RRF fusion |\n| **Prototype to Production** | Same code from dev to prod (just scale up) |\n\n### Not Recommended For ❌\n\n| Scenario | Better Alternative |\n|:---|:---|\n| \u003e100M vectors | Milvus, Qdrant (distributed) |\n| \u003c10ms latency requirements | Redis-based vector DB |\n| Multi-region HA | Cloud-native vector DB (Pinecone) |\n| Non-Go teams | Chroma (Python), Weaviate |\n\n### Real-World Examples\n\n- **AI Agent Memory**: Long-term memory for agents using Hindsight (World, Bank, Opinion, Observation)\n- **Legal Document Analysis**: Store contracts, clauses, and case law with metadata filters\n- **Customer Support Chatbot**: Persistent conversation history + knowledge base search\n- **Code Search Engine**: Semantic code search + syntax-aware filtering\n- **Research Paper Graph**: Citation network + vector similarity\n- **E-commerce Recommendations**: User embeddings + product graph\n\n## 📊 Comparison with Alternatives\n\n### Vector Database Comparison\n\n| Feature | sqvect | Chroma | Weaviate | Milvus | Qdrant |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| **Architecture** | Embedded | Server | Server | Distributed | Server |\n| **Language** | Go | Python | Go | Go | Rust |\n| **Dependencies** | SQLite only | DuckDB | Vector+Obj | Many | Many |\n| **Setup Time** | ~1 sec | ~5 min | ~10 min | ~30 min | ~10 min |\n| **Vector Search** | ✅ | ✅ | ✅ | ✅ | ✅ |\n| **Keyword Search** | ✅ FTS5 | ❌ | ⚠️ | ❌ | ❌ |\n| **Graph DB** | ✅ Built-in | ❌ | ❌ | ❌ | ❌ |\n| **RAG Tables** | ✅ Ready | ❌ DIY | ❌ DIY | ❌ DIY | ❌ DIY |\n| **ACL/Security** | ✅ Row-level | ❌ | ⚠️ | ⚠️ | ⚠️ |\n| **Quantization** | SQ8/PQ/Binary | ❌ | ✅ | ✅ | ✅ |\n| **Scalability** | \u003c10M | \u003c100M | \u003c1B | \u003e1B | \u003c1B |\n| **Backup** | Copy file | Export | Snapshot | Complex | Snapshot |\n| **Ideal For** | Edge/Local | Python ML | Enterprise | Big Data | Production |\n\n### When to Choose sqvect?\n\n**Choose sqvect if:**\n- ✅ You want a **single-file** database (no separate services)\n- ✅ You're building **local-first** or **edge AI** applications\n- ✅ You need **built-in RAG schemas** (docs, sessions, messages)\n- ✅ You want **graph algorithms** without Neo4j\n- ✅ You value **simplicity** over horizontal scalability\n- ✅ You're targeting **\u003c10 million vectors**\n\n**Choose alternatives if:**\n- ❌ You need **distributed** deployment across multiple nodes\n- ❌ You have **\u003e100M vectors** and need horizontal scaling\n- ❌ You require **sub-10ms** query latency\n- ❌ Your team doesn't use Go (prefer Python/TypeScript SDKs)\n\n### Unique Differentiators\n\n🎯 **No other vector DB combines:**\n1. Vector + Graph + Document + Chat + **Agent Memory (Hindsight)** in ONE file\n2. Built-in RAG schemas (zero design work)\n3. **Hindsight**: biomimetic memory system for AI agents (TEMPR retrieval)\n4. Row-Level Security without external auth\n5. Edge deployment ready (no network/containers)\n6. Pure Go (cross-compile to any platform)\n\n## ⚖️ License\n\nMIT License. See [LICENSE](LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliliang-cn%2Fsqvect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliliang-cn%2Fsqvect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliliang-cn%2Fsqvect/lists"}