{"id":36236492,"url":"https://github.com/reasonkit/reasonkit-mem","last_synced_at":"2026-01-11T06:00:51.072Z","repository":{"id":331376486,"uuid":"1126026341","full_name":"reasonkit/reasonkit-mem","owner":"reasonkit","description":"High-performance vector database \u0026 RAG memory layer - hybrid search, embeddings, RAPTOR trees, BM25 fusion for AI systems.","archived":false,"fork":false,"pushed_at":"2026-01-05T23:43:39.000Z","size":26099,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-07T05:20:24.521Z","etag":null,"topics":["ai-memory","bm25","developer-tools","embedding","embeddings","hnsw","hybrid-search","information-retrieval","knowledge-base","llm","long-term-memory","qdrant","rag","raptor","reranking","retrieval","rust","semantic-search","tantivy","vector-database"],"latest_commit_sha":null,"homepage":"https://reasonkit.sh","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reasonkit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["reasonkit"],"open_collective":"reasonkit","custom":["https://reasonkit.sh/sponsor"]}},"created_at":"2025-12-31T22:48:01.000Z","updated_at":"2026-01-03T18:38:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/reasonkit/reasonkit-mem","commit_stats":null,"previous_names":["reasonkit/reasonkit-mem"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/reasonkit/reasonkit-mem","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasonkit%2Freasonkit-mem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasonkit%2Freasonkit-mem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasonkit%2Freasonkit-mem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasonkit%2Freasonkit-mem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reasonkit","download_url":"https://codeload.github.com/reasonkit/reasonkit-mem/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reasonkit%2Freasonkit-mem/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28293188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-11T04:44:51.577Z","status":"ssl_error","status_checked_at":"2026-01-11T04:44:44.232Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-memory","bm25","developer-tools","embedding","embeddings","hnsw","hybrid-search","information-retrieval","knowledge-base","llm","long-term-memory","qdrant","rag","raptor","reranking","retrieval","rust","semantic-search","tantivy","vector-database"],"created_at":"2026-01-11T06:00:27.639Z","updated_at":"2026-01-11T06:00:51.048Z","avatar_url":"https://github.com/reasonkit.png","language":"Rust","funding_links":["https://github.com/sponsors/reasonkit","https://opencollective.com/reasonkit","https://reasonkit.sh/sponsor"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# ReasonKit Mem\n\n**Memory \u0026 Retrieval Infrastructure for ReasonKit**\n\n[![CI](https://badges.reasonkit.sh/github/actions/workflow/status/reasonkit/reasonkit-mem/ci.yml?branch=main\u0026style=flat-square\u0026logo=github\u0026label=CI\u0026color=06b6d4\u0026logoColor=06b6d4)](https://github.com/reasonkit/reasonkit-mem/actions/workflows/ci.yml)\n[![Security](https://badges.reasonkit.sh/github/actions/workflow/status/reasonkit/reasonkit-mem/security.yml?branch=main\u0026style=flat-square\u0026logo=github\u0026label=Security\u0026color=10b981\u0026logoColor=10b981)](https://github.com/reasonkit/reasonkit-mem/actions/workflows/security.yml)\n[![Crates.io](https://badges.reasonkit.sh/crates/v/reasonkit-mem?style=flat-square\u0026logo=rust\u0026color=10b981\u0026logoColor=f9fafb)](https://crates.io/crates/reasonkit-mem)\n[![docs.rs](https://badges.reasonkit.sh/docsrs/reasonkit-mem?style=flat-square\u0026logo=docs.rs\u0026color=06b6d4\u0026logoColor=f9fafb)](https://docs.rs/reasonkit-mem)\n[![Downloads](https://badges.reasonkit.sh/crates/d/reasonkit-mem?style=flat-square\u0026color=ec4899\u0026logo=rust\u0026logoColor=f9fafb)](https://crates.io/crates/reasonkit-mem)\n[![License](https://badges.reasonkit.sh/static/v1?label=license\u0026message=Apache%202.0\u0026color=a855f7\u0026style=flat-square\u0026labelColor=030508)](./LICENSE)\n[![Rust](https://badges.reasonkit.sh/static/v1?label=rust\u0026message=1.75%2B\u0026color=f97316\u0026style=flat-square\u0026logo=rust\u0026logoColor=f9fafb)](https://www.rust-lang.org/)\n\n_The Long-Term Memory Layer (\"Hippocampus\") for AI Reasoning_\n\n[Documentation](https://docs.rs/reasonkit-mem) | [ReasonKit Core](https://github.com/ReasonKit/reasonkit-core) | [Website](https://reasonkit.sh)\n\n\u003c/div\u003e\n\n---\n\n**ReasonKit Mem** is the memory layer (\"Hippocampus\") for ReasonKit. It provides vector storage, hybrid search, RAPTOR trees, and embedding support.\n\n## Features\n\n- **Vector Storage** - Qdrant-based dense vector storage with embedded mode\n- **Hybrid Search** - Dense (Qdrant) + Sparse (Tantivy BM25) fusion\n- **RAPTOR Trees** - Hierarchical retrieval for long-form QA\n- **Embeddings** - Local (BGE-M3) and remote (OpenAI) embedding support\n- **Reranking** - Cross-encoder reranking for precision\n\n## Installation\n\n### Universal Installer (Recommended)\n\n**Installs all 4 ReasonKit projects together:**\n\n```bash\ncurl -fsSL https://get.reasonkit.sh | bash -s -- --with-memory\n```\n\n**Platform \u0026 Shell Support:**\n\n- ✅ All platforms (Linux/macOS/Windows/WSL)\n- ✅ All shells (Bash/Zsh/Fish/Nu/PowerShell/Elvish)\n- ✅ Auto-detects shell and configures PATH\n- ✅ Beautiful progress visualization\n\n### Cargo (Rust Library)\n\nAdd to your `Cargo.toml`:\n\n```toml\n[dependencies]\nreasonkit-mem = \"0.1\"\ntokio = { version = \"1\", features = [\"full\"] }\n```\n\n## Usage\n\n### Basic Usage (Embedded Mode)\n\n```rust,ignore\nuse reasonkit_mem::storage::Storage;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Create embedded storage (automatic file storage fallback)\n    let storage = Storage::new_embedded().await?;\n\n    // Use storage...\n    Ok(())\n}\n```\n\n### Storage with Custom Configuration\n\n```rust,ignore\nuse reasonkit_mem::storage::{Storage, EmbeddedStorageConfig};\nuse std::path::PathBuf;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Create storage with custom file path\n    let config = EmbeddedStorageConfig::file_only(PathBuf::from(\"./data\"));\n    let storage = Storage::new_embedded_with_config(config).await?;\n\n    // Or use Qdrant (requires running server)\n    let qdrant_config = EmbeddedStorageConfig::with_qdrant(\n        \"http://localhost:6333\",\n        \"my_collection\",\n        1536,\n    );\n    let qdrant_storage = Storage::new_embedded_with_config(qdrant_config).await?;\n\n    Ok(())\n}\n```\n\n### Hybrid Search with KnowledgeBase\n\n```rust,ignore\nuse reasonkit_mem::retrieval::KnowledgeBase;\nuse reasonkit_mem::{Document, DocumentType, Source, SourceType};\nuse chrono::Utc;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Create in-memory knowledge base\n    let kb = KnowledgeBase::in_memory()?;\n\n    // Create a document\n    let source = Source {\n        source_type: SourceType::Local,\n        url: None,\n        path: Some(\"notes.md\".to_string()),\n        arxiv_id: None,\n        github_repo: None,\n        retrieved_at: Utc::now(),\n        version: None,\n    };\n\n    let doc = Document::new(DocumentType::Note, source)\n        .with_content(\"Machine learning is a subset of artificial intelligence.\".to_string());\n\n    // Add document to knowledge base\n    kb.add(\u0026doc).await?;\n\n    // Search using sparse retrieval (BM25)\n    let results = kb.retriever().search_sparse(\"machine learning\", 5).await?;\n\n    for result in results {\n        println!(\"Score: {:.3}, Text: {}\", result.score, result.text);\n    }\n\n    Ok(())\n}\n```\n\n### Using Embeddings\n\n```rust,ignore\nuse reasonkit_mem::embedding::{EmbeddingConfig, EmbeddingPipeline, OpenAIEmbedding};\nuse reasonkit_mem::retrieval::KnowledgeBase;\nuse std::sync::Arc;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Create OpenAI embedding provider (requires OPENAI_API_KEY env var)\n    let embedding_provider = OpenAIEmbedding::openai()?;\n    let pipeline = Arc::new(EmbeddingPipeline::new(Arc::new(embedding_provider)));\n\n    // Create knowledge base with embedding support\n    let kb = KnowledgeBase::in_memory()?\n        .with_embedding_pipeline(pipeline);\n\n    // Now hybrid search will use both dense (vector) and sparse (BM25)\n    // let results = kb.query(\"semantic search query\", 10).await?;\n\n    Ok(())\n}\n```\n\n### Embedded Mode Documentation\n\nFor detailed information about embedded mode, see [docs/EMBEDDED_MODE_GUIDE.md](docs/EMBEDDED_MODE_GUIDE.md).\n\n## Architecture\n\n![ReasonKit Mem Hybrid Architecture](./brand/readme/hybrid_architecture.png)\n![ReasonKit Mem Hybrid Architecture Technical Diagram](./brand/readme/hybrid_retrieval_engine.svg)\n\n### The RAPTOR Algorithm (Hierarchical Indexing)\n\nReasonKit Mem implements **RAPTOR** (Recursive Abstractive Processing for Tree-Organized Retrieval) to answer high-level questions across large document sets.\n\n![ReasonKit Mem RAPTOR Tree Structure](./brand/readme/raptor_tree_structure.svg)\n\n![ReasonKit Mem RAPTOR Tree](./brand/readme/raptor_tree.png)\n\n### The Memory Dashboard\n\n![ReasonKit Mem Dashboard](./brand/readme/memory_dashboard.png)\n\n### Integration Ecosystem\n\n![ReasonKit Mem Ecosystem](./brand/readme/mem_ecosystem.png)\n\n## Technology Stack\n\n| Component      | Technology          | Purpose                |\n| -------------- | ------------------- | ---------------------- |\n| **Qdrant**     | qdrant-client 1.10+ | Dense vector storage   |\n| **Tantivy**    | tantivy 0.22+       | BM25 sparse search     |\n| **RAPTOR**     | Custom Rust         | Hierarchical retrieval |\n| **Embeddings** | BGE-M3 / OpenAI     | Dense representations  |\n| **Reranking**  | Cross-encoder       | Final precision boost  |\n\n## Project Structure\n\n```text\nreasonkit-mem/\n├── src/\n│   ├── storage/      # Qdrant vector + file-based storage\n│   ├── embedding/    # Dense vector embeddings\n│   ├── retrieval/    # Hybrid search, fusion, reranking\n│   ├── raptor/       # RAPTOR hierarchical tree structure\n│   ├── indexing/     # BM25/Tantivy sparse indexing\n│   └── rag/          # RAG pipeline orchestration\n├── benches/          # Performance benchmarks\n├── examples/         # Usage examples\n├── docs/             # Additional documentation\n└── Cargo.toml\n```\n\n## Feature Flags\n\n| Feature            | Description                              |\n| ------------------ | ---------------------------------------- |\n| `default`          | Core functionality                       |\n| `python`           | Python bindings via PyO3                 |\n| `local-embeddings` | Local BGE-M3 embeddings via ONNX Runtime |\n\n## API Reference\n\n### Core Types (re-exported at crate root)\n\n```rust,ignore\nuse reasonkit_mem::{\n    // Documents\n    Document, DocumentType, DocumentContent,\n    // Chunks\n    Chunk, EmbeddingIds,\n    // Sources\n    Source, SourceType,\n    // Metadata\n    Metadata, Author,\n    // Search\n    SearchResult, MatchSource, RetrievalConfig,\n    // Processing\n    ProcessingStatus, ProcessingState, ContentFormat,\n    // Errors\n    MemError, MemResult,\n};\n```\n\n### Storage Module\n\n```rust,ignore\nuse reasonkit_mem::storage::{\n    Storage,\n    EmbeddedStorageConfig,\n    StorageBackend,\n    InMemoryStorage,\n    FileStorage,\n    QdrantStorage,\n    AccessContext,\n    AccessLevel,\n};\n```\n\n### Embedding Module\n\n```rust,ignore\nuse reasonkit_mem::embedding::{\n    EmbeddingProvider,      // Trait for embedding backends\n    OpenAIEmbedding,        // OpenAI API embeddings\n    EmbeddingConfig,        // Configuration\n    EmbeddingPipeline,      // Batch processing pipeline\n    EmbeddingResult,        // Single embedding result\n    EmbeddingVector,        // Vec\u003cf32\u003e alias\n    cosine_similarity,      // Utility function\n    normalize_vector,       // Utility function\n};\n```\n\n### Retrieval Module\n\n```rust,ignore\nuse reasonkit_mem::retrieval::{\n    HybridRetriever,        // Main retrieval engine\n    KnowledgeBase,          // High-level API\n    HybridResult,           // Search result\n    RetrievalStats,         // Statistics\n    // Fusion\n    FusionEngine,\n    FusionStrategy,\n    // Reranking\n    Reranker,\n    RerankerConfig,\n};\n```\n\n## Version \u0026 Maturity\n\n| Component            | Status    | Notes                                                |\n| -------------------- | --------- | ---------------------------------------------------- |\n| **Vector Storage**   | ✅ Stable | Qdrant integration production-ready                  |\n| **Hybrid Search**    | ✅ Stable | Dense + Sparse fusion working                        |\n| **RAPTOR Trees**     | ✅ Stable | Hierarchical retrieval implemented                   |\n| **Embeddings**       | ✅ Stable | OpenAI API fully supported                           |\n| **Local Embeddings** | 🔶 Beta   | BGE-M3 ONNX (enable with `local-embeddings` feature) |\n| **Python Bindings**  | 🔶 Beta   | Build from source with `--features python`           |\n\n**Current Version:** v0.1.2 | [CHANGELOG](CHANGELOG.md) | [Releases](https://github.com/reasonkit/reasonkit-mem/releases)\n\n### Verify Installation\n\n```rust\nuse reasonkit_mem::storage::Storage;\n\n#[tokio::main]\nasync fn main() -\u003e anyhow::Result\u003c()\u003e {\n    // Quick verification - creates in-memory storage\n    let storage = Storage::new_embedded().await?;\n    println!(\"ReasonKit Mem initialized successfully!\");\n    Ok(())\n}\n```\n\n## License\n\nApache License 2.0 - see [LICENSE](https://github.com/reasonkit/reasonkit-mem/blob/main/LICENSE)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\n![ReasonKit Ecosystem Connection](./brand/readme/ecosystem_connection.png)\n\n**Part of the ReasonKit Ecosystem**\n\n[ReasonKit Core](https://github.com/reasonkit/reasonkit-core) | [ReasonKit Web](https://github.com/reasonkit/reasonkit-web) | [Website](https://reasonkit.sh)\n\n_\"See How Your AI Thinks\"_\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freasonkit%2Freasonkit-mem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freasonkit%2Freasonkit-mem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freasonkit%2Freasonkit-mem/lists"}