{"id":18941151,"url":"https://github.com/couchbase-ecosystem/couchbase-haystack","last_synced_at":"2025-04-15T20:31:48.506Z","repository":{"id":254086361,"uuid":"839754940","full_name":"Couchbase-Ecosystem/couchbase-haystack","owner":"Couchbase-Ecosystem","description":"Couchbase haystack document store","archived":false,"fork":false,"pushed_at":"2025-04-11T11:15:30.000Z","size":546,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-11T12:58:20.019Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Couchbase-Ecosystem.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-08T09:05:24.000Z","updated_at":"2025-04-11T11:15:30.000Z","dependencies_parsed_at":"2024-08-27T06:23:34.012Z","dependency_job_id":"8962cdf8-4aba-4f75-ad70-a0c2bd46fc66","html_url":"https://github.com/Couchbase-Ecosystem/couchbase-haystack","commit_stats":null,"previous_names":["couchbase-ecosystem/couchbase-haystack"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fcouchbase-haystack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fcouchbase-haystack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fcouchbase-haystack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Couchbase-Ecosystem%2Fcouchbase-haystack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Couchbase-Ecosystem","download_url":"https://codeload.github.com/Couchbase-Ecosystem/couchbase-haystack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248409337,"owners_count":21098768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T12:26:19.249Z","updated_at":"2025-04-15T20:31:48.470Z","avatar_url":"https://github.com/Couchbase-Ecosystem.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003ecouchbase-haystack\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003eA \u003ca href=\"https://docs.haystack.deepset.ai/docs/document_store\"\u003e\u003ci\u003eHaystack\u003c/i\u003e\u003c/a\u003e Document Store for \u003ca href=\"https://www.couchbase.com\"\u003e\u003ci\u003eCouchbase\u003c/i\u003e\u003c/a\u003e.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/Couchbase-Ecosystem/couchbase-haystack/actions?query=workflow%3Aci\"\u003e\n    \u003cimg alt=\"ci\" src=\"https://github.com/Couchbase-Ecosystem/couchbase-haystack/actions/workflows/ci.yml/badge.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://couchbase-ecosystem.github.io/couchbase-haystack/\"\u003e\n    \u003cimg alt=\"documentation\" src=\"https://img.shields.io/badge/docs-mkdocs%20material-blue.svg?style=flat\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/couchbase-haystack/\"\u003e\n    \u003cimg alt=\"pypi version\" src=\"https://img.shields.io/pypi/v/couchbase-haystack.svg\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/haystack-ai/\"\u003e\n    \u003cimg alt=\"haystack version\" src=\"https://img.shields.io/pypi/v/haystack-ai.svg?label=haystack\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n----\n\n**Table of Contents**\n\n- [Overview](#overview)\n- [Installation](#installation)\n- [Usage](#usage)\n- [License](#license)\n\n## Overview\n\nAn integration of [Couchbase](https://www.couchbase.com) NoSQL database with [Haystack v2.0](https://docs.haystack.deepset.ai/v2.0/docs/intro)\nby [deepset](https://www.deepset.ai). In Couchbase [Vector search index](https://docs.couchbase.com/server/current/vector-search/vector-search.html)\nis being used for indexing document embeddings and dense retrievals.\n\nThe library allows using Couchbase as a [DocumentStore](https://docs.haystack.deepset.ai/v2.0/docs/document-store), and implements the required [Protocol](https://docs.haystack.deepset.ai/v2.0/docs/document-store#documentstore-protocol) methods. You can start working with the implementation by importing it from `couchbase_haystack` package:\n\n```python\nfrom couchbase_haystack import CouchbaseSearchDocumentStore\n```\n\nIn addition to the `CouchbaseSearchDocumentStore` the library includes the following haystack components which can be used in a pipeline:\n\n- [CouchbaseSearchEmbeddingRetriever]() - is a typical [retriever component](https://docs.haystack.deepset.ai/v2.0/docs/retrievers) which can be used to query vector store index and find related Documents. The component uses `CouchbaseSearchDocumentStore` to query embeddings.\n\nThe `couchbase-haystack` library uses [Python Driver](https://docs.couchbase.com/python-sdk/current/hello-world/start-using-sdk.html).\n\n`CouchbaseSearchDocumentStore` will store Documents as JSON documents in Couchbase. Embeddings are stored as part of the document, with indexing and querying of vector embeddings managed by Couchbase's dedicated [Vector Search Index](https://docs.couchbase.com/server/current/vector-search/vector-search.html). The document store supports both scope-level and global-level vector search indexes:\n\n- Scope-level indexes (default): The vector search index is created at the scope level and only searches documents within that scope\n- Global-level indexes: The vector search index is created at the bucket level and can search across all scopes and collections in the bucket\n\n```text\n                                   +-----------------------------+\n                                   |       Couchbase Database    |\n                                   +-----------------------------+\n                                   |                             |\n                                   |      +----------------+     |\n                                   |      |  Data service  |     |\n                write_documents    |      +----------------+     |\n          +------------------------+-----\u003e|   properties   |     |\n          |                        |      |                |     |\n+---------+--------------+         |      |   embedding    |     |\n|                        |         |      +--------+-------+     |\n| CouchbaseSearchDocumentStore |         |               |             |\n|                        |         |               |index        |\n+---------+--------------+         |               |             |\n          |                        |      +--------+--------+    |\n          |                        |      |  Search service |    |\n          |                        |      +-----------------+    |\n          +-----------------------\u003e|      |       FTS       |    |\n               query_embeddings    |      |   Vector Index  |    |\n                                   |      | (for embedding) |    |\n                                   |      +-----------------+    |\n                                   |                             |\n                                   +-----------------------------+\n```\n\nIn the above diagram:\n\n- `Data service` Supports the storing, setting, and retrieving of documents, specified by key. Basically where the documents are stored in key value.\n- `properties` are Document [attributes](https://docs.haystack.deepset.ai/v2.0/docs/data-classes#document) stored as part of the Document.\n- `embedding` is also a property of the Document (just shown separately in the diagram for clarity) which is a vector of type `LIST[FLOAT]`.\n- `Search service` Where indexes specially purposed for Full Text Search and Vector search are created. The Search Service allows for efficient querying \nand retrieval based on both text content and vector embeddings.\n\n`CouchbaseSearchDocumentStore` requires the vector index to be created manually either by sdk or UI. Before writing documents you should make sure Documents are embedded by one of the provided [embedders](https://docs.haystack.deepset.ai/v2.0/docs/embedders). For example [SentenceTransformersDocumentEmbedder](https://docs.haystack.deepset.ai/v2.0/docs/sentencetransformersdocumentembedder) can be used in indexing pipeline to calculate document embeddings before writing those to Couchbase.\n\n## Installation\n\n`couchbase-haystack` can be installed as any other Python library, using pip:\n\n```bash\npip install --upgrade pip # optional\npip install sentence-transformers # required in order to run pipeline examples given below\npip install couchbase-haystack\n```\n\n## Usage\n\n### Running Couchbase\n\nYou will need a running instance of Couchbase to use the components from this package. There are several options available:\n\n- [Docker](https://docs.couchbase.com/server/current/getting-started/do-a-quick-install.html)\n- [Couchbase Cloud](https://www.couchbase.com/products/capella) - a fully managed cloud service\n- [Couchbase Server](https://www.couchbase.com/downloads) - installable on various operating systems\n\nThe simplest way to start the database locally is with a Docker container:\n\n```bash\ndocker run \\\n    --restart always \\\n    --publish=8091-8096:8091-8096 --publish=11210:11210 \\\n    --env COUCHBASE_ADMINISTRATOR_USERNAME=admin \\\n    --env COUCHBASE_ADMINISTRATOR_PASSWORD=passw0rd \\\n    couchbase:enterprise-7.6.2\n```\n\nIn this example, the container is started using Couchbase Server version `7.6.2`. The `COUCHBASE_ADMINISTRATOR_USERNAME` and `COUCHBASE_ADMINISTRATOR_PASSWORD` environment variables set the default credentials for authentication.\n\n\u003e **Note:**  \n\u003e Assuming you have a Docker container running, navigate to \u003chttp://localhost:8091\u003e to open the Couchbase Web Console and explore your data.\n\n### Document Store\n\nOnce you have the package installed and the database running, you can start using `CouchbaseSearchDocumentStore` as any other document stores that support embeddings.\n\n```python\nfrom couchbase_haystack import CouchbaseSearchDocumentStore, CouchbasePasswordAuthenticator\nfrom haystack.utils.auth import Secret\n\ndocument_store = CouchbaseSearchDocumentStore(\n    cluster_connection_string=Secret.from_env_var(\"CB_CONNECTION_STRING\"),\n    authenticator=CouchbasePasswordAuthenticator(\n      username=Secret.from_env_var(\"CB_USERNAME\"),\n      password=Secret.from_env_var(\"CB_PASSWORD\")\n    ),\n    bucket = \"haystack_bucket_name\",\n    scope=\"haystack_scope_name\",\n    collection=\"haystack_collection_name\",\n    vector_search_index = \"vector_search_index\",\n    is_global_level_index=False  # Enables scope-level vector search index by default\n)\n```\n\nAssuming there is a list of documents available and a running couchbase database you can write/index those in Couchbase, e.g.:\n\n```python\nfrom haystack import Document\n\ndocuments = [Document(content=\"Alice has been living in New York City for the past 5 years.\")]\n\ndocument_store.write_documents(documents)\n```\n\nIf you intend to obtain embeddings before writing documents use the following code:\n\n```python\nfrom haystack import Document\n\n# import one of the available document embedders\nfrom haystack.components.embedders import SentenceTransformersDocumentEmbedder \n\ndocuments = [Document(content=\"Alice has been living in New York City for the past 5 years.\")]\n\ndocument_embedder = SentenceTransformersDocumentEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\")\ndocument_embedder.warm_up() # will download the model during first run\ndocuments_with_embeddings = document_embedder.run(documents)\n\ndocument_store.write_documents(documents_with_embeddings.get(\"documents\"))\n```\n\nMake sure embedding model produces vectors of same size as it has been set on `Couchbase Vector Index`, e.g. setting `embedding_dim=384` would comply with the \"sentence-transformers/all-MiniLM-L6-v2\" model.\n\n\u003e **Note**\n\u003e Most of the time you will be using [Haystack Pipelines](https://docs.haystack.deepset.ai/v2.0/docs/pipelines) to build both indexing and querying RAG scenarios.\n\nIt is important to understand how haystack Documents are stored in Couchbase after you call `write_documents`.\n\n```python\nfrom random import random\n\nsample_embedding = [random() for _ in range(384)]  # using fake/random embedding for brevity here to simplify example\ndocument = Document(\n    content=\"Alice has been living in New York City for the past 5 years.\", embedding=sample_embedding, meta={\"num_of_years\": 5}\n)\ndocument.to_dict()\n```\n\nThe above code converts a Document to a dictionary and will render the following output:\n\n```bash\n\u003e\u003e\u003e output:\n{\n    \"id\": \"11c255ad10bff4286781f596a5afd9ab093ed056d41bca4120c849058e52f24d\",\n    \"content\": \"Alice has been living in New York City for the past 5 years.\",\n    \"dataframe\": None,\n    \"blob\": None,\n    \"score\": None,\n    \"embedding\": [0.025010755222666936, 0.27502931836911926, 0.22321073814882275, ...], # vector of size 384\n    \"num_of_years\": 5,\n}\n```\n\nThe data from the dictionary will be used to create a document in COuchbase after you write the document with `document_store.write_documents([document])`. You could query it with Cypher, e.g. `MATCH (doc:Document) RETURN doc`. Below is a json document Couchbase:\n\n```js\n{\n  \"id\": \"11c255ad10bff4286781f596a5afd9ab093ed056d41bca4120c849058e52f24d\",\n  \"embedding\": [0.6394268274307251, 0.02501075528562069,0.27502933144569397, ...], // vector of size 384\n  \"content\": \"Alice has been living in New York City for the past 5 years.\",\n  \"meta\": {\n    \"num_of_years\": 5\n  }\n}\n```\n\nThe full list of parameters accepted by `CouchbaseSearchDocumentStore` can be found in\n[API documentation](https://couchbase-ecosystem.github.io/couchbase-haystack/reference/couchbase_document_store).\n\n### Indexing documents\n\nWith Haystack you can use [DocumentWriter](https://docs.haystack.deepset.ai/v2.0/docs/documentwriter) component to write Documents into a Document Store. In the example below we construct pipeline to write documents to Couchbase using `CouchbaseSearchDocumentStore`:\n\n```python\nfrom haystack import Document\nfrom haystack.components.embedders import SentenceTransformersDocumentEmbedder\nfrom haystack.components.writers import DocumentWriter\nfrom haystack.pipeline import Pipeline\nfrom haystack.utils.auth import Secret\n\nfrom couchbase_haystack import CouchbaseSearchDocumentStore, CouchbasePasswordAuthenticator\n\ndocuments = [Document(content=\"This is document 1\"), Document(content=\"This is document 2\")]\n\ndocument_store = CouchbaseSearchDocumentStore(\n    cluster_connection_string=Secret.from_env_var(\"CB_CONNECTION_STRING\"),\n    authenticator=CouchbasePasswordAuthenticator(\n      username=Secret.from_env_var(\"CB_USERNAME\"),\n      password=Secret.from_env_var(\"CB_PASSWORD\")\n    ),\n    bucket = \"haystack_bucket_name\",\n    scope=\"haystack_scope_name\",\n    collection=\"haystack_collection_name\",\n    vector_search_index = \"vector_search_index\"\n)\nembedder = SentenceTransformersDocumentEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\")\ndocument_writer = DocumentWriter(document_store=document_store)\n\nindexing_pipeline = Pipeline()\nindexing_pipeline.add_component(instance=embedder, name=\"embedder\")\nindexing_pipeline.add_component(instance=document_writer, name=\"writer\")\n\nindexing_pipeline.connect(\"embedder\", \"writer\")\nindexing_pipeline.run({\"embedder\": {\"documents\": documents}})\n```\n\n```bash\n\u003e\u003e\u003e output:\n`{'writer': {'documents_written': 2}}`\n```\n\n### Retrieving documents\n\n`CouchbaseSearchEmbeddingRetriever` component can be used to retrieve documents from Couchbase by querying vector index using an embedded query. Below is a pipeline which finds documents using query embedding:\n\n```python\nfrom typing import List\nfrom haystack.utils.auth import Secret\nfrom haystack import Document, Pipeline\nfrom haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder\n\nfrom couchbase_haystack.document_store import CouchbaseSearchDocumentStore, CouchbasePasswordAuthenticator\nfrom couchbase_haystack.component.retriever import CouchbaseSearchEmbeddingRetriever\n\ndocument_store = CouchbaseSearchDocumentStore(\n    cluster_connection_string=Secret.from_env_var(\"CB_CONNECTION_STRING\"),\n    authenticator=CouchbasePasswordAuthenticator(\n      username=Secret.from_env_var(\"CB_USERNAME\"),\n      password=Secret.from_env_var(\"CB_PASSWORD\")\n    ),\n    bucket = \"haystack_bucket_name\",\n    scope=\"haystack_scope_name\",\n    collection=\"haystack_collection_name\",\n    vector_search_index = \"vector_search_index\"\n)\n\ndocuments = [\n    Document(content=\"Alice has been living in New York City for the past 5 years.\", meta={\"num_of_years\": 5, \"city\": \"New York\"}),\n    Document(content=\"John moved to Los Angeles 2 years ago and loves the sunny weather.\", meta={\"num_of_years\": 2, \"city\": \"Los Angeles\"}),\n]\n\n# Same model is used for both query and Document embeddings\nmodel_name = \"sentence-transformers/all-MiniLM-L6-v2\"\n\ndocument_embedder = SentenceTransformersDocumentEmbedder(model=model_name)\ndocument_embedder.warm_up()\ndocuments_with_embeddings = document_embedder.run(documents)\n\ndocument_store.write_documents(documents_with_embeddings.get(\"documents\"))\n\nprint(\"Number of documents written: \", document_store.count_documents())\n\npipeline = Pipeline()\npipeline.add_component(\"text_embedder\", SentenceTransformersTextEmbedder(model=model_name))\npipeline.add_component(\"retriever\", CouchbaseSearchEmbeddingRetriever(document_store=document_store))\npipeline.connect(\"text_embedder.embedding\", \"retriever.query_embedding\")\n\nresult = pipeline.run(\n    data={\n        \"text_embedder\": {\"text\": \"What cities do people live in?\"},\n        \"retriever\": {\n            \"top_k\": 5\n        },\n    }\n)\n\ndocuments: List[Document] = result[\"retriever\"][\"documents\"]\n```\n\n```bash\n\u003e\u003e\u003e output:\n[Document(id=3e35fa03aff6e3c45e6560f58adc4fde3c436c111a8809c30133b5cb492e8694, content: 'Alice has been living in New York City for the past 5 years.', meta: {'num_of_years': 5, 'city': 'New York'}, score: 0.36796408891677856, embedding: \"embedding\": vector of size 384), Document(id=ca4d7d7d7ff6c13b950a88580ab134b2dc15b48a47b8f571a46b354b5344e5fa, content: 'John moved to Los Angeles 2 years ago and loves the sunny weather.', meta: {'num_of_years': 2, 'city': 'Los Angeles'}, score: 0.3126790523529053, embedding: vector of size 384)]\n```\n\n### More examples\n\nYou can find more examples in the implementation [repository](examples):\n\n- [indexing_pipeline.py](examples/indexing_pipeline.py) - Indexing text files (documents) from a remote http location.\n- [rag_pipeline.py](examples/rag_pipeline.py) - Generative question answering RAG pipeline using `CouchbaseSearchEmbeddingRetriever` to fetch documents from Couchbase document store and answer question using [HuggingFaceAPIGenerator](https://docs.haystack.deepset.ai/v2.0/docs/huggingfacetgigenerator).\n\n## License\n\n`couchbase-haystack` is distributed under the terms of the [MIT](https://spdx.org/licenses/MIT.html) license.\n---\n\n## 📢 Support Policy\n\nWe truly appreciate your interest in this project!  \nThis project is **community-maintained**, which means it's **not officially supported** by our support team.\n\nIf you need help, have found a bug, or want to contribute improvements, the best place to do that is right here — by [opening a GitHub issue](https://github.com/Couchbase-Ecosystem/couchbase-haystack/issues).  \nOur support portal is unable to assist with requests related to this project, so we kindly ask that all inquiries stay within GitHub.\n\nYour collaboration helps us all move forward together — thank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcouchbase-ecosystem%2Fcouchbase-haystack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcouchbase-ecosystem%2Fcouchbase-haystack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcouchbase-ecosystem%2Fcouchbase-haystack/lists"}