{"id":34024723,"url":"https://github.com/awinml/voyage-embedders-haystack","last_synced_at":"2026-02-07T15:27:49.502Z","repository":{"id":207335518,"uuid":"718963006","full_name":"awinml/voyage-embedders-haystack","owner":"awinml","description":"Custom components for Haystack for creating embeddings and reranking documents using the VoyageAI Models.","archived":false,"fork":false,"pushed_at":"2025-12-23T07:15:00.000Z","size":126,"stargazers_count":5,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-24T08:41:34.975Z","etag":null,"topics":["embeddings","haystack","voyageai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/awinml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-15T06:37:22.000Z","updated_at":"2025-12-23T05:07:31.000Z","dependencies_parsed_at":"2023-11-18T10:25:08.617Z","dependency_job_id":"e1adb296-381a-4fec-baa3-47e23d74573a","html_url":"https://github.com/awinml/voyage-embedders-haystack","commit_stats":null,"previous_names":["awinml/voyage-embedders-haystack"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/awinml/voyage-embedders-haystack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awinml%2Fvoyage-embedders-haystack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awinml%2Fvoyage-embedders-haystack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awinml%2Fvoyage-embedders-haystack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awinml%2Fvoyage-embedders-haystack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/awinml","download_url":"https://codeload.github.com/awinml/voyage-embedders-haystack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awinml%2Fvoyage-embedders-haystack/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28124763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-30T02:00:05.476Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embeddings","haystack","voyageai"],"created_at":"2025-12-13T16:36:03.106Z","updated_at":"2026-02-07T15:27:49.493Z","avatar_url":"https://github.com/awinml.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e \u003ca href=\"https://github.com/awinml/voyage-embedders-haystack\"\u003e Voyage Embedders and Rankers - Haystack \u003c/a\u003e \u003c/h1\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[![PyPI](https://img.shields.io/pypi/v/voyage-embedders-haystack)](https://pypi.org/project/voyage-embedders-haystack/)\n[![Downloads](https://img.shields.io/pypi/dm/voyage-embedders-haystack?color=blue\u0026logo=pypi\u0026logoColor=gold)](https://pypi.org/project/voyage-embedders-haystack/)\n[![License](https://img.shields.io/github/license/awinml/voyage-embedders-haystack?color=green)](LICENSE)\n[![Tests](https://github.com/awinml/voyage-embedders-haystack/workflows/Test/badge.svg)](https://github.com/awinml/voyage-embedders-haystack/actions)\n[![Coverage](https://coveralls.io/repos/github/awinml/voyage-embedders-haystack/badge.svg?branch=main)](https://coveralls.io/github/awinml/voyage-embedders-haystack?branch=main)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/awinml/voyage-embedders-haystack/main.svg)](https://results.pre-commit.ci/latest/github/awinml/voyage-embedders-haystack/main)\n[![Types](https://img.shields.io/badge/types-ty-blue.svg)](https://docs.astral.sh/ty/)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n\n\u003c/div\u003e\n\nCustom components for [Haystack](https://github.com/deepset-ai/haystack) for creating embeddings and reranking documents using the [Voyage Models](https://voyageai.com/).\n\nVoyage’s embedding models are state-of-the-art in retrieval accuracy. These models outperform top performing embedding models like `intfloat/e5-mistral-7b-instruct` and `OpenAI/text-embedding-3-large` on the [MTEB Benchmark](https://github.com/embeddings-benchmark/mteb).\n\n#### What's New (v1.9.1)\n\n- Serialize `chunk_fn` for `VoyageContextualizedDocumentEmbedder` using Haystack's `serialize_callable`/`deserialize_callable`.\n- Improved typing across all components (explicit `run()` return types, tighter `chunk_fn` type).\n- Developer experience improvements: dotenv support for examples, `pyproject.toml` cleanup, updated `CONTRIBUTING.md`.\n\nSee the full [Changelog](CHANGELOG.md) for all releases.\n\n## Requirements\n\n- Python 3.10 or higher\n- [Voyage AI API Key](https://voyageai.com/)\n\n## Installation\n\n```bash\npip install voyage-embedders-haystack\n```\n\n## Usage\n\nYou can use Voyage Embedding models with multiple components:\n\n- **[VoyageTextEmbedder](https://github.com/awinml/voyage-embedders-haystack/blob/main/src/voyage_embedders/voyage_text_embedder.py)**: For generating embeddings for queries.\n- **[VoyageDocumentEmbedder](https://github.com/awinml/voyage-embedders-haystack/blob/main/src/voyage_embedders/voyage_document_embedder.py)**: For creating semantic embeddings for documents in your indexing pipeline.\n- **[VoyageContextualizedDocumentEmbedder](https://github.com/awinml/voyage-embedders-haystack/blob/main/src/haystack_integrations/components/embedders/voyage_embedders/voyage_contextualized_document_embedder.py)**: For creating contextualized embeddings where document chunks are embedded together to preserve context and improve retrieval accuracy.\n- **[VoyageMultimodalEmbedder](https://github.com/awinml/voyage-embedders-haystack/blob/main/src/haystack_integrations/components/embedders/voyage_embedders/voyage_multimodal_embedder.py)**: For creating multimodal embeddings that can encode text, images, and videos into a shared vector space.\n\nThe Voyage Reranker models can be used with the [VoyageRanker](https://github.com/awinml/voyage-embedders-haystack/blob/main/src/haystack_integrations/components/rankers/voyage/ranker.py) component.\n\n### Multimodal Embeddings\n\nThe `VoyageMultimodalEmbedder` uses Voyage's multimodal embedding model (`voyage-multimodal-3.5`) to encode text, images, and videos into a shared vector space. This enables cross-modal similarity search where you can find images using text queries or find related content across different modalities.\n\n**Key features:**\n\n- Supports text, images (PIL Images, ByteStream), and videos\n- Inputs can combine multiple modalities (e.g., text + image)\n- Variable output dimensions: 256, 512, 1024 (default), 2048\n- Recommended model: `voyage-multimodal-3.5`\n\n**Usage example:**\n\n```python\nfrom haystack.dataclasses import ByteStream\nfrom haystack_integrations.components.embedders.voyage_embedders import VoyageMultimodalEmbedder\nfrom voyageai.video_utils import Video\n\n# Text-only embedding\nembedder = VoyageMultimodalEmbedder(model=\"voyage-multimodal-3.5\")\nresult = embedder.run(inputs=[[\"What is in this image?\"]])\n\n# Mixed text and image embedding\nimage_bytes = ByteStream.from_file_path(\"image.jpg\")\nresult = embedder.run(inputs=[[\"Describe this image:\", image_bytes]])\n\n# Video embedding\nvideo = Video.from_path(\"video.mp4\", model=\"voyage-multimodal-3.5\")\nresult = embedder.run(inputs=[[\"Describe this video:\", video]])\n```\n\n### Contextualized Chunk Embeddings\n\nThe `VoyageContextualizedDocumentEmbedder` uses Voyage's contextualized embedding models to encode document chunks \"in context\" with other chunks from the same document. This approach preserves semantic relationships between chunks and reduces context loss, leading to improved retrieval accuracy.\n\n**Key features:**\n\n- Documents are grouped by a metadata field (default: `source_id`)\n- Chunks from the same source document are embedded together\n- Maintains semantic connections between related chunks\n- Recommended model: `voyage-context-3`\n\nFor detailed usage examples, see the [contextualized embedder example](https://github.com/awinml/voyage-embedders-haystack/blob/main/examples/contextualized_embedder_example.py).\n\nOnce you've selected the suitable component for your specific use case, initialize the component with the model name and VoyageAI API key. You can also\nset the environment variable `VOYAGE_API_KEY` instead of passing the API key as an argument.\nTo get an API key, please see the [Voyage AI website.](https://www.voyageai.com/)\n\nInformation about the supported models, can be found on the [Voyage AI Documentation.](https://docs.voyageai.com/)\n\n## Example\n\nYou can find all the examples in the [`examples`](https://github.com/awinml/voyage-embedders-haystack/tree/main/examples) folder.\n\nBelow is the example Semantic Search pipeline that uses the [Simple Wikipedia](https://huggingface.co/datasets/pszemraj/simple_wikipedia) Dataset from HuggingFace.\n\nLoad the dataset:\n\n```python\n# Install HuggingFace Datasets using \"pip install datasets\"\nfrom datasets import load_dataset\nfrom haystack import Pipeline\nfrom haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever\nfrom haystack.components.writers import DocumentWriter\nfrom haystack.dataclasses import Document\nfrom haystack.document_stores.in_memory import InMemoryDocumentStore\n\n# Import Voyage Embedders\nfrom haystack_integrations.components.embedders.voyage_embedders import VoyageDocumentEmbedder, VoyageTextEmbedder\n\n# Load first 100 rows of the Simple Wikipedia Dataset from HuggingFace\ndataset = load_dataset(\"pszemraj/simple_wikipedia\", split=\"validation[:100]\")\n\ndocs = [\n    Document(\n        content=doc[\"text\"],\n        meta={\n            \"title\": doc[\"title\"],\n            \"url\": doc[\"url\"],\n        },\n    )\n    for doc in dataset\n]\n```\n\nIndex the documents to the `InMemoryDocumentStore` using the `VoyageDocumentEmbedder` and `DocumentWriter`:\n\n```python\ndoc_store = InMemoryDocumentStore(embedding_similarity_function=\"cosine\")\nretriever = InMemoryEmbeddingRetriever(document_store=doc_store)\ndoc_writer = DocumentWriter(document_store=doc_store)\n\ndoc_embedder = VoyageDocumentEmbedder(\n    model=\"voyage-2\",\n    input_type=\"document\",\n)\ntext_embedder = VoyageTextEmbedder(model=\"voyage-2\", input_type=\"query\")\n\n# Indexing Pipeline\nindexing_pipeline = Pipeline()\nindexing_pipeline.add_component(instance=doc_embedder, name=\"DocEmbedder\")\nindexing_pipeline.add_component(instance=doc_writer, name=\"DocWriter\")\nindexing_pipeline.connect(\"DocEmbedder\", \"DocWriter\")\n\nindexing_pipeline.run({\"DocEmbedder\": {\"documents\": docs}})\n\nprint(f\"Number of documents in Document Store: {len(doc_store.filter_documents())}\")\nprint(f\"First Document: {doc_store.filter_documents()[0]}\")\nprint(f\"Embedding of first Document: {doc_store.filter_documents()[0].embedding}\")\n```\n\nQuery the Semantic Search Pipeline using the `InMemoryEmbeddingRetriever` and `VoyageTextEmbedder`:\n\n```python\ntext_embedder = VoyageTextEmbedder(model=\"voyage-2\", input_type=\"query\")\n\n# Query Pipeline\nquery_pipeline = Pipeline()\nquery_pipeline.add_component(instance=text_embedder, name=\"TextEmbedder\")\nquery_pipeline.add_component(instance=retriever, name=\"Retriever\")\nquery_pipeline.connect(\"TextEmbedder.embedding\", \"Retriever.query_embedding\")\n\n# Search\nresults = query_pipeline.run({\"TextEmbedder\": {\"text\": \"Which year did the Joker movie release?\"}})\n\n# Print text from top result\ntop_result = results[\"Retriever\"][\"documents\"][0].content\nprint(\"The top search result is:\")\nprint(top_result)\n```\n\n## Contributing\n\nWe welcome contributions from the community! Please take a look at our [contributing guide](CONTRIBUTING.md) for more details on how to get started.\n\nPull requests are welcome. For major changes, please open an issue first to discuss the proposed changes.\n\n## License\n\n`voyage-embedders-haystack` is distributed under the terms of the [Apache-2.0 license](https://github.com/awinml/voyage-embedders-haystack/blob/main/LICENSE).\n\nMaintained by [Ashwin Mathur](https://github.com/awinml).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawinml%2Fvoyage-embedders-haystack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fawinml%2Fvoyage-embedders-haystack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawinml%2Fvoyage-embedders-haystack/lists"}