{"id":13510714,"url":"https://github.com/docarray/docarray","last_synced_at":"2026-01-12T00:55:04.672Z","repository":{"id":37092101,"uuid":"438303578","full_name":"docarray/docarray","owner":"docarray","description":"Represent, send, store and search multimodal data","archived":false,"fork":false,"pushed_at":"2025-04-24T16:36:17.000Z","size":253598,"stargazers_count":3055,"open_issues_count":102,"forks_count":232,"subscribers_count":45,"default_branch":"main","last_synced_at":"2025-05-07T21:59:56.440Z","etag":null,"topics":["cross-modal","data-structures","dataclass","deep-learning","docarray","elasticsearch","fastapi","machine-learning","multi-modal","multimodal","nearest-neighbor-search","nested-data","neural-search","protobuf","pydantic","pytorch","qdrant","semantic-search","weaviate"],"latest_commit_sha":null,"homepage":"https://docs.docarray.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/docarray.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-14T15:26:24.000Z","updated_at":"2025-05-06T19:04:23.000Z","dependencies_parsed_at":"2023-12-22T12:24:05.615Z","dependency_job_id":"f82ee31d-74c2-4e52-b263-642bf7efd51c","html_url":"https://github.com/docarray/docarray","commit_stats":{"total_commits":914,"total_committers":59,"mean_commits":"15.491525423728813","dds":0.6444201312910285,"last_synced_commit":"9c95f3adfd329f1b11671ca3b9e8c10163021206"},"previous_names":["jina-ai/docarray"],"tags_count":168,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docarray%2Fdocarray","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docarray%2Fdocarray/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docarray%2Fdocarray/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docarray%2Fdocarray/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/docarray","download_url":"https://codeload.github.com/docarray/docarray/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253580003,"owners_count":21930863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cross-modal","data-structures","dataclass","deep-learning","docarray","elasticsearch","fastapi","machine-learning","multi-modal","multimodal","nearest-neighbor-search","nested-data","neural-search","protobuf","pydantic","pytorch","qdrant","semantic-search","weaviate"],"created_at":"2024-08-01T02:01:51.159Z","updated_at":"2026-01-12T00:55:04.665Z","avatar_url":"https://github.com/docarray.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/docarray/docarray/blob/main/docs/assets/logo-dark.svg?raw=true\" alt=\"DocArray logo: The data structure for unstructured data\" width=\"150px\"\u003e\n\u003cbr\u003e\n\u003cb\u003eThe data structure for multimodal data\u003c/b\u003e\n\u003c/p\u003e\n\n\u003cp align=center\u003e\n\u003ca href=\"https://pypi.org/project/docarray/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/docarray?style=flat-square\u0026amp;label=Release\" alt=\"PyPI\"\u003e\u003c/a\u003e\n\u003ca href=\"https://bestpractices.coreinfrastructure.org/projects/6554\"\u003e\u003cimg src=\"https://bestpractices.coreinfrastructure.org/projects/6554/badge\"\u003e\u003c/a\u003e\n\u003ca href=\"https://codecov.io/gh/docarray/docarray\"\u003e\u003cimg alt=\"Codecov branch\" src=\"https://img.shields.io/codecov/c/github/docarray/docarray/main?\u0026logo=Codecov\u0026logoColor=white\u0026style=flat-square\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypistats.org/packages/docarray\"\u003e\u003cimg alt=\"PyPI - Downloads from official pypistats\" src=\"https://img.shields.io/pypi/dm/docarray?style=flat-square\"\u003e\u003c/a\u003e\n\u003ca href=\"https://discord.gg/WaMp6PVPgR\"\u003e\u003cimg src=\"https://dcbadge.vercel.app/api/server/WaMp6PVPgR?theme=default-inverted\u0026style=flat-square\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003e **Note**\n\u003e The README you're currently viewing is for DocArray\u003e0.30, which introduces some significant changes from DocArray 0.21. If you wish to continue using the older DocArray \u003c=0.21, ensure you install it via `pip install docarray==0.21`. Refer to its [codebase](https://github.com/docarray/docarray/tree/v0.21.0), [documentation](https://docarray.jina.ai), and [its hot-fixes branch](https://github.com/docarray/docarray/tree/docarray-v1-fixes) for more information.\n\n\nDocArray is a Python library expertly crafted for the [representation](#represent), [transmission](#send), [storage](#store), and [retrieval](#retrieve) of multimodal data. Tailored for the development of multimodal AI applications, its design guarantees seamless integration with the extensive Python and machine learning ecosystems. As of January 2022, DocArray is openly distributed under the [Apache License 2.0](https://github.com/docarray/docarray/blob/main/LICENSE.md) and currently enjoys the status of a sandbox project within the [LF AI \u0026 Data Foundation](https://lfaidata.foundation/).\n\n\n\n- :fire: Offers native support for **[NumPy](https://github.com/numpy/numpy)**, **[PyTorch](https://github.com/pytorch/pytorch)**, **[TensorFlow](https://github.com/tensorflow/tensorflow)**, and **[JAX](https://github.com/google/jax)**, catering specifically to **model training scenarios**.\n- :zap: Based on **[Pydantic](https://github.com/pydantic/pydantic)**, and instantly compatible with web and microservice frameworks like **[FastAPI](https://github.com/tiangolo/fastapi/)** and **[Jina](https://github.com/jina-ai/jina/)**.\n- :package: Provides support for vector databases such as **[Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [ElasticSearch](https://www.elastic.co/de/elasticsearch/), **[Redis](https://redis.io/)**, **[Mongo Atlas](https://www.mongodb.com/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**.\n- :chains: Allows data transmission as JSON over **HTTP** or as **[Protobuf](https://protobuf.dev/)** over **[gRPC](https://grpc.io/)**.\n\n## Installation\n\nTo install DocArray from the CLI, run the following command:\n\n```shell\npip install -U docarray\n```\n\n\u003e **Note**\n\u003e To use DocArray \u003c=0.21, make sure you install via `pip install docarray==0.21` and check out its [codebase](https://github.com/docarray/docarray/tree/v0.21.0) and [docs](https://docarray.jina.ai) and [its hot-fixes branch](https://github.com/docarray/docarray/tree/docarray-v1-fixes).\n\n## Get Started\nNew to DocArray? Depending on your use case and background, there are multiple ways to learn about DocArray:\n \n- [Coming from pure PyTorch or TensorFlow](#coming-from-pytorch)\n- [Coming from Pydantic](#coming-from-pydantic)\n- [Coming from FastAPI](#coming-from-fastapi)\n- [Coming from Jina](#coming-from-jina)\n- [Coming from a vector database](#coming-from-a-vector-database)\n- [Coming from Langchain](#coming-from-langchain)\n\n\n## Represent\n\nDocArray empowers you to **represent your data** in a manner that is inherently attuned to machine learning.\n\nThis is particularly beneficial for various scenarios:\n\n- :running: You are **training a model**: You're dealing with tensors of varying shapes and sizes, each signifying different elements. You desire a method to logically organize them.\n- :cloud: You are **serving a model**: Let's say through FastAPI, and you wish to define your API endpoints precisely.\n- :card_index_dividers: You are **parsing data**: Perhaps for future deployment in your machine learning or data science projects.\n\n\u003e :bulb: **Familiar with Pydantic?** You'll be pleased to learn\n\u003e that DocArray is not only constructed atop Pydantic but also maintains complete compatibility with it!\n\u003e Furthermore, we have a [specific section](#coming-from-pydantic) dedicated to your needs!\n\nIn essence, DocArray facilitates data representation in a way that mirrors Python dataclasses, with machine learning being an integral component:\n\n\n```python\nfrom docarray import BaseDoc\nfrom docarray.typing import TorchTensor, ImageUrl\nimport torch\n\n\n# Define your data model\nclass MyDocument(BaseDoc):\n    description: str\n    image_url: ImageUrl  # could also be VideoUrl, AudioUrl, etc.\n    image_tensor: TorchTensor[1704, 2272, 3]  # you can express tensor shapes!\n\n\n# Stack multiple documents in a Document Vector\nfrom docarray import DocVec\n\nvec = DocVec[MyDocument](\n    [\n        MyDocument(\n            description=\"A cat\",\n            image_url=\"https://example.com/cat.jpg\",\n            image_tensor=torch.rand(1704, 2272, 3),\n        ),\n    ]\n    * 10\n)\nprint(vec.image_tensor.shape)  # (10, 1704, 2272, 3)\n```\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick for more details\u003c/summary\u003e\n\nLet's take a closer look at how you can represent your data with DocArray:\n\n```python\nfrom docarray import BaseDoc\nfrom docarray.typing import TorchTensor, ImageUrl\nfrom typing import Optional\nimport torch\n\n\n# Define your data model\nclass MyDocument(BaseDoc):\n    description: str\n    image_url: ImageUrl  # could also be VideoUrl, AudioUrl, etc.\n    image_tensor: Optional[\n        TorchTensor[1704, 2272, 3]\n    ] = None  # could also be NdArray or TensorflowTensor\n    embedding: Optional[TorchTensor] = None\n```\n\nSo not only can you define the types of your data, you can even **specify the shape of your tensors!**\n\n```python\n# Create a document\ndoc = MyDocument(\n    description=\"This is a photo of a mountain\",\n    image_url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n)\n\n# Load image tensor from URL\ndoc.image_tensor = doc.image_url.load()\n\n\n# Compute embedding with any model of your choice\ndef clip_image_encoder(image_tensor: TorchTensor) -\u003e TorchTensor:  # dummy function\n    return torch.rand(512)\n\n\ndoc.embedding = clip_image_encoder(doc.image_tensor)\n\nprint(doc.embedding.shape)  # torch.Size([512])\n```\n\n### Compose nested Documents\n\nOf course, you can compose Documents into a nested structure:\n\n```python\nfrom docarray import BaseDoc\nfrom docarray.documents import ImageDoc, TextDoc\nimport numpy as np\n\n\nclass MultiModalDocument(BaseDoc):\n    image_doc: ImageDoc\n    text_doc: TextDoc\n\n\ndoc = MultiModalDocument(\n    image_doc=ImageDoc(tensor=np.zeros((3, 224, 224))), text_doc=TextDoc(text='hi!')\n)\n```\n\nYou rarely work with a single data point at a time, especially in machine learning applications. That's why you can easily collect multiple `Documents`:\n\n### Collect multiple `Documents`\n\nWhen building or interacting with an ML system, usually you want to process multiple Documents (data points) at once.\n\nDocArray offers two data structures for this:\n\n- **`DocVec`**: A vector of `Documents`. All tensors in the documents are stacked into a single tensor. **Perfect for batch processing and use inside of ML models**.\n- **`DocList`**: A list of `Documents`. All tensors in the documents are kept as-is. **Perfect for streaming, re-ranking, and shuffling of data**.\n\nLet's take a look at them, starting with `DocVec`:\n\n```python\nfrom docarray import DocVec, BaseDoc\nfrom docarray.typing import AnyTensor, ImageUrl\nimport numpy as np\n\n\nclass Image(BaseDoc):\n    url: ImageUrl\n    tensor: AnyTensor  # this allows torch, numpy, and tensor flow tensors\n\n\nvec = DocVec[Image](  # the DocVec is parametrized by your personal schema!\n    [\n        Image(\n            url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n            tensor=np.zeros((3, 224, 224)),\n        )\n        for _ in range(100)\n    ]\n)\n``` \n\nIn the code snippet above, `DocVec` is **parametrized by the type of document** you want to use with it: `DocVec[Image]`.\n\nThis may look weird at first, but we're confident that you'll get used to it quickly!\nBesides, it lets us do some cool things, like having **bulk access to the fields that you defined** in your document:\n\n```python\ntensor = vec.tensor  # gets all the tensors in the DocVec\nprint(tensor.shape)  # which are stacked up into a single tensor!\nprint(vec.url)  # you can bulk access any other field, too\n```\n\nThe second data structure, `DocList`, works in a similar way:\n\n```python\nfrom docarray import DocList\n\ndl = DocList[Image](  # the DocList is parametrized by your personal schema!\n    [\n        Image(\n            url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n            tensor=np.zeros((3, 224, 224)),\n        )\n        for _ in range(100)\n    ]\n)\n```\n\nYou can still bulk access the fields of your document:\n\n```python\ntensors = dl.tensor  # gets all the tensors in the DocList\nprint(type(tensors))  # as a list of tensors\nprint(dl.url)  # you can bulk access any other field, too\n```\n\nAnd you can insert, remove, and append documents to your `DocList`:\n\n```python\n# append\ndl.append(\n    Image(\n        url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n        tensor=np.zeros((3, 224, 224)),\n    )\n)\n# delete\ndel dl[0]\n# insert\ndl.insert(\n    0,\n    Image(\n        url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n        tensor=np.zeros((3, 224, 224)),\n    ),\n)\n```\n\nAnd you can seamlessly switch between `DocVec` and `DocList`:\n\n```python\nvec_2 = dl.to_doc_vec()\nassert isinstance(vec_2, DocVec)\n\ndl_2 = vec_2.to_doc_list()\nassert isinstance(dl_2, DocList)\n```\n\n\u003c/details\u003e\n\n## Send\n\nDocArray facilitates the **transmission of your data** in a manner inherently compatible with machine learning.\n\nThis includes native support for **Protobuf and gRPC**, along with **HTTP** and serialization to JSON, JSONSchema, Base64, and Bytes.\n\nThis feature proves beneficial for several scenarios:\n\n- :cloud: You are **serving a model**, perhaps through frameworks like **[Jina](https://github.com/jina-ai/jina/)** or **[FastAPI](https://github.com/tiangolo/fastapi/)**\n- :spider_web: You are **distributing your model** across multiple machines and need an efficient means of transmitting your data between nodes\n- :gear: You are architecting a **microservice** environment and require a method for data transmission between microservices\n\n\u003e :bulb: **Are you familiar with FastAPI?** You'll be delighted to learn\n\u003e that DocArray maintains full compatibility with FastAPI!\n\u003e Plus, we have a [dedicated section](#coming-from-fastapi) specifically for you!\n\nWhen it comes to data transmission, serialization is a crucial step. Let's delve into how DocArray streamlines this process:\n\n\n```python\nfrom docarray import BaseDoc\nfrom docarray.typing import ImageTorchTensor\nimport torch\n\n\n# model your data\nclass MyDocument(BaseDoc):\n    description: str\n    image: ImageTorchTensor[3, 224, 224]\n\n\n# create a Document\ndoc = MyDocument(\n    description=\"This is a description\",\n    image=torch.zeros((3, 224, 224)),\n)\n\n# serialize it!\nproto = doc.to_protobuf()\nbytes_ = doc.to_bytes()\njson = doc.json()\n\n# deserialize it!\ndoc_2 = MyDocument.from_protobuf(proto)\ndoc_4 = MyDocument.from_bytes(bytes_)\ndoc_5 = MyDocument.parse_raw(json)\n```\n\nOf course, serialization is not all you need. So check out how DocArray integrates with **[Jina](https://github.com/jina-ai/jina/)** and **[FastAPI](https://github.com/tiangolo/fastapi/)**.\n\n## Store\n\nAfter modeling and possibly distributing your data, you'll typically want to **store it** somewhere. That's where DocArray steps in!\n\n**Document Stores** provide a seamless way to, as the name suggests, store your Documents. Be it locally or remotely, you can do it all through the same user interface:\n\n- :cd: **On disk**, as a file in your local filesystem\n- :bucket: On **[AWS S3](https://aws.amazon.com/de/s3/)**\n- :cloud: On **[Jina AI Cloud](https://cloud.jina.ai/)**\n\nThe Document Store interface lets you push and pull Documents to and from multiple data sources, all with the same user interface.\n\nFor example, let's see how that works with on-disk storage:\n\n```python\nfrom docarray import BaseDoc, DocList\n\n\nclass SimpleDoc(BaseDoc):\n    text: str\n\n\ndocs = DocList[SimpleDoc]([SimpleDoc(text=f'doc {i}') for i in range(8)])\ndocs.push('file://simple_docs')\n\ndocs_pull = DocList[SimpleDoc].pull('file://simple_docs')\n```\n\n## Retrieve\n\n**Document Indexes** let you index your Documents in a **vector database** for efficient similarity-based retrieval.\n\nThis is useful for:\n\n- :left_speech_bubble: Augmenting **LLMs and Chatbots** with domain knowledge ([Retrieval Augmented Generation](https://arxiv.org/abs/2005.11401))\n- :mag: **Neural search** applications\n- :bulb: **Recommender systems**\n\nCurrently, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**,  **[Redis](https://redis.io/)**, **[Mongo Atlas](https://www.mongodb.com/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come!\n\nThe Document Index interface lets you index and retrieve Documents from multiple vector databases, all with the same user interface.\n\nIt supports ANN vector search, text search, filtering, and hybrid search.\n\n```python\nfrom docarray import DocList, BaseDoc\nfrom docarray.index import HnswDocumentIndex\nimport numpy as np\n\nfrom docarray.typing import ImageUrl, ImageTensor, NdArray\n\n\nclass ImageDoc(BaseDoc):\n    url: ImageUrl\n    tensor: ImageTensor\n    embedding: NdArray[128]\n\n\n# create some data\ndl = DocList[ImageDoc](\n    [\n        ImageDoc(\n            url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n            tensor=np.zeros((3, 224, 224)),\n            embedding=np.random.random((128,)),\n        )\n        for _ in range(100)\n    ]\n)\n\n# create a Document Index\nindex = HnswDocumentIndex[ImageDoc](work_dir='/tmp/test_index')\n\n\n# index your data\nindex.index(dl)\n\n# find similar Documents\nquery = dl[0]\nresults, scores = index.find(query, limit=10, search_field='embedding')\n```\n\n---\n\n## Learn DocArray\n\nDepending on your background and use case, there are different ways for you to understand DocArray.\n\n### Coming from DocArray \u003c=0.21\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nIf you are using DocArray version 0.30.0 or lower, you will be familiar with its [dataclass API](https://docarray.jina.ai/fundamentals/dataclass/).\n\n_DocArray \u003e=0.30 is that idea, taken seriously._ Every document is created through a dataclass-like interface,\ncourtesy of [Pydantic](https://pydantic-docs.helpmanual.io/usage/models/).\n\nThis gives the following advantages:\n- **Flexibility:** No need to conform to a fixed set of fields -- your data defines the schema\n- **Multimodality:** At their core, documents are just dictionaries. This makes it easy to create and send them from any language, not just Python.\n\nYou may also be familiar with our old Document Stores for vector DB integration.\nThey are now called **Document Indexes** and offer the following improvements (see [here](#store) for the new API):\n\n- **Hybrid search:** You can now combine vector search with text search, and even filter by arbitrary fields\n- **Production-ready:** The new Document Indexes are a much thinner wrapper around the various vector DB libraries, making them more robust and easier to maintain\n- **Increased flexibility:** We strive to support any configuration or setting that you could perform through the DB's first-party client\n\nFor now, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**, **[Redis](https://redis.io/)**, **[Mongo Atlas](https://www.mongodb.com/)**, Exact Nearest Neighbour search and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come.\n\n\u003c/details\u003e\n\n### Coming from Pydantic\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nIf you come from Pydantic, you can see DocArray documents as juiced up Pydantic models, and DocArray as a collection of goodies around them.\n\nMore specifically, we set out to **make Pydantic fit for the ML world** - not by replacing it, but by building on top of it!\n\nThis means you get the following benefits:\n\n- **ML-focused types**: Tensor, TorchTensor, Embedding, ..., including **tensor shape validation**\n- Full compatibility with **FastAPI**\n- **DocList** and **DocVec** generalize the idea of a model to a _sequence_ or _batch_ of models. Perfect for **use in ML models** and other batch processing tasks.\n- **Types that are alive**: ImageUrl can `.load()` a URL to image tensor, TextUrl can load and tokenize text documents, etc.\n- Cloud-ready: Serialization to **Protobuf** for use with microservices and **gRPC**\n- **Pre-built multimodal documents** for different data modalities: Image, Text, 3DMesh, Video, Audio and more. Note that all of these are valid Pydantic models!\n- **Document Stores** and **Document Indexes** let you store your data and retrieve it using **vector search**\n\nThe most obvious advantage here is **first-class support for ML centric data**, such as `{Torch, TF, ...}Tensor`, `Embedding`, etc.\n\nThis includes handy features such as validating the shape of a tensor:\n\n```python\nfrom docarray import BaseDoc\nfrom docarray.typing import TorchTensor\nimport torch\n\n\nclass MyDoc(BaseDoc):\n    tensor: TorchTensor[3, 224, 224]\n\n\ndoc = MyDoc(tensor=torch.zeros(3, 224, 224))  # works\ndoc = MyDoc(tensor=torch.zeros(224, 224, 3))  # works by reshaping\n\ntry:\n    doc = MyDoc(tensor=torch.zeros(224))  # fails validation\nexcept Exception as e:\n    print(e)\n    # tensor\n    # Cannot reshape tensor of shape (224,) to shape (3, 224, 224) (type=value_error)\n\n\nclass Image(BaseDoc):\n    tensor: TorchTensor[3, 'x', 'x']\n\n\nImage(tensor=torch.zeros(3, 224, 224))  # works\n\ntry:\n    Image(\n        tensor=torch.zeros(3, 64, 128)\n    )  # fails validation because second dimension does not match third\nexcept Exception as e:\n    print()\n\n\ntry:\n    Image(\n        tensor=torch.zeros(4, 224, 224)\n    )  # fails validation because of the first dimension\nexcept Exception as e:\n    print(e)\n    # Tensor shape mismatch. Expected(3, 'x', 'x'), got(4, 224, 224)(type=value_error)\n\ntry:\n    Image(\n        tensor=torch.zeros(3, 64)\n    )  # fails validation because it does not have enough dimensions\nexcept Exception as e:\n    print(e)\n    # Tensor shape mismatch. Expected (3, 'x', 'x'), got (3, 64) (type=value_error)\n```\n\n\u003c/details\u003e\n\n### Coming from PyTorch\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nIf you come from PyTorch, you can see DocArray mainly as a way of _organizing your data as it flows through your model_.\n\nIt offers you several advantages:\n\n- Express **tensor shapes in type hints**\n- **Group tensors that belong to the same object**, e.g. an audio track and an image\n- **Go directly to deployment**, by re-using your data model as a [FastAPI](https://fastapi.tiangolo.com/) or [Jina](https://github.com/jina-ai/jina) API schema\n- Connect model components between **microservices**, using Protobuf and gRPC\n\nDocArray can be used directly inside ML models to handle and represent multimodaldata.\nThis allows you to reason about your data using DocArray's abstractions deep inside of `nn.Module`,\nand provides a FastAPI-compatible schema that eases the transition between model training and model serving.\n\nTo see the effect of this, let's first observe a vanilla PyTorch implementation of a tri-modal ML model:\n\n```python\nimport torch\nfrom torch import nn\n\n\ndef encoder(x):\n    return torch.rand(512)\n\n\nclass MyMultiModalModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.audio_encoder = encoder()\n        self.image_encoder = encoder()\n        self.text_encoder = encoder()\n\n    def forward(self, text_1, text_2, image_1, image_2, audio_1, audio_2):\n        embedding_text_1 = self.text_encoder(text_1)\n        embedding_text_2 = self.text_encoder(text_2)\n\n        embedding_image_1 = self.image_encoder(image_1)\n        embedding_image_2 = self.image_encoder(image_2)\n\n        embedding_audio_1 = self.image_encoder(audio_1)\n        embedding_audio_2 = self.image_encoder(audio_2)\n\n        return (\n            embedding_text_1,\n            embedding_text_2,\n            embedding_image_1,\n            embedding_image_2,\n            embedding_audio_1,\n            embedding_audio_2,\n        )\n```\n\nNot very easy on the eyes if you ask us. And even worse, if you need to add one more modality you have to touch every part of your code base, changing the `forward()` return type and making a whole lot of changes downstream from that.\n\nSo, now let's see what the same code looks like with DocArray:\n\n```python\nfrom docarray import DocList, BaseDoc\nfrom docarray.documents import ImageDoc, TextDoc, AudioDoc\nfrom docarray.typing import TorchTensor\nfrom torch import nn\nimport torch\n\n\ndef encoder(x):\n    return torch.rand(512)\n\n\nclass Podcast(BaseDoc):\n    text: TextDoc\n    image: ImageDoc\n    audio: AudioDoc\n\n\nclass PairPodcast(BaseDoc):\n    left: Podcast\n    right: Podcast\n\n\nclass MyPodcastModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.audio_encoder = encoder()\n        self.image_encoder = encoder()\n        self.text_encoder = encoder()\n\n    def forward_podcast(self, docs: DocList[Podcast]) -\u003e DocList[Podcast]:\n        docs.audio.embedding = self.audio_encoder(docs.audio.tensor)\n        docs.text.embedding = self.text_encoder(docs.text.tensor)\n        docs.image.embedding = self.image_encoder(docs.image.tensor)\n\n        return docs\n\n    def forward(self, docs: DocList[PairPodcast]) -\u003e DocList[PairPodcast]:\n        docs.left = self.forward_podcast(docs.left)\n        docs.right = self.forward_podcast(docs.right)\n\n        return docs\n```\n\nLooks much better, doesn't it?\nYou instantly win in code readability and maintainability. And for the same price you can turn your PyTorch model into a FastAPI app and reuse your Document\nschema definition (see [below](#coming-from-fastapi)). Everything is handled in a pythonic manner by relying on type hints.\n\n\u003c/details\u003e\n\n\n### Coming from TensorFlow\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nLike the [PyTorch approach](#coming-from-pytorch), you can also use DocArray with TensorFlow to handle and represent multimodal data inside your ML model.\n\nFirst off, to use DocArray with TensorFlow we first need to install it as follows:\n\n```\npip install tensorflow==2.12.0\npip install protobuf==3.19.0\n```\n\nCompared to using DocArray with PyTorch, there is one main difference when using it with TensorFlow:\nWhile DocArray's `TorchTensor` is a subclass of `torch.Tensor`, this is not the case for the `TensorFlowTensor`: Due to some technical limitations of `tf.Tensor`, DocArray's `TensorFlowTensor` is not a subclass of `tf.Tensor` but rather stores a `tf.Tensor` in its `.tensor` attribute. \n\nHow does this affect you? Whenever you want to access the tensor data to, let's say, do operations with it or hand it to your ML model, instead of handing over your `TensorFlowTensor` instance, you need to access its `.tensor` attribute.\n\nThis would look like the following:\n\n```python\nfrom typing import Optional\n\nfrom docarray import DocList, BaseDoc\n\nimport tensorflow as tf\n\n\nclass Podcast(BaseDoc):\n    audio_tensor: Optional[AudioTensorFlowTensor] = None\n    embedding: Optional[AudioTensorFlowTensor] = None\n\n\nclass MyPodcastModel(tf.keras.Model):\n    def __init__(self):\n        super().__init__()\n        self.audio_encoder = AudioEncoder()\n\n    def call(self, inputs: DocList[Podcast]) -\u003e DocList[Podcast]:\n        inputs.audio_tensor.embedding = self.audio_encoder(\n            inputs.audio_tensor.tensor\n        )  # access audio_tensor's .tensor attribute\n        return inputs\n```\n\n\u003c/details\u003e\n\n### Coming from FastAPI\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nDocuments are Pydantic Models (with a twist), and as such they are fully compatible with FastAPI!\n\nBut why should you use them, and not the Pydantic models you already know and love?\nGood question!\n\n- Because of the ML-first features, types and validations, [here](#coming-from-pydantic)\n- Because DocArray can act as an [ORM for vector databases](#coming-from-a-vector-database), similar to what SQLModel does for SQL databases\n\nAnd to seal the deal, let us show you how easily documents slot into your FastAPI app:\n\n```python\nimport numpy as np\nfrom fastapi import FastAPI\nfrom docarray.base_doc import DocArrayResponse\nfrom docarray import BaseDoc\nfrom docarray.documents import ImageDoc\nfrom docarray.typing import NdArray, ImageTensor\n\n\nclass InputDoc(BaseDoc):\n    img: ImageDoc\n    text: str\n\n\nclass OutputDoc(BaseDoc):\n    embedding_clip: NdArray\n    embedding_bert: NdArray\n\n\napp = FastAPI()\n\n\ndef model_img(img: ImageTensor) -\u003e NdArray:\n    return np.zeros((100, 1))\n\n\ndef model_text(text: str) -\u003e NdArray:\n    return np.zeros((100, 1))\n\n\n@app.post(\"/embed/\", response_model=OutputDoc, response_class=DocArrayResponse)\nasync def create_item(doc: InputDoc) -\u003e OutputDoc:\n    doc = OutputDoc(\n        embedding_clip=model_img(doc.img.tensor), embedding_bert=model_text(doc.text)\n    )\n    return doc\n\n\ninput_doc = InputDoc(text='', img=ImageDoc(tensor=np.random.random((3, 224, 224))))\n\nasync with AsyncClient(app=app, base_url=\"http://test\") as ac:\n    response = await ac.post(\"/embed/\", data=input_doc.json())\n```\n\nJust like a vanilla Pydantic model!\n\n\u003c/details\u003e\n\n### Coming from Jina\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nJina has adopted docarray as their library for representing and serializing Documents.\n\nJina allows to serve models and services that are built with DocArray allowing you to serve and scale these applications\nmaking full use of DocArray's serialization capabilites. \n\n```python\nimport numpy as np\nfrom jina import Deployment, Executor, requests\nfrom docarray import BaseDoc, DocList\nfrom docarray.documents import ImageDoc\nfrom docarray.typing import NdArray, ImageTensor\n\n\nclass InputDoc(BaseDoc):\n    img: ImageDoc\n    text: str\n\n\nclass OutputDoc(BaseDoc):\n    embedding_clip: NdArray\n    embedding_bert: NdArray\n\n\ndef model_img(img: ImageTensor) -\u003e NdArray:\n    return np.zeros((100, 1))\n\n\ndef model_text(text: str) -\u003e NdArray:\n    return np.zeros((100, 1))\n\n\nclass MyEmbeddingExecutor(Executor):\n    @requests(on='/embed')\n    def encode(self, docs: DocList[InputDoc], **kwargs) -\u003e DocList[OutputDoc]:\n        ret = DocList[OutputDoc]()\n        for doc in docs:\n            output = OutputDoc(\n                embedding_clip=model_img(doc.img.tensor),\n                embedding_bert=model_text(doc.text),\n            )\n            ret.append(output)\n        return ret\n\n\nwith Deployment(\n    protocols=['grpc', 'http'], ports=[12345, 12346], uses=MyEmbeddingExecutor\n) as dep:\n    resp = dep.post(\n        on='/embed',\n        inputs=DocList[InputDoc](\n            [InputDoc(text='', img=ImageDoc(tensor=np.random.random((3, 224, 224))))]\n        ),\n        return_type=DocList[OutputDoc],\n    )\n    print(resp)\n```\n\n\u003c/details\u003e\n\n### Coming from a vector database\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nIf you came across DocArray as a universal vector database client, you can best think of it as **a new kind of ORM for vector databases**.\nDocArray's job is to take multimodal, nested and domain-specific data and to map it to a vector database,\nstore it there, and thus make it searchable:\n\n```python\nfrom docarray import DocList, BaseDoc\nfrom docarray.index import HnswDocumentIndex\nimport numpy as np\n\nfrom docarray.typing import ImageUrl, ImageTensor, NdArray\n\n\nclass ImageDoc(BaseDoc):\n    url: ImageUrl\n    tensor: ImageTensor\n    embedding: NdArray[128]\n\n\n# create some data\ndl = DocList[ImageDoc](\n    [\n        ImageDoc(\n            url=\"https://upload.wikimedia.org/wikipedia/commons/2/2f/Alpamayo.jpg\",\n            tensor=np.zeros((3, 224, 224)),\n            embedding=np.random.random((128,)),\n        )\n        for _ in range(100)\n    ]\n)\n\n# create a Document Index\nindex = HnswDocumentIndex[ImageDoc](work_dir='/tmp/test_index2')\n\n\n# index your data\nindex.index(dl)\n\n# find similar Documents\nquery = dl[0]\nresults, scores = index.find(query, limit=10, search_field='embedding')\n```\n\nCurrently, DocArray supports the following vector databases:\n\n- [Weaviate](https://www.weaviate.io/)\n- [Qdrant](https://qdrant.tech/)\n- [Elasticsearch](https://www.elastic.co/elasticsearch/) v8 and v7\n- [Redis](https://redis.io/)\n- [Milvus](https://milvus.io)\n- ExactNNMemorySearch as a local alternative with exact kNN search.\n- [HNSWlib](https://github.com/nmslib/hnswlib) as a local-first ANN alternative\n- [Mongo Atlas](https://www.mongodb.com/)\n\nAn integration of [OpenSearch](https://opensearch.org/) is currently in progress.\n\nOf course this is only one of the things that DocArray can do, so we encourage you to check out the rest of this readme!\n\n\u003c/details\u003e\n\n\n### Coming from Langchain\n\n\u003cdetails markdown=\"1\"\u003e\n  \u003csummary\u003eClick to expand\u003c/summary\u003e\n\nWith DocArray, you can connect external data to LLMs through Langchain. DocArray gives you the freedom to establish \nflexible document schemas and choose from different backends for document storage.\nAfter creating your document index, you can connect it to your Langchain app using [DocArrayRetriever](https://python.langchain.com/docs/modules/data_connection/retrievers/integrations/docarray_retriever).\n\nInstall Langchain via:\n```shell\npip install langchain\n```\n\n1. Define a schema and create documents:\n```python\nfrom docarray import BaseDoc, DocList\nfrom docarray.typing import NdArray\nfrom langchain.embeddings.openai import OpenAIEmbeddings\n\nembeddings = OpenAIEmbeddings()\n\n\n# Define a document schema\nclass MovieDoc(BaseDoc):\n    title: str\n    description: str\n    year: int\n    embedding: NdArray[1536]\n\n\nmovies = [\n    {\"title\": \"#1 title\", \"description\": \"#1 description\", \"year\": 1999},\n    {\"title\": \"#2 title\", \"description\": \"#2 description\", \"year\": 2001},\n]\n\n# Embed `description` and create documents\ndocs = DocList[MovieDoc](\n    MovieDoc(embedding=embeddings.embed_query(movie[\"description\"]), **movie)\n    for movie in movies\n)\n```\n\n2. Initialize a document index using any supported backend:\n```python\nfrom docarray.index import (\n    InMemoryExactNNIndex,\n    HnswDocumentIndex,\n    WeaviateDocumentIndex,\n    QdrantDocumentIndex,\n    ElasticDocIndex,\n    RedisDocumentIndex,\n    MongoDBAtlasDocumentIndex,\n)\n\n# Select a suitable backend and initialize it with data\ndb = InMemoryExactNNIndex[MovieDoc](docs)\n```\n\n3. Finally, initialize a retriever and integrate it into your chain!\n```python\nfrom langchain.chat_models import ChatOpenAI\nfrom langchain.chains import ConversationalRetrievalChain\nfrom langchain.retrievers import DocArrayRetriever\n\n\n# Create a retriever\nretriever = DocArrayRetriever(\n    index=db,\n    embeddings=embeddings,\n    search_field=\"embedding\",\n    content_field=\"description\",\n)\n\n# Use the retriever in your chain\nmodel = ChatOpenAI()\nqa = ConversationalRetrievalChain.from_llm(model, retriever=retriever)\n```\n\nAlternatively, you can use built-in vector stores. Langchain supports two vector stores: [DocArrayInMemorySearch](https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/docarray_in_memory) and [DocArrayHnswSearch](https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/docarray_hnsw). \nBoth are user-friendly and are best suited to small to medium-sized datasets.\n\n\u003c/details\u003e\n\n\n## See also\n\n- [Documentation](https://docs.docarray.org)\n- [DocArray\u003c=0.21 documentation](https://docarray.jina.ai/)\n- [Join our Discord server](https://discord.gg/WaMp6PVPgR)\n- [Donation to Linux Foundation AI\u0026Data blog post](https://jina.ai/news/donate-docarray-lf-for-inclusive-standard-multimodal-data-model/)\n- [Roadmap](https://github.com/docarray/docarray/issues/1714)\n\n\u003e DocArray is a trademark of LF AI Projects, LLC\n\u003e \n","funding_links":[],"categories":["Python","向量数据库、向量搜索、最近邻搜索","Data Containers \u0026 Dataframes","🔍 Libraries \u0026 Frameworks","Industry Strength Information Retrieval","pytorch","Machine Learning"],"sub_categories":["网络服务_其他"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdocarray%2Fdocarray","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdocarray%2Fdocarray","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdocarray%2Fdocarray/lists"}