{"id":21482084,"url":"https://github.com/vitrivr/vitrivr-engine","last_synced_at":"2026-03-10T07:31:47.351Z","repository":{"id":207723398,"uuid":"679355326","full_name":"vitrivr/vitrivr-engine","owner":"vitrivr","description":"vitrivr's next-generation retrieval engine.  It is capable of extracting and retrieving a wider range of multimedia objects such as audio, video, images or 3d models.","archived":false,"fork":false,"pushed_at":"2026-03-05T13:55:21.000Z","size":5470,"stargazers_count":9,"open_issues_count":14,"forks_count":4,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-03-05T17:33:01.687Z","etag":null,"topics":["feature-extraction","multimedia","multimedia-retrieval","query","retrieval","retrieval-systems"],"latest_commit_sha":null,"homepage":"https://vitrivr.org","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vitrivr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-08-16T16:45:02.000Z","updated_at":"2026-03-05T13:55:26.000Z","dependencies_parsed_at":"2024-03-25T10:53:01.881Z","dependency_job_id":"88abb998-32e4-477a-b2a3-7fcd112ab1ed","html_url":"https://github.com/vitrivr/vitrivr-engine","commit_stats":null,"previous_names":["vitrivr/vitrivr-engine"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/vitrivr/vitrivr-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitrivr%2Fvitrivr-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitrivr%2Fvitrivr-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitrivr%2Fvitrivr-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitrivr%2Fvitrivr-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vitrivr","download_url":"https://codeload.github.com/vitrivr/vitrivr-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vitrivr%2Fvitrivr-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30326908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["feature-extraction","multimedia","multimedia-retrieval","query","retrieval","retrieval-systems"],"created_at":"2024-11-23T12:29:50.903Z","updated_at":"2026-03-10T07:31:47.322Z","avatar_url":"https://github.com/vitrivr.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca id=\"btop\"\u003e\u003c/a\u003e\n\u003c!-- README setup inspired by https://github.com/othneildrew/Best-README-Template --\u003e\n\n\u003c!-- === PROJECT LOGO === --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\u003c!-- github does not allow css, hence we use html --\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg src=\"images/vitrivr_512.png\" width=\"512\" alt=\"vitrivr log\"/\u003e\u003cbr /\u003e\u003cbr /\u003e\u003cbr /\u003e\n    \u003cimg src=\"images/vengine-256.png\" alt=\"vitrivr-engine logo\"/\u003e\n  \u003c/a\u003e\n \u003cbr/\u003e\n \u003cbr/\u003e\n\n[![GitHub Release](https://img.shields.io/github/release/vitrivr/vitrivr-engine?include_prereleases=\u0026sort=semver\u0026color=blue\u0026style=for-the-badge\u0026label=Release)](https://github.com/vitrivr/vitrivr-engine/releases)\n[![License](https://img.shields.io/badge/License-MIT-blue?style=for-the-badge)](#license)\n[![issues - vitrivr-engine](https://img.shields.io/github/issues/vitrivr/vitrivr-engine?style=for-the-badge)](https://github.com/vitrivr/vitrivr-engine/issues)\n\n[vitrivr](https://vitrivr.org)'s next-generation retrieval engine.\n\n[Read The Docs](https://github.com/vitrivr/vitrivr-engine/wiki)\n\n\u003c/div\u003e\n\n## vitrivr-engine\n\nvitrivr-engine is [vitrivr](https://vitrivr.org)'s next generation retrieval engine with a flexible, modular\narchitecture.\nBased on the experiences with its predecessor, [Cineast](https://github.com/vitrivr/cineast),\nvitrivr engine's data model, ingestion pipeline and retrieval logic have been reworked from the ground.\nEssentially, vitrivr-engine enables the analysis (i.e. ingestion) and querying (i.e. retrieval ) of\nmultimedia data.\n\n## Built With\n\n* [Kotlin](https://kotlinlang.org) for the JVM, e.g. [OpenJDK](https://openjdk.org/)\n* [OpenApi](https://www.openapis.org/)\n* [PosgreSQL with pgvector](https://www.postgresql.org/)\n* [CottontailDB](https://github.com/vitrivr/CottontailDB)\n* ... and more ...\n\n## Getting Started and Example Usage\n\nThis guide shows how to build and run vitrivr-engine either by [manually building](#build-and-extract) it or using the [Docker setup](#docker-setup).\n\n### Getting Started\n\nThis guide shows how to build and run `vitrivr-engine` using the provided ZIP distribution and a schema configuration\nfile.\n\n#### Prerequisites\n\n* JDK 21 or higher (e.g., [OpenJDK](https://openjdk.org))\n* A running database:\n    * [PostgreSQL with `pgVector`](https://github.com/pgvector/pgvector) (recommended) **or**\n    * [CottontailDB](https://github.com/vitrivr/cottontaildb) (v0.16.5+)\n\n* Some multimedia content (images or video)\n\n---\n\n##### Build and Extract\n\n1. Clone the repository and build the distribution:\n\n   ```bash\n   git clone https://github.com/vitrivr/vitrivr-engine.git\n   cd vitrivr-engine\n   ./gradlew distZip\n   ```\n\n2. Unzip the distribution:\n\n   ```bash\n   unzip -d vitrivr-engine-server/build/distributions/vitrivr-engine-server-*.zip\n   ```\n\n3. Prepare a folder structure with your media (e.g. `sandbox/media/`):\n\n   ```\n   vitrivr-engine/\n   └── vitrivr-engine-server-\u003cversion\u003e/\n       ├── bin/\n       └── lib/\n   └── sandbox/\n       └── media/\n           ├── image1.png\n           └── video.mp4\n   ```\n4. Clone and run the external feature extraction service if you want to use CLIP or DINO for feature extraction:\n    ```bash\n   git clone https://github.com/vitrivr/vitrivr-python-descriptor-server\n   cd vitrivr-python-descriptor-server\n   python3 -m venv features\n   source features/bin/activate\n   pip install -r requirements.txt\n   python3 main.py\n   ```\n\n---\n\n##### Docker Setup\n\nAlternatively, you can use the Docker image to run `vitrivr-engine`: With the help of the docker-compose file, you can\nset up a PostgreSQL database with `pgVector`, an external instance to extract features and the `vitrivr-engine` server. \nNote: The video feature extraction using the docker image is currently not supported when running on Apple Silicon \n(e.g., M1, M2, M3) due to architecture compatibility issues.\n\nRun the following command to start the Docker containers:\n\n   ```bash\n   docker compose up --build\n   ```\n\nThis command will start the PostgreSQL database, the feature extraction service, and the `vitrivr-engine` server.\nAll configuration files are mounted from the `vitrivr-engine-server/config` directory, so you can easily modify them and\nrestart the containers to apply changes. Also, all data is mounted to the `vitrivr-engine-server/data` directory, so you\ncan easily access the data files.\n\n#### Setup Configuration\n\nCreate a `config-schema.json` file to define your schema, fields, and backend. Make sure to adjust the database\nconnection parameters to match your PostgreSQL or CottontailDB setup. The following example uses PostgreSQL with\n`pgVector` and defines fields for image and video ingestion, including a CLIP feature extractor and a thumbnail\nexporter.\n\nIf you are using Docker, please review the config files in `example-configs/docker`.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ccode\u003econfig-schema.json\u003c/code\u003e\u003c/summary\u003e\n\n```json\n{\n  \"schemas\": {\n    \"sandbox\": {\n      \"connection\": {\n        \"database\": \"PgVectorConnectionProvider\",\n        \"parameters\": {\n          \"host\": \"127.0.0.1\",\n          \"port\": \"5432\",\n          \"database\": \"postgres\",\n          \"username\": \"postgres\",\n          \"password\": \"secret\"\n        }\n      },\n      \"fields\": {\n        \"averagecolor\": {\n          \"factory\": \"AverageColor\"\n        },\n        \"file\": {\n          \"factory\": \"FileSourceMetadata\"\n        },\n        \"time\": {\n          \"factory\": \"TemporalMetadata\"\n        },\n        \"clip\": {\n          \"factory\": \"CLIP\",\n          \"parameters\": {\n            \"host\": \"http://127.0.0.1:8888\"\n          }\n        }\n      },\n      \"resolvers\": {\n        \"disk\": {\n          \"factory\": \"DiskResolver\",\n          \"parameters\": {\n            \"location\": \"./sandbox/thumbnails/\",\n            \"mimeType\": \"JPG\"\n          }\n        }\n      },\n      \"exporters\": {\n        \"thumbnail\": {\n          \"factory\": \"ThumbnailExporter\",\n          \"parameters\": {\n            \"maxSideResolution\": \"400\",\n            \"mimeType\": \"JPG\",\n            \"resolver\": \"disk\"\n          }\n        }\n      },\n      \"extractionPipelines\": {\n        \"image\": {\n          \"path\": \"/vitrivr-engine/example-configs/native/image-ingest.json\"\n        },\n        \"video\": {\n          \"path\": \"/vitrivr-engine/example-configs/native/video-ingest.json\"\n        }\n      }\n    }\n  }\n}\n\n\n```\n\n\u003c/details\u003e\n\n---\n\n#### Setup Pipelines\n\nDepending on your media type, you can create individual ingestion pipelines for images and videos. The following example\nshows how to set up pipelines for image and video ingestion:\nThe fields used here are defined in the `config-schema.json` file, and the pipelines are already defined in the\n`extractionPipelines` section of the `config-schema.json`.\n\u003cdetails\u003e\n\u003csummary\u003e\u003ccode\u003eimage-ingest.json\u003c/code\u003e\u003c/summary\u003e\n\n```json\n{\n  \"schema\": \"sandbox\",\n  \"context\": {\n    \"contentFactory\": \"InMemoryContentFactory\",\n    \"resolvers\": [\n      \"disk\"\n    ],\n    \"local\": {\n      \"enumerator\": {\n        \"path\": \"./media\",\n        \"depth\": \"1\"\n      },\n      \"thumbnail\": {\n        \"maxSideResolution\": \"400\",\n        \"mimeType\": \"JPG\"\n      }\n    }\n  },\n  \"operators\": {\n    \"enumerator\": {\n      \"type\": \"ENUMERATOR\",\n      \"factory\": \"FileSystemEnumerator\",\n      \"mediaTypes\": [\n        \"IMAGE\"\n      ]\n    },\n    \"decoder\": {\n      \"type\": \"DECODER\",\n      \"factory\": \"ImageDecoder\"\n    },\n    \"persister\": {\n      \"type\": \"TRANSFORMER\",\n      \"factory\": \"PersistRetrievableTransformer\"\n    },\n    \"thumbnail\": {\n      \"type\": \"EXPORTER\",\n      \"exporterName\": \"thumbnail\"\n    },\n    \"clip\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"clip\"\n    },\n    \"file\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"file\"\n    },\n    \"averagecolor\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"averagecolor\"\n    }\n  },\n  \"operations\": {\n    \"enumerator\": {\n      \"operator\": \"enumerator\"\n    },\n    \"decoder\": {\n      \"operator\": \"decoder\",\n      \"inputs\": [\n        \"enumerator\"\n      ]\n    },\n    \"persist\": {\n      \"operator\": \"persister\",\n      \"inputs\": [\n        \"decoder\"\n      ]\n    },\n    \"thumbnail\": {\n      \"operator\": \"thumbnail\",\n      \"inputs\": [\n        \"persist\"\n      ]\n    },\n    \"clip\": {\n      \"operator\": \"clip\",\n      \"inputs\": [\n        \"thumbnail\"\n      ]\n    },\n    \"averagecolor\": {\n      \"operator\": \"averagecolor\",\n      \"inputs\": [\n        \"clip\"\n      ]\n    },\n    \"file\": {\n      \"operator\": \"file\",\n      \"inputs\": [\n        \"averagecolor\"\n      ]\n    }\n  },\n  \"output\": [\n    \"file\"\n  ]\n}\n\n\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ccode\u003evideo-ingest.json\u003c/code\u003e\u003c/summary\u003e\n\n```json\n{\n  \"schema\": \"sandbox\",\n  \"context\": {\n    \"contentFactory\": \"InMemoryContentFactory\",\n    \"resolvers\": [\n      \"disk\"\n    ],\n    \"local\": {\n      \"enumerator\": {\n        \"path\": \"/sandbox/media\",\n        \"depth\": \"1\"\n      },\n      \"thumbnail\": {\n        \"maxSideResolution\": \"400\",\n        \"mimeType\": \"JPG\"\n      },\n      \"filter\": {\n        \"type\": \"SOURCE:VIDEO\"\n      },\n      \"decoder\": {\n        \"timeWindowMs\": \"5000\"\n      }\n    }\n  },\n  \"operators\": {\n    \"enumerator\": {\n      \"type\": \"ENUMERATOR\",\n      \"factory\": \"FileSystemEnumerator\",\n      \"mediaTypes\": [\n        \"VIDEO\"\n      ]\n    },\n    \"decoder\": {\n      \"type\": \"DECODER\",\n      \"factory\": \"VideoDecoder\"\n    },\n    \"persister\": {\n      \"type\": \"TRANSFORMER\",\n      \"factory\": \"PersistRetrievableTransformer\"\n    },\n    \"aggregator\": {\n      \"type\": \"TRANSFORMER\",\n      \"factory\": \"LastContentAggregator\"\n    },\n    \"time\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"time\"\n    },\n    \"thumbnail\": {\n      \"type\": \"EXPORTER\",\n      \"exporterName\": \"thumbnail\"\n    },\n    \"clip\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"clip\"\n    },\n    \"file\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"file\"\n    },\n    \"averagecolor\": {\n      \"type\": \"EXTRACTOR\",\n      \"fieldName\": \"averagecolor\"\n    },\n    \"filter\": {\n      \"type\": \"TRANSFORMER\",\n      \"factory\": \"TypeFilterTransformer\"\n    }\n  },\n  \"operations\": {\n    \"enumerator\": {\n      \"operator\": \"enumerator\"\n    },\n    \"decoder\": {\n      \"operator\": \"decoder\",\n      \"inputs\": [\n        \"enumerator\"\n      ]\n    },\n    \"aggregator\": {\n      \"operator\": \"aggregator\",\n      \"inputs\": [\n        \"decoder\"\n      ]\n    },\n    \"persist\": {\n      \"operator\": \"persister\",\n      \"inputs\": [\n        \"aggregator\"\n      ]\n    },\n    \"thumbnail\": {\n      \"operator\": \"thumbnail\",\n      \"inputs\": [\n        \"persist\"\n      ]\n    },\n    \"clip\": {\n      \"operator\": \"clip\",\n      \"inputs\": [\n        \"thumbnail\"\n      ]\n    },\n    \"averagecolor\": {\n      \"operator\": \"averagecolor\",\n      \"inputs\": [\n        \"clip\"\n      ]\n    },\n    \"time\": {\n      \"operator\": \"time\",\n      \"inputs\": [\n        \"averagecolor\"\n      ]\n    },\n    \"file\": {\n      \"operator\": \"file\",\n      \"inputs\": [\n        \"time\"\n      ]\n    }\n  },\n  \"output\": [\n    \"file\"\n  ]\n}\n\n```\n\n\u003c/details\u003e\n\n---\n\n#### Start the Server\n\nNavigate to the distribution:\n\n```bash\ncd ../vitrivr-engine-server-\u003cversion\u003e \n./bin/vitrivr-engine-server config-schema.json\n```\n\nInside the interactive CLI:\n\n```bash\nv\u003e sandbox init         # Initializes the schema (sandbox = your schema name)\n```\n\n\n```bash\nv\u003e sandbox about         # Check the initialization state the schema (sandbox = your schema name)\n```\n\nIf you want to ingest media files, you can use the following commands:\n\n```bash\nv\u003e sandbox extract -n image # Ingest all images through the specified pipeline\nv\u003e sandbox extract -n video # Ingest all videos through the specified pipeline\n```\n\nOr if you are using Docker, you can run the following command:\n\n```bash\ndocker attach vitrivr-engine-vitrivr-engine-1\nsandbox init\nsandbox extract -n image\nsandbox extract -n video\n```\n\nYou can now access the REST API at [http://localhost:7070](http://localhost:7070) and query the system.\n\n---\n\n#### Example Query\n\nUse the OpenAPI Swagger UI or send a query. A simple example query to retrieve the multimedia content based on a text\ninput could look like this. For this, the CLIP feature is used. Inside `input-text`, you can provide a description of\nthe content you want to retrieve, such as \"an orange starfish on the seafloor\". The query will return the relevant media\nfiles based on the CLIP feature extraction.\n\n```bash\n\n```json\n{\n  \"inputs\": {\n    \"input-text\": {\"type\": \"TEXT\", \"data\": \"an orange starfish on the seafloor\"}\n  },\n  \"operations\": {\n    \"clip\" : {\"type\": \"RETRIEVER\", \"field\": \"clip\", \"input\": \"input-text\"},\n    \"relations\" : {\"type\": \"TRANSFORMER\", \"transformerName\": \"RelationExpander\", \"input\": \"clip\"},\n    \"lookup\" : {\"type\": \"TRANSFORMER\", \"transformerName\": \"FieldLookup\", \"input\": \"relations\"},\n    \"aggregator\" : {\"type\": \"TRANSFORMER\", \"transformerName\": \"ScoreAggregator\",  \"input\": \"lookup\"},\n    \"filelookup\" : {\"type\": \"TRANSFORMER\", \"transformerName\": \"FieldLookup\", \"input\": \"aggregator\"}\n  },\n  \"context\": {\n    \"global\": {\n      \"limit\": \"1000\"\n    },\n    \"local\" : {\n      \"lookup\":{\"field\": \"time\", \"keys\": \"start, end\"},\n      \"relations\" :{\"outgoing\": \"partOf\"},\n      \"filelookup\": {\"field\": \"file\", \"keys\": \"path\"}\n    }\n  },\n  \"output\": \"filelookup\"\n}\n```\n\n---\n\n## Project Structure\n\nThe project is set up as a multi-module Kotlin project:\n\n| Module                                                                                                                | Description                                                                                                                                                                                                                              | Maven Dependency |\n|-----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|\n| [`vitrivr-engine-core`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-core)                               | The core library of the project, which provides basic interfaces \u0026 classes.                                                                                                                                                              | Yes              |\n| [`vtirivr-engine-query`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-query)                             | Query / retrieval related extension to the core library with various retrievers and data manipulation operators.                                                                                                                         | Yes              |\n| [`vitrivr-engine-index`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-index)                             | Indexing / ingestion related extension to the core library with various decoders and segmenters.                                                                                                                                         | Yes              |\n| [`vitrivr-engine-module-cottontaildb`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-plugin-cottontaildb) | The database driver for the [CottontailDB](https://github.com/vitrivr/cottontaildb) database, used for NNNS and other queries.                                                                                                           | Yes              |\n| [`vitrivr-engine-module-features`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-plugin-features)         | Extension that contains specific indexing and retrieval implementations such as fulltext, colour, etc.                                                                                                                                   | Yes              |\n| [`vitrivr-engine-module-m3d`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-module-m3d)                   | Extension related to 3d model indexing and retrieval. Contains various feature modules and capability to process meshes.                                                                                                                 | Yes              |\n| [`vitrivr-engine-module-fes`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-module-fes)                   | Extension that can be used to harnes feature extraction provided by an external ML model server. **Requires local generation of bindings:** `./gradlew :vitrivr-engine-module-fes:generateFESClient`                                     | Yes              |\n| [`vitrivr-engine-server`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-server)                           | A [Javalin](https://javalin.io) powered server providing an [OpenApi](https://openapis.org) [documented REST API](vitrivr-engine-server/doc/oas.json) for both, ingestion and querying and a CLI, essentially the runtime of the project | No               |\n\n## Contributing\n\nWe welcome contributors. Please fork the repo and open a pull-request with your work.\nA good starting point are the 'good first issue' issues.\n\n## Contributors\n\n* @ppanopticon\n* @lucaro\n* @sauterl\n* @net-csscience-raphel\n* @rahelarnold98\n* @faberf\n\n## Citation\n\nSee [Citation](https://github.com/vitrivr/vitrivr-engine/wiki/Home#Citation)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitrivr%2Fvitrivr-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitrivr%2Fvitrivr-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitrivr%2Fvitrivr-engine/lists"}