{"id":19984487,"url":"https://github.com/intellabs/vdms","last_synced_at":"2025-05-16T09:06:49.504Z","repository":{"id":40792253,"uuid":"120660606","full_name":"IntelLabs/vdms","owner":"IntelLabs","description":"VDMS: Your Favorite Visual Data Management System","archived":false,"fork":false,"pushed_at":"2025-05-15T20:17:07.000Z","size":58258,"stargazers_count":88,"open_issues_count":46,"forks_count":33,"subscribers_count":16,"default_branch":"develop","last_synced_at":"2025-05-16T09:06:39.570Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IntelLabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"Security.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-02-07T19:22:35.000Z","updated_at":"2025-05-14T16:55:40.000Z","dependencies_parsed_at":"2024-08-08T07:02:45.277Z","dependency_job_id":"f9fa4b93-7105-4dee-a5c7-ad716ef8de34","html_url":"https://github.com/IntelLabs/vdms","commit_stats":{"total_commits":560,"total_committers":15,"mean_commits":"37.333333333333336","dds":0.6642857142857144,"last_synced_commit":"577dd37d9cbe4e2242cd834174a4477f87f0b9d8"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fvdms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fvdms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fvdms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IntelLabs%2Fvdms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IntelLabs","download_url":"https://codeload.github.com/IntelLabs/vdms/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254501558,"owners_count":22081528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T04:19:09.795Z","updated_at":"2025-05-16T09:06:44.492Z","avatar_url":"https://github.com/IntelLabs.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VDMS: Your Favorite Visual Data Management System\n\n[![GitHub License](https://img.shields.io/github/license/IntelLabs/vdms)](https://github.com/IntelLabs/vdms/blob/master/LICENSE)\n[![Dependency Status](https://img.shields.io/librariesio/github/IntelLabs/vdms?style=flat-square)](https://libraries.io/github/IntelLabs/vdms)\n[![Coverity Scan](https://img.shields.io/coverity/scan/30010)](https://scan.coverity.com/projects/intellabs-vdms)\n[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/IntelLabs/vdms/badge)](https://securityscorecards.dev/viewer/?uri=github.com/IntelLabs/vdms)\n\n[![Release Notes](https://img.shields.io/github/release/IntelLabs/vdms?style=flat-square)](https://github.com/IntelLabs/vdms/releases)\n[![Open Issues](https://img.shields.io/github/issues-raw/IntelLabs/vdms?style=flat-square)](https://github.com/IntelLabs/vdms/issues)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/vdms?style=flat-square)](https://pypistats.org/packages/vdms)\n[![Docker Pulls](https://img.shields.io/docker/pulls/intellabs/vdms)](https://hub.docker.com/r/intellabs/vdms)\n\nVDMS is a storage solution for efficient access of big-”visual”-data that aims\nto achieve cloud scale by searching for relevant visual data via visual\nmetadata stored as a graph and enabling machine friendly enhancements to\nvisual data for faster access.  We use an in-persistent-memory graph database\ndeveloped in our team called Persistent Memory Graph Database (PMGD) as the\nmetadata tier and we are exploring the use of an array data manager, TileDB\nand other formats for images, visual descriptors, and videos as part of our\nVisual Compute Library (VCL). VDMS is run as a server listening for client\nrequests and we provide client side bindings to enable communication between (\npython, C++) applications and the server. Hence, it also has a Request Server\ncomponent defined to implement the VDMS API, handle concurrent client\nrequests, and coordinate the request execution across its metadata and data\ncomponents to return unified responses. This project aims to research the use\nof a scalable multi-node graph based metadata store as part of a hierarchical\nstorage framework specifically aimed at processing visual data, and also it\nincludes an investigation into the right hardware and software optimizations\nto store and efficiently access large scale (pre-processed) visual data.\n\n## Motivation\n\nData access is swiftly becoming a bottleneck in visual data processing,\nproviding an opportunity to influence the way visual data is treated in the\nstorage system. To foster this discussion, we identify two key areas where\nstorage research can strongly influence visual processing run-times:\nefficient metadata storage and new storage formats for visual data. We\npropose a storage architecture designed for efficient visual data access\nthat exploits next generation hardware and give preliminary results showing\nhow it enables efficient vision analytics.\n\n\n## Get Started\n\nTo get started, take a look at the [INSTALL.md](INSTALL.md) file, where\nyou will find instructions on how to install and run the server.\n\nAlso, visit our [wiki](https://github.com/IntelLabs/vdms/wiki)\nto learn more about the VDMS API, and take a look at some of\nthe examples/tutorials.\n\n## Academic Papers\n\nConference | Links, Cite | Description\n------------ | ------------- | -------------\nIndustrial and Applications @ VLDB 2021 | [Paper](http://vldb.org/pvldb/vol14/p3240-remis.pdf) | Demonstrate VDMS capabilities in image search application\n2nd USENIX Workshop @ HotEdge 2019 | [Paper](https://www.usenix.org/system/files/hotedge19-paper-altarawneh.pdf), [Presentation](https://www.usenix.org/sites/default/files/conference/protected-files/hotedge19_slides_altarawneh.pdf) | VDMS in Edge-to-cloud architecture for video streaming application\nLearning Systems @ NIPS 2018 | [Paper](https://export.arxiv.org/abs/1810.11832), [Cite](https://dblp.uni-trier.de/rec/bibtex/journals/corr/abs-1810-11832) | Systems for Machine Learning [Workshop](http://learningsys.org/nips18/cfp.html) @ NIPS\nHotStorage @ ATC 2017 | [Paper](https://www.usenix.org/conference/hotstorage17/program/presentation/gupta-cledat), [Presentation](https://www.usenix.org/conference/hotstorage17/program/presentation/gupta-cledat), [Cite](https://www.usenix.org/biblio/export/bibtex/203374)| Positioning Paper at USENIX ATC 2017 Workshop\n\n\n## Contributing\n\nThank you for your interest in contributing to VDMS.\nAs an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes.\n\nTo learn how to contribute to VDMS, please follow the [Developer Guide](https://github.com/IntelLabs/vdms/wiki/Developer-Guide).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintellabs%2Fvdms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintellabs%2Fvdms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintellabs%2Fvdms/lists"}