{"id":13423664,"url":"https://github.com/microsoft/SPTAG","last_synced_at":"2025-03-15T17:32:04.891Z","repository":{"id":37324761,"uuid":"148464499","full_name":"microsoft/SPTAG","owner":"microsoft","description":"A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.","archived":false,"fork":false,"pushed_at":"2025-02-17T01:26:27.000Z","size":25825,"stargazers_count":4878,"open_issues_count":133,"forks_count":585,"subscribers_count":136,"default_branch":"main","last_synced_at":"2025-03-13T01:21:53.186Z","etag":null,"topics":["approximate-nearest-neighbor-search","distributed-serving","fresh-update","neighborhood-graph","space-partition-tree","vector-search"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-12T10:42:51.000Z","updated_at":"2025-03-11T03:27:01.000Z","dependencies_parsed_at":"2024-01-06T02:45:28.539Z","dependency_job_id":"eeb8a46c-6fea-4fba-963b-eccb1296dbc4","html_url":"https://github.com/microsoft/SPTAG","commit_stats":{"total_commits":214,"total_committers":41,"mean_commits":5.219512195121951,"dds":0.6542056074766356,"last_synced_commit":"4ecf2495eb6187b555c0a6ee7d3b5394d5362183"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSPTAG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSPTAG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSPTAG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSPTAG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/SPTAG/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243767101,"owners_count":20344872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-nearest-neighbor-search","distributed-serving","fresh-update","neighborhood-graph","space-partition-tree","vector-search"],"created_at":"2024-07-31T00:00:40.145Z","updated_at":"2025-03-15T17:32:04.827Z","avatar_url":"https://github.com/microsoft.png","language":"C++","funding_links":[],"categories":["Model Serving \u0026 Inference","C++","Sdks \u0026 Libraries","others","*Ops for AI","C++ (70)","Azure AI Search","Awesome Vector Search Engine","5. Retrieval-Augmented Generation (RAG) \u0026 Knowledge"],"sub_categories":["Vector Databases \u0026 Retrieval Infrastructure","Model Serving \u0026 Inference","Library"],"readme":"# SPTAG: A library for fast approximate nearest neighbor search\n\n[![MIT licensed](https://img.shields.io/badge/license-MIT-yellow.svg)](https://github.com/Microsoft/SPTAG/blob/master/LICENSE)\n[![Build status](https://sysdnn.visualstudio.com/SPTAG/_apis/build/status/SPTAG-GITHUB)](https://sysdnn.visualstudio.com/SPTAG/_build/latest?definitionId=2)\n\n## **SPTAG**\n SPTAG (Space Partition Tree And Graph) is a library for large scale vector approximate nearest neighbor search scenario released by [Microsoft Research (MSR)](https://www.msra.cn/) and [Microsoft Bing](http://bing.com). \n\n \u003cp align=\"center\"\u003e\n \u003cimg src=\"docs/img/sptag.png\" alt=\"architecture\" width=\"500\"/\u003e\n \u003c/p\u003e\n\n## What's NEW\n* Result Iterator with Relaxed Monotonicity Signal Support\n* New Research Paper [SPFresh: Incremental In-Place Update for Billion-Scale Vector Search](https://dl.acm.org/doi/10.1145/3600006.3613166) - _published in SOSP 2023_\n* New Research Paper [VBASE: Unifying Online Vector Similarity Search and Relational Queries via Relaxed Monotonicity](https://www.usenix.org/system/files/osdi23-zhang-qianxi_1.pdf) - _published in OSDI 2023_\n\n## **Introduction**\n \nThis library assumes that the samples are represented as vectors and that the vectors can be compared by L2 distances or cosine distances. \nVectors returned for a query vector are the vectors that have smallest L2 distance or cosine distances with the query vector. \n\nSPTAG provides two methods: kd-tree and relative neighborhood graph (SPTAG-KDT) \nand balanced k-means tree and relative neighborhood graph (SPTAG-BKT).\nSPTAG-KDT is advantageous in index building cost, and SPTAG-BKT is advantageous in search accuracy in very high-dimensional data.\n\n\n\n## **How it works**\n\nSPTAG is inspired by the NGS approach [[WangL12](#References)]. It contains two basic modules: index builder and searcher. \nThe RNG is built on the k-nearest neighborhood graph [[WangWZTG12](#References), [WangWJLZZH14](#References)] \nfor boosting the connectivity. Balanced k-means trees are used to replace kd-trees to avoid the inaccurate distance bound estimation in kd-trees for very high-dimensional vectors.\nThe search begins with the search in the space partition trees for \nfinding several seeds to start the search in the RNG. \nThe searches in the trees and the graph are iteratively conducted. \n\n ## **Highlights**\n  * Fresh update: Support online vector deletion and insertion\n  * Distributed serving: Search over multiple machines\n\n ## **Build**\n\n### **Requirements**\n\n* swig \u003e= 4.0.2\n* cmake \u003e= 3.12.0\n* boost \u003e= 1.67.0\n\n### **Fast clone**\n\n```\nset GIT_LFS_SKIP_SMUDGE=1\ngit clone --recurse-submodules https://github.com/microsoft/SPTAG\n\nOR\n\ngit config --global filter.lfs.smudge \"git-lfs smudge --skip -- %f\"\ngit config --global filter.lfs.process \"git-lfs filter-process --skip\"\n```\n\n### **Install**\n\n\u003e For Linux:\n```bash\nmkdir build\ncd build \u0026\u0026 cmake .. \u0026\u0026 make\n```\nIt will generate a Release folder in the code directory which contains all the build targets.\n\n\u003e For Windows:\n```bash\nmkdir build\ncd build \u0026\u0026 cmake -A x64 ..\n```\nIt will generate a SPTAGLib.sln in the build directory. \nCompiling the ALL_BUILD project in the Visual Studio (at least 2019) will generate a Release directory which contains all the build targets.\n\nFor detailed instructions on installing Windows binaries, please see [here](docs/WindowsInstallation.md)\n\n\u003e Using Docker:\n```bash\ndocker build -t sptag .\n```\nWill build a docker container with binaries in `/app/Release/`.\n\n### **Verify** \n\nRun the SPTAGTest (or Test.exe) in the Release folder to verify all the tests have passed.\n\n### **Usage**\n\nThe detailed usage can be found in [Get started](docs/GettingStart.md). There is also an end-to-end tutorial for building vector search online service using Python Wrapper in [Python Tutorial](docs/Tutorial.ipynb).\nThe detailed parameters tunning can be found in [Parameters](docs/Parameters.md).\n\n## **References**\nPlease cite SPTAG in your publications if it helps your research:\n```\n@inproceedings{xu2023spfresh,\n  title={SPFresh: Incremental In-Place Update for Billion-Scale Vector Search},\n  author={Xu, Yuming and Liang, Hengyu and Li, Jin and Xu, Shuotao and Chen, Qi and Zhang, Qianxi and Li, Cheng and Yang, Ziyue and Yang, Fan and Yang, Yuqing and others},\n  booktitle={Proceedings of the 29th Symposium on Operating Systems Principles},\n  pages={545--561},\n  year={2023}\n}\n\n@inproceedings{zhang2023vbase,\n  title={$\\{$VBASE$\\}$: Unifying Online Vector Similarity Search and Relational Queries via Relaxed Monotonicity},\n  author={Zhang, Qianxi and Xu, Shuotao and Chen, Qi and Sui, Guoxin and Xie, Jiadong and Cai, Zhizhen and Chen, Yaoqi and He, Yinxuan and Yang, Yuqing and Yang, Fan and others},\n  booktitle={17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)},\n  year={2023}\n}\n\n@inproceedings{ChenW21,\n  author = {Qi Chen and \n            Bing Zhao and \n            Haidong Wang and \n            Mingqin Li and \n            Chuanjie Liu and \n            Zengzhong Li and \n            Mao Yang and \n            Jingdong Wang},\n  title = {SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search},\n  booktitle = {35th Conference on Neural Information Processing Systems (NeurIPS 2021)},\n  year = {2021}\n}\n\n@manual{ChenW18,\n  author    = {Qi Chen and\n               Haidong Wang and\n               Mingqin Li and \n               Gang Ren and\n               Scarlett Li and\n               Jeffery Zhu and\n               Jason Li and\n               Chuanjie Liu and\n               Lintao Zhang and\n               Jingdong Wang},\n  title     = {SPTAG: A library for fast approximate nearest neighbor search},\n  url       = {https://github.com/Microsoft/SPTAG},\n  year      = {2018}\n}\n\n@inproceedings{WangL12,\n  author    = {Jingdong Wang and\n               Shipeng Li},\n  title     = {Query-driven iterated neighborhood graph search for large scale indexing},\n  booktitle = {ACM Multimedia 2012},\n  pages     = {179--188},\n  year      = {2012}\n}\n\n@inproceedings{WangWZTGL12,\n  author    = {Jing Wang and\n               Jingdong Wang and\n               Gang Zeng and\n               Zhuowen Tu and\n               Rui Gan and\n               Shipeng Li},\n  title     = {Scalable k-NN graph construction for visual descriptors},\n  booktitle = {CVPR 2012},\n  pages     = {1106--1113},\n  year      = {2012}\n}\n\n@article{WangWJLZZH14,\n  author    = {Jingdong Wang and\n               Naiyan Wang and\n               You Jia and\n               Jian Li and\n               Gang Zeng and\n               Hongbin Zha and\n               Xian{-}Sheng Hua},\n  title     = {Trinary-Projection Trees for Approximate Nearest Neighbor Search},\n  journal   = {{IEEE} Trans. Pattern Anal. Mach. Intell.},\n  volume    = {36},\n  number    = {2},\n  pages     = {388--403},\n  year      = {2014\n}\n```\n\n## **Contribute**\n\nThis project welcomes contributions and suggestions from all the users.\n\nWe use [GitHub issues](https://github.com/Microsoft/SPTAG/issues) for tracking suggestions and bugs.\n\n## **License**\nThe entire codebase is under [MIT license](https://github.com/Microsoft/SPTAG/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2FSPTAG","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2FSPTAG","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2FSPTAG/lists"}