{"id":18798580,"url":"https://github.com/liteobject/demo-semantic-search","last_synced_at":"2026-04-27T18:32:12.119Z","repository":{"id":249853336,"uuid":"832755624","full_name":"LiteObject/demo-semantic-search","owner":"LiteObject","description":"Semantic search demonstrations using embeddings and distance metrics. Includes similarity queries and nearest-neighbor search with local models.","archived":false,"fork":false,"pushed_at":"2025-12-12T05:20:34.000Z","size":25,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-13T13:51:29.770Z","etag":null,"topics":["dense-retrieval","embeddings","llm","machine-learning","python","semantic-search","slm","vector-database","weaviate"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LiteObject.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-23T16:54:33.000Z","updated_at":"2025-12-12T05:22:14.000Z","dependencies_parsed_at":"2024-07-23T19:53:54.663Z","dependency_job_id":"931fbc4e-f510-4c06-b2cf-54bb5df534ab","html_url":"https://github.com/LiteObject/demo-semantic-search","commit_stats":null,"previous_names":["liteobject/demo-semantic-search"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LiteObject/demo-semantic-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiteObject%2Fdemo-semantic-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiteObject%2Fdemo-semantic-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiteObject%2Fdemo-semantic-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiteObject%2Fdemo-semantic-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LiteObject","download_url":"https://codeload.github.com/LiteObject/demo-semantic-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiteObject%2Fdemo-semantic-search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32349445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T17:12:42.749Z","status":"ssl_error","status_checked_at":"2026-04-27T17:12:41.658Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dense-retrieval","embeddings","llm","machine-learning","python","semantic-search","slm","vector-database","weaviate"],"created_at":"2024-11-07T22:12:24.459Z","updated_at":"2026-04-27T18:32:12.112Z","avatar_url":"https://github.com/LiteObject.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Semantic Search Demo\n\nThis repository demonstrates practical implementations of semantic search concepts using embeddings and various distance metrics. It includes examples of how to perform similarity comparisons and nearest-neighbor queries using local embedding models.\n\n## What is Semantic Search?\n\nTraditional search engines rely on keyword matching - they look for exact words or phrases in documents. Semantic search goes deeper by trying to understand the actual meaning and context of both the search query and the documents being searched.\n\nThe key insight is that text can be converted into numerical vectors called embeddings, where semantically similar texts end up close to each other in the vector space. This allows us to find relevant documents even when they don't contain the exact same words as the query.\n\nFor example, if you search for \"how to prepare pasta\", a semantic search engine might also return documents about \"cooking noodles\" or \"preparing Italian food\", because these concepts are semantically related.\n\n## Core Concepts\n\n### Embeddings\n\nAn embedding is a numerical representation of text - a vector of numbers that captures the semantic meaning of that text. Modern embedding models, like those from Sentence Transformers, convert text into these vectors automatically. The important property is that semantically similar texts produce vectors that are close together in the vector space.\n\n### Distance Metrics\n\nOnce you have embeddings, you need a way to measure similarity between them. This project demonstrates several common distance metrics:\n\n**Cosine Similarity** - Measures the angle between two vectors. It ranges from -1 to 1, where 1 means the vectors point in the same direction (most similar). This is widely used because it's invariant to vector magnitude.\n\n**Dot Product** - The sum of element-wise products. For normalized vectors, it's equivalent to cosine similarity, but it's faster to compute on large batches.\n\n**Euclidean Distance** - The straight-line distance between two points in the embedding space. Lower values indicate more similarity. Useful when the magnitude of vectors matters.\n\n**Manhattan Distance** - The sum of absolute differences between vector components. Less commonly used than Euclidean, but sometimes useful for high-dimensional data.\n\n### Vector Databases and Indexing\n\nFor practical applications with large document collections, you need efficient ways to find the nearest neighbors. Exact methods like brute-force search are simple but slow when you have millions of documents. Approximate methods use techniques like locality-sensitive hashing or random projections to find candidates quickly, then refine the results.\n\n## Files in This Repository\n\n### similarity_query.py\n\nDemonstrates direct similarity comparisons between pairs of texts using all supported distance metrics. This is useful for:\n- Understanding how different metrics behave\n- Comparing two specific documents or concepts\n- Evaluating the quality of your embedding model\n\nRun it to see how different metrics rate the similarity between various text pairs.\n\n### nearest_neighbor_query.py\n\nImplements nearest-neighbor search over a collection of documents. Includes both:\n- **Exact NN** - Computes distances to all documents and returns the best matches. Guaranteed to find the true nearest neighbors.\n- **Approximate NN** - Uses random projections to reduce dimensionality and speed up search, then re-ranks results using the full embeddings.\n\nThis demonstrates how to build a semantic search engine that can handle larger document collections efficiently.\n\n## Getting Started\n\n### Prerequisites\n\nYou'll need Python 3.8+ and the following packages:\n- numpy - For numerical computations\n- sentence-transformers - For generating embeddings with local models\n\n### Installation\n\nCreate and activate a virtual environment:\n```bash\npython -m venv .venv\n.venv\\Scripts\\activate  # On Windows\nsource .venv/bin/activate  # On Unix/Mac\n```\n\nInstall dependencies:\n```bash\npip install numpy sentence-transformers\n```\n\n### Running the Demos\n\nExecute the similarity query demo:\n```bash\npython similarity_query.py\n```\n\nExecute the nearest-neighbor query demo:\n```bash\npython nearest_neighbor_query.py\n```\n\nThe scripts will download the default embedding model (`all-MiniLM-L6-v2`) on first run, which is a fast and efficient model suitable for most use cases.\n\n## Understanding the Results\n\nWhen you run the similarity demo, you'll see how different metrics score the same pairs of texts. Notice that:\n- Related texts consistently score higher across all metrics\n- Different metrics emphasize different aspects of similarity\n- Unrelated texts get uniformly low scores\n\nThe nearest-neighbor demo shows how the system finds relevant documents from a collection. The benchmark function helps you understand the performance characteristics of different approaches.\n\n## Choosing the Right Metric\n\n- Use **cosine similarity** as your default for most semantic search tasks - it's robust and computationally efficient\n- Use **dot product** for very large-scale batch operations where speed is critical\n- Use **Euclidean distance** when vector magnitude carries semantic meaning (less common in NLP)\n- Use **Manhattan distance** for sparse vectors or when you need a more robust metric for outliers\n\n## Vector Databases and Production Systems\n\nFor production use cases with large document collections, consider vector databases like:\n- Weaviate - Open-source vector database with semantic search capabilities\n- Milvus - High-performance vector similarity search\n- Pinecone - Managed vector database service\n- FAISS - Facebook's library for efficient similarity search\n\nThese systems optimize storage, indexing, and retrieval at scale and often include HNSW or other advanced approximate nearest-neighbor algorithms.\n\n## Further Reading\n\n- [Sentence Transformers Documentation](https://www.sbert.net/)\n- [Understanding Word Embeddings](https://towardsdatascience.com/word-embeddings-explained-4bef3666e842)\n- [Approximate Nearest Neighbors](https://en.wikipedia.org/wiki/Nearest_neighbor_search#Approximate_nearest_neighbor)\n- [Weaviate - Vector Database](https://weaviate.io/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliteobject%2Fdemo-semantic-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fliteobject%2Fdemo-semantic-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fliteobject%2Fdemo-semantic-search/lists"}