{"id":14156589,"url":"https://github.com/EulerSearch/embedding_studio","last_synced_at":"2025-08-06T03:30:55.149Z","repository":{"id":213950734,"uuid":"712179847","full_name":"EulerSearch/embedding_studio","owner":"EulerSearch","description":" Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.","archived":false,"fork":false,"pushed_at":"2024-03-15T11:35:44.000Z","size":10654,"stargazers_count":377,"open_issues_count":5,"forks_count":5,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-08-18T08:07:40.296Z","etag":null,"topics":["embeddings","embeddings-similarity","fine-tuning","llm-inference","query-parser","search-algorithm","search-engine","search-query-parser","semantic-similarity","unstructured-data","unstructured-search","vector-database"],"latest_commit_sha":null,"homepage":"https://embeddingstud.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EulerSearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-31T00:33:13.000Z","updated_at":"2024-08-07T10:28:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"513a009b-7bb0-42a4-985b-03f4dba248b4","html_url":"https://github.com/EulerSearch/embedding_studio","commit_stats":null,"previous_names":["eulersearch/embedding_studio"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EulerSearch%2Fembedding_studio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EulerSearch%2Fembedding_studio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EulerSearch%2Fembedding_studio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EulerSearch%2Fembedding_studio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EulerSearch","download_url":"https://codeload.github.com/EulerSearch/embedding_studio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228835539,"owners_count":17979162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embeddings","embeddings-similarity","fine-tuning","llm-inference","query-parser","search-algorithm","search-engine","search-query-parser","semantic-similarity","unstructured-data","unstructured-search","vector-database"],"created_at":"2024-08-17T08:06:44.170Z","updated_at":"2025-08-06T03:30:55.125Z","avatar_url":"https://github.com/EulerSearch.png","language":"Python","readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/embedding_studio_logo.svg\" alt=\"Embedding Studio\" /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://hugsearch.demo.embeddingstud.io/\" style=\"font-size: 20px;\"\u003e\u003cstrong\u003e👉 Try the Live Demo\u003c/strong\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"#\"\u003e\u003cimg src=\"https://img.shields.io/badge/version-1.0.0-green.svg\" alt=\"version\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://www.python.org/downloads/release/python-31014/\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3.10-blue.svg\" alt=\"Python 3.10\"\u003e\u003c/a\u003e\n    \u003ca href=\"#\"\u003e\u003cimg src=\"https://img.shields.io/badge/CUDA-11.7.1-green.svg\" alt=\"CUDA 11.7.1\"\u003e\u003c/a\u003e\n    \u003ca href=\"#\"\u003e\u003cimg src=\"https://img.shields.io/badge/docker--compose-2.17.0-blue.svg\" alt=\"Docker Compose Version\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://embeddingstud.io/\"\u003eWebsite\u003c/a\u003e •\n    \u003ca href=\"https://embeddingstud.io/tutorial/getting_started/\"\u003eDocumentation\u003c/a\u003e •\n    \u003ca href=\"https://embeddingstud.io/challenges/\"\u003eChallenges \u0026 Solutions\u003c/a\u003e •\n    \u003ca href=\"https://embeddingstud.io/challenges/\"\u003eUse Cases\u003c/a\u003e\n\u003c/p\u003e\n\n**Embedding Studio** is an innovative open-source framework designed to transform embedding models and vector databases into comprehensive, self-improving search engines. With built-in clickstream collection, continuous model refinement, and intelligent vector optimization, it creates a feedback loop that enhances search quality over time based on real user interactions.\n\n\u003ctable style=\"margin-left: auto; margin-right: auto\"\u003e\n    \u003ctr align=\"center\"\u003e\u003ctd\u003eCommunity Support\u003c/td\u003e\u003c/tr\u003e\n    \u003ctr align=\"center\"\u003e\n        \u003ctd\u003e\n            Embedding Studio grows with our team's enthusiasm. Your \u003cb\u003estar on the repository\u003c/b\u003e helps us keep developing. \u003cbr\u003e\n            Join us in reaching our goal:\n            \u003cp align=\"center\"\u003e\n                \u003ca href=\"#\"\u003e\u003cimg src=\"https://embeddingstud.io/badge?title=Stars%20Goal\u0026scale=500\u0026width=200\u0026color=5d5d5d\" alt=\"Progress\"/\u003e\u003c/a\u003e\n            \u003c/p\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Features\n\n### Core Capabilities\n\n1. 🔄 **Full-Cycle Search Engine** - Transform your vector database into a complete search solution\n2. 🖱️ **User Feedback Collection** - Automatically gather clickstream and session data\n3. 🚀 **Continuous Improvement** - Enhance search quality on-the-fly without long waiting periods\n4. 📊 **Performance Monitoring** - Track search quality metrics through comprehensive dashboards\n5. 🎯 **Iterative Fine-Tuning** - Improve your embedding model through user interaction data\n6. 🔍 **Blue-Green Deployment** - Zero-downtime deployment of improved embedding models\n7. 💾 **Multi-Source Integration** - Connect to various data sources (S3, GCP, PostgreSQL, etc.)\n8. 🧠 **Vector Optimization** - Apply post-training adjustments for incremental improvements\n\n### Specialized Features\n\n- 📈 **Personalization Support** - Create user-specific vector adjustments based on individual behavior\n- 💬 **Suggestion System** - Generate intelligent query autocompletions based on user patterns\n- 🔎 **Category Prediction** - Automatically identify relevant categories for search queries\n- 🔤 **Multi-Modal Support** - Work with text, images, and structured data in one framework\n- 🧩 **Plugin Architecture** - Extend functionality through a comprehensive plugin system\n\n### In Development (*)\n\n- 📑 **Zero-Shot Query Parser** - Mix structured and unstructured search queries\n- 📚 **Catalog Pre-Training** - Fine-tune embedding models on your specific content before deployment\n- 📊 **Advanced Analytics** - More detailed insights into search performance and user behavior\n\n(*) - Features in active development\n\n## When is Embedding Studio the Best Fit?\n\nMore about it [here](docs/when-to-use-the-embeddingstudio.md).\n\n- 📚💼 **Rich Content Collections** - Businesses with extensive catalogs and unstructured data\n- 🛍️🤝 **Customer-Centric Platforms** - Applications prioritizing personalized user experiences\n- 🔄📊 **Dynamic Content** - Platforms with evolving content and changing user preferences\n- 🔍🧠 **Complex Queries** - Systems handling nuanced and multifaceted search needs\n- 🔄📊 **Mixed Data Types** - Applications integrating different data formats in search\n- 🔄🚀 **Continuous Improvement** - Platforms seeking ongoing optimization through user interactions\n- 💵💡 **Cost-Conscious Organizations** - Teams looking for powerful yet affordable solutions\n\n## Challenges Solved\n\n**Disclaimer:** Embedding Studio is not another Vector Database - it's a framework that transforms your Vector Database into a complete Search Engine with all necessary components.\n\n- ✅ **Cold Start Problems** - Jump-start search quality with minimal data\n- ✅ **Static Search Quality** - Create systems that improve automatically over time\n- ✅ **Long Improvement Cycles** - Reduce frustration with fast feedback loops\n- ✅ **Resource-Heavy Reindexing** - Optimize the updating process for better performance\n- ✅ **Hybrid Search Complexity** - Seamlessly combine structured and unstructured search\n- ✅ **Query Understanding** - Parse natural language queries more effectively\n- ✅ **New Content Discovery** - Ensure fresh items get proper visibility\n\nMore about challenges and solutions [here](https://embeddingstud.io/challenges/)\n\n## System Architecture\n\nEmbedding Studio uses a modular, service-based architecture:\n\n### Core Components\n\n- **API Service** - Central coordination point for applications\n- **Vector Database** - PostgreSQL with pgvector for embedding storage\n- **Clickstream System** - Captures and processes user interactions\n- **Worker Services**:\n  - **Fine-Tuning Worker** - Handles model training and improvement\n  - **Inference Worker** - Manages Triton Inference Server for embeddings\n  - **Improvement Worker** - Processes incremental vector adjustments\n  - **Upsertion Worker** - Manages content updates and indexing\n\n### Data Flow\n\n1. **Content Ingestion** - Load data from various sources\n2. **User Interaction** - Collect clickstream data through API endpoints\n3. **Fine-Tuning** - Use interaction data to improve embedding models\n4. **Model Deployment** - Update inference service with improved models\n5. **Search and Retrieval** - Deliver better results based on fine-tuned models\n\n## Comparison with Traditional Approaches\n\n![Embedding Studio Chart](assets/embedding_studio_chart.png)\n\nOur framework enables you to continuously fine-tune your model based on user experience, allowing you to form search \nresults for user queries faster and more accurately.\n\n$${\\color{red}RED:}$$ On the graph, typical search solutions without enhancements, \nsuch as Full Text Searching (FTS), Nearest Neighbor Search (NNS), and others, are marked in red. Without the use of \nadditional tools, the search quality remains unchanged over time.\n\n$${\\color{orange}ORANGE:}$$ Solutions are depicted that accumulate some feedback (clicks, reviews, votes, discussions, etc.) and then\ninitiate a full model retraining. The primary issue with these solutions is that full model retraining is a\ntime-consuming and expensive procedure, thus lacking reactive adjustments (for example, when a product suddenly\nexperiences increased demand, and the search system has not yet adapted to it).\n\n$${\\color{#6666ff}INDIGO:}$$ We propose a solution that allows collecting user feedback and rapidly retraining the model on the difference between\nthe old and new versions. This enables a smoother and more relevant search quality curve for your system.\n\n## Getting Started\n\n### Prerequisites\n\n- Docker Compose v2.17.0+\n- For fine-tuning: NVIDIA GPU with CUDA support\n- Minimum 8GB RAM allocated to Docker\n\n## Documentation\n\nFor comprehensive documentation:\n\n- [Core Concepts](docs/tutorial/getting_started/core_concepts.md)\n- [Architecture Overview](docs/tutorial/getting_started/architecture_overview.md)\n- [Docker Quick Start](docs/tutorial/getting_started/docker_quickstart.md)\n- [Configuration Guide](docs/tutorial/getting_started/configurations.md)\n- [Plugin Development](docs/tutorial/plugins/understanding_plugin_system.md)\n- [Vector Database Integration](docs/tutorial/vectordb/integration.md)\n- [Code Documentation](docs/embedding_studio)\n\n## Plugin System\n\nEmbedding Studio features a powerful plugin architecture allowing extension of:\n\n- Data loaders for different sources\n- Text and image processors\n- Fine-tuning methods\n- Vector optimization strategies\n- Query processing logic\n\nCreate custom plugins by extending base classes and implementing your specific logic.\n\n## Contributing\n\nWe welcome contributions to Embedding Studio! To contribute:\n\n1. Fork the repository\n2. Create a feature branch\n3. Submit a pull request\n\nPlease check our contributing guidelines for detailed information.\n\n## 📬 Contact Us\n\u003cstrong\u003eEulerSearch Inc.\u003c/strong\u003e\u003cbr/\u003e 3416, 1007 N Orange St. 4th Floor,\u003cbr/\u003e Wilmington, DE, New Castle, US, 19801\u003cbr/\u003e Contact Email: \u003ca href=\"mailto:aleksandr.iudaev@eulersearch.com\"\u003ealeksandr.iudaev@eulersearch.com\u003c/a\u003e\u003cbr/\u003e Phone: +34 (691) 454 148\u003cbr/\u003e LinkedIn: \u003ca href=\"https://www.linkedin.com/in/alexanderyudaev/\"\u003ehttps://www.linkedin.com/in/alexanderyudaev/\u003c/a\u003e\n\n## License\n\nEmbedding Studio is licensed under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text.","funding_links":[],"categories":["vector-database"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEulerSearch%2Fembedding_studio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEulerSearch%2Fembedding_studio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEulerSearch%2Fembedding_studio/lists"}