{"id":26333037,"url":"https://github.com/astrabert/sentrev-demo","last_synced_at":"2025-06-25T23:38:30.495Z","repository":{"id":272282841,"uuid":"916070192","full_name":"AstraBert/SenTrEv-demo","owner":"AstraBert","description":"Demo for SenTrEv python package","archived":false,"fork":false,"pushed_at":"2025-01-20T22:25:34.000Z","size":1471,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-07T17:58:28.692Z","etag":null,"topics":["dense-retrieval","embedding-models","embeddings","evaluation","fastembed","metrics","qdrant","sentence-transformers","sparse-retrieval","vector-database","vector-search"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/sentrev/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AstraBert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-13T11:58:00.000Z","updated_at":"2025-01-20T22:25:38.000Z","dependencies_parsed_at":"2025-01-13T13:26:09.880Z","dependency_job_id":"f5481b6e-f960-4ac0-a399-b4dd551e3e6a","html_url":"https://github.com/AstraBert/SenTrEv-demo","commit_stats":null,"previous_names":["astrabert/sentrev-demo"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraBert%2FSenTrEv-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraBert%2FSenTrEv-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraBert%2FSenTrEv-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AstraBert%2FSenTrEv-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AstraBert","download_url":"https://codeload.github.com/AstraBert/SenTrEv-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806048,"owners_count":20350773,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dense-retrieval","embedding-models","embeddings","evaluation","fastembed","metrics","qdrant","sentence-transformers","sparse-retrieval","vector-database","vector-search"],"created_at":"2025-03-15T23:37:43.389Z","updated_at":"2025-03-15T23:37:44.380Z","avatar_url":"https://github.com/AstraBert.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/AstraBert/SenTrEv-demo/blob/main/SenTrEvDemoNotebook.ipynb\"\u003e\n  \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\n\u003c/a\u003e\n\u003cbr\u003e\n\u003cbr\u003e\n\n# SenTrEv📊\n\n![SenTrEv logo](https://raw.githubusercontent.com/AstraBert/SenTrEv/main/logo.png)\n\n## Concepts📚\n\n![Dictionary](https://positek.net/wp-content/uploads/2013/10/shutterstock_63985306_result.png)\n\n\n### 1. Embeddings\n\n\n\u003e _\"In [natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing \"Natural language processing\"), a **word embedding** is a representation of a word. The embedding is used in [text analysis](https://en.wikipedia.org/wiki/Content_analysis \"Content analysis\"). Typically, the representation is a [real-valued](https://en.wikipedia.org/wiki/Real_number \"Real number\") vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning\"._ \n\n**Source**: [Wikipedia - Word embedding](https://en.wikipedia.org/wiki/Word_embedding)\n\n\n### 2. Vector database\n\n\n\u003e _\"A [Vector Database](https://qdrant.tech/qdrant-vector-database/) is a specialized system designed to efficiently handle high-dimensional vector data. It excels at indexing, querying, and retrieving this data, enabling advanced analysis and similarity searches\"._\n\nSource: [Qdrant - What is a vector database?](https://qdrant.tech/articles/what-is-a-vector-database/)\n\n\n### 3. Sparse embeddings\n\n\n\u003e_\"Sparse vectors [...] focus only on the essentials. In most sparse vectors, a large number of elements are zeros. When a feature or token is present, it’s marked—otherwise, zero. Sparse vectors, are used for **exact matching** and specific token-based identification\"._\n\nSource: [Qdrant - What is a vector database?](https://qdrant.tech/articles/what-is-a-vector-database/#1-dense-vectors)\n\n\n### 4. Dense embeddings\n\n\u003e _\"Dense vectors are, quite literally, dense with information. Every element in the vector contributes to the **semantic meaning**, **relationships** and **nuances** of the data. [...] Together, they convey the overall meaning of the sentence, and are better for identifying contextually similar items\"._\n\nSource: [Qdrant - What is a vector database?](https://qdrant.tech/articles/what-is-a-vector-database/#1-dense-vectors)\n\n\n### 5. Retrieval\n\n\u003e _\"**Information retrieval** (**IR**) in [computing](https://en.wikipedia.org/wiki/Computing \"Computing\") and [information science](https://en.wikipedia.org/wiki/Information_science \"Information science\") is the task of identifying and retrieving [information system](https://en.wikipedia.org/wiki/Information_system \"Information system\") resources that are relevant to an [information need](https://en.wikipedia.org/wiki/Information_needs \"Information needs\")\"_\n\nSource: [Wikipedia - Information Retrieval](https://en.wikipedia.org/wiki/Information_retrieval)\n\n\n## The problem🤔\n\n![Problem](https://tse1.mm.bing.net/th?id=OIP.Jlxk2GuV8_hcloTZglA0CwHaHa\u0026pid=Api)\n\n\n- **Retrieval Augmented Generation** (RAG) is growing in importance\n- As of December 2024, more than **10,000 models** are available within the _sentence-transformers_ python library\n- It's not always easy to understand what's the **best embedding technique** (sparse or dense) and the **best embedding model** for our use case\n- It's often complicated to evaluate embeddings on **different data types**\n\n\n## SenTrEv - A potential solution✅\n\n\u003e [!IMPORTANT]\n\u003e _**SenTrEv** (**Sen**tence **Tr**ansformers **Ev**aluator) is a python package that is aimed at running simple evaluation tests to help you choose the best embedding model for Retrieval Augmented Generation (RAG) with your text-based documents._\n\n\n## Installation⬇️\n\n```bash\npython3 -m pip install sentrev\n```\n\n## Applicability💡\n\n- Highly integrated within the **Qdrant environment**\n- **FastEmbed** sparse encoding models\n- **Sentence Transformers** dense encoding models\n\n\u003e [!IMPORTANT]\n\u003e _Supports most of the **text-based file formats** (.docx, .pptx, .pdf, .md, .html, .xml, .csv, .xlsx)_\n\n\n## The workflow🔁\n\n\u003cdiv\u003e\n\t\u003cimg src=\"https://raw.githubusercontent.com/AstraBert/SenTrEv/main/workflow.png\" width=750 height=450 alt=\"SenTrEv workflow\"\u003e\n\u003c/div\u003e\n\n\n## Metrics👑\n\n![metrics](https://opensource.com/sites/default/files/lead-images/metrics_graph_stats_blue.png)\n\n\n## Ranking\n\n- **Success rate**: defined as the number retrieval operation in which the correct context was retrieved ranking top among all the retrieved contexts, out of the total retrieval operations\n- **Mean Reciprocal Ranking (MRR)**: MRR defines how high in ranking the correct context is placed among the retrieved results.\n\n\n### Relevance\n\n- **Precision**: Number of relevant documents out of the total number of retrieved documents. .\n- **Non-Relevant Ratio**: Number of non-relevant documents out of the total number of retrieved documents\n\n\u003e[!NOTE]\n\u003e_Relevance is based on the \"page\" metadata entry: if the retrieved document comes from the same page of the query, the document is considered relevant._\n  \n\n### Time\n\n- **Time performance**: Average duration for a retrieval operation\n\n### Carbon emissions\n\n- **Carbon emissions**: Carbon emissions are calculated in gCO2eq (grams of CO2 equivalent) through the Python library [`codecarbon`](https://codecarbon.io/).\n\n![footprint](https://4.bp.blogspot.com/_bKdsX12OGu4/StySMnyHWAI/AAAAAAAACKo/W94GzkdWqu8/s400/carbon_footprint.jpg)\n\n\n### CO2 equivalent?\n\n\u003e[!NOTE]\n\u003e _This simply means that there are lots of other greenhouse gases (methane, clorofluorocarbons, nitric oxide…) which all have global warming potential: despite our emissions being mainly made up by CO2, they encompass also these other gases, and it is easier for us to express everything in terms of CO2. For example: 1 kg of emitted methane can be translated into producing 25 kg of CO2e._\n\n\n### How do we measure it?\n-  [`codecarbon`](https://codecarbon.io/) works with a scheduler and every 15s measures the **carbon intensity** of your running code based on the local grid (country or regional if on cloud)\n\n\u003e _\"Carbon Intensity of the consumed electricity is calculated as a weighted average of the emissions from the different energy sources that are used to generate electricity, including fossil fuels and renewables.\"_ See [here](https://mlco2.github.io/codecarbon/methodology.html#carbon-intensity)\n\n\n## Example usage🔎\n\n![Talk is cheap](https://quotesaga.s3.amazonaws.com/quote/QS_be152af2851e4e4e8d5049b0d5cbaed9.jpg)\n\n\n### 1. Import necessary dependencies\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom qdrant_client import QdrantClient\nfrom fastembed import SparseTextEmbedding\nfrom sentrev.evaluator import evaluate_dense_retrieval, evaluate_sparse_retrieval\nimport os\n```\n\n### 2. Prepare the embedding models\n\n```python\n# Load all the dense embedding models\ndense_encoder1 = SentenceTransformer('sentence-transformers/all-mpnet-base-v2', device=\"cuda\")\ndense_encoder2 = SentenceTransformer('sentence-transformers/all-MiniLM-L12-v2', device=\"cuda\")\ndense_encoder3 = SentenceTransformer('sentence-transformers/LaBSE', device=\"cuda\")\n\n# Create a list of the dense encoders\ndense_encoders = [dense_encoder1, dense_encoder2, dense_encoder3]\n\n# Create a dictionary that maps each encoder to its name\ndense_encoder_to_names = { dense_encoder1: 'all-mpnet-base-v2', dense_encoder2: 'all-MiniLM-L12-v2', dense_encoder3: 'LaBSE'}\n\n# Load all the sparse embedding models\nsparse_encoder1 = SparseTextEmbedding(\"Qdrant/bm25\")\nsparse_encoder2 = SparseTextEmbedding(\"prithivida/Splade_PP_en_v1\")\nsparse_encoder3 = SparseTextEmbedding(\"Qdrant/bm42-all-minilm-l6-v2-attentions\")\n\n# Create a list of the sparse encoders\nsparse_encoders = [sparse_encoder1, sparse_encoder2, sparse_encoder3]\n\n# Create a dictionary that maps each sparse encoder to its name\nsparse_encoder_to_names = { sparse_encoder1: 'BM25', sparse_encoder2: 'Splade', sparse_encoder3: 'BM42'}\n```\n\n### 3. Collect data\n\n```python\n# Collect data\nfiles = [\"~/data/attention_is_all_you_need.pdf\", \"~/data/generative_adversarial_nets.pdf\", \"~/data/narration.docx\", \"~/data/call-to-action.html\", \"~/data/test.xml\"]\n```\n\n\n### 4. Create Qdrant Client\n\n1. Pull and run Qdrant locally with **Docker**:\n\n```bash\ndocker pull qdrant/qdrant\ndocker run -p 6333:6333 -p 6334:6334 qdrant/qdrant\n```\n\n2. Use the python API to create the client:\n\n```python\n# Create Qdrant client\nclient = QdrantClient(\"http://localhost:6333\")\n```\n\n\n### 5. Evaluate retrieval!\n\n```python\n# Define CSV path where the stats will be saved\ncsv_path_dense = \"~/evals/dense_stats.csv\"\ncsv_path_sparse = \"~/evals/sparse_stats.csv\"\n\n# Run evaluation for dense retrieval\nevaluate_dense_retrieval(files, dense_encoders, dense_encoder_to_names, client, csv_path_dense, chunking_size = 1500, text_percentage=0.3, distance=\"dot\", mrr=10, carbon_tracking=\"AUT\", plot=True)\n\n# Run evaluation for sparse retrieval\nevaluate_sparse_retrieval(files, sparse_encoders, sparse_encoder_to_names, client, csv_path_sparse, chunking_size = 1200, text_percentage=0.4, distance=\"euclid\", mrr=10, carbon_tracking=\"AUT\", plot=True)\n\n```\n\n\n### 6. Code reference\n\n\u003e [!IMPORTANT]\n\u003e _Find the full code reference on [the GitHub repository](https://github.com/AstraBert/SenTrEv/blob/main/REFERENCE.md)_\n\n\n## What to expect👀\n\n![expect](https://tse3.mm.bing.net/th?id=OIP.I6OGEoDOqBYJ78__eOMKLQHaBs\u0026pid=Api)\n\n\n### 1. Stats in a CSV\n\n![csv_file](https://qpjsuxneowmprnoqndhu.supabase.co/storage/v1/object/public/screenshots/csvfile.png?t=2025-01-12T23%3A28%3A21.877Z)\n\n\n### 2. Plots\n\n![Plots](https://qpjsuxneowmprnoqndhu.supabase.co/storage/v1/object/public/screenshots/stats_mrr.png)\n\n## SenTrEv is in active development🚀\n\n\u003e [!IMPORTANT]\n\u003e **[Feedback](https://github.com/AstraBert/SenTrEv/discussions) and [contributions](https://github.com/AstraBert/SenTrEv/blob/main/CONTRIBUTING.md) are always welcome!**\n\n\n## Thanks for the attention!🤗\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrabert%2Fsentrev-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastrabert%2Fsentrev-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrabert%2Fsentrev-demo/lists"}