{"id":19947356,"url":"https://github.com/firstbatchxyz/rag-evaluations","last_synced_at":"2025-09-19T11:32:27.879Z","repository":{"id":227721726,"uuid":"771957643","full_name":"firstbatchxyz/rag-evaluations","owner":"firstbatchxyz","description":"Evaluation repository of wikipedia index with Dria","archived":false,"fork":false,"pushed_at":"2024-03-14T18:43:17.000Z","size":17,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-13T00:35:39.136Z","etag":null,"topics":["dria","evaluation","hotpot"],"latest_commit_sha":null,"homepage":"https://www.dria.co","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firstbatchxyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-14T09:11:43.000Z","updated_at":"2024-10-18T03:28:59.000Z","dependencies_parsed_at":"2024-03-14T21:25:51.345Z","dependency_job_id":"9055cad2-73b0-40ba-8ed6-8fb57596537b","html_url":"https://github.com/firstbatchxyz/rag-evaluations","commit_stats":null,"previous_names":["firstbatchxyz/rag-evaluations"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Frag-evaluations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Frag-evaluations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Frag-evaluations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Frag-evaluations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firstbatchxyz","download_url":"https://codeload.github.com/firstbatchxyz/rag-evaluations/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233570243,"owners_count":18695849,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dria","evaluation","hotpot"],"created_at":"2024-11-13T00:35:45.852Z","updated_at":"2025-09-19T11:32:22.600Z","avatar_url":"https://github.com/firstbatchxyz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HotpotQA Model Evaluator with Dria's Public RAG Model\n\nThis project is designed to evaluate different models on the HotpotQA dataset. It uses a multi-threaded approach to\nevaluate models concurrently, providing a comprehensive and efficient evaluation process.\n\n## Methodology\n\nThe dataset for evaluation is [HotpotQA](https://huggingface.co/datasets/hotpot_qa), a dataset with 113k Wikipedia-based question-answer pairs. \n\nWe evaluate the validation subset of the dataset, which contains 7405 question-answer pairs.\n\n### Used Models\n\n- Mixtral 8x7B\n- Command R\n- Meta Llama 70B\n- Meta Llama 13B\n- GPT3.5-Turbo\n\nProject includes a Judge class for [GPT-as-a-judge](https://arxiv.org/pdf/2306.05685.pdf). This class is used to evaluate generations by LLMs \nusing ```gpt-4-0125-preview```\n\nThe HotpotQA dataset is loaded and split into rows, with each row being evaluated concurrently by a worker thread.\n\n\n## Evaluation Process\n\n### System Prompts of evaluators\n\nFor Replicate and OpenAI:\n\n```\n\"Step 1: Analyze context for answering questions.\\n\"\n\"Step 2: Decide context is relevant with question or not relevant with question.\\n \"\n\"Step 3: If any topic about question mentioned in context, use that information for question.\\n \"\n\"Step 4: If context has not mention on question, ignore that context I give you and use your self knowledge.\\n \"\n\"Step 5: Answer the question.\\n \"\n```\n\nFor Cohere, there is no system prompt.\n\n### Query Prompts for evaluators\n\nFor all models, the query prompt is as follows:\n\n```\n'''\n{context}\n'''\n**Question**: {question}\"\n```\n\nFor Cohere, context is not included in the query prompt. Instead, it is passed as a separate parameter.\n\n### The evaluation process is as follows:\n\n1. Initially, we analyzed vanilla model responses, assessed by Judge LLM. Next, we extracted context for each question using Dria, then calculated similarity scores. Questions below the threshold were excluded from the evaluation cluster.\n\n2. For included questions:\n    - Context is retrieved from the Local Wikipedia Index using Dria.\n    - Number of article retrieved from Dria is 1. \n    - Context is split into smaller chunks. \n    - Select two chunks with maximum number of shared keywords.\n3. The context is then used to evaluate the RAG model.\n4. The response generated by the RAG model and Simple model is then evaluated by the GPT 4.5 determine if the\n   response aligns with the correct answer.\n\n\n\n## Getting Started\n\nThese instructions will get you a copy of the project up and running on your local machine for development and testing\npurposes.\n\n### Installing Dria CLI\n\nThis project also requires the Dria CLI. You can install it by following the instructions on\nits [GitHub page](https://github.com/firstbatchxyz/dria-cli/tree/master).\n\n### Setup Local Wikipedia with Dria CLI\n\nAfter installing the Dria CLI, you should fetch Wikipedia Index with Dria and serve it locally. You can do this by\nrunning the following commands in your terminal:\n\n```bash\ndria fetch uaBIB4kh7gYh6vSNL7V2eygfbyRu9vGZ_nJ6jKVn_x8 # Transaction/Contract ID of Wikipedia\ndria serve uaBIB4kh7gYh6vSNL7V2eygfbyRu9vGZ_nJ6jKVn_x8\n```\n\n### Project Dependencies\n\nTo install the necessary dependencies, run the following command in your terminal:\n\n```bash\npip install -r requirements.txt\n```\n\n### Used APIs and Services\n\nUsing the project requires access to the following APIs and services:\n- Cohere API: Required for Cohere model evaluations.\n- OpenAI API: Required for any evaluation.\n- Replicate API: Required for any evaluation.\n\nYou will need to obtain API keys for these services and set them as environment variables in your terminal. The\nenvironment variables are as follows:\n\n- COHERE_API_KEY\n- OPENAI_API_KEY\n- REPLICATE_API_KEY\n\n### Usage\n\nTo run the main script, use the following command:\n\n```bash\npython main.py --max_worker \u003cmax_worker\u003e --output_dir \u003coutput_dir\u003e --dataset_slice \u003cdataset_slice\u003e\n```\n\nReplace \u003cmax_worker\u003e, \u003coutput_dir\u003e, and \u003cdataset_slice\u003e with your desired values.\n\n- \u003cmax_worker\u003e: The maximum number of worker threads for concurrent model evaluation.\n- \u003coutput_dir\u003e: The directory where the evaluation results will be saved.\n- \u003cdataset_slice\u003e: The percentage of the HotpotQA dataset to be used for evaluation.\n\n### Contributing\n\nPlease read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Frag-evaluations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirstbatchxyz%2Frag-evaluations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Frag-evaluations/lists"}