{"id":23821016,"url":"https://github.com/truesparrowsystems/prompt-eval-be","last_synced_at":"2026-06-29T00:32:11.802Z","repository":{"id":176830780,"uuid":"623399020","full_name":"TrueSparrowSystems/prompt-eval-be","owner":"TrueSparrowSystems","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-02T21:45:38.000Z","size":183,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-11-12T00:24:06.664Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TrueSparrowSystems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-04T09:44:07.000Z","updated_at":"2024-03-06T16:13:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"0aea5b01-abd4-49bd-9bb6-89a3067a1923","html_url":"https://github.com/TrueSparrowSystems/prompt-eval-be","commit_stats":null,"previous_names":["truesparrowsystems/prompt-eval-be"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TrueSparrowSystems/prompt-eval-be","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fprompt-eval-be","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fprompt-eval-be/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fprompt-eval-be/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fprompt-eval-be/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TrueSparrowSystems","download_url":"https://codeload.github.com/TrueSparrowSystems/prompt-eval-be/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueSparrowSystems%2Fprompt-eval-be/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34909137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-02T08:19:46.399Z","updated_at":"2026-06-29T00:32:11.797Z","avatar_url":"https://github.com/TrueSparrowSystems.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prompt Evaluator Backend\n\n## Introduction\n\nThe Prompt Evaluator is a test suite that helps evaluate prompt templates and AI models. It enables Product Managers and Developers to create prompt templates with custom variables, define test cases with specific variable values and expected responses, and match the generated responses exactly or fuzzily. The suite also allows for comparing GraphQL query responses and measuring the accuracy of prompt templates against different AI models. By leveraging the capabilities of the Prompt Evaluator, Product Managers and Developers can make informed decisions, iterate on their prompt designs, and enhance the overall quality and accuracy of their AI-powered applications.\n\n## Features\n\n- **Experiments** - The experiment feature in our product allows users to create collections of prompt templates. Users can define their own conversations with various roles and prompts, incorporating variables where necessary. Users can evaluate the performance of prompts by executing them with different OpenAI models and associated test cases.\n\n- **Prompt Templates** - Prompt templates are the building blocks of an Experiment which allow users to define their own prompts. They are highly customizable, allowing users the flexibility to modify the content, format, and variables according to their requirements.\n\n- **Test Cases** - These are the cases on which the accuracy of a prompt is evaluated. Users can define their own test cases and associate them with prompts. Test cases can be defined as a list of inputs and expected outputs.\n\nBy running prompt templates with different models and test cases, users gain valuable insights into the performance and suitability of their prompts for different scenarios. For detailed information on the features, please refer to the [**product guide**](./docs/productGuide.md).\n\n## Architecture\n\nPrompt Evaluator has two components:\n\n- [**Frontend**](https://github.com/TrueSparrowSystems/prompt-eval-fe)\n- [**Backend**](https://github.com/TrueSparrowSystems/prompt-eval-be)\n\nThis is the backend component of the Prompt Evaluator tool. It is built using Django and MongoDB. It exposes a GraphQL API for the frontend to consume. The frontend is built using Next.js. The frontend and backend communicate with each other using the GraphQL API. It is a standalone application that can be deployed separately. For better understanding of the architecture, please refer to the following diagrams:\n\n- [**Sequence diagrams**](./docs/sequenceDiagram.mermaid)\n- [**DBML diagrams**](./docs/db.dbml)\n- [**DevOps Documentation**](./docs/deployment.md)\n\n## Tech Stack\n\n- **Language**: - Python 3.9\n- **Framework**: - Django 3.2.7\n- **Database**: - MongoDB 5.0.3\n- **API**: - GraphQL\n\n## Installation\n\nFollow the instructions below for installation:\n\n1. Check you have installed python version greater than or equal to 3.9\n2. [Install MongoDB](https://www.mongodb.com/docs/manual/administration/install-community/): If MongoDB is not installed on your system, you can download the MongoDB installer from the official website, run it, and follow the installation instructions. Make sure to have MongoDB installed before proceeding.\n\n3. Go to the project directory and copy the contents of .env.sample file in .env file and add the values for all env variables.\n\n```sh { language=sh }\ncd prompt-eval-be\n\n# For Linux/macOS\ncp .env.sample .env\n# For Windows\ncopy .env.sample .env\n```\n\n4. Generate `OPENAI_API_KEY` using this [link](https://platform.openai.com/account/api-keys) and update the same in .env file.\n\n5. Run below command to specify the management of large files with Git.\n\n```sh { language=sh }\nbrew install git-lfs\n```\n\n6. Create a python virtual environment\n\n```sh { language=sh }\npython3 -m venv .venv\n```\n\n7. Activate the virtual environment\n\n```sh { language=sh }\nsource .venv/bin/activate\n```\n\n8. Upgrade to the latest pip version\n\n```sh { language=sh }\npython -m pip install --upgrade pip\n```\n\n9. Clone the [evals](https://github.com/openai/evals) submodule\n\n```sh { language=sh }\ngit submodule update --init --recursive\npip install -e evals_framework\n```\n\n10. Install the dependencies\n\n```sh { language=sh }\npip install -r requirements.txt\n```\n\n## Run Tests\n\nRun the test cases using following command\n\n```sh { language=sh }\npython manage.py test graphQL\n```\n\n## Generate Report\n\nRun the following command to generate test coverage report\n\n```sh { language=sh }\ncoverage run manage.py test\ncoverage report\n```\n\n## Start the server\n\n1. Activate the virtual environment if not already activated\n\n```sh { language=sh }\nsource .venv/bin/activate\n```\n\n2. Run the api server using following command\n\n```sh { language=sh }\npython manage.py runserver 8000\n```\n\n## Documents\n\n1. Sequence diagrams: - [docs/sequenceDiagram.mermaid](./docs/sequenceDiagram.mermaid)\n2. DBML diagrams: - [docs/db.dbml](./docs/db.dbml)\n3. Product usage guide: - [docs/productGuide.md](./docs/productGuide.md)\n4. Deployment to AWS guide: - [docs/deployment.md](./docs/deployment.md)\n\n## Contribution\n\nWe welcome more helping hands to make **Prompt Evaluator** better. Feel free to report issues, raise PRs for fixes \u0026 enhancements. We are constantly working towards addressing broader, more generic issues to provide a clear and user-centric solution that unleashes your full potential. Stay tuned for exciting updates as we continue to enhance our tool.\n\n\u003cp align=\"left\"\u003eBuilt with :heart: by \u003ca href=\"https://truesparrow.com/\" target=\"_blank\"\u003eTrue Sparrow\u003c/a\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruesparrowsystems%2Fprompt-eval-be","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftruesparrowsystems%2Fprompt-eval-be","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftruesparrowsystems%2Fprompt-eval-be/lists"}