{"id":18339902,"url":"https://github.com/mxagar/chatbot_evaluation","last_synced_at":"2025-04-09T20:38:53.482Z","repository":{"id":225864478,"uuid":"767081895","full_name":"mxagar/chatbot_evaluation","owner":"mxagar","description":"This repository contains a simple chatbot evaluation package which can be used to score chatbots using a dataset of predefined and rated chat sessions.","archived":false,"fork":false,"pushed_at":"2024-05-22T15:15:56.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T12:49:45.794Z","etag":null,"topics":["bert","chatbot","evaluation","llm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mxagar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-04T17:09:26.000Z","updated_at":"2024-05-22T15:15:59.000Z","dependencies_parsed_at":"2024-03-04T18:41:00.431Z","dependency_job_id":"4a8eefd0-674b-46d8-ac39-38854b56fe21","html_url":"https://github.com/mxagar/chatbot_evaluation","commit_stats":null,"previous_names":["mxagar/chatbot_evaluation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxagar%2Fchatbot_evaluation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxagar%2Fchatbot_evaluation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxagar%2Fchatbot_evaluation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mxagar%2Fchatbot_evaluation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mxagar","download_url":"https://codeload.github.com/mxagar/chatbot_evaluation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248109286,"owners_count":21049281,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","chatbot","evaluation","llm"],"created_at":"2024-11-05T20:19:47.503Z","updated_at":"2025-04-09T20:38:53.461Z","avatar_url":"https://github.com/mxagar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Chatbot Evaluation\n\nThis repository contains a simple chatbot evaluation package which can be used to score chatbots using a dataset of predefined and rated chat sessions.\n\n## Table of Contents\n\n- [Chatbot Evaluation](#chatbot-evaluation)\n  - [Table of Contents](#table-of-contents)\n  - [File Structure and Package Overview](#file-structure-and-package-overview)\n  - [Setup](#setup)\n  - [Data Input, Dataset](#data-input-dataset)\n  - [Chatbot Integration](#chatbot-integration)\n  - [Scorer Integration](#scorer-integration)\n  - [Configuration YAML](#configuration-yaml)\n  - [Tests and Contonuous Integration](#tests-and-contonuous-integration)\n  - [Further Implementation Notes](#further-implementation-notes)\n    - [Metrics](#metrics)\n    - [Parameters to Consider](#parameters-to-consider)\n  - [Intereting Links](#intereting-links)\n  - [Authorship](#authorship)\n\n## File Structure and Package Overview\n\nThe package is composed by the following files:\n\n```\n.\n│   config_eval.yaml                        # Cofiguration YAML\n│   evaluate.py                             # User script-based entrypoint\n│   pyproject.toml\n│   README.md\n│   requirements.in                         # Dependencies\n│   requirements.txt\n│\n├───assets/\n├───chatbot_evaluation/                     # Package folder\n│   │   chatbot.py                          # Chatbot definitions\n│   │   core.py                             # Common structures\n│   │   dataset.py                          # Dataset loading\n│   │   evaluation.py                       # Evaluation\n│   │   persistence.py                      # Store results\n│   │   scoring.py                          # Scorer definitions\n│   └───__init__.py\n│\n├───data/\n│   │   chat_history_dummy.csv              # Dummy dataset with chat history format\n│   │   qa_pairs_dummy.csv                  # Dummy dataset with Q-A pair format\n│   │   check_data.py\n│   │   convert_data.py\n│   │   create_history_dummy_dataset.py\n│   └───README.md\n│\n├───notebooks/\n│       README.md\n│       result_evaluation.ipynb\n│       scoring_tests.ipynb\n│       simple_service_tests.ipynb\n│\n├───results\n│       README.md\n│\n├───tests/                                  # Pytest testing\n│   │   config_test.yaml\n│   │   conftest.py\n│   │   test_evaluation.py\n│   └───__init__.py\n│\n└───utils\n        README.md\n```\n\nThe package is contained mainly in `chatbot_evaluation/`; the modules of the package are extendable.\n\nThe tests are implemented in [`./tests/test_evaluation.py`](./tests/test_evaluation.py).\n\nA simple usage script is provided in [`evaluate.py`](./evaluate.py).\n\n## Setup\n\nIn order to use the package, first, you need to set a Python environment and then install the dependencies.\nYou can install the package, although it's not necessary if you use it from the repository folder.\nA quick recipe to getting started by using [conda](https://conda.io/projects/conda/en/latest/index.html) is the following:\n\n```bash\n# Set proxy, if required\n\n# Create environment, e.g., with conda, to control Python version\nconda create -n chat-eval python=3.10 pip\nconda activate chat-eval\n\n# Install pip-tools\npython -m pip install -U pip-tools\n\n# Generate pinned requirements.txt\npip-compile requirements.in\n\n# Install pinned requirements, as always\npython -m pip install -r requirements.txt\n\n# If required, add new dependencies to requirements.in and sync\n# i.e., update environment\npip-compile requirements.in\npip-sync requirements.txt\npython -m pip install -r requirements.txt\n\n# Optional: To install the package\npython -m pip pip install .\n\n# Optional: if you's like to export you final conda environment config\nconda env export \u003e environment.yml\n# Optional: If required, to delete the conda environment\nconda remove --name chat-eval --all\n```\n\nOnce everything is installed, we can use the package as follows:\n\n```bash\n# Get a dataset with the propper format -\u003e see data/README.md\n# Set a configuration YAML -\u003e config_eval.yaml\n# Run evaluate.py\npython evaluate.py --config_path ./config_eval.yaml --dataset_path ./data/chat_history_dummy.csv\n```\n\nThe results should be placed in the `./results/` folder.\n\n## Data Input, Dataset\n\nCurrently two dataset formats are supported.\nSee [`./data/README.md`](./data/README.md) for more information.\n\n## Chatbot Integration\n\nAt the moment, these chatbots have been defined after deriving `AbstractChatBot`:\n\n- `ChatBotDummy`: it randomly selects an answer from a predefined list, i.e., a list of answers taken from the fed dataset.\n- `ChatBotAPI`: it sends questions to a remote API, e.g. OpenAI's ChatGPT. However, this class needs to be slightly adapted to the use-case.\n- `ChatBotLib`: it can connect to a local LLM using a library; **note that this class needs to be implemented yet**.\n\nFurther chatbots can be easily defined by copy-pasting any of the above; then, the `evaluation.load_chatbot()` interface needs to be extended.\n\n## Scorer Integration\n\nAt the moment, these scorers have been defined after deriving `AbstractScorer`:\n\n- `ScorerDummy`: it randomly scores the similarity between a reference and predicted answer/string.\n- `ScorerBERT`: it scores the similarity between a reference and predicted answer/string using the [BERT-score](https://github.com/Tiiiger/bert_score) package.\n- `ScorerBERT`: it scores the similarity between a reference and predicted answer/string using the [sentence-transformers](https://www.sbert.net/) package; text embeddings are computed and then the cosine similarity is obtained.\n- `ScorerLLM`: This scorer predicts the similarity score between a predicted and reference string by asking a LLM; **note that this class needs to be implemented yet**.\n\nFurther scorers can be easily defined by copy-pasting any of the above; then, the `evaluation.load_scorers()` interface needs to be extended.\n\n## Configuration YAML\n\nIn the following, an exemplary configuration YAML is shown:\n\n```yaml\nchatbot_type: \"api\"  # Options: \"dummy\", \"api\", \"lib\" (if implemented)\ndataset_path: \"./data/chat_history_dummy.csv\"\n\n# Configuration for ChatBotAPI\napi:\n  url: \"api.example.com\"\n  token_type: \"Bearer\"\n\n# \"dummy\", bert-score: \"bert\", sentence-transformers: \"sbert\", llm: \"llm\" (if implemented)\nscorers:\n  - \"dummy\"\n  - \"bert\"\n  - \"sbert\"\n\n# Configuration for BERT-scorer\nbert:\n  lang: \"en\"  # \"de\", \"en\"\n\n# Configuration for SBERT-scorer (sentence-transformers)\nsbert:\n  model_name: \"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\"\n\n# Optional chatbot parameters\nchatbot_params:\n  param_1: \"value_1\"\n  param_2: \"value_2\"\n  param_3: \"value_3\"\n```\n\nThe configuration file is loaded in [`evaluation.py`](./chatbot_evaluation/evaluation.py) and its content can be easily extended.\nThe values of the configuration file are copied to the results CSV.\n\n## Tests and Contonuous Integration\n\nWe can run tests with [pytest](https://docs.pytest.org) from the root folder:\n\n```bash\ncd ... # root folder where tests/ is located\npytest tests\n```\n\nAdditionally, a Github action/workflow is defined in [`.github/workflows/pytest_workflow.yaml`](.github/workflows/pytest_workflow.yaml) which installs the dependencies and the package, and runs the tests.\n\n## Further Implementation Notes\n\n### Metrics\n\n- [x] Time to provide the answer (several queries necessary?)\n- [x] Answer BERT-score wrt. target/reference\n- [x] Answer SBERT-score wrt. target\n- [ ] Answer length (tokens)\n\n### Parameters to Consider\n\n- [ ] App service type (e.g., capacity, vCPU, RAM, ...)\n- [ ] Chatbot model and version (thus, context length), e.g., gpt-35-turbo-0613, ...\n- [ ] Temperature: 0.7, ...\n- [ ] Prompts\n\n## Intereting Links\n\n- [Azure OpenAI Service REST API reference](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)\n- [Metric: bert_score](https://huggingface.co/spaces/evaluate-metric/bertscore)\n- [OpenAI Prompting guide](https://platform.openai.com/docs/guides/prompt-engineering/strategy-test-changes-systematically)\n- [CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models](https://arxiv.org/abs/2401.17043)\n- [Ragas: framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines](https://docs.ragas.io/en/latest/index.html)\n\n## Authorship\n\nMikel Sagardia, 2024.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmxagar%2Fchatbot_evaluation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmxagar%2Fchatbot_evaluation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmxagar%2Fchatbot_evaluation/lists"}