{"id":18513329,"url":"https://github.com/jndiogo/combi-tester","last_synced_at":"2025-05-14T12:20:42.871Z","repository":{"id":244088461,"uuid":"814146806","full_name":"jndiogo/combi-tester","owner":"jndiogo","description":"A combination tester for LLMs","archived":false,"fork":false,"pushed_at":"2024-06-13T15:19:20.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-14T12:19:56.550Z","etag":null,"topics":["ai","llm","local-ai","test-automation","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jndiogo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-12T12:33:49.000Z","updated_at":"2024-06-13T15:19:24.000Z","dependencies_parsed_at":"2024-06-12T21:54:52.179Z","dependency_job_id":"463fe05b-5ac9-484a-afdd-85e91c276e73","html_url":"https://github.com/jndiogo/combi-tester","commit_stats":null,"previous_names":["jndiogo/combi-tester"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jndiogo%2Fcombi-tester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jndiogo%2Fcombi-tester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jndiogo%2Fcombi-tester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jndiogo%2Fcombi-tester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jndiogo","download_url":"https://codeload.github.com/jndiogo/combi-tester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254140805,"owners_count":22021225,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","llm","local-ai","test-automation","testing"],"created_at":"2024-11-06T15:37:36.208Z","updated_at":"2025-05-14T12:20:42.809Z","avatar_url":"https://github.com/jndiogo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Combi-tester\n\nThis is a simple way to test an LLM from combinations of prompts and expected results.\n\nFor example, suppose you'd want test a situation where a model would extract element and level information from input prompts like this:\n\n```\nSet element {valid_element*} to level {valid_level*}\n```\n\nIf we'd define variable {valid_element*} to be one of A,B,C,D and {valid_level*} to be a number from 0 to 9, we could then use Combi-tester to see if the model correctly extracted all combinations like:\n\n```\nSet element A to level 0 -\u003e element=A level=0\nSet element A to level 1 -\u003e element=A level=1\nSet element A to level 2 -\u003e element=A level=2\n...\nSet element B to level 0 -\u003e element=B level=0\n... and so on ...\n```\n\nIn this manner we can measure LLM models' output, test different prompts, system instructions, etc.\n\n\nCombi-tester requires [Sibila](https://github.com/jndiogo/sibila) to access local and remote LLMs. Install Sibila with:\n\n```\npip install --upgrade sibila\n```\n\nEach test is driven by an YAML config file, like the following example of an LLM for NLP processing of user commands, controlling a system of four elements (A,B,C,D), with each having levels between 0 and 9. See the inlined comments for more info:\n\n\n``` YAML\nsetup:\n  # optional instruction/system message:\n  inst: |-\n    You will parse user commands and emit actions to control the state of a system, which is listed after \"STATE:\"\n    The system has 4 elements named A, B, C, D, which can be set to a level between 0 and 9, where level 0 means off, while level 9 means maximum or full.\n    If the user requests a non-existent element (not one of A, B, C, D), emit a special action setting any element to special level -1.\n    If the user requests a level which is not between 0 and 9, emit a special action setting any element to special level -1.\n    For example: \n    - if the user enters \"set element A to level 7\", you should emit an action with element=A, level=7\n    - if the user enters \"set element H to level 4\", you should emit an action with element=A, level=-1, because H is not a valid element\n    - if the user enters \"set element B to level 16\", you should emit an action setting element=A, level=-2, because level 16 is outside 0-9 range\n    \n  # the script responsible for doing the inference (whatever you put in the generate() function) and scoring/evaluation (evaluate()):\n  script: |\n    from typing import Any, Optional, Union, Literal\n    from pydantic import BaseModel, Field\n    \n    Element = Literal[\"A\",\"B\",\"C\",\"D\"]\n    \n    class Action(BaseModel):\n        #thinking: str = Field(description=\"Reasoning behind the action\")\n        element: Element = Field(description=\"Element to set\")\n        level: int = Field(description=\"Level to set, between 0 and 9, or special level -1\")\n    \n\n    def generate(model, inst_text, in_text):\n        return model.extract(list[Action], in_text, inst=inst_text)\n\n    def evaluate(value: Action, expected: dict):\n        sub_scores = []\n\n        for field in expected:\n             score = getattr(value,field) == expected[field]\n             sub_scores.append(float(score))\n\n        return sum(sub_scores) / len(sub_scores)\n          \n        \n  vars: # vars defined here will be replaced into the \"in\" text for each test run:\n    off_state: {'A': 0, 'B': 0, 'C': 0, 'D': 0}\n    valid_element*: [\"A\",\"B\",\"C\",\"D\"] # vars ending with * will have their values combined to form each individual prompt\n    invalid_element*: [\"E\",\"F\",\"G\",\"H\",\"I\",\"J\",\"K\",\"L\",\"M\",\"N\",\"O\",\"P\",\"Q\",\"R\",\"S\",\"T\",\"U\",\"V\",\"X\",\"Y\",\"W\",\"Z\"]\n    valid_level*: [0,1,2,3,4,5,6,7,8,9]\n    invalid_level*: [10,11,12,13,14,15,24,36,41,59,67,73,84,99,563,999,-1,-5,-20]\n\ntests: # the actual test input (aka user prompt):\n  - in: |-\n      STATE: {off_state}\n      Set element {valid_element*} to level {valid_level*}\n    expected: # the expected generated values, each key is used in the evaluate() function:\n      element: \"valid_element*\"\n      level: \"valid_level*\"\n\n  - in: |-\n      STATE: {off_state}\n      Set {invalid_element*} to {valid_level*}\n    expected:\n      level: \"-1\" # direct value to compare with LLM result\n\n  - in: |-\n      STATE: {off_state}\n      Set {valid_element*} to {invalid_level*}\n    expected:\n      level: \"-1\"\n\n```\n\nTo test local GGUF models (llama.cpp based) place the files in a \"models\" folder. Or use any remote models (OpenAI, Anthropic, Mistral, etc) supported by Sibila.\n\nRun the test from the YAML config:\n\n\n``` python\ntry: from dotenv import load_dotenv; load_dotenv()\nexcept: ...\n\nfrom tester import TestSet\n\nfrom sibila import Models\n\n\nModels.setup(\"models\", clear=True)\n\nmodels = [\n    \"llamacpp:Phi-3-mini-4k-instruct-q4.gguf\",\n    \"llamacpp:Meta-Llama-3-8B-Instruct-Q4_K_M.gguf\",\n    \"llamacpp:openchat-3.6-8b-20240522-Q4_K_M.gguf\",\n    \"llamacpp:Mistral-7B-Instruct-v0.3.Q4_K_M.gguf\",\n    \"openai:gpt-3.5-turbo-0125\",\n    \"openai:gpt-4o-2024-05-13\",\n]\n\ntest_path = \"abcd4-mult.yaml\"\ntester = TestSet(test_path)\n\nres = tester.run_tests_for_models(models, \n                                  options={\"only_tests\": [], # if set: the indices of the tests to run\n                                           \"first_n_runs\": None, # if set: only do first n combinations\n                                           \"model_delays\": {\"anthropic\": 5} # if model name in keys, delay n seconds before next call (for simple rate limiting) \n                                          })\n\nwith open(\"last_result.txt\", \"w\", encoding=\"utf-8\") as f:\n  f.write(str(res))\n\nprint(TestSet.report(res))\n```\n\nGiving these (resumed) results:\n\n```\nMean score for all models: 0.918\nModel scores:\n  llamacpp:Phi-3-mini-4k-instruct-q4.gguf: 0.877\n  llamacpp:Meta-Llama-3-8B-Instruct-Q4_K_M.gguf: 0.930\n  llamacpp:openchat-3.6-8b-20240522-Q4_K_M.gguf: 0.926\n  llamacpp:Mistral-7B-Instruct-v0.3.Q4_K_M.gguf: 0.837\n  openai:gpt-3.5-turbo-0125: 0.939\n  openai:gpt-4o-2024-05-13: 0.998\nTotal 14 tests, 454 runs.\n\nIncorrect answers per model:\n= llamacpp:Phi-3-mini-4k-instruct-q4.gguf: 0.877 ===================\n{'in_text': \"STATE: {'A': 0, 'B': 0, 'C': 0, 'D': 0}\\nSet E to 2\", 'result': [Action(element='A', level=2)], 'expected': [{'level': -1}], 'score': 0.0}\n{'in_text': \"STATE: {'A': 0, 'B': 0, 'C': 0, 'D': 0}\\nSet E to 3\", 'result': [Action(element='A', level=3)], 'expected': [{'level': -1}], 'score': 0.0}\n{'in_text': \"STATE: {'A': 0, 'B': 0, 'C': 0, 'D': 0}\\nSet E to 4\", 'result': [Action(element='A', level=4)], 'expected': [{'level': -1}], 'score': 0.0}\n{'in_text': \"STATE: {'A': 0, 'B': 0, 'C': 0, 'D': 0}\\nSet E to 5\", 'result': [Action(element='A', level=5)], 'expected': [{'level': -1}], 'score': 0.0}\n{'in_text': \"STATE: {'A': 0, 'B': 0, 'C': 0, 'D': 0}\\nSet E to 6\", 'result': [Action(element='A', level=6)], 'expected': [{'level': -1}], 'score': 0.0}\n(...)\n```\n\n## To do\n\n- Load driving variables from files.\n- Accept message threads as input.\n- An HTML results viewer.\n- Results caching and persistence.\n- Document the compare functions.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjndiogo%2Fcombi-tester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjndiogo%2Fcombi-tester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjndiogo%2Fcombi-tester/lists"}