{"id":13620424,"url":"https://github.com/dezoito/ollama-grid-search","last_synced_at":"2025-05-15T10:03:06.910Z","repository":{"id":214220695,"uuid":"735982439","full_name":"dezoito/ollama-grid-search","owner":"dezoito","description":"A multi-platform desktop application to evaluate and compare LLM models, written in Rust and React.","archived":false,"fork":false,"pushed_at":"2025-01-11T18:39:05.000Z","size":6316,"stargazers_count":699,"open_issues_count":5,"forks_count":39,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-04-14T16:56:11.207Z","etag":null,"topics":["ab-testing","ai","grid-search","gridsearch","llm","ollama","rust","testing-tools"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dezoito.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-26T16:35:23.000Z","updated_at":"2025-04-13T14:36:23.000Z","dependencies_parsed_at":"2024-01-27T23:22:39.960Z","dependency_job_id":"de6293ec-ca67-4a6c-8aa9-e6eb0b9fb635","html_url":"https://github.com/dezoito/ollama-grid-search","commit_stats":{"total_commits":335,"total_committers":8,"mean_commits":41.875,"dds":0.06268656716417909,"last_synced_commit":"bc806f975ce7fb55b21e56e24ee7b282401c5299"},"previous_names":["dezoito/ollama-grid-search"],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dezoito%2Follama-grid-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dezoito%2Follama-grid-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dezoito%2Follama-grid-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dezoito%2Follama-grid-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dezoito","download_url":"https://codeload.github.com/dezoito/ollama-grid-search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254303496,"owners_count":22048209,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","ai","grid-search","gridsearch","llm","ollama","rust","testing-tools"],"created_at":"2024-08-01T21:00:55.649Z","updated_at":"2025-05-15T10:03:04.475Z","avatar_url":"https://github.com/dezoito.png","language":"TypeScript","funding_links":[],"categories":["TypeScript","Model Evaluation and Testing Tools","A01_文本生成_文本对话","ai","Datasets","Evaluation and Monitoring","rust"],"sub_categories":["大语言对话模型及数据"],"readme":"# Ollama Grid Search: Instantly Evaluate Multiple LLMs and Prompts.\n\nThis project automates the process of selecting the best models, prompts, or inference parameters for a given use-case, allowing you to iterate over their combinations and to visually inspect the results.\n\nIt assumes [Ollama](https://www.ollama.ai) is installed and serving endpoints, either in `localhost` or in a remote server.\n\nHere's what an experiment for a simple prompt, tested on 3 different models, looks like:\n\n[\u003cimg src=\"./screenshots/main.png?raw=true\" alt=\"Main Screenshot\" width=\"720\"\u003e](./screenshots/main.png?raw=true)\n\n(For a more in-depth look at an evaluation process assisted by this tool, please check https://dezoito.github.io/2023/12/27/rust-ollama-grid-search.html).\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Features](#features)\n- [Grid Search Concept](#grid-search-or-something-similar)\n- [A/B Testing](#ab-testing)\n- [Prompt Archive](#prompt-archive)\n- [Experiment Logs](#experiment-logs)\n- [Future Features](#future-features)\n- [Contributing](#contributing)\n- [Development](#development)\n- [Citations](#citations)\n- [Acknowledgements](#thank-you)\n\n## Installation\n\nCheck the [releases page](https://github.com/dezoito/ollama-grid-search/releases) for the project, or on the sidebar.\n\n## Features\n\n- Automatically fetches models from local or remote Ollama servers;\n- Iterates over multiple different models, prompts and parameters to generate inferences;\n- A/B test different prompts on several models simultaneously;\n- Allows multiple iterations for each combination of parameters;\n- Allows [limited concurrency](https://dezoito.github.io/2024/03/21/react-limited-concurrency.html) **or** synchronous inference calls (to prevent spamming servers);\n- Optionally outputs inference parameters and response metadata (inference time, tokens and tokens/s);\n- Refetching of individual inference calls;\n- Model selection can be filtered by name;\n- List experiments which can be downloaded in JSON format;\n- Experiments can be inspected in readable views;\n- Re-run past experiments, cloning or modifying the parameters used in the past;\n- Configurable inference timeout;\n- Custom default parameters and system prompts can be defined in settings\n- Fully functional prompt database with examples;\n- Prompts can be selected and \"autocompleted\" by typing \"/\" in the inputs\n\n## Grid Search (or something similar...)\n\nTechnically, the term \"grid search\" refers to iterating over a series of different model hyperparams to optimize model performance, but that usually means parameters like `batch_size`, `learning_rate`, or `number_of_epochs`, more commonly used in training.\n\nBut the concept here is similar:\n\nLets define a selection of models, a prompt and some parameter combinations:\n\n[\u003cimg src=\"./screenshots/gridparams-animation.gif?raw=true\" alt=\"gridparams\" width=\"400\"\u003e](./screenshots/gridparams-animation.gif?raw=true)\n\nThe prompt will be submitted once for each parameter **value**, for each one of the selected models, generating a set of responses.\n\n## A/B Testing\n\nSimilarly, you can perform A/B tests by selecting different models and compare results for the same prompt/parameter combination, or test different prompts under similar configurations:\n\n[\u003cimg src=\"./screenshots/ab-animation.gif?raw=true\" alt=\"A/B testing\" width=\"720\"\u003e](./screenshots/ab-animation.gif?raw=true)\n\n\u003csmall\u003eComparing the results of different prompts for the same model\u003c/small\u003e\n\n## Prompt Archive\n\nYou can save and manage your prompts (we want to make prompts compatible with [Open WebUI](https://github.com/open-webui/open-webui))\n\n[\u003cimg src=\"./screenshots/prompt-archive.png?raw=true\" alt=\"Settings\" width=\"720\"\u003e](./screenshots/prompt-archive.png?raw=true)\n\nYou can **autocomplete** prompts by typing \"/\" (inspired by Open WebUI, as well):\n\n[\u003cimg src=\"./screenshots/autocomplete.gif?raw=true\" alt=\"A/B testing\" width=\"720\"\u003e](./screenshots/autocomplete.gif?raw=true)\n\n## Experiment Logs\n\nYou can list, inspect, or download your experiments:\n\n[\u003cimg src=\"./screenshots/experiments.png?raw=true\" alt=\"Settings\" width=\"720\"\u003e](./screenshots/experiments.png?raw=true)\n\n## Future Features\n\n- Grading results and filtering by grade\n- Importing, exporting and sharing prompt lists and experiment files.\n\n## Contributing\n\n- For obvious bugs and spelling mistakes, please go ahead and submit a PR.\n\n- If you want to propose a new feature, change existing functionality, or propose anything more complex, please open an issue for discussion, **before** getting work done on a PR.\n\n## Development\n\nThe [development notes](./docs/DEVELOPMENT.md) provide setup instructions, sequence diagrams, and workflow charts that should make it easier to understand the project and get started.\n\n## Citations\n\nThe following works and theses have cited this repository:\n\nInouye, D \u0026 Lindo, L, \u0026 Lee, R \u0026 Allen, E; Computer Science and Engineering Senior Theses: **Applied Auto-tuning on LoRA Hyperparameters**\nSanta Clara University, 2024\n\u003chttps://scholarcommons.scu.edu/cgi/viewcontent.cgi?article=1271\u0026context=cseng_senior\u003e\n\n## Thank you!\n\nHuge thanks to [@FabianLars](https://github.com/FabianLars), [@peperroni21](https://github.com/pepperoni21) and [@TomReidNZ](https://github.com/TomReidNZ).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdezoito%2Follama-grid-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdezoito%2Follama-grid-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdezoito%2Follama-grid-search/lists"}