https://github.com/3ndetz/expert-union
Improving LLM reponse quality experiment using expert agreement approach
https://github.com/3ndetz/expert-union
Last synced: 3 months ago
JSON representation
Improving LLM reponse quality experiment using expert agreement approach
- Host: GitHub
- URL: https://github.com/3ndetz/expert-union
- Owner: 3ndetz
- License: other
- Created: 2025-05-27T10:44:05.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-27T23:15:51.000Z (about 1 year ago)
- Last Synced: 2025-05-28T00:22:13.045Z (about 1 year ago)
- Language: Python
- Homepage:
- Size: 40 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Support: SUPPORT.md
Awesome Lists containing this project
README
# LLM expert union
Upperleveled "mixture of experts" approach without router and LLM architecture changing.
This is a simple experiment for achieving better LLM response quality by using multiple LLMs and a simple union of their responses.
## Idea
Simple example: when you want to diagnose a disease, you can ask multiple doctors and get their opinions. Then you can combine these opinions to find the repeating answers and get the most probable diagnosis.
Here's the same idea, but for LLMs. You can ask multiple LLMs the same question and combine their agreements to get the most probable answer.
## Current implementation
Here's only the simpliest implementation using only 1 LLM, but with many experts defined in role-prompts (`resources/expert_roles.yml`).
Seems like it works well (but not too good), but I need to perform some benchmarks to be sure. Interesting fact: final expert role very often chooses really correct answer, even if other experts disagree with it.
TODO:
- [x] 1 LLM & multi roles pipeline
- [ ] find little and suitable benchmark
- [ ] perform benchmark on 1-LLM pipeline
- [ ] implement multi-LLM support
- [ ] benchmark it