{"id":31765199,"url":"https://github.com/leonericsson/llmcontext","last_synced_at":"2025-10-10T00:14:14.575Z","repository":{"id":210221564,"uuid":"726039668","full_name":"LeonEricsson/llmcontext","owner":"LeonEricsson","description":":anger: Pressure testing the context window of open LLMs","archived":false,"fork":false,"pushed_at":"2024-08-25T15:45:32.000Z","size":918,"stargazers_count":21,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-08-25T16:59:10.813Z","etag":null,"topics":["deep-learning","huggingface-transformers","jupyter-notebook","llm","mistral-7b"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LeonEricsson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-01T12:05:43.000Z","updated_at":"2024-08-25T15:45:35.000Z","dependencies_parsed_at":"2023-12-20T12:50:02.086Z","dependency_job_id":"0c7f2bfd-142e-4ada-be7d-23c374a977e2","html_url":"https://github.com/LeonEricsson/llmcontext","commit_stats":null,"previous_names":["leonericsson/llmcontext"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/LeonEricsson/llmcontext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeonEricsson%2Fllmcontext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeonEricsson%2Fllmcontext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeonEricsson%2Fllmcontext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeonEricsson%2Fllmcontext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LeonEricsson","download_url":"https://codeload.github.com/LeonEricsson/llmcontext/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LeonEricsson%2Fllmcontext/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002386,"owners_count":26083356,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","huggingface-transformers","jupyter-notebook","llm","mistral-7b"],"created_at":"2025-10-10T00:14:01.771Z","updated_at":"2025-10-10T00:14:14.569Z","avatar_url":"https://github.com/LeonEricsson.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pressure Testing: Open LLMs 💢\n\nThis is derivative work of [Needle In A Haystack - Pressure Testing LLMs](https://github.com/gkamradt/LLMTest_NeedleInAHaystack), a project where @gkamradt explored the in-context retrieval abilities of GPT-4 and Claude 2. I was impressed by the insights gained from this test, and as an open-source enthusiast, I felt compelled to extend the experiment to the broader open-source LLM market. As such, this project examines the in-context retrieval capabilities of popular open-source models. My primary aim is to evaluate how these widely-used models in the LLM community perform in terms of simple retrieval within their context window. I welcome suggestions for additional models to include in our study, particularly those with larger context windows and the ability to run with 24GB VRAM + 64GB RAM.\n\n**Note:** As a response to @gkamradt's work, Anthropic ran their own pressure tests, covered in [this](https://www.anthropic.com/index/claude-2-1-prompting) blog post. They were able to massivively improve in-context retrieval performance by priming the model response with `Here is the most relevant sentence in the text:`. All my tests using this retrieval priming technique will be suffixed with `rp`.\n\n## The Test 📝\n\n1. Place a random fact or statement (the 'needle') in the middle of a long context window (the 'haystack')\n2. Ask the model to retrieve this statement using the following prompt format\n\n```\nYou are provided with a text of some essays, admist these essays is a sentence\nthat contains the answer to the user's question. I will now provide the text\n(delimited with XML tags) followed by the user question.\n\n[TEXT]\n{content}\n[/TEXT]\n\n\nUser: {prompt}\n\nAssistant: {retrival primer}\n```\n\n3. Iterate over various document depths (where the needle is placed) and context lengths to measure performance\n\n## Roadmap 🛣️\n\nAn ongoing list of models to pressure test.\n\n```\n1. Mistral 7B Instruct v0.2\n```\n\n## Results 📊\n\nEach test consists of a retrieval, at certain depth percentage, for a given context length. The results are combined into a pivot table illustrating how well the model response was, judged by GPT-4. The scoring system is defined as\n\n```\nScore 1: The answer is completely unrelated to the reference.\nScore 3: The answer has minor relevance but does not align with the reference.\nScore 5: The answer has moderate relevance but contains inaccuracies.\nScore 7: The answer aligns with the reference but has minor omissions.\nScore 10: The answer is completely accurate and aligns perfectly with the reference.\n```\n\nI have slightly adjusted @gkamradt's visualization code to work for this project. The code can be found [here](/utils/visualize.ipynb). The raw results are found in `results/`.\n\n### Qwen-1.5-4B @ 7k [RP]\n\nQwen doesn't have any attention optimizations (SHA, MHA, MQA, GQA), hence scaling contexts is super expensive in terms of VRAM :( \n\n![](/img/qwen1.5-4b-rp.png)\n\n### Qwen-1.5-7B @ 7k [RP]\n\nWish I could test how well this does at higher contexts, all Qwen 1.5 support contexts up to 32k in practice.\n\n![](/img/qwen1.5-7b.png)\n\n### Mistral-7B-Instruct-v0.2 @ 16k\n\nThis model is trained on 8k context but features a theoretical context window of up to 128k, made possible through sliding window attention.\n\n![](/img/mistral_7b_16k.png)\n\n### Mistral-7B-Instruct-v0.2 @ 16k [RP]\n\nUsing the retrieval priming technique from Anthropic, results improve **tremendously**.The model is capable of handling contexts exceeding 8k. However, its performance is characterized by volatility; it tends to either achieve flawless success or encounter complete failure.\n\n![](/img/mistral_7b_16k_rp.png)\n\n### OpenChat 7B 3.5-1210 @ 8k\n\n![](/img/openchat-3.5-1210_8k.png)\n\n### OpenChat 7B 3.5-1210 @ 8k [RP]\n\n![](/img/openchat-3.5-1210_8k_rp.png)\n\n### Starling LM 7B Alpha @ 8k\n\nStarling is finetuned from OpenChat 3.5 and is one of the best 7B models on Chatbot Arena.\n\n![](/img/starling-lm-7b-alpha.png)\n\n### Starling LM 7B Alpha @ 8k [RP]\n\n![](/img/starling-lm-7b-alpha_rp.png)\n\n### Toppy 7B @ 16k\n\n![](/img/toppy-7b.png)\n\n## Implementation\n\nJust a quick note on the implementation. @gkamradt refactored and cleaned the code significantly since I originally started working on this. I don't plan to sync this with his more polished version. This code works fine but it's hacky.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleonericsson%2Fllmcontext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleonericsson%2Fllmcontext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleonericsson%2Fllmcontext/lists"}