{"id":13682349,"url":"https://github.com/rhohndorf/Auto-Llama-cpp","last_synced_at":"2025-04-30T09:32:00.889Z","repository":{"id":153230785,"uuid":"626537624","full_name":"rhohndorf/Auto-Llama-cpp","owner":"rhohndorf","description":"Uses Auto-GPT with Llama.cpp","archived":false,"fork":false,"pushed_at":"2024-04-08T23:15:16.000Z","size":593,"stargazers_count":384,"open_issues_count":5,"forks_count":68,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-11-09T18:41:57.582Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rhohndorf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-11T17:03:58.000Z","updated_at":"2024-11-07T18:28:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"217678d0-0a87-47a6-aa65-661c173b5d1e","html_url":"https://github.com/rhohndorf/Auto-Llama-cpp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhohndorf%2FAuto-Llama-cpp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhohndorf%2FAuto-Llama-cpp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhohndorf%2FAuto-Llama-cpp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rhohndorf%2FAuto-Llama-cpp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rhohndorf","download_url":"https://codeload.github.com/rhohndorf/Auto-Llama-cpp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224206008,"owners_count":17273376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T13:01:44.599Z","updated_at":"2024-11-12T02:30:23.419Z","avatar_url":"https://github.com/rhohndorf.png","language":"Python","funding_links":[],"categories":["GitHub projects","Python"],"sub_categories":[],"readme":"\u003e [!NOTE]\n\u003e If you are interested in locally running autonomous agents, also have a look at this [project](https://github.com/rhohndorf/beezle-bug) of mine. It's much cleaner, more stable and under more active development.\n\n# Auto-Llama-cpp: An Autonomous Llama Experiment\nThis is a fork of Auto-GPT with added support for locally running llama models through llama.cpp.\nThis is more of a proof of concept. It's sloooow and most of the time you're fighting with the too small context window size or the models answer is not valid JSON. But sometimes it works and then it's really quite magical what even such a small model comes up with. \nBut obviously don't expect GPT-4 brilliance here.\n\n\n## Supported Models\n---\nSince this uses [llama.cpp](https://github.com/ggerganov/llama.cpp) under the hood it should work with all models they support. As of writing this is \n* LLaMA\n* Alpaca\n* GPT4All\n* Chinese LLaMA / Alpaca\n* Vigogne (French)\n* Vicuna\n* Koala\n\n## Model Performance (the experience so far)\n---\n\n### Response Quality\nSo far I have tried \n* Vicuna-13b-4BIT \n* LLama-13B-4BIT\n\nOverall the Vicuna model performed much better than the original LLama model in terms of answering in the required JSON format and how much sense the answers make. I just couldn't get it to stop starting every answer with ### ASSISTANT.\nI am very curious to hear how well others models perform. The 7B models seemed have problems with grasping what's asked of them in the prompt, but I tried very little in this direction since the inference speed didn't seem to be much faster for me.\n\n### Inference Speed\nThe biggest problem at the moment is indeed inference speed. As the agent is self prompting a lot, a few seconds of infernce that are acceptable in a chatbot scenario become minutes and more. \nTesting things like different prompts etc is a pain under these conditions. \n\n## Discussion\nFell free to add your thoughts and experiences in the [discussion](https://github.com/rhohndorf/Auto-Llama-cpp/discussions) area. What models did you try? How well did they work ou for you? \n\n## Future Plans\n---\n\n1. Add GPU Support via GPTQ\n2. Improve Prompts\n3. Remove external API support (This is supposed to be completely self-contained agent)\n4. Add support for [Open Assistent](https://github.com/LAION-AI/Open-Assistant) models\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhohndorf%2FAuto-Llama-cpp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frhohndorf%2FAuto-Llama-cpp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frhohndorf%2FAuto-Llama-cpp/lists"}