{"id":13722330,"url":"https://github.com/OpenLemur/Lemur","last_synced_at":"2025-05-07T15:30:32.304Z","repository":{"id":190201758,"uuid":"682124318","full_name":"OpenLemur/Lemur","owner":"OpenLemur","description":"[ICLR 2024] Lemur: Open Foundation Models for Language Agents","archived":false,"fork":false,"pushed_at":"2023-10-28T08:54:35.000Z","size":3815,"stargazers_count":536,"open_issues_count":6,"forks_count":35,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-11-14T11:39:59.891Z","etag":null,"topics":["code-generation","language-model","machine-learning","natural-language-processing","nlp","text-reasoning"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2310.06830","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenLemur.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-23T13:48:25.000Z","updated_at":"2024-11-13T14:09:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5566c77-9af2-4972-98d6-4abef89eed1e","html_url":"https://github.com/OpenLemur/Lemur","commit_stats":null,"previous_names":["openlemur/lemur"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenLemur%2FLemur","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenLemur%2FLemur/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenLemur%2FLemur/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenLemur%2FLemur/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenLemur","download_url":"https://codeload.github.com/OpenLemur/Lemur/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252905515,"owners_count":21822818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code-generation","language-model","machine-learning","natural-language-processing","nlp","text-reasoning"],"created_at":"2024-08-03T01:01:27.504Z","updated_at":"2025-05-07T15:30:31.729Z","avatar_url":"https://github.com/OpenLemur.png","language":"Python","funding_links":[],"categories":["Project List","A01_文本生成_文本对话"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e","大语言对话模型及数据"],"readme":"# [Lemur: Open Foundation Models for Language Agents](https://arxiv.org/abs/2310.06830)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://i.imgur.com/Tga8kHW.jpeg\" alt=\"Lemur\"\u003e\n\u003c/p\u003e\n   \u003ca href=\"https://huggingface.co/OpenLemur\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Models\" src=\"https://img.shields.io/badge/🤗-Models-blue\" /\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://xlang.ai/blog/openlemur\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Blog\" src=\"https://img.shields.io/badge/📖-Blog-red\" /\u003e\n   \u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2310.06830\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Paper\" src=\"https://img.shields.io/badge/📜-Paper-purple\" /\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/OpenLemur/lemur\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Stars\" src=\"https://img.shields.io/github/stars/OpenLemur/lemur?style=social\" /\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://github.com/OpenLemur/lemur/issues\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Open Issues\" src=\"https://img.shields.io/github/issues-raw/OpenLemur/lemur\" /\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://twitter.com/XLangNLP\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Twitter Follow\" src=\"https://img.shields.io/twitter/follow/XLANG NLP Lab\" /\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://join.slack.com/t/xlanggroup/shared_invite/zt-20zb8hxas-eKSGJrbzHiPmrADCDX3_rQ\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Join Slack\" src=\"https://img.shields.io/badge/Slack-join-blueviolet?logo=slack\u0026amp\" /\u003e\n   \u003c/a\u003e\n   \u003ca href=\"https://discord.gg/4Gnw7eTEZR\" target=\"_blank\"\u003e\n      \u003cimg alt=\"Discord\" src=\"https://dcbadge.vercel.app/api/server/4Gnw7eTEZR?compact=true\u0026style=flat\" /\u003e\n   \u003c/a\u003e\n\nLemur is an openly accessible language model optimized for both natural language and coding capabilities to serve as the backbone of versatile language agents.\nAs language models continue to evolve from conversational chatbots to functional agents that can act in the real world, they need both strong language understanding and the ability to execute actions. Lemur balances natural language and coding skills to enable agents to follow instructions, reason for tasks, and take grounded actions.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/interface.png\"\u003e\n\u003c/div\u003e\n\nPlease refer to our paper and code for more details:\n- [[Paper](https://arxiv.org/abs/2310.06830)] Lemur: Harmonizing Natural Language and Code for Language Agents\n- [[Blog](https://www.xlang.ai/blog/openlemur)] Introducing Lemur: Open Foundation Models for Language Agents\n\n\n## 🔥 News\n* **[18 October, 2023]:** 🎉 We open-sourced [code for OpenAgents](https://github.com/xlang-ai/OpenAgents): An Open Platform for Language Agents in the Wild. \n* **[11 October, 2023]:** 🎉 We released the research paper and codebase. We will continue updating this repository.\n* **[23 August, 2023]:** 🎉 We released the weights of [`OpenLemur/lemur-70b-v1`](https://huggingface.co/OpenLemur/lemur-70b-v1), and [`OpenLemur/lemur-70b-chat-v1`](https://huggingface.co/OpenLemur/lemur-70b-chat-v1)! Check it out in [HuggingFace Hub](https://huggingface.co/OpenLemur).\n\n## Models\nWe released our models on the HuggingFace Hub:\n* [OpenLemur/lemur-70b-v1](https://huggingface.co/OpenLemur/lemur-70b-v1)\n* [OpenLemur/lemur-70b-chat-v1](https://huggingface.co/OpenLemur/lemur-70b-chat-v1)\n\n## Table of Contents\n- [Lemur: Open Foundation Models for Language Agents](#lemur-open-foundation-models-for-language-agents)\n  - [🔥 News](#-news)\n  - [Models](#models)\n  - [Table of Contents](#table-of-contents)\n  - [Why Lemur?](#why-lemur)\n  - [Quickstart](#quickstart)\n    - [Setup](#setup)\n    - [Lemur-70B](#lemur-70b)\n    - [Lemur-70B-Chat](#lemur-70b-chat)\n  - [Training](#training)\n  - [Evaluation](#evaluation)\n    - [Foundational Abilities](#foundational-abilities)\n    - [Interactive Agent Skills](#interactive-agent-skills)\n      - [Deploy](#deploy)\n      - [MINT](#mint)\n      - [WebArena](#webarena)\n      - [InterCode](#intercode)\n  - [Citation](#citation)\n  - [Acknowledgements](#acknowledgements)\n\n\n\n## Why Lemur?\nMost existing open-source models specialize in either natural language or code. Lemur combines both strengths by:\n\n- Pretraining Llama-2-70B on a 90B token corpus with the 10:1 ratio of code to text and obtaining Lemur-70B-v1\n- Instruction tuning Lemur-70B-v1 on 300K examples covering both text and code and obtaining Lemur-70B-Chat-v1\n\nThis two-stage training produces state-of-the-art performance averaged across diverse language and coding benchmarks, surpassing other available open-source models and narrowing the gap between open-source and commercial models on agent abilities.\n\n## Quickstart\n\n### Setup\nFirst, we have to install all the libraries listed in `requirements.txt`\n\n```bash\nconda create -n xchat python=3.10\nconda activate xchat\nconda install pytorch==2.0.1 pytorch-cuda=11.8 -c pytorch -c nvidia\nconda install -c \"nvidia/label/cuda-11.8.0\" cuda-nvcc\n```\nThen, install the xchat package:\n```bash\ngit clone git@github.com:OpenLemur/Lemur.git\ncd Lemur\npip install -e .\n```\n\n### Lemur-70B\nFor the base model lemur-70b-v1, you can use it in this way:\n\n\u003cdetails\u003e\n\u003csummary\u003eClick me\u003c/summary\u003e\n\u003cp\u003e\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"OpenLemur/lemur-70b-v1\")\nmodel = AutoModelForCausalLM.from_pretrained(\"OpenLemur/lemur-70b-v1\", device_map=\"auto\", load_in_8bit=True)\n\n# Text Generation Example\nprompt = \"The world is \"\ninput = tokenizer(prompt, return_tensors=\"pt\")\noutput = model.generate(**input, max_length=50, num_return_sequences=1)\ngenerated_text = tokenizer.decode(output[0], skip_special_tokens=True)\nprint(generated_text)\n\n# Code Generation Example\nprompt = \"\"\"\ndef factorial(n):\n   if n == 0:\n      return 1\n\"\"\"\ninput = tokenizer(prompt, return_tensors=\"pt\")\noutput = model.generate(**input, max_length=200, num_return_sequences=1)\ngenerated_code = tokenizer.decode(output[0], skip_special_tokens=True)\nprint(generated_code)\n```\n\n\u003c/p\u003e\n\n\u003c/details\u003e\n\n\n### Lemur-70B-Chat\nWe instruction-finetune lemur-70b-v1 model with ChatML format to obtain lemur-70b-chat-v1. You can use lemur-70b-chat-v1 in this way:\n\n\u003cdetails\u003e\n\u003csummary\u003eClick me\u003c/summary\u003e\n\u003cp\u003e\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\n\ntokenizer = AutoTokenizer.from_pretrained(\"OpenLemur/lemur-70b-chat-v1\")\nmodel = AutoModelForCausalLM.from_pretrained(\"OpenLemur/lemur-70b-chat-v1\", device_map=\"auto\", load_in_8bit=True)\n\n# Text Generation Example\nprompt = \"\"\"\u003c|im_start|\u003esystem\nYou are a helpful, respectful, and honest assistant.\n\u003c|im_end|\u003e\n\u003c|im_start|\u003euser\nWhat's a lemur's favorite fruit?\u003c|im_end|\u003e\n\u003c|im_start|\u003eassistant\n\"\"\"\ninput = tokenizer(prompt, return_tensors=\"pt\")\noutput = model.generate(**input, max_length=50, num_return_sequences=1)\ngenerated_text = tokenizer.decode(output[0], skip_special_tokens=True)\nprint(generated_text)\n\n# Code Generation Example\nprompt = \"\"\"\u003c|im_start|\u003esystem\nBelow is an instruction that describes a task. Write a response that appropriately completes the request.\n\u003c|im_end|\u003e\n\u003c|im_start|\u003euser\nWrite a Python function to merge two sorted lists into one sorted list without using any built-in sort functions.\u003c|im_end|\u003e\n\u003c|im_start|\u003eassistant\n\"\"\"\ninput = tokenizer(prompt, return_tensors=\"pt\")\noutput = model.generate(**input, max_length=200, num_return_sequences=1)\ngenerated_code = tokenizer.decode(output[0], skip_special_tokens=True)\nprint(generated_code)\n```\n\n\u003c/p\u003e\n\n\u003c/details\u003e\n\n## Training\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/training.png\"\u003e\n\u003c/div\u003e\n\n\n## Evaluation\nWe evaluated Lemur across:\n- 8 language and code datasets like MMLU, BBH, GSM8K, HumanEval, and Spider to validate balanced capabilities\n- 13 interactive agent datasets to test skills like tool usage, adapting to feedback from environments or humans, and exploring partially observable digital or physical environments.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/agent-scenarios.png\"\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./assets/overall-perform.png\"\u003e\n\u003c/div\u003e\n\n### Foundational Abilities\nWe build the evaluation suite based on [open-instruct](https://github.com/allenai/open-instruct). We will keep updating more tasks and models.\n\nCurrently, we support the following tasks:\n- [✅] [MMLU](./scripts/eval/mmlu.sh)\n- [✅] [BBH](./scripts/eval/bbh.sh)\n- [✅] [GSM8K](./scripts/eval/gsm8k.sh)\n- [✅] [HumanEval](./scripts/eval/human_eval.sh)\n- [✅] [MBPP](./scripts/eval/mbpp.sh)\n- [🚧] [Spider]()\n- [🚧] [MultiPL-E]()\n- [🚧] [DS-1000]()\n- [🚧] ...\n\n### Interactive Agent Skills\nWe use the evaluation frameworks provided by [MINT](https://github.com/xingyaoww/mint-bench), [InterCode](https://github.com/princeton-nlp/intercode), and [WebArena](https://github.com/web-arena-x/webarena) to evaluate interactive agent skills.\n\n#### Deploy\nWe use vLLM to serve the Lemur model. However, the official FastChat codebase does not yet support Lemur-Chat. Therefore, we provide a docker to serve vLLM for Lemur. Please refer to [vllm_lemur.sh](./scripts/deploy/vllm_lemur.sh) for more detailed information.\n\n```bash\nbash scripts/deploy/vllm_lemur.sh\n```\n\n#### MINT\nWe [fork MINT](https://github.com/OpenLemur/mint-bench) codebase to share the configs we used. Please refer to [this config folder](https://github.com/OpenLemur/mint-bench/tree/main/configs) for more details. Please run vllm with [`vllm_lemur.sh`](./scripts/deploy/vllm_lemur.sh) script.\n\n#### WebArena\nWe [fork WebArena](https://github.com/OpenLemur/webarena) codebase to enable vLLM evaluation. To run the evaluation on WebArena, please refer to our [forked WebArena codebase](https://github.com/OpenLemur/webarena).\n\n#### InterCode\nWe [fork InterCode](https://github.com/OpenLemur/intercode) codebase and do modifications to enable Lemur evaluation. Please refer to [this script folder](https://github.com/OpenLemur/intercode/tree/master/scripts) for more details. Please run `text-generation-inference` with [`tgi_lemur.sh`](scripts/deploy/tgi_lemur.sh) script.\n\n\n## Citation\nIf you find our work helpful, please cite us:\n```\n@misc{xu2023lemur,\n      title={Lemur: Harmonizing Natural Language and Code for Language Agents}, \n      author={Yiheng Xu and Hongjin Su and Chen Xing and Boyu Mi and Qian Liu and Weijia Shi and Binyuan Hui and Fan Zhou and Yitao Liu and Tianbao Xie and Zhoujun Cheng and Siheng Zhao and Lingpeng Kong and Bailin Wang and Caiming Xiong and Tao Yu},\n      year={2023},\n      eprint={2310.06830},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n\n## Acknowledgements\n\nThe Lemur project is an open collaborative research effort between [XLang Lab](https://www.xlang.ai/) and [Salesforce Research](https://www.salesforceairesearch.com/). We thank the following institutions for their gift support:\n\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"assets/transparent.png\" width=\"20\" style=\"pointer-events: none;\"\u003e\n\n\u003ca href=\"https://www.salesforceairesearch.com/\"\u003e\n    \u003cimg src=\"assets/salesforce.webp\" alt=\"Salesforce Research\" height = 30/\u003e\n\u003c/a\u003e\n\n\u003cimg src=\"assets/transparent.png\" width=\"20\" style=\"pointer-events: none;\"\u003e\n\n\u003ca href=\"https://research.google/\"\u003e\n    \u003cimg src=\"assets/google_research.svg\" alt=\"Google Research\" height = 30/\u003e\n\u003c/a\u003e\n\n\u003cimg src=\"assets/transparent.png\" width=\"25\" style=\"pointer-events: none;\"\u003e\n\n\u003ca href=\"https://www.amazon.science/\" style=\"display: inline-block; margin-bottom: -100px;\"\u003e\n    \u003cimg src=\"assets/amazon.svg\" alt=\"Amazon AWS\" height = 20 /\u003e\n\u003c/a\u003e\n\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenLemur%2FLemur","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenLemur%2FLemur","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenLemur%2FLemur/lists"}