{"id":14964950,"url":"https://github.com/jpmanson/llm_templates","last_synced_at":"2025-08-21T11:20:33.960Z","repository":{"id":222922232,"uuid":"758745205","full_name":"jpmanson/llm_templates","owner":"jpmanson","description":"Instruction/chat prompts creation library for text generation LLMs. It supports local and Hugging Face models.","archived":false,"fork":false,"pushed_at":"2024-08-10T01:34:31.000Z","size":309,"stargazers_count":20,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-20T22:50:22.304Z","etag":null,"topics":["chatbot","cohere","gemma","huggingface","jinja2","library","llama2","llama3","llm","mistral","nlp","nlp-library","phi3","template"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jpmanson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-17T01:10:25.000Z","updated_at":"2025-01-14T05:32:46.000Z","dependencies_parsed_at":"2024-02-17T02:27:36.093Z","dependency_job_id":"09bf8f29-d996-47d2-9e8d-187f43671216","html_url":"https://github.com/jpmanson/llm_templates","commit_stats":{"total_commits":32,"total_committers":2,"mean_commits":16.0,"dds":0.0625,"last_synced_commit":"d3a2c968c4db23384bd7ec827bfa30ca03612c67"},"previous_names":["jpmanson/llm_templates"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmanson%2Fllm_templates","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmanson%2Fllm_templates/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmanson%2Fllm_templates/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jpmanson%2Fllm_templates/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jpmanson","download_url":"https://codeload.github.com/jpmanson/llm_templates/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235208944,"owners_count":18953003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","cohere","gemma","huggingface","jinja2","library","llama2","llama3","llm","mistral","nlp","nlp-library","phi3","template"],"created_at":"2024-09-24T13:34:00.914Z","updated_at":"2025-01-24T23:58:42.169Z","avatar_url":"https://github.com/jpmanson.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## `llm-templates`\n![PyPI - Version](https://img.shields.io/pypi/v/llm-templates?color=%252334D058)\n\nIt's a conversation formatter for chat models. The library allows you to easily format a chat conversation, with the same format with which the language model was trained. The library is installed with pip:\n\n```\npip install llm-templates\n```\n\nYou can quickly start with the library using the following [Colab notebook](https://colab.research.google.com/drive/187sA0rGwVLiP6A3IyvV0ICglsJQmvQa3#scrollTo=sUjN9PDx-dcp):\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/187sA0rGwVLiP6A3IyvV0ICglsJQmvQa3#scrollTo=sUjN9PDx-dcp)\n\nThe library has built in templates for the following models:\n- `zephyr`\n- `llama2`\n- `llama3`\n- `llama31`\n- `mistral`\n- `gemma`\n- `cohere`\n- `phi3`\n\n...and HuggingFace models, using [Jinja2](https://github.com/pallets/jinja) templates when tokenizer_config.json file is available.\n\nThis is a quick example with Llama3 model:\n```python\nfrom llm_templates import Formatter, Conversation, Content\n\nmessages = [Content(role=\"user\", content=\"Hello!\"),\n            Content(role=\"assistant\", content=\"How can I help you?\"),\n            Content(role=\"user\", content=\"Write a poem about the sea\")]\n\nconversation = Conversation(model='llama3', messages=messages)\nconversation_str = Formatter().render(conversation, add_assistant_prompt=True)\n\nprint(conversation_str)\n```\nAnd the result will be:\n```\n\u003c|begin_of_text|\u003e\u003c|start_header_id|\u003euser\u003c|end_header_id|\u003e\nHello!\u003c|eot_id|\u003e\n\u003c|start_header_id|\u003eassistant\u003c|end_header_id|\u003e\nHow can I help you?\u003c|eot_id|\u003e\n\u003c|start_header_id|\u003euser\u003c|end_header_id|\u003e\nWrite a poem about the sea\u003c|eot_id|\u003e\n\u003c|start_header_id|\u003eassistant\u003c|end_header_id|\u003e\n```\n\n## Introduction\n\nMany models are based on foundational or pre-trained LLMs, which are then retrained (fin-tuning) with specially designed instruction datasets to improve and refine the abilities of these models on specific tasks:\n\n![transfer_learning](images/transfer_learning.png)\n\nThese data sets typically include a variety of text examples, which can range from questions and answers to instructions and answers. The main purpose is to teach the model how to follow instructions or how to appropriately respond to certain types of requests.\n\nWhen a language model like GPT-3 or GPT-4 is “fine-tuned” on these data sets, it learns to better understand and perform the tasks presented to it. For example, you may get better at understanding complex instructions, generating more relevant and accurate responses, or adapting to specific communication styles. This is particularly useful for specialized applications, where the model is required to understand and respond appropriately to a specific set of instructions or questions related to a particular field or topic, or for use in dialog systems.\n\nThe process of adapting basic LLMs to models trained in following instructions (instruction-following) is called alignment. https://openai.com/research/instruction-following:\n\n![alignment](images/alignment.png)\n\nInstruction datasets are used for fine-tuning Large Language Models (LLMs). This fine-tuning typically uses supervised learning and includes both an input string and an expected output string. The input and output strings follow a template known as the instruction dataset format (for example, `[INST]` `\u003c\u003cSYS\u003e\u003e`). OpenAI's ChatML and Stanford's Alpaca are examples of Instruction datasets. Below is the instruction data format used by Alpaca for fine-tuning that includes context information (the input field below):\n\n```\nBelow is an instruction that describes a task, paired with an input that provides \nfurther context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n```\n\nBecause the models have been fine-tuned (trained) to generate text in dialogue or query contexts, at the time of inference, we will need to format our prompts in the same way, to not deteriorate the quality of our queries or dialogues.\n\nIn conversations or in the request for instructions, messages have a role and content, the latter being the actual text of the message. Commonly, the roles are \"user\" for messages sent by the user, \"assistant\" for responses written by the model, and optionally, \"system\" for high-level directives given at the beginning of the conversation.\n\nIf all this seems a bit abstract, here is a chat example to make it more concrete:\n\n```\n[\n{\"role\": \"user\", \"content\": \"Hello!\"},\n{\"role\": \"assistant\", \"content\": \"Nice to meet you!\"}\n]\n```\n\nThis sequence of messages needs to be converted into a text string before it can be tokenized and used as input to a model. The problem, however, is that there are many ways to do this conversion. We could, for example, convert the message list into an Instant Messenger format:\n\n```\nUser: Hello!\nAssistant: Nice to meet you!\n```\n\nOr we could add special tokens to indicate the roles:\n\n```\n[USER] Hello! [/USER]\n[ASST] Nice to meet you! [/ASST]\n```\n\nOr we could also add tokens to indicate boundaries between messages, but insert the role information as a string:\n\n```\n\u003c|im_start|\u003euser\nHello!\u003c|im_end|\u003e\n\u003c|im_start|\u003eassistant\nNice to meet you!\u003c|im_end|\u003e\n```\n\nThere are many ways to do this, and none of them are best or correct. The way depends on how the different models have been trained. The previous examples are not invented; but they are real and used in some popular models. Once a model has been trained with a certain format, we want to ensure that future inputs use the same format, otherwise we can deteriorate the performance of the model.\n\n## Library usage\n\nThis is where `llm-templates` comes in. It is a Python package that provides a simple and flexible way to convert a list of messages into a string that can be used as input to a model. It also provides a way to convert the model's output back into a list of messages. The package is designed to be easy to use and to work with a wide range of models and formats. The package is designed to be easy to use and to work with a wide range of models and formats. It is also designed to be flexible and to allow you to customize the conversion process to suit your needs. The package is designed to be easy to use and to work with a wide range of models and formats. It is also designed to be flexible and to allow you to customize the conversion process to suit your needs.\n\n```python\nfrom llm_templates import Formatter, Conversation\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": \"Hello!\"\n    },\n    {\n        \"role\": \"assistant\",\n        \"content\": \"How can I help you?\"\n    },\n    {\n        \"role\": \"user\",\n        \"content\": \"Write a poem about the sea\"\n    }\n]\n\nformatter = Formatter()\n\n# Local model\nconversation = Conversation(model=\"zephyr\", messages=messages)\nprint(formatter.render(conversation, add_assistant_prompt=True))\n\n```\n\nAnd the output using zephyr model template will be:\n\n``` \n\u003c|user|\u003eHello!\u003c/s\u003e\n\u003c|assistant|\u003eHow can I help you?\u003c/s\u003e\n\u003c|user|\u003eWrite a poem about the sea\u003c/s\u003e\n\u003c|assistant|\u003e\u003c/s\u003e\n```\n\nAnother example using `llama2` model:\n\n```python\nformatter = Formatter()\n\n# Local model\nconversation = Conversation(model=\"zephyr\", messages=messages)\nprint(formatter.render(conversation))\n\n```\nThe output will be:\n\n```\n\u003cs\u003e[INST] Hello! [/INST]\nHow can I help you? \u003c/s\u003e\n\u003cs\u003e[INST] Write a poem about the sea [/INST]\n\n```\n\nYou can also use HuggingFace models:\n\n```python\nfrom llm_templates import Formatter, Conversation, Content\n\nmessages = [Content(role=\"user\", content=\"Hello!\"),\n            Content(role=\"assistant\", content=\"How can I help you?\"),\n            Content(role=\"user\", content=\"Write a poem about the sea\")]\n\nformatter = Formatter()\n\n# Apply Hugging Face Mixtral model template\nmodel = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n# model = \"HuggingFaceH4/zephyr-7b-beta\"\nconversation = Conversation(model=model, messages=messages)\nconversation_str = formatter.render(conversation)\n\nprint(conversation_str)\n\n# And then call the model in HuggingFace via API\nfrom huggingface_hub import InferenceClient\nclient = InferenceClient()\n\nresult = client.text_generation(prompt=conversation_str, model=model, max_new_tokens=768, temperature=0.7, top_p=0.9,\n                                top_k=50)\n\n```\nAnd the result will be something like this:\n\n```\nThe sea, a vast and endless blue,\nA world of wonder, forever new.\nIts waves crash down with gentle might,\nA symphony of nature's sight.\n\nBeneath the surface, secrets lie,\nA realm where creatures roam and fly.\nCoral castles, home to life,\nA world at peace, amidst the strife.\n\n....\n```\n\n## References\n\n- [Chat Templates](https://huggingface.co/blog/chat-templates)\n- [stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca#data-release)\n- [Gemma Formatting](https://ai.google.dev/gemma/docs/formatting?hl=es-419)\n- [Meta Llama 3 chat](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpmanson%2Fllm_templates","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjpmanson%2Fllm_templates","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjpmanson%2Fllm_templates/lists"}