{"id":19815958,"url":"https://github.com/replicate/cog-llama-template","last_synced_at":"2025-04-05T22:09:15.194Z","repository":{"id":182190937,"uuid":"659624312","full_name":"replicate/cog-llama-template","owner":"replicate","description":"LLaMA Cog template","archived":false,"fork":false,"pushed_at":"2024-02-12T22:15:33.000Z","size":40193,"stargazers_count":308,"open_issues_count":26,"forks_count":50,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-03-29T21:06:21.109Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/replicate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-06-28T08:13:41.000Z","updated_at":"2024-12-25T11:33:03.000Z","dependencies_parsed_at":"2023-12-21T22:32:39.501Z","dependency_job_id":"b87e5350-54f0-4bc2-8e04-a543bd368a1e","html_url":"https://github.com/replicate/cog-llama-template","commit_stats":null,"previous_names":["a16z-infra/cog-llama-template","replicate/cog-llama-template"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-llama-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-llama-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-llama-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/replicate%2Fcog-llama-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/replicate","download_url":"https://codeload.github.com/replicate/cog-llama-template/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247406091,"owners_count":20933803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T10:07:51.573Z","updated_at":"2025-04-05T22:09:15.167Z","avatar_url":"https://github.com/replicate.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLaMA Cog template 🦙\n\nThis is a monorepo for building multiple Llama models using Cog:\n\n- llama-2-13b\n- llama-2-13b-chat\n- llama-2-13b-transformers\n- llama-2-70b\n- llama-2-70b-chat\n- llama-2-7b\n- llama-2-7b-chat\n- llama-2-7b-transformers\n- llama-2-7b-vllm\n\nSee [replicate.com/meta](https://replicate.com/meta).\n\n---\n\n\n**NOTE: This is an experimental branch that depends on exllama**\n\nFor now, you should:\n```sh\ngit clone https://github.com/turboderp/exllama\ncd exllama\ngit checkout e8a544f95b3fd64dfa5549eeeafb85b1ac71a793\n```\n\nWe're working on a proper integration.\n\n**This Cog template works with LLaMA 1 \u0026 2 versions.**\n\nLLaMA is a [new open-source language model from Meta Research](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) that performs as well as closed-source models. \n\nThis is a guide to running LLaMA using in the cloud using Replicate. You'll use the [Cog](https://github.com/replicate/cog) command-line tool to package the model and push it to Replicate as a web interface and API.\n\nThis template can be used to run the `7B`, `13B`, and `70B` versions of LLaMA and LLaMA2 and it also works with fine-tuned models.\n\n**Note: Please verify the system prompt for LLaMA or LLAMA2 and update it accordingly.**\n\n**Note: LLaMA is for research purposes only. It is not intended for commercial use. Check the license of LLaMA \u0026 LLaMA2 on the official LLaMA website of Meta Platforms, Inc.**\n\n## Prerequisites\n\n- **LLaMA weights**. The weights for LLaMA have not yet been released publicly. To apply for access, fill out the Meta Research form to be able to download the weights.\n- **GPU machine**. You'll need a Linux machine with an NVIDIA GPU attached and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker) installed. If you don't already have access to a machine with a GPU, check out our [guide to getting a GPU machine](https://replicate.com/docs/guides/get-a-gpu-machine).\n- **Docker**. You'll be using the [Cog](https://github.com/replicate/cog) command-line tool to build and push a model. Cog uses Docker to create containers for models.\n\n## Step 0: Install Cog\n\nFirst, [install Cog](https://github.com/replicate/cog#install):\n\n```\nsudo curl -o /usr/local/bin/cog -L \"https://github.com/replicate/cog/releases/latest/download/cog_$(uname -s)_$(uname -m)\"\nsudo chmod +x /usr/local/bin/cog\n```\n\n## Step 1: Set up weights\n\nReplicate currently supports the `7B` model size.\n\nPut your downloaded weights in a folder called `unconverted-weights`. The folder hierarchy should look something like this: \n\n```\nunconverted-weights\n├── 7B\n│   ├── checklist.chk\n│   ├── consolidated.00.pth\n│   └── params.json\n├── tokenizer.model\n└── tokenizer_checklist.chk\n```\n\nConvert the weights from a PyTorch checkpoint to a transformers-compatible format using the this command:\n\n```\ncog run python -m transformers.models.llama.convert_llama_weights_to_hf --input_dir unconverted-weights --model_size 7B --output_dir weights\n```\n\nYou final directory structure should look like this:\n\n```\nweights\n├── config.json\n├── generation_config.json\n├── pytorch_model-00001-of-00002.bin\n├── pytorch_model-00002-of-00002.bin\n├── pytorch_model.bin.index.json\n├── special_tokens_map.json\n├── tokenizer.model\n└── tokenizer_config.json\n```\n\nOnce you've done this, you should uncomment `unconverted-weights` in your `.dockerignore` file. This ensures that `unconverted-weights` aren't built into the resulting cog image.\n\n## Step 2: Tenzorize the weights from the transformers-compatible/huggingface format (this will allow cold-starts to happen much faster):\n\nRun convert_to_tensors.py to tenzorize the weights from the previous transformers-compatible/huggingface format:\n```\ncog run python convert_to_tensors.py\n```\nThis will tensorize your weights and write the tensorized weights to `./llama_weights/llama-7b/llama_7b_fp16.tensors` if you have a GPU available and `.../llama_7b_fp32.tensors` if you don't.\n(To load the tensorized model instead of the transformers-compatible/huggingface weights, verify that `DEFAULT_MODEL_NAME` in `config.py` is set to the path of your tensorized weights.) \n\n- Make sure `**.tensors` is not in your `.dockerignore`:\n  In your `.dockerignore` file, remove `**.tensors`. This line will ignore all files that end with .tensors, no matter where they are in the directory structure.\n\n## Step 3: Run the model\n\nYou can run the model locally to test it:\n\n```\ncog predict -i prompt=\"Simply put, the theory of relativity states that\"\n```\n\nLLaMA is not fine-tuned to answer questions. You should construct your prompt so that the expected answer is the natural continuation of your prompt. \n\nHere are a few examples from the [LLaMA FAQ](https://github.com/facebookresearch/llama/blob/57b0eb62de0636e75af471e49e2f1862d908d9d8/FAQ.md#2-generations-are-bad):\n\n- Do not prompt with \"What is the meaning of life? Be concise and do not repeat yourself.\" but with \"I believe the meaning of life is\"\n- Do not prompt with \"Explain the theory of relativity.\" but with \"Simply put, the theory of relativity states that\"\n- Do not prompt with \"Ten easy steps to build a website...\" but with \"Building a website can be done in 10 simple steps:\\n\"\n\n## Step 4: Create a model on Replicate\n\nGo to [replicate.com/create](https://replicate.com/create) to create a Replicate model.\n\nMake sure to specify \"private\" to keep the model private.\n\n## Step 5: Configure the model to run on A100 GPUs\n\nReplicate supports running models on a variety of GPUs. The default GPU type is a T4, but for best performance you'll want to configure your model to run on an A100.\n\nClick on the \"Settings\" tab on your model page, scroll down to \"GPU hardware\", and select \"A100\". Then click \"Save\".\n\n## Step 6: Push the model to Replicate\n\nLog in to Replicate:\n\n```\nsudo cog login\n```\n\nPush the contents of your current directory to Replicate, using the model name you specified in step 3:\n\n```\nsudo cog push r8.im/username/modelname\n```\n\nNote: if you get an error while pushing your model indicating that your model does not exist on Replicate (even if it was successfully created on the Replicate dashboard), make sure to use the \"sudo\" command in the \"cog login\" in terminal.\n\n[Learn more about pushing models to Replicate.](https://replicate.com/docs/guides/push-a-model)\n\n\n## Step 7: Run the model on Replicate\n\nNow that you've pushed the model to Replicate, you can run it from the website or with an API.\n\nTo use your model in the browser, go to your model page.\n\nTo use your model with an API, click on the \"API\" tab on your model page. You'll see commands to run the model with cURL, Python, etc.\n\nTo learn more about how to use Replicate, [check out our documentation](https://replicate.com/docs).\n\n## Contributors ✨\nThis template was generated by Marco Mascorro (@mascobot), with some modifications to the original cog LLaMA template and with the help of the cog and Replicate documentation that wonderful people put together. See all contributors below.\n\nThis project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fcog-llama-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freplicate%2Fcog-llama-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freplicate%2Fcog-llama-template/lists"}