{"id":15164767,"url":"https://github.com/gpustack/gguf-packer-go","last_synced_at":"2026-05-16T15:34:35.319Z","repository":{"id":252379292,"uuid":"835643768","full_name":"gpustack/gguf-packer-go","owner":"gpustack","description":"Deliver LLMs of GGUF format via Dockerfile.","archived":false,"fork":false,"pushed_at":"2024-10-24T01:59:44.000Z","size":730,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-30T00:51:57.981Z","etag":null,"topics":["gguf","go","llama"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpustack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-30T08:48:34.000Z","updated_at":"2024-10-24T01:51:31.000Z","dependencies_parsed_at":"2024-08-27T03:48:13.036Z","dependency_job_id":"96b14578-f791-430d-bdd2-05e9c282e5fd","html_url":"https://github.com/gpustack/gguf-packer-go","commit_stats":{"total_commits":57,"total_committers":1,"mean_commits":57.0,"dds":0.0,"last_synced_commit":"27a42ffdb01603a38b7aa603794da44557f701e2"},"previous_names":["thxcode/gguf-packer-go","gpustack/gguf-packer-go"],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/gpustack/gguf-packer-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fgguf-packer-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fgguf-packer-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fgguf-packer-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fgguf-packer-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpustack","download_url":"https://codeload.github.com/gpustack/gguf-packer-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fgguf-packer-go/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33108191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gguf","go","llama"],"created_at":"2024-09-27T04:00:31.774Z","updated_at":"2026-05-16T15:34:35.282Z","avatar_url":"https://github.com/gpustack.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GGUF Packer\n\n\u003e tl;dr, Deliver LLMs of [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format via Dockerfile.\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/gpustack/gguf-packer-go)](https://goreportcard.com/report/github.com/gpustack/gguf-packer-go)\n[![CI](https://img.shields.io/github/actions/workflow/status/gpustack/gguf-packer-go/cmd.yml?label=ci)](https://github.com/gpustack/gguf-packer-go/actions)\n[![License](https://img.shields.io/github/license/gpustack/gguf-packer-go?label=license)](https://github.com/gpustack/gguf-packer-go#license)\n[![Download](https://img.shields.io/github/downloads/gpustack/gguf-packer-go/total)](https://github.com/gpustack/gguf-packer-go/releases)\n[![Docker Pulls](https://img.shields.io/docker/pulls/gpustack/gguf-packer)](https://hub.docker.com/r/gpustack/gguf-packer)\n[![Release](https://img.shields.io/github/v/release/gpustack/gguf-packer-go)](https://github.com/gpustack/gguf-packer-go/releases/latest)\n\n[GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) is a file format for storing models for inference\nwith GGML and executors based on GGML. GGUF is a binary format that is designed for fast loading and saving of models,\nand for ease of reading. Models are traditionally developed using PyTorch or another framework, and then converted to\nGGUF for use in GGML.\n\nGGUF Packer aids in building Large Language Model (LLM) distributions. All you need is [Docker](https://www.docker.com/)\n(or [BuildKit daemon](https://github.com/moby/buildkit?tab=readme-ov-file#quick-start)).\n\n## Key Features\n\n- **Efficient Distribution**: GGUF Packer uses the [BuildKit](https://github.com/moby/buildkit) frontend to streamline\n  the building of LLM distributions.\n- **Docker Integration**: It leverages Docker and BuildKit for seamless build process, allowing the use of Dockerfile\n  directly instead of the [Ollama Model File](https://github.com/ollama/ollama/blob/main/docs/modelfile.md).\n- **[Cloud-Native](https://www.cncf.io/)  Support**: It aligns with cloud-native practices,\n  referencing [KEP-4639 OCI VolumeSource PoC](https://github.com/kubernetes/kubernetes/issues/125463).\n\n## Agenda\n\n- [Quick Start](#quick-start)\n    + [Prerequisites](#prerequisites)\n    + [Write Dockerfile](#write-dockerfile)\n    + [Build Model](#build-model)\n    + [Estimate Model Memory Usage](#estimate-model-memory-usage)\n    + [Build Model with other Quantize Type](#build-model-with-other-quantize-type)\n    + [Pull Model from Container Image Registry](#pull-model-from-container-image-registry)\n    + [Run Model](#run-model)\n    + [Refer Model](#refer-model)\n- [GGUFPackerfile](#ggufpackerfile)\n- [Overview](#overview)\n    + [Format](#format)\n    + [Instructions](#instructions)\n        * [ADD](#add)\n        * [ARG](#arg)\n        * [CAT](#cat)\n        * [CMD](#cmd)\n        * [COPY](#copy)\n        * [CONVERT](#convert)\n        * [FROM](#from)\n        * [LABEL](#label)\n        * [QUANTIZE](#quantize)\n- [Motivation](#motivation)\n    + [Docker Image](#docker-image)\n    + [OCI Distribution](#oci-distribution)\n    + [Ollama Model](#ollama-model)\n\n## Quick Start\n\n### Prerequisites\n\nInstall [Docker](https://docs.docker.com/engine/install/)\nand [GGUF Packer](https://github.com/gpustack/gguf-packer-go/releases).\n\n### Write Dockerfile\n\nTo get started, create a `Dockefile` file with the following content:\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\nARG        BASE=scratch\nARG        QUANTIZE_TYPE=Q5_K_M\nARG        CHAT_TEMPLATE=\"{% for message in messages %}{{'\u003c|im_start|\u003e' + message['role'] + '\\n' + message['content'] + '\u003c|im_end|\u003e' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '\u003c|im_start|\u003eassistant\\n' }}{% endif %}\"\n\nFROM       scratch AS f16\nADD        https://huggingface.co/Qwen/Qwen2-0.5B-Instruct.git  Qwen2-0.5B-Instruct\nCONVERT    --type=F16  Qwen2-0.5B-Instruct  Qwen2-0.5B-Instruct.F16.gguf\n\nFROM       ${BASE}\nLABEL      gguf.model.from=\"Hugging Face\"\nQUANTIZE   --from=f16 --type=${QUANTIZE_TYPE}  Qwen2-0.5B-Instruct.F16.gguf  Qwen2-0.5B-Instruct.${QUANTIZE_TYPE}.gguf\nCAT        \u003c\u003cEOF system-prompt.txt\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\nIn answering questions, follow these steps:\n1. Entity Identification: Identify the main entities involved in the query. Entities can include people, objects, events, or abstract concepts.\n2. Relationship Comprehension: Determine the relationships between these entities. Relationships may be explicit in the text or may need to be inferred based on context and general knowledge.\n3. Implication Understanding: Understand the general implications of these relationships. These implications can be based on established facts, principles, or rules related to the identified relationships.\n4. Question Contextualization: Align the implications of the relationships with the context of the query. This alignment should guide your response to the query.\n5. Answer Generation: Based on the understanding of the entities, their relationships, and implications, generate an appropriate response to the query.\nEOF\nCMD        [\"-m\", \"Qwen2-0.5B-Instruct.${QUANTIZE_TYPE}.gguf\", \"-c\", \"8192\", \"--system-prompt-file\", \"system-prompt.txt\", \"--chat-template\", \"${CHAT_TEMPLATE}\"]\n```\n\nThe provided `Dockerfile` will build a distribution package for\nthe [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct/tree/main) large language model (LLM). The\nmodel has been quantized using the `Q5_K_M` quantization technique, which reduces the model size and inference latency\nwithout significantly impacting accuracy.\n\n- `ARG BASE=...`: The base image for the build, default is `scratch`.\n- `ARG QUANTIZE_TYPE=...`: The quantize type for the model, default is `Q5_K_M`.\n- `ARG CHAT_TEMPLATE=\"...\"`: The chat template for the model, default is the OpenAI GPT-3 chat template.\n- `FROM scratch AS f16`: The first stage to convert the model to `F16` format, named `f16`.\n    + `ADD ...`: Add the model from the Hugging Face repository.\n    + `CONVERT ...`: Convert the model to `F16` format.\n- `FROM ${BASE}`: The second stage to quantize the model from the first stage, and set the system prompt and chat\n  template.\n    + `LABEL ...`: Add metadata to the image.\n    + `QUANTIZE ...`: Quantize the model from the first stage.\n    + `CAT ...`: Concatenate the system prompt to a file.\n    + `CMD ...`: Specify the default commands.\n\n### Build Model\n\nThe `ADD` instruction allows you to clone the model from a Git repository. However, please note that the Git LFS (Large\nFile Storage) is not yet supported(see [moby/buildkit#5212](https://github.com/moby/buildkit/pull/5212)). To achieve\nthis functionality, you can use a development version of BuildKit.\n\nFirst, set up a BuildKit daemon by running the command:\n\n```shell\n$ docker buildx create --name \"git-lfs\" --driver \"docker-container\" --driver-opt \"image=thxcode/buildkit:v0.15.1-git-lfs\" --buildkitd-flags \"--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host\" --bootstrap \n```\n\nNext, build and publish your model. By including the `--push` argument, the built model will be automatically published\nto the Docker registry:\n\n```shell\n$ export REPO=\"YOUR_REPOSITORY\"\n$ docker build --builder git-lfs --tag ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo --load --push $(pwd)\n```\n\n### Estimate Model Memory Usage\n\nOnce the building process is complete, we can utilize `gguf-packer` to estimate the model:\n\n```shell\n$ gguf-packer estimate ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo\n+-------+--------------+--------------------+-----------------+-----------+----------------+---------------+----------------+----------------+---------------------------------------------+---------------------------------------+\n|  ARCH | CONTEXT SIZE | BATCH SIZE (L / P) | FLASH ATTENTION | MMAP LOAD | EMBEDDING ONLY | DISTRIBUTABLE | OFFLOAD LAYERS | FULL OFFLOADED |                     RAM                     |                 VRAM 0                |\n|       |              |                    |                 |           |                |               |                |                +--------------------+-----------+------------+----------------+-----------+----------+\n|       |              |                    |                 |           |                |               |                |                | LAYERS (I + T + O) |    UMA    |   NONUMA   | LAYERS (T + O) |    UMA    |  NONUMA  |\n+-------+--------------+--------------------+-----------------+-----------+----------------+---------------+----------------+----------------+--------------------+-----------+------------+----------------+-----------+----------+\n| qwen2 |     8192     |     2048 / 512     |     Disabled    |  Enabled  |       No       |  Unsupported  |   25 (24 + 1)  |       Yes      |      1 + 0 + 0     | 89.19 MiB | 239.19 MiB |     24 + 1     | 96.58 MiB | 1.03 GiB |\n+-------+--------------+--------------------+-----------------+-----------+----------------+---------------+----------------+----------------+--------------------+-----------+------------+----------------+-----------+----------+\n```\n\n### Build Model with other Quantize Type\n\nYou can build the model using various quantization types by setting the `QUANTIZE_TYPE` argument:\n\n```shell\n$ export QUANTIZE_TYPE=\"Q4_K_M\" \n$ docker build --builder git-lfs --tag ${REPO}/qwen2:0.5b-instruct-$(echo \"${QUANTIZE_TYPE}\" | tr '[:upper:]' '[:lower:]' | sed 's/_/-/g')-demo --build-arg QUANTIZE_TYPE=${QUANTIZE_TYPE} --load --push $(pwd)\n```\n\nWith build cache, the total build time will be reduced.\n\n### Pull Model from Container Image Registry\n\nYou can retrieve the published models from the Docker registry using `gguf-packer`:\n\n```shell\n$ gguf-packer pull ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo\n$ gguf-packer pull ${REPO}/qwen2:0.5b-instruct-q4-k-m-demo\n$ gguf-packer list\n\n      NAME                  TAG                  ID       ARCH    PARAMS     BPW          TYPE          CREATED         SIZE     \n  ${REPO}/qwen2  0.5b-instruct-q4-k-m-demo  a0d46ab8fd9f  qwen2  494.03 M  6.35 bpw  IQ2_XXS/Q4_K_M  19 minutes ago  379.38 MiB  \n  ${REPO}/qwen2  0.5b-instruct-q5-k-m-demo  269bac3c0e20  qwen2  494.03 M  6.71 bpw  IQ3_XXS/Q5_K_M  30 minutes ago  400.62 MiB\n```\n\n### Run Model\n\nTo run a local model\nusing [ghcr.io/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp/pkgs/container/llama.cpp), you can utilize\nthe gguf-packer:\n\n```shell\n$ gguf-packer run ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo -- --flash-attn\n```\n\nYou can preview the running command by using the `--dry-run` option:\n\n```shell\n$ gguf-packer run ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo --dry-run -- --flash-attn\n\ndocker run --rm --interactive --tty --privileged --publish 8080:8080 --volume ${GGUF_PACKER_STORE_PATH}/models/layers/sha256/269bac3c0e202559a2e75f88d087df3324f95b6aaf108e9e70e8b8895aaa8561:/gp-849d4691 ghcr.io/ggerganov/llama.cpp:server -m /gp-849d4691/Qwen2-0.5B-Instruct.Q5_K_M.gguf -c 8192 --system-prompt-file /gp-849d4691/system-prompt.txt --chat-template \"{% for message in messages %}{{'\u003c|im_start|\u003e' + message['role'] + '\\\\n' + message['content'] + '\u003c|im_end|\u003e' + '\\\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '\u003c|im_start|\u003eassistant\\\\n' }}{% endif %}\" --flash-attn --host 0.0.0.0\n```\n\n### Refer Model\n\nSince the `${REPO}/qwen2:0.5b-instruct-q5-k-m-demo` is a standard OCI Artifact, you can refer it using the `FROM`\ninstruction in other Dockerfiles.\n\nYou can rebuild a model based on [Ubuntu:22.04](https://hub.docker.com/_/ubuntu/tags).\n\n```shell\n$ docker build --builder git-lfs --tag ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo2 --build-arg BASE=ubuntu:22.04 --load --push $(pwd)\n```\n\nTo proceed, create a file named `Dockerfile.infer` with the following content:\n\n```dockerfile\n# syntax=docker/dockerfile:1.7-labs\nARG  REPO=\"\"\nFROM ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo2\nRUN  apt-get update \u0026\u0026 \\\n     apt-get install -y libcurl4-openssl-dev libgomp1 curl \nENV LC_ALL=C.utf8\nCOPY --from=ghcr.io/ggerganov/llama.cpp:server /llama-server /\nENTRYPOINT [ \"/llama-server\" ]\n# reuse model file and system prompt file from the base image\nCMD [\"-m\", \"Qwen2-0.5B-Instruct.Q5_K_M.gguf\", \"-c\", \"8192\", \"--system-prompt-file\", \"system-prompt.txt\"]\n```\n\n- `ARG REPO=...`: The repository of the model image.\n- `FROM ...`: The base image for the build.\n- `RUN ...`: Install the dependencies.\n- `ENV ...`: Set the local.\n- `COPY --from=... ...`: Copy the llama-server binary from the llama.cpp\n  image.\n- `ENTRYPOINT ...`: Specify the default commands.\n- `CMD ...`: Specify the default commands.\n\nOnce the `Dockerfile.infer` is created, you can build the container image using the following command:\n\n```shell\n$ docker build --builder git-lfs --tag ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo2-infer --build-arg REPO=${REPO} --file Dockerfile.infer --load $(pwd) \n```\n\nAnd, you can run the built image with `docker run`:\n\n```shell\n$ docker run --rm --interactive --tty ${REPO}/qwen2:0.5b-instruct-q5-k-m-demo2-infer\n```\n\n## GGUFPackerfile\n\n`GGUFPackerfile` is the preferred file name of the GGUF Packer frontend. It can be simply understood that when a\n`Dockerfile` is added with a specific syntax, this `Dockerfile` is equivalent to `GGUFPackerfile`.\n\n\u003ctable\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth\u003e \u003c/th\u003e\n\u003cth\u003e Command \u003c/th\u003e\n\u003cth\u003e Content \u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd\u003e \n\n**Dockerfile**\n\n\u003c/td\u003e\n\u003ctd\u003e \n\n``` shell\n$ docker build --tag ${TAG}\n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\nFROM scratch\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e \n\n**GGUFPackerfile**\n\n\u003c/td\u003e\n\u003ctd\u003e \n\n```\n$ docker build --tag ${TAG} \\\n    --build-arg BUILDKIT_SYNTAX=gpustack/gguf-packer:latest \\\n    --file GGUFPackerfile \n```\n\n\u003c/td\u003e\n\u003ctd\u003e\n\n``` dockerfile\nFROM scratch\n```\n\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\nSee [examples](./examples) for more details.\n\n## Overview\n\nGGUF Packer is a high-level languages have been implemented\nfor [BuildKit LLB](https://github.com/moby/buildkit?tab=readme-ov-file#exploring-llb), which supports the following\ninstructions:\n\n| Instruction             | Description                                                                                      |\n|-------------------------|--------------------------------------------------------------------------------------------------|\n| [`ADD`](#add)           | Add local or remote files and directories.                                                       |\n| [`ARG`](#arg)           | Use build-time variables. \u003cbr/\u003e Allow one GGUFPackerfile to build various models.                |\n| [`CAT`](#cat)           | Concatenate content to a file. \u003cbr/\u003e Be able to create system prompt by hand.                    |\n| [`CMD`](#cmd)           | Specify default commands. \u003cbr/\u003e Declare the main model, drafter, multimodal projector and so on. |\n| [`COPY`](#copy)         | Copy files and directories.                                                                      |\n| [`CONVERT`](#convert)   | Convert safetensors model files to a GGUF model file.                                            |\n| [`FROM`](#from)         | Set the base image for the build.                                                                |\n| [`LABEL`](#label)       | Add metadata to an image.                                                                        |\n| [`QUANTIZE`](#quantize) | Quantize a GGUF file.                                                                            |\n\n### Format\n\nThe format follows the definition\nof [Dockerfile](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#format), here is an\nexample:\n\n```dockerfile\n# Comment\nINSTRUCTION arguments\n```\n\nBuildKit treats lines that begin with `#` as a comment, unless the line is a\nvalid [parser directive](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#parser-directives).\nWhen using `Dockerfile` file, A `# syntax=gpustack/gguf-packer:latest` must add to the top of the file,\nsee [Usage](#usage).\n\n### Instructions\n\n#### ADD\n\nThe `ADD` instruction copies new files or directories from `\u003csrc\u003e` and adds them to the filesystem of the image at the\npath `\u003cdest\u003e`. Files and directories can be copied from the build context, a remote URL, or a Git repository.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\n# add from http\nADD https://huggingface.co/QuantFactory/Qwen2-0.5B-Instruct-GGUF/resolve/main/Qwen2-0.5B-Instruct.Q5_K_M.gguf /app/Qwen2-0.5B-Instruct.Q5_K_M.gguf\n\n# add from git repository\nADD https://huggingface.co/Qwen/Qwen2-0.5B-Instruct.git /app/Qwen2-0.5B-Instruct\n```\n\n##### Available Options\n\n- `ADD [--keep-git-dir=\u003cboolean\u003e] \u003csrc\u003e ... \u003cdir\u003e`, preserve the `.git` directory when adding from a Git repository.\n- `ADD [--checksum=\u003chash\u003e] \u003csrc\u003e ... \u003cdir\u003e`, only support HTTP/HTTPS URLs, the checksum is formatted\n  as \u003calgorithm\u003e:\u003chash\u003e. The supported algorithms are sha256, sha384, and sha512.\n- `ADD [--chown=\u003cuser\u003e:\u003cgroup\u003e] [--chmod=\u003cperms\u003e ...] \u003csrc\u003e ... \u003cdest\u003e`,\n  referring [Dockerfile/COPY --chown --chmod](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---chown---chmod).\n- `ADD [--link[=\u003cboolean\u003e]] \u003csrc\u003e ... \u003cdest\u003e`,\n  referring [Dockerfile/COPY --link](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---link).\n- `ADD [--exclude=\u003cpath\u003e ...] \u003csrc\u003e ... \u003cdest\u003e`,\n  referring [Dockerfile/COPY --exclude](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---exclude).\n\n#### ARG\n\nThe `ARG` instruction defines a variable that users can pass at build-time to the builder with the `docker build`\ncommand using the `--build-arg \u003cvarname\u003e=\u003cvalue\u003e` flag.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\nARG REPO=QuantFactory\nARG MODEL=Qwen2-0.5B-Instruct\nARG QUANTIZE_TYPE=Q5_K_M\n\nADD https://huggingface.co/${REPO}/${MODEL}-GGUF/resolve/main/${MODEL}.${QUANTIZE_TYPE}.gguf /app/${MODEL}.${QUANTIZE_TYPE}.gguf\n```\n\nGGUF Packer supports global `ARG`s, which means you can use the same `ARG` in multiple stages.\n\n#### CAT\n\nThe `CAT` instruction allows you to concatenate content to a file.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\nCAT \u003c\u003cEOF /app/system-prompt.txt\nYou are an AI assistant that follows instruction extremely well. Help as much as you can.\nIn answering questions, follow these steps:\n1. Entity Identification: Identify the main entities involved in the query. Entities can include people, objects, events, or abstract concepts.\n2. Relationship Comprehension: Determine the relationships between these entities. Relationships may be explicit in the text or may need to be inferred based on context and general knowledge.\n3. Implication Understanding: Understand the general implications of these relationships. These implications can be based on established facts, principles, or rules related to the identified relationships.\n4. Question Contextualization: Align the implications of the relationships with the context of the query. This alignment should guide your response to the query.\n5. Answer Generation: Based on the understanding of the entities, their relationships, and implications, generate an appropriate response to the query.\nEOF\n\n# llama-server like application's arguments\nCMD [\"-c\", \"8192\", \"--system-prompt-file\", \"/app/system-prompt.txt\"]\n```\n\n#### CMD\n\nThe `CMD` instruction sets the command to be executed. There can only be one `CMD` instruction in a Dockerfile. If you\nlist more than one `CMD`, only the last one takes effect.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\nARG CHAT_TEMPLATE=\"{% for message in messages %}{{'\u003c|im_start|\u003e' + message['role'] + '\\n' + message['content'] + '\u003c|im_end|\u003e' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '\u003c|im_start|\u003eassistant\\n' }}{% endif %}\"\n\nCMD [\"-m\", \"/app/Qwen2-0.5B-Instruct.Q5_K_M.gguf\", \"-c\", \"8192\", \"--system-prompt-file\", \"/app/system-prompt.txt\", \"--chat-template\", \"${CHAT_TEMPLATE}\"]\n```\n\n#### COPY\n\nThe `COPY` instruction copies new files or directories from `\u003csrc\u003e` and adds them to the filesystem of the image at the\npath `\u003cdest\u003e`. Files and directories can be copied from the build context, build stage, named context, or an image.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\n# copy from local\nCOPY Qwen2-0.5B-Instruct.Q5_K_M.gguf /app/\n\n# copy from other stage\nCOPY --from=original /app/Qwen2-0.5B-Instruct.Q5_K_M.gguf /app/\n```\n\n##### Available Options\n\n- `COPY [--from=\u003cimage|stage|context\u003e] \u003csrc\u003e ... \u003cdest\u003e`, by default, the `COPY` instruction copies files from the build\n  context. The `COPY --from` flag lets you copy files from an image, a build stage, or a named context instead.\n- `COPY [--parents[=\u003cboolean\u003e]] \u003csrc\u003e ... \u003cdest\u003e`, preserves parent directories for `\u003csrc\u003e` entries.\n- `COPY [--chown=\u003cuser\u003e:\u003cgroup\u003e] [--chmod=\u003cperms\u003e ...] \u003csrc\u003e ... \u003cdest\u003e`,\n  referring [Dockerfile/COPY --chown --chmod](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---chown---chmod).\n- `COPY [--link[=\u003cboolean\u003e]] \u003csrc\u003e ... \u003cdest\u003e`,\n  referring [Dockerfile/COPY --link](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---link).\n- `COPY [--exclude=\u003cpath\u003e ...] \u003csrc\u003e ... \u003cdest\u003e`,\n  referring [Dockerfile/COPY --exclude](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#copy---exclude).\n\n#### CONVERT\n\nThe `CONVERT` instruction allows you to convert safetensors model files to a GGUF model file.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\n# convert safetensors model files from current stage\nADD     https://huggingface.co/Qwen/Qwen2-0.5B-Instruct.git /app/Qwen2-0.5B-Instruct\nCONVERT --type=F16 /app/Qwen2-0.5B-Instruct /app/Qwen2-0.5B-Instruct.F16.gguf\n\n# convert from other stage\nCONVERT --from=other-stage --type=F16 /app/Qwen2-0.5B-Instruct /app/Qwen2-0.5B-Instruct.F16.gguf\n\n# convert from build context\nCONVERT --from=context --type=F16 /app/Qwen2-0.5B-Instruct /app/Qwen2-0.5B-Instruct.F16.gguf\n\n# convert a PEFT LoRA adapter to GGUF file\nADD        https://huggingface.co/inflaton/Qwen2-1.5B-MAC-lora.git Qwen2-1.5B-MAC-lora\nADD        https://huggingface.co/Qwen/Qwen2-1.5B.git Qwen2-1.5B\nCONVERT    --type=F16 --class=lora --base=Qwen2-1.5B Qwen2-1.5B-MAC-lora Qwen2-1.5B-MAC-lora.F16.gguf\n```\n\n##### Available Options\n\n- `CONVERT [--from=\u003cimage|stage|context\u003e] \u003csrc\u003e \u003cdest\u003e`, by default, the `CONVERT` instruction converts file from the\n  build context. The `CONVERT --from` flag lets you convert file from an image, a build stage, or a named context\n  instead.\n- `CONVERT [--class=\u003cmodel|lora\u003e] \u003csrc\u003e \u003cdest\u003e`, specify the class for the model, default is `model`.\n    + `CONVERT --class=lora --base=\u003cpath\u003e \u003csrc\u003e \u003cdest\u003e`, convert a PEFT LoRA adapter to GGUF file, must provide the\n      `base` model.\n- `CONVERT [--type=\u003ctype\u003e] \u003csrc\u003e \u003cdest\u003e`, specify the output type for `\u003cdest\u003e`, select from `F32`, `F16`, `BF16`,\n  `Q8_0`, `TQ1_0`, and `TQ2_0`, default is `F16`.\n\n#### FROM\n\nThe `FROM` instruction initializes a new build stage and sets\nthe [base image](https://docs.docker.com/reference/glossary/#base-image) for subsequent instructions. As such, a valid\nDockerfile must start with a `FROM` instruction. The image can be any valid image.\n\n`FROM` can appear multiple times within a single Dockerfile to create multiple images or use one build stage as a\ndependency for another. Simply make a note of the last image ID output by the commit before each new `FROM` instruction.\nEach `FROM` instruction clears any state created by previous instructions.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\nFROM scratch \n\n# reference another image\nFROM thxcode/qwen2:0.5b-instruct-q5-k-m\n```\n\n#### LABEL\n\nThe `LABEL` instruction adds metadata to an image. A `LABEL` is a key-value pair. To include spaces within a `LABEL`\nvalue, use quotes and backslashes as you would in command-line parsing.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\nLABEL org.opencontainers.image.title=\"Qwen2-0.5B-Instruct\" \\\n      org.opencontainers.image.description=\"Qwen2 0.5B Instruct model\" \\\n      org.opencontainers.image.url=\"https://huggingface.co/Qwen/Qwen2-0.5B-Instruct\" \\\n      org.opencontainers.image.source=\"https://huggingface.co/Qwen/Qwen2-0.5B-Instruct\"\n```\n\n##### Export Labels\n\nSince GGUF format model files will\nrecord [the general metadata](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md#general-metadata), GGUF Packer\ncan retrieve these values and export them as labels.\n\n- `gguf.model.architecture`: The architecture of the model.\n- `gguf.model.parameters`: The parameters of the model.\n- `gguf.model.bpw`: The bits per weight of the model.\n- `gguf.model.filetype`: The file type of the model.\n- `gguf.model.name`: The name of the model, if specified.\n- `gguf.model.vendor`: The vendor of the model, if specified.\n- `gguf.model.authors`: The authors of the model, if specified.\n- `gguf.model.url`: The URL of the model, if specified.\n- `gguf.model.description`: The description of the model, if specified.\n- `gguf.model.licenses`: The licenses of the model, if specified.\n- `gguf.model.usage`: The usage of the model, default is `text-to-text`.\n\nAll labels can be overridden by the Dockerfile/GGUFPackerfile.\n\n#### QUANTIZE\n\nThe `QUANTIZE` instruction allows you to quantize a GGUF file.\n\n```dockerfile\n# syntax=gpustack/gguf-packer:latest\n\n# quantize a GGUF file from current stage\nCONVERT  --type=F16 /app/Qwen2-0.5B-Instruct /app/Qwen2-0.5B-Instruct.F16.gguf\nQUANTIZE --type=Q5_K_M /app/Qwen2-0.5B-Instruct.F16.gguf /app/Qwen2-0.5B-Instruct.Q5_K_M.gguf\n\n# quantize from other stage\nQUANTIZE --from=other-stage --type=Q5_K_M /app/Qwen2-0.5B-Instruct.F16.gguf /app/Qwen2-0.5B-Instruct.Q5_K_M.gguf\n\n# quantize from build context\nQUANTIZE --from=context --type=Q5_K_M /app/Qwen2-0.5B-Instruct.F16.gguf /app/Qwen2-0.5B-Instruct.Q5_K_M.gguf\n```\n\n##### Available Options\n\n- `QUANTIZE [--from=\u003cimage|stage|context\u003e] \u003csrc\u003e \u003cdest\u003e`, by default, the `QUANTIZE` instruction quantizes file from the\n  build context. The `QUANTIZE --from` flag lets you quantize file from an image, a build stage, or a named context\n  instead.\n- `QUANTIZE [--type=\u003ctype\u003e] \u003csrc\u003e \u003cdest\u003e`, specify the output type for `\u003cdest\u003e`,\n  referring [llama.cpp/quantize](https://github.com/ggerganov/llama.cpp/blob/c887d8b01726b11ea03dbcaa9d44fa74422d0076/examples/quantize/quantize.cpp#L19-L51),\n  upper case, default is `Q5_K_M`.\n- `QUANTIZE [--pure] \u003csrc\u003e \u003cdest\u003e`, indicate to disable k-quant mixtures and quantize all tensors to the same type.\n- `QUANTIZE [--imatrix=\u003cpath\u003e] \u003csrc\u003e \u003cdest\u003e`, introduce a file as importance matrix for quant optimizations.\n    + `QUANTIZE --imatrix=\u003cpath\u003e [--include-weights=\u003ctensor_name,...\u003e] \u003csrc\u003e \u003cdest\u003e`, specify to use the importance\n      matrix for this/these tensors.\n    + `QUANTIZE --imatrix=\u003cpath\u003e [--exclude-weights=\u003ctensor_name,...\u003e] \u003csrc\u003e \u003cdest\u003e`, specify to use the importance\n      matrix, but exclude for this/these tensors.\n- `QUANTIZE [--leave-output-tensor] \u003csrc\u003e \u003cdest\u003e`, indicate to not quantize the `output.weight` tensor.\n- `QUANTIZE [--output-tensor-type=\u003ctype\u003e] \u003csrc\u003e \u003cdest\u003e`, indicate the output tensor type,\n  referring [llama.cpp/ggml](https://github.com/ggerganov/llama.cpp/blob/c887d8b01726b11ea03dbcaa9d44fa74422d0076/ggml/src/ggml.c#L579-L974),\n  upper case.\n- `QUANTIZE [--token-embedding-type=\u003ctype\u003e] \u003csrc\u003e \u003cdest\u003e`, indicate the token embedding type,\n  referring [llama.cpp/ggml](https://github.com/ggerganov/llama.cpp/blob/c887d8b01726b11ea03dbcaa9d44fa74422d0076/ggml/src/ggml.c#L579-L974),\n  upper case.\n\n## Motivation\n\nIn the realm of Large Language Model (LLM) world, three projects stand\nout: [GGML](https://github.com/ggerganov/ggml), [LLaMA.Cpp](https://github.com/ggerganov/llama.cpp),\nand [Ollama](https://github.com/ollama/ollama). LLaMA.Cpp is built on GGML, and Ollama extends LLaMA.Cpp.\n\nGGML presents an alternative for engineers who prefer avoiding Python due to common issues like environment\nconfiguration, regional limitations, and installation complexities: a tensor computing library rooted in C/C++. With\nGGML's quantized model file, the [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) format, GGML\nempowers edge devices to run LLMs efficiently.\n\nLLaMa.cpp encapsulates various prominent LLM architectures and, with its\nflagship [llama-server](https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md), enables users to\nset up\nan OpenAI GPT-like service on their laptops with ease.\n\nHowever, managing a Chat LLM service involves numerous parameters and model file management challenges. Ollama addresses\nthis by introducing a [Model File](https://github.com/ollama/ollama/blob/main/docs/modelfile.md) that facilitates the\ndistribution of the model file and its parameters, much like a\nDockerfile.\n\nWhile the Ollama Model File is a Dockerfile-like tool for building and distributing Ollama models only, it does not\nalign well with the Cloud Native ecosystem. Let's explore the reasons why.\n\n### Docker Image\n\nTake, for example, the renowned DockerHub registry's [alpine](https://hub.docker.com/_/alpine/tags) image. We can\nretrieve its manifest using [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md):\n\n```shell\n$ crane manifest docker.io/library/alpine:latest | jq .\n{\n  \"manifests\": [\n    {\n      \"digest\": \"sha256:eddacbc7e24bf8799a4ed3cdcfa50d4b88a323695ad80f317b6629883b2c2a78\",\n      \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n      \"platform\": {\n        \"architecture\": \"amd64\",\n        \"os\": \"linux\"\n      },\n      \"size\": 528\n    },\n    {\n      \"digest\": \"sha256:5c7e326e3c8a8c51654a6c5d94dac98d7f6fc4b2a762d86aaf67b7e76a6aee46\",\n      \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n      \"platform\": {\n        \"architecture\": \"arm\",\n        \"os\": \"linux\",\n        \"variant\": \"v6\"\n      },\n      \"size\": 528\n    },\n    ...\n  ],\n  \"mediaType\": \"application/vnd.docker.distribution.manifest.list.v2+json\",\n  \"schemaVersion\": 2\n}\n```\n\nThe `mediaType` of `alpine:latest` image manifest\nis `application/vnd.docker.distribution.manifest.list.v2+json`, indicating a manifest list for multiple platforms.\nFor [OCI](https://opencontainers.org/) compatibility,\nthe corresponding `mediaType`\nis [\n`application/vnd.oci.image.index.v1+json`](https://github.com/opencontainers/image-spec/blob/main/media-types.md#applicationvndociimageindexv1json).\n\nDelving deeper into the `linux/amd64` platform manifest for `alpine:latest`:\n\n```shell\n$ crane manifest docker.io/library/alpine@sha256:eddacbc7e24bf8799a4ed3cdcfa50d4b88a323695ad80f317b6629883b2c2a78 | jq .\n{\n  \"schemaVersion\": 2,\n  \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n  \"config\": {\n    \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n    \"size\": 1471,\n    \"digest\": \"sha256:324bc02ae1231fd9255658c128086395d3fa0aedd5a41ab6b034fd649d1a9260\"\n  },\n  \"layers\": [\n    {\n      \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n      \"size\": 3622892,\n      \"digest\": \"sha256:c6a83fedfae6ed8a4f5f7cbb6a7b6f1c1ec3d86fea8cb9e5ba2e5e6673fde9f6\"\n    }\n  ]\n}\n``` \n\nHere, the `mediaType` is `application/vnd.docker.distribution.manifest.v2+json`, which translates\nto [\n`application/vnd.oci.image.manifest.v1+json`](https://github.com/opencontainers/image-spec/blob/main/media-types.md#applicationvndociimagemanifestv1json)\nfor [OCI](https://opencontainers.org/) compatibility.\n\nThe manifest includes a special `config` field, referencing the image configuration as a JSON object detailing the\nimage's settings.\n\n```shell\n$ crane blob docker.io/library/alpine@sha256:324bc02ae1231fd9255658c128086395d3fa0aedd5a41ab6b034fd649d1a9260 | jq .\n{\n  \"architecture\": \"amd64\",\n  \"config\": {\n    \"Hostname\": \"\",\n    \"Domainname\": \"\",\n    \"User\": \"\",\n    \"AttachStdin\": false,\n    \"AttachStdout\": false,\n    \"AttachStderr\": false,\n    \"Tty\": false,\n    \"OpenStdin\": false,\n    \"StdinOnce\": false,\n    \"Env\": [\n      \"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"\n    ],\n    \"Cmd\": [\n      \"/bin/sh\"\n    ],\n    \"Image\": \"sha256:3e25db883ea289c8b0d3006e7a6a03c56be09c50f03f6b228ba2fe74fd8194d2\",\n    \"Volumes\": null,\n    \"WorkingDir\": \"\",\n    \"Entrypoint\": null,\n    \"OnBuild\": null,\n    \"Labels\": null\n  },\n  \"container\": \"fc33bc50844a0d6cdfc682fcf85647ec60783decbd3850e58ab7e013ef892cfb\",\n  \"container_config\": {...},\n  \"created\": \"2024-07-22T22:26:43.778747613Z\",\n  \"docker_version\": \"23.0.11\",\n  \"history\": [...],\n  \"os\": \"linux\",\n  \"rootfs\": {...}\n}\n```\n\n### OCI Distribution\n\nThe [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/main/spec.md) defines a\nstandard for image registries that store and serve images. Examples of OCI Registries\ninclude [Docker Registry](https://docs.docker.com/registry/), [GitHub Container Registry](https://docs.github.com/en/packages/guides/about-github-container-registry), [Harbor](https://goharbor.io/), [Quay](https://www.quay.io/), [Azure Container Registry](https://azure.microsoft.com/en-us/services/container-registry/), [Google Container Registry](https://cloud.google.com/container-registry).\n\nInitially designed for storing container images, OCI Registries now also\nsupport [OCI Artifacts]((https://github.com/opencontainers/artifacts)), with Helm charts being a prime example. Helm\ncharts, once managed through [Git repositories](https://github.com/rancher/charts)\nor [released independently](https://github.com/prometheus-community/helm-charts/releases), can now\nbe [distributed as OCI Artifacts](https://helm.sh/docs/topics/registries/),\nstreamlining operations to a single OCI registry management task.\n\nWe can use `crane` to retrieve a Helm chart's manifest and download it as below:\n\n```shell\n$ crane manifest ghcr.io/argoproj/argo-helm/argo-cd:7.3.11 | jq .\n{\n  \"schemaVersion\": 2,\n  \"config\": {\n    \"mediaType\": \"application/vnd.cncf.helm.config.v1+json\",\n    \"digest\": \"sha256:42242c5441612b0cedb4cfc87ad5c257ec062ff6fab8c27557a072739eff0d71\",\n    \"size\": 940\n  },\n  \"layers\": [\n    {\n      \"mediaType\": \"application/vnd.cncf.helm.chart.provenance.v1.prov\",\n      \"digest\": \"sha256:2920df17e16b736156075e5859b7cb09d127d91bcaefdafd63860fb775609df9\",\n      \"size\": 1870\n    },\n    {\n      \"mediaType\": \"application/vnd.cncf.helm.chart.content.v1.tar+gzip\",\n      \"digest\": \"sha256:4249ea76c915bb04f4dda095e608004c08f13a7d0e0da2d1836ffc57a8592f7b\",\n      \"size\": 168713\n    }\n  ]\n}\n\n$ crane pull ghcr.io/argoproj/argo-helm/argo-cd:7.3.11 argo-cd.tar\n\n$ tar xf argo-cd.tar\n\n$ ls -alth .\ntotal 760\ndrwxr-xr-x   7 gpustack  wheel   224B Jul 26 13:26 .\ndrwxrwxrwt  53 root     wheel   1.7K Jul 26 13:26 ..\n-rw-r--r--@  1 gpustack  wheel   172K Jul 26 13:23 argo-cd.tar\n-rw-r--r--   1 gpustack  wheel   1.8K Jan  1  1970 2920df17e16b736156075e5859b7cb09d127d91bcaefdafd63860fb775609df9.tar.gz\n-rw-r--r--   1 gpustack  wheel   165K Jan  1  1970 4249ea76c915bb04f4dda095e608004c08f13a7d0e0da2d1836ffc57a8592f7b.tar.gz\n-rw-r--r--   1 gpustack  wheel   302B Jan  1  1970 manifest.json\n-rw-r--r--   1 gpustack  wheel   940B Jan  1  1970 sha256:42242c5441612b0cedb4cfc87ad5c257ec062ff6fab8c27557a072739eff0d71\n```\n\n### Ollama Model\n\nExamining the Ollama model, specifically the [LLaMa3.1:8B](https://ollama.com/library/llama3.1) model, we initially\nassumed it conformed to the standard OCI Registry.\n\nHowever, attempts to retrieve its manifest with `crane` resulted in a 404 error, indicating non-compliance with OCI\nstandards.\n\n```shell\n$ crane manifest ollama.com/library/llama3.1:8b | jq .\nError: fetching manifest ollama.com/library/llama3.1:8b: GET https://ollama.com/v2/: unexpected status code 404 Not Found: 404 page not found\n\n$ curl https://ollama.com/v2/library/llama3.1/manifests/8b | jq .\n{\n  \"schemaVersion\": 2,\n  \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n  \"config\": {\n    \"digest\": \"sha256:e711233e734332fe5f8a09b2407fb5a083e39ca7e0ba90788026414cd4c059af\",\n    \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n    \"size\": 485\n  },\n  \"layers\": [\n    {\n      \"digest\": \"sha256:87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87\",\n      \"mediaType\": \"application/vnd.ollama.image.model\",\n      \"size\": 4661211808\n    },\n    {\n      \"digest\": \"sha256:8cf247399e57085e6b34c345ebea38c1aa3e2b25c8294eecb746dd7b01dd9079\",\n      \"mediaType\": \"application/vnd.ollama.image.template\",\n      \"size\": 1692\n    },\n    {\n      \"digest\": \"sha256:f1cd752815fcf68c3c2e73b2b00b5396c5dffb9eebe49567573f275f9ec85fcd\",\n      \"mediaType\": \"application/vnd.ollama.image.license\",\n      \"size\": 12321\n    },\n    {\n      \"digest\": \"sha256:56bb8bd477a519ffa694fc449c2413c6f0e1d3b1c88fa7e3c9d88d3ae49d4dcb\",\n      \"mediaType\": \"application/vnd.ollama.image.params\",\n      \"size\": 96\n    }\n  ]\n}\n\n$ crane blob ollama.com/library/llama3.1@sha256:e711233e734332fe5f8a09b2407fb5a083e39ca7e0ba90788026414cd4c059af | jq .\nError: pulling layer ollama.com/library/llama3.1@sha256:e711233e734332fe5f8a09b2407fb5a083e39ca7e0ba90788026414cd4c059af: GET https://ollama.com/v2/: unexpected status code 404 Not Found: 404 page not found\n\n$ curl https://ollama.com/v2/library/llama3.1/blobs/sha256:e711233e734332fe5f8a09b2407fb5a083e39ca7e0ba90788026414cd4c059af\n\u003ca href=\"https://dd20bb891979d25aebc8bec07b2b3bbc.r2.cloudflarestorage.com/ollama/docker/registry/v2/blobs/sha256/e7/e711233e734332fe5f8a09b2407fb5a083e39ca7e0ba90788026414cd4c059af/data?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026amp;X-Amz-Credential=66040c77ac1b787c3af820529859349a%2F20240725%2Fauto%2Fs3%2Faws4_request\u0026amp;X-Amz-Date=20240725T145758Z\u0026amp;X-Amz-Expires=86400\u0026amp;X-Amz-SignedHeaders=host\u0026amp;X-Amz-Signature=262290faed14709a9c0faf9f8b8d567d3fec84731b2877852d275bc979f0530f\"\u003eTemporary Redirect\u003c/a\u003e.\n\n$ curl -L https://ollama.com/v2/library/llama3.1/blobs/sha256:e711233e734332fe5f8a09b2407fb5a083e39ca7e0ba90788026414cd4c059af\n{\"model_format\":\"gguf\",\"model_family\":\"llama\",\"model_families\":[\"llama\"],\"model_type\":\"8.0B\",\"file_type\":\"Q4_0\",\"architecture\":\"amd64\",\"os\":\"linux\",\"rootfs\":{\"type\":\"layers\",\"diff_ids\":[\"sha256:87048bcd55216712ef14c11c2c303728463207b165bf18440b9b84b07ec00f87\",\"sha256:11ce4ee3e170f6adebac9a991c22e22ab3f8530e154ee669954c4bc73061c258\",\"sha256:f1cd752815fcf68c3c2e73b2b00b5396c5dffb9eebe49567573f275f9ec85fcd\",\"sha256:56bb8bd477a519ffa694fc449c2413c6f0e1d3b1c88fa7e3c9d88d3ae49d4dcb\"]}}\n```\n\n**The Ollama model's distribution method is not a good practice.** Why?\n\nHelm chart packages are usually very small, and the network cost of deploying OCI in a nearby network is very low, so\nwe don't need to disguise the Helm chart as a container image.\n\nHowever, large model files, such as the 4.3GB LLaMa3.1:8B model, incur significant network costs when distributed\nwithout compression.\n\nMoreover, we can see many Ollama **pre-download** images in DockerHub, which is inefficient, wasting storage and network\nresources.\n\n[![](./docs/assets/dockerhub-ollama-model-cache.jpg)](https://hub.docker.com/search?q=ollama)\n\nIn conclusion, while Ollama has gained popularity in managing LLM distributions, its approach diverges from best\npractices for OCI Artifacts. GGUF Packer, on the other hand, offers a contemporary solution that adheres to OCI\nstandards, reducing both network and storage overhead.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpustack%2Fgguf-packer-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpustack%2Fgguf-packer-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpustack%2Fgguf-packer-go/lists"}