{"id":13808609,"url":"https://github.com/noizu-labs-ml/ex_llama","last_synced_at":"2025-05-14T02:32:10.320Z","repository":{"id":232888014,"uuid":"785361299","full_name":"noizu-labs-ml/ex_llama","owner":"noizu-labs-ml","description":"Elixir NIFs for interacting with llama_cpp.rust managed GGUF models.","archived":false,"fork":false,"pushed_at":"2024-04-12T09:30:22.000Z","size":47,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-04T01:09:52.289Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/noizu-labs-ml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-11T18:27:35.000Z","updated_at":"2024-07-21T15:31:49.000Z","dependencies_parsed_at":"2024-04-16T13:05:08.755Z","dependency_job_id":null,"html_url":"https://github.com/noizu-labs-ml/ex_llama","commit_stats":null,"previous_names":["noizu-labs-ml/ex_llama"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noizu-labs-ml%2Fex_llama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noizu-labs-ml%2Fex_llama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noizu-labs-ml%2Fex_llama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noizu-labs-ml%2Fex_llama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/noizu-labs-ml","download_url":"https://codeload.github.com/noizu-labs-ml/ex_llama/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225273250,"owners_count":17448074,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T01:01:47.352Z","updated_at":"2024-11-19T00:30:53.468Z","avatar_url":"https://github.com/noizu-labs-ml.png","language":"Elixir","funding_links":[],"categories":["Artificial Intelligence","Clients/Wrappers/Nifs"],"sub_categories":["ExLLama: llama.cpp nif extensions"],"readme":"ExLLama: LlammaCpp.rs NIF wrapper for Elixir/Erlang.\n=======\n\nThis is an Alpha Library for loading and interacting with models via the llama_cpp rust client exposed as nif extensions. \nInspired By [llama_cpp_ex](https://github.com/jeregrine/llama_cpp_ex)\n\n\n## Getting Started\n1. Add the `ex_llama` dependency to your `mix.exs` file:\n\n```elixir\ndef deps do \n  [\n    {:ex_llama, \"~\u003e 0.0.1\"}\n  ]\n\nend\n```\n\n\n## Chat Completion \nAs of this build only `\u003c|role|\u003emesssage\u003c/s\u003e` format chat completion is supported, such as used by tiny llama. \n\n\n```elixir \n\n    {:ok, llama} = ExLLama.load_model(\"./test/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf\")\n    thread = [\n      %{role: :user, content: \"Say Hello. And only hello. Example \\\"Hello\\\".\"},\n      %{role: :assistant, content: \"Hello\"},\n      %{role: :user, content: \"Repeat what you just said.\"},\n      %{role: :assistant, content: \"Hello\"},\n      %{role: :user, content: \"Say Goodbye.\"},\n      %{role: :assistant, content: \"Goodbye\"},\n      %{role: :user, content: \"Say Apple.\"},\n      %{role: :assistant, content: \"Apple\"},\n      %{role: :user, content: \"What did you just say?.\"},\n    ]\n\n    {:ok, response} = ExLLama.chat_completion(llama, thread, %{seed: 2})\n    # response = %{\n    #         choices: [\n    #           %{reason: :end, role: \"assistant\", content: \"Apple\"},\n    #           %{reason: :end, role: \"assistant\", content: \"Apple\"},\n    #           %{reason: :end, role: \"assistant\", content: \"Apple\"}\n    #         ]\n    # }\n\n```\n\n\n## Simple Completion (direct)\n```elixir\n    {:ok, llama} = ExLLama.load_model(\"./test/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf\")\n    {:ok, options} = ExLLama.Session.default_options()\n    {:ok, session} = ExLLama.create_session(llama, %{options| seed: 2})\n    ExLLama.advance_context(session, \"\u003c|user|\u003e\\n Say Hello. And only hello. Example \\\"Hello\\\".\u003c/s\u003e\\n\u003c|assistant|\u003e\\n Hello\u003c/s\u003e\\n\u003c|user|\u003e\\n Repeat what you just said.\u003c/s\u003e\\n\u003c|assistant|\u003e\\n Hello\u003c/s\u003e\\n\u003c|user|\u003e\\n Say Goodbye.\u003c/s\u003e\\n\u003c|assistant|\u003e\\n\")\n    {:ok, response} = ExLLama.completion(session, 512, \"\u003c/s\u003e\\n*\")\n    response = String.trim_leading(response)\n    # \"Goodbye.\u003c/s\u003e\"\n```\n\n## Streaming Completion (final mechanism will be replaced with a Stream\n```elixir\n\n  def receive_text(acc \\\\ []) do\n    receive do\n      x = {:ok, _} -\u003e Enum.reverse([x|acc])\n      x = {:error, _} -\u003e Enum.reverse([x|acc])\n      :fin -\u003e\n        Enum.reverse(acc)\n      x -\u003e\n        receive_text([x | acc])\n    end\n  end\n\n#...\n    {:ok, llama} = ExLLama.load_model(\"./test/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf\")\n    {:ok, options} = ExLLama.Session.default_options()\n    {:ok, session} = ExLLama.create_session(llama, %{options| seed: 2})\n    ExLLama.advance_context(session, \"\u003c|user|\u003e\\n Say Hello. And only hello. Example \\\"Hello\\\".\u003c/s\u003e\\n\u003c|assistant|\u003e\\n Hello\u003c/s\u003e\\n\u003c|user|\u003e\\n Repeat what you just said.\u003c/s\u003e\\n\u003c|assistant|\u003e\\n Hello\u003c/s\u003e\\n\u003c|user|\u003e\\n Say Goodbye.\u003c/s\u003e\\n\u003c|assistant|\u003e\\n\")\n    ExLLama.Session.start_completing_with(session, %{max_tokens: 512})\n    receive_text()\n\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoizu-labs-ml%2Fex_llama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnoizu-labs-ml%2Fex_llama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoizu-labs-ml%2Fex_llama/lists"}