{"id":18093789,"url":"https://github.com/onicai/llama_cpp_canister","last_synced_at":"2025-03-29T06:31:15.447Z","repository":{"id":239415052,"uuid":"770576576","full_name":"onicai/llama_cpp_canister","owner":"onicai","description":"llama.cpp for the Internet Computer","archived":false,"fork":false,"pushed_at":"2025-03-08T00:19:55.000Z","size":31620,"stargazers_count":42,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-08T00:25:44.589Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onicai.png","metadata":{"files":{"readme":"README-0001-b841d0.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-11T19:38:40.000Z","updated_at":"2025-03-08T00:19:59.000Z","dependencies_parsed_at":"2025-01-16T21:19:21.191Z","dependency_job_id":"02cb56cb-c44e-45b9-afdd-4dc9a0761c64","html_url":"https://github.com/onicai/llama_cpp_canister","commit_stats":null,"previous_names":["onicai/llama_cpp_canister"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onicai%2Fllama_cpp_canister","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onicai%2Fllama_cpp_canister/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onicai%2Fllama_cpp_canister/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onicai%2Fllama_cpp_canister/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onicai","download_url":"https://codeload.github.com/onicai/llama_cpp_canister/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246150270,"owners_count":20731414,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-31T19:01:38.225Z","updated_at":"2025-03-29T06:31:15.431Z","avatar_url":"https://github.com/onicai.png","language":"C++","readme":"# DETAILS FOR UPGRADE from llama.cpp sha `615212` to `b841d0`\n\n### cpp_paths\n\n#### main_.cpp\n`meld main_.cpp llama_cpp_onicai_fork/examples/main/main.cpp`\n- use `main_` instead of `main`\n- A few items related to console \u0026 ctrl+C need to be outcommented\n\n\n#### llama_cpp_onicai_fork/src/llama.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw std::runtime_error(format` with `IC_API::trap(std::string(\"RUNTIME ERROR: \") + format`\n- replace `throw` with `IC_API::trap`\n- outcomment `try - catch`. The program will abrupt in case of thrown exceptions.\n- outcomment threading related items:\n  - `#include \u003cfuture\u003e`\n  - `#include \u003cmutex\u003e`\n  - `#include \u003cthread\u003e`\n- outcomment these functions completely:\n  - `llama_tensor_quantize_internal`\n  - `llama_model_quantize_internal`\n\n\n#### llama_cpp_onicai_fork/src/llama-vocab.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw std::runtime_error(format` with `IC_API::trap(std::string(\"RUNTIME ERROR: \") + format`\n- outcomment `try - catch`. The program will abrupt in case of thrown exceptions.\n- add a check on `llama_token_bos(model)`, else the llama2.c models never stop generating:\n  ```\n  bool llama_token_is_eog_impl(const struct llama_vocab \u0026 vocab, llama_token token) {\n      return token != -1 \u0026\u0026 (\n          token == llama_token_eos_impl(vocab) ||\n          token == llama_token_eot_impl(vocab) || \n          token == llama_token_bos_impl(vocab) // ICPP-PATCH: the llama2.c model predicts bos without first predicting an eos\n      );\n  }\n  ```\n\n#### llama_cpp_onicai_fork/src/llama-grammar.cpp\nNo changes needed\n\n#### llama_cpp_onicai_fork/src/llama-sampling.cpp\nNo changes needed\n\n#### llama_cpp_onicai_fork/src/unicode-data.cpp\n- no modifications needed for the IC\n\n#### llama_cpp_onicai_fork/src/unicode.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw` with `IC_API::trap`\n\n#### llama_cpp_onicai_fork/common/json-schema-to-grammar.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw` with `IC_API::trap`\n- outcomment `try - catch`. The program will abrupt in case of thrown exceptions.\n\n\n#### llama_cpp_onicai_fork/common/build-info.cpp\n- run this command to create it:\n```\nmake build-info-cpp-wasm\n``` \n\n#### llama_cpp_onicai_fork/common/grammar-parser.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw` with `IC_API::trap`\n- outcomment `try - catch`. The program will abrupt in case of thrown exceptions.\n\n#### llama_cpp_onicai_fork/common/sampling.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw` with `IC_API::trap`\n\n#### llama_cpp_onicai_fork/common/common.cpp\n- add `#include \"ic_api.h\"`\n- replace `throw` with `IC_API::trap`\n- outcomment all code related to `\u003cpthread.h\u003e`\n- outcomment `try - catch`. The program will abrupt in case of thrown exceptions.\n- outcomment `std::getenv`\n\n\n---\n### c_paths\n\n#### llama_cpp_onicai_fork/ggml/src/ggml.c\n- outcomment all code related to signals\n  - `#include \u003csignal.h\u003e`\n- Many threading outcomments. \n\n#### llama_cpp_onicai_fork/ggml/src/ggml-alloc.c\nNo updates needed for icpp-pro\n\n#### llama_cpp_onicai_fork/ggml/src/ggml-backend.c\nNo updates needed for icpp-pro\n\n#### llama_cpp_onicai_fork/ggml/src/ggml-quants.c\nNo updates needed for icpp-pro\n\n#### llama_cpp_onicai_fork/ggml/src/ggml-aarch64.c\nNo updates needed for icpp-pro\n\n---\n### headers to modify\n\n#### llama_cpp_onicai_fork/common/log.h\n- `#include \u003cthread\u003e`\n- Some other threading code\n\n#### llama_cpp_onicai_fork/common/common.h\n- `#include \u003cthread\u003e`","funding_links":[],"categories":["Decentralized AI"],"sub_categories":["TON"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonicai%2Fllama_cpp_canister","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonicai%2Fllama_cpp_canister","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonicai%2Fllama_cpp_canister/lists"}