{"id":50720102,"url":"https://github.com/barrel-platform/barrel_inference","last_synced_at":"2026-06-09T23:01:44.295Z","repository":{"id":359871033,"uuid":"1247807971","full_name":"barrel-platform/barrel_inference","owner":"barrel-platform","description":"OTP-native LLM inference runtime with token-exact tiered KV cache, plus an OpenAI/Anthropic/Ollama-compatible HTTP daemon. Erlang/OTP + llama.cpp.","archived":false,"fork":false,"pushed_at":"2026-06-09T20:42:50.000Z","size":7726,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-09T21:09:36.512Z","etag":null,"topics":["erlang","inference","kv-cache","llama-cpp","llm","otp"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/barrel-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-23T20:07:01.000Z","updated_at":"2026-06-09T20:43:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/barrel-platform/barrel_inference","commit_stats":null,"previous_names":["barrel-platform/barrel_inference"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/barrel-platform/barrel_inference","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrel-platform%2Fbarrel_inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrel-platform%2Fbarrel_inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrel-platform%2Fbarrel_inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrel-platform%2Fbarrel_inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/barrel-platform","download_url":"https://codeload.github.com/barrel-platform/barrel_inference/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barrel-platform%2Fbarrel_inference/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34129072,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlang","inference","kv-cache","llama-cpp","llm","otp"],"created_at":"2026-06-09T23:01:43.539Z","updated_at":"2026-06-09T23:01:44.289Z","avatar_url":"https://github.com/barrel-platform.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Barrel Inference\n\nOTP-native LLM inference for the BEAM: dirty NIFs over `llama.cpp`, supervised\nper-model processes, and a byte-exact tiered KV cache, with an\nOpenAI/Anthropic/Ollama-compatible HTTP daemon on top.\n\nInference as a first-class OTP citizen, not a Python sidecar. The wedge is\nsupervision, per-model queues, and cancel-on-disconnect, with the cache more\nwarm state than fits in RAM.\n\n## Layout\n\nA rebar3 umbrella; each app is a separately publishable Hex package and the\nrepo is versioned as a whole.\n\n| App | What it is |\n|-----|------------|\n| [`apps/barrel_inference`](apps/barrel_inference) | The runtime: dirty NIFs over llama.cpp, supervised model processes, byte-exact tiered KV cache. |\n| [`apps/barrel_inference_server`](apps/barrel_inference_server) | The API daemon: OpenAI-, Anthropic-, and Ollama-compatible HTTP, model registry, per-model queues, keep-alive, metrics. |\n| [`apps/barrel_inference_cli`](apps/barrel_inference_cli) | The `barrel-inference` CLI: `serve` boots the daemon; `pull`/`run`/`ps`/`rm` drive a running one over HTTP. |\n\nA distributed control plane (`barrel_inference_cluster`: routing, cache-aware\nplacement, node discovery) is a planned follow-up.\n\n## Build\n\n    rebar3 compile              # builds the NIF (vendored llama.cpp via cmake)\n    rebar3 as prod release      # the barrel_inference_server daemon release\n    rebar3 escriptize           # the barrel-inference CLI\n\nRequires Erlang/OTP 28 and rebar3 3.25+, plus cmake and a C/C++ toolchain for\nthe NIF. See each app's README for the public API and configuration.\n\n## Run\n\n    barrel-inference serve                 # start the API server\n    barrel-inference pull \u003cmodel\u003e          # fetch a model\n    barrel-inference run \u003cmodel\u003e \"hello\"   # one-shot completion\n    barrel-inference ps                    # list loaded models\n\nOr with Docker:\n\n    docker compose up\n\n## Documentation\n\nOne site covers the whole project, for both operators and contributors:\n\u003chttps://barrel-platform.github.io/barrel_inference/\u003e. It is a mkdocs build at\nthe repo root (`mkdocs.yml`, `docs/`) that surfaces each app's guides in place.\n\nFunction-level API reference ships per package on hexdocs:\n[barrel_inference](https://hexdocs.pm/barrel_inference) and\n[barrel_inference_server](https://hexdocs.pm/barrel_inference_server).\n\nBuild the site locally:\n\n    pip install -r docs-requirements.txt\n    mkdocs serve\n\n## License\n\nMIT. Part of the [barrel-platform](https://github.com/barrel-platform) project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarrel-platform%2Fbarrel_inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbarrel-platform%2Fbarrel_inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarrel-platform%2Fbarrel_inference/lists"}