{"id":25122617,"url":"https://github.com/derekwin/oneinfer","last_synced_at":"2025-10-27T10:02:36.465Z","repository":{"id":274953484,"uuid":"924541985","full_name":"derekwin/OneInfer","owner":"derekwin","description":"Inference model all in one.","archived":false,"fork":false,"pushed_at":"2025-02-20T02:48:06.000Z","size":977,"stargazers_count":3,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-20T03:29:40.861Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derekwin.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-30T07:46:54.000Z","updated_at":"2025-02-20T02:48:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"d5acffa4-b3ec-476c-8bd1-f4162f6b6591","html_url":"https://github.com/derekwin/OneInfer","commit_stats":null,"previous_names":["derekwin/oneinfer"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2FOneInfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2FOneInfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2FOneInfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derekwin%2FOneInfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derekwin","download_url":"https://codeload.github.com/derekwin/OneInfer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246850329,"owners_count":20844036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-08T07:14:51.391Z","updated_at":"2025-10-27T10:02:36.387Z","avatar_url":"https://github.com/derekwin.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](./assets/logos/oneinferlogo-long.png)\n\n# OneInfer \n\nInference model all in one. Easily Large Model Inference on Your Local Machine.\n\nA lightweight CLI tool that simplifies deploying and managing machine learning models locally. Designed for seamless integration of popular model sources (Hugging Face Hub, ModelScope, local files) with optimized CPU/GPU execution via llama.cpp backend (more backends coming soon).\n\n[中文文档](./readme_zh.md)\n\n## Difference from Ollama\n\nCompared to Ollama, OneInfer offers more flexibility and choices:\n\n1. **Broader model platform support**: OneInfer supports downloading models from multiple platforms, including Hugging Face and ModelScope, whereas Ollama is limited to its platform.\n2. **Support for diverse inference backends**: OneInfer will support various inference backends, not only for language models but also for visual models and other non-LLM models, providing users with more options and freedom.\n\nWith OneInfer, users can enjoy the convenience of portable local deployment while benefiting from more platform and model choices, offering a richer and more flexible experience.\n\n## Roadmap\n- [x] Ollama-like model management\n- [x] Using any pre-trained models from Hugging Face or ModelScope\n- [x] Support for serving LLM models using `llama.cpp` (.gguf models)\n- [x] Web UI\n- [x] Ready-to-use packaged application, allowing users to download and use it immediately without the need for compilation.\n- [ ] More inference backends to be supported\n- [ ] More types of models to be supported\n\n## Requirements\n- Python3 for downloading models from Hugging Face and ModelScope.\n- Go 1.18+ for building oneinfer.\n- git for downloading other repos.\n\n## Install pre-built Package\nCheckout https://github.com/derekwin/OneInfer/releases and install oneinfer by following the instructions provided.\n\n## Build and Install\nTo build and install OneInfer:\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/derekwin/oneinfer.git\n   cd oneinfer\n   ```\n2. Run the `make` command:\n   ```bash\n   make USE_CUDA=1\n   ```\n\n   for different GPU/backend:\n   - **USE_BLAS=1**: For general CPU usage.\n   - **USE_CUDA=1**: For NVIDIA GPUs.\n   - **USE_MUSA=1**: For Meta's AI accelerators.\n   - **USE_HIP=1**: For AMD GPUs.\n   - **USE_CANN=1**: For Huawei Ascend AI accelerators.\n   - **USE_Vulkan=1**: For GPUs that support Vulkan, offering efficient parallel computation.\n   - **USE_Metal=1**: For Apple devices.\n   - **USE_SYCL=1**: For a variety of heterogeneous devices (including CPU, GPU, FPGA) using oneAPI.\n\n3. Install/Uninstall the binary:\n   ```bash\n   sudo bash install.sh/uninstall.sh\n   ```\n\nor you can run `bash allinnoe.sh` directly.\n\n## Usage\n\n### model add\nAdd a model to OneInfer. This can either be from ModelScope, Hugging Face, or a local file.\n(You can get repo name and file name from huggingface or modelscope website.)\n\n```bash\noneinfer add \u003cmodel_repo\u003e \u003cplatform_name\u003e \u003cfile_name\u003e\n```\n\n#### Download a model from ModelScope\nExample for downloading the `DeepSeek` model from ModelScope:\n\n```bash\n# deepseek r1 from unsloth\noneinfer add unsloth/DeepSeek-R1-Distill-Qwen-32B-GGUF modelscope DeepSeek-R1-Distill-Qwen-32B-Q5_K_M.gguf\noneinfer add unsloth/DeepSeek-R1-Distill-Qwen-7B-GGUF modelscope DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf\n```\n\n#### Download a model from Hugging Face\nExample for downloading a model from Hugging Face:\n\n```bash\noneinfer add RepoId huggingface modelname\n```\n\n#### Add a local model\nExample for adding a local model file:\n\n```bash\noneinfer add localmodelname local \n# then type file path\n./test/fakemodel.bin\n```\n\n### model list\nList all available models that have been added to OneInfer.\n\n```bash\noneinfer ls\n```\n\n### model remove\nRemove a specific model by its name.\n\n```bash\noneinfer rm \u003cmodel_name\u003e\n```\n\n## Run as Server\nFirst run OneInfer as a background server to manage model serving:\n\n```bash\nnohup oneinfer serve \u0026\n```\n\nThis will start a OneInfer server with a web UI in the background for managing model serving. Open your browser and navigate to \"http://\u003cyour_server_ip\u003e:9090\" to access the web UI.\n\n![](./assets/webui.png)\n\n## Manage as Client\n\n### Start a Model\nStart a specific model by specifying its name. You can also define the host and port for the model server.\n\n```bash\noneinfer run modelname [-p (default 8080)] [-H (default 127.0.0.1)]\n```\n\nFor example:\n\n```bash\noneinfer run DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf\n```\n\nThis will call the OneInfer server and start the model server.\n\n### Status of All Running Models\nView the status of all running models:\n\n```bash\noneinfer ps\n```\n\nThis will list the currently running models along with their status.\n\n### Stop a Model\nStop a running model by its unique identifier (UID):\n\n```bash\noneinfer stop \u003cmodel_uid\u003e\n```\n\n### Stop the Server\nStop the entire OneInfer server:\n\n```bash\noneinfer stop serve\n```\n\nThis will stop the server and all running models.\n\n---\n\n## Troubleshooting\n\n- If you encounter any issues with model downloads, ensure that Python 3 is installed and working properly for the ModelScope and Hugging Face integrations.\n- If a model cannot be started, check if the port is already in use or if any dependency is missing.\n\nFor detailed help on each command, use the `--help` flag:\n```bash\noneinfer --help\n```\n\n---\n\n## License\nOneInfer is open-source software licensed under the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderekwin%2Foneinfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderekwin%2Foneinfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderekwin%2Foneinfer/lists"}