{"id":29092406,"url":"https://github.com/parisneo/openllmserverscript","last_synced_at":"2025-06-28T07:06:46.791Z","repository":{"id":293252075,"uuid":"983428711","full_name":"ParisNeo/openllmserverscript","owner":"ParisNeo","description":"A simple script to install and run openllm on a server","archived":false,"fork":false,"pushed_at":"2025-05-14T11:32:44.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-14T12:48:06.154Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ParisNeo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-14T11:18:56.000Z","updated_at":"2025-05-14T11:32:47.000Z","dependencies_parsed_at":"2025-05-15T02:46:34.088Z","dependency_job_id":null,"html_url":"https://github.com/ParisNeo/openllmserverscript","commit_stats":null,"previous_names":["parisneo/openllmserverscript"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ParisNeo/openllmserverscript","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fopenllmserverscript","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fopenllmserverscript/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fopenllmserverscript/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fopenllmserverscript/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ParisNeo","download_url":"https://codeload.github.com/ParisNeo/openllmserverscript/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ParisNeo%2Fopenllmserverscript/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262389494,"owners_count":23303343,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-28T07:06:42.602Z","updated_at":"2025-06-28T07:06:46.784Z","avatar_url":"https://github.com/ParisNeo.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenLLM Server Setup Script\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Bash Version](https://img.shields.io/badge/bash-%3E%3D4.x-blue.svg)](https://www.gnu.org/software/bash/)\n[![Contributions Welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/ParisNeo/openllmserverscript/issues)\n\u003c!-- Optional: Add a badge for your latest release if you create tags/releases --\u003e\n\u003c!-- [![Latest Release](https://img.shields.io/github/v/release/ParisNeo/openllmserverscript)](https://github.com/ParisNeo/openllmserverscript/releases) --\u003e\n\nThis repository contains `setup_openllm.sh`, a Bash script designed to automate the installation and configuration of [OpenLLM](https://github.com/bentoml/OpenLLM) as a server on Debian/Ubuntu-based systems. It simplifies setting up multiple LLMs to run simultaneously, each as a dedicated systemd service.\n\nThe script supports:\n*   Serving models downloaded from Hugging Face Hub.\n*   Importing and serving your own local model files (GGUF or Hugging Face directory format).\n\n## Features\n\n*   **Dedicated User \u0026 Group:** Creates a system user `openllm` and group `openllm_data` for running services and managing models securely.\n*   **Custom Model Directory:** Allows you to specify a directory for storing models and the Python virtual environment.\n*   **Python Virtual Environment:** Isolates OpenLLM and its dependencies.\n*   **Local Model Import:** Prompts to import local GGUF files or Hugging Face model directories.\n*   **Hugging Face Hub Integration:** Lists available models from the Hub and allows selection for installation.\n*   **Multiple Model Services:** Configures each selected model to run as a separate systemd service on a unique port (starting from 3000).\n*   **Offline-First for Local Models:** Sets `TRANSFORMERS_OFFLINE=1` and `HF_HUB_OFFLINE=1` for services running local models to prevent unnecessary network calls.\n*   **Permission Management:** Sets appropriate permissions for the model directory, granting access to the `openllm` user and the user who ran the script (via the `openllm_data` group).\n*   **Dependency Installation:** Automatically installs `python3-venv` and `python3-pip` if not present (for Debian/Ubuntu).\n\n## Prerequisites\n\n*   A Debian/Ubuntu-based Linux system.\n*   `sudo` or root privileges to run the script.\n*   `curl` or `wget` to download the script (or clone the repository).\n*   **Recommended:** `jq` for robust parsing of model lists from Hugging Face Hub (`sudo apt-get install jq`). The script has a basic fallback if `jq` is not found.\n*   Sufficient disk space for models and Python environment.\n*   Sufficient RAM/VRAM depending on the models you intend to run.\n\n## Installation \u0026 Usage\n\n1.  **Download the script:**\n    ```bash\n    curl -LO https://raw.githubusercontent.com/ParisNeo/openllmserverscript/main/setup_openllm.sh\n    # OR using wget:\n    # wget https://raw.githubusercontent.com/ParisNeo/openllmserverscript/main/setup_openllm.sh\n    ```\n\n2.  **Make it executable:**\n    ```bash\n    chmod +x setup_openllm.sh\n    ```\n\n3.  **Run the script with sudo:**\n    ```bash\n    sudo ./setup_openllm.sh\n    ```\n\n4.  **Follow the prompts:**\n    *   **Model Directory:** You'll be asked to specify the directory where models and the OpenLLM virtual environment will be stored (default: `/opt/openllm_models`).\n    *   **Import Local Models:** The script will ask if you want to import local models.\n        *   If \"yes\", provide a custom ID for OpenLLM to refer to the model, the full path to the model file (e.g., `/path/to/model.gguf`) or directory, and specify if it's a GGUF model.\n    *   **Hugging Face Hub Models:** The script will then ask if you want to list and select models from Hugging Face Hub.\n        *   If \"yes\", a list of available models will be shown (if `jq` is installed, this list is more reliable). Enter the space-separated IDs of the models you wish to install (e.g., `mistralai/Mistral-7B-Instruct-v0.1 meta-llama/Llama-2-7b-chat-hf`).\n        *   **Note:** For gated models on Hugging Face Hub, ensure you have logged in via `huggingface-cli login` or set the `HF_TOKEN` environment variable *before* running the script if you intend to download them. The script itself does not handle `HF_TOKEN` input.\n\n5.  **Service Creation:**\n    The script will create systemd service files (e.g., `openllm-your-model-id.service`) for each selected model in `/etc/systemd/system/`. It will then enable and attempt to start these services.\n\n## Post-Installation\n\n*   **Check Service Status:**\n    ```bash\n    systemctl status openllm-\u003cyour-sanitized-model-id\u003e.service\n    # To list all openllm services:\n    systemctl list-units 'openllm-*.service'\n    ```\n\n*   **View Logs:**\n    Model downloads (for Hub models) and initialization can take time. Monitor the logs:\n    ```bash\n    journalctl -u openllm-\u003cyour-sanitized-model-id\u003e.service -f\n    ```\n    Replace `\u003cyour-sanitized-model-id\u003e` with the model ID used in the service name (e.g., `openllm-mistralai-Mistral-7B-Instruct-v0.1.service`).\n\n*   **Accessing Models:**\n    Each model service will be running on a different port, starting from `3000` and incrementing (3001, 3002, etc.). You can typically access the OpenAI-compatible API at `http://localhost:\u003cport\u003e/v1/`.\n\n*   **File Access Permissions:**\n    The user who ran the `setup_openllm.sh` script (the `sudo` user) is added to the `openllm_data` group. This grants them read/write access to the model directory (e.g., `/opt/openllm_models`). You may need to **log out and log back in** for these group changes to take effect for your user.\n\n*   **Local Model File Permissions:**\n    If you imported local models, ensure the `openllm` user has persistent read access to their original paths, as OpenLLM might symlink or reference them. The imported model data itself (or symlinks) will be managed within the `OPENLLM_HOME` directory (your chosen model directory).\n\n## Managing Services\n\nYou can use standard `systemctl` commands to manage the services:\n*   `sudo systemctl stop openllm-\u003cmodel-id\u003e.service`\n*   `sudo systemctl start openllm-\u003cmodel-id\u003e.service`\n*   `sudo systemctl restart openllm-\u003cmodel-id\u003e.service`\n*   `sudo systemctl enable openllm-\u003cmodel-id\u003e.service` (to start on boot)\n*   `sudo systemctl disable openllm-\u003cmodel-id\u003e.service` (to prevent starting on boot)\n\n## Configuration Details\n\n*   **User:** `openllm`\n*   **Group:** `openllm_data`\n*   **Default Model Directory:** `/opt/openllm_models`\n*   **Virtual Environment:** Located at `\u003cMODEL_DIR\u003e/.venv`\n*   **OpenLLM Executable in venv:** `\u003cMODEL_DIR\u003e/.venv/bin/openllm`\n*   **Environment Variables for Services:**\n    *   `OPENLLM_HOME`: Set to your chosen model directory.\n    *   `TRANSFORMERS_OFFLINE=1`: Ensures OpenLLM doesn't try to fetch from Hugging Face Hub unnecessarily.\n    *   `HF_HUB_OFFLINE=1`: Similar to above, for Hugging Face Hub specific operations.\n\n## Troubleshooting\n\n*   **Service Fails to Start:** Check the logs (`journalctl -u \u003cservice-name\u003e`) for errors. Common issues include:\n    *   Port conflicts (another service using the assigned port).\n    *   Insufficient RAM/VRAM.\n    *   Incorrect model ID or model files.\n    *   Permissions issues for local model paths.\n    *   Missing `CUDA_VISIBLE_DEVICES` environment variable in the service file if you have multiple GPUs and the model requires a specific one (e.g., for `bitsandbytes`).\n*   **`jq: command not found`:** The script will try to parse model lists without `jq`, but it's less reliable. Install `jq` for best results: `sudo apt-get install jq`.\n*   **Access Denied to Model Directory:** If your user cannot access the model directory after running the script, ensure you have logged out and logged back in for group membership changes to apply.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request or open an Issue for any bugs, feature requests, or improvements.\n\n1.  Fork the repository.\n2.  Create your feature branch (`git checkout -b feature/AmazingFeature`).\n3.  Commit your changes (`git commit -m 'Add some AmazingFeature'`).\n4.  Push to the branch (`git push origin feature/AmazingFeature`).\n5.  Open a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details (you'll need to create this file if you haven't).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparisneo%2Fopenllmserverscript","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparisneo%2Fopenllmserverscript","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparisneo%2Fopenllmserverscript/lists"}