{"id":38673795,"url":"https://github.com/wi2trier/gpu-server","last_synced_at":"2026-01-17T10:00:00.116Z","repository":{"id":210890390,"uuid":"727384929","full_name":"wi2trier/gpu-server","owner":"wi2trier","description":"System configuration for a CUDA-based GPU server using Nix","archived":false,"fork":false,"pushed_at":"2025-12-15T02:00:18.000Z","size":364,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-17T18:27:44.132Z","etag":null,"topics":["cuda","gpu","nix","server","system-config","ubuntu"],"latest_commit_sha":null,"homepage":"","language":"Nix","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wi2trier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-04T19:00:36.000Z","updated_at":"2025-12-15T02:00:22.000Z","dependencies_parsed_at":"2024-12-16T03:23:41.870Z","dependency_job_id":"683442d7-6640-4aca-a540-36498eeb22f7","html_url":"https://github.com/wi2trier/gpu-server","commit_stats":{"total_commits":106,"total_committers":3,"mean_commits":"35.333333333333336","dds":0.08490566037735847,"last_synced_commit":"bb918bdca0d0d49edd322e22fc9af7daa408dc38"},"previous_names":["mirkolenz/gpu-server","wi2trier/gpu-server"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wi2trier/gpu-server","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wi2trier%2Fgpu-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wi2trier%2Fgpu-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wi2trier%2Fgpu-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wi2trier%2Fgpu-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wi2trier","download_url":"https://codeload.github.com/wi2trier/gpu-server/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wi2trier%2Fgpu-server/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuda","gpu","nix","server","system-config","ubuntu"],"created_at":"2026-01-17T09:59:59.976Z","updated_at":"2026-01-17T10:00:00.102Z","avatar_url":"https://github.com/wi2trier.png","language":"Nix","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WI2 GPU Server\n\n## Overview\n\nOur GPU server is provided for research in the field of machine learning and deep learning.\nPlease do not run/host CPU-based applications, we will happily provide you with a virtual machine for that.\nIt shall also not be used as a development machine, please test your code locally and only use the server for training and evaluation.\n\nThe server has the following hardware configuration:\n\n- CPU: 2x Intel Xeon Gold 6138 @ 2.00GHz (20 cores, 40 threads)\n- GPU: 8x NVIDIA Tesla V100 (32 GB VRAM)\n- RAM: 12x Micron 64 GB DDR4-2666 ECC\n\nTo run your code on the server, we provide two container runtimes:\n[Podman](https://podman.io) (a general-purpose Docker replacement) and [Apptainer](https://apptainer.org) (aimed specifically at scientific computing).\nDue to its ease of use, we generally recommend Apptainer for users unfamiliar with container engines like Docker.\nAmong others, it automatically forwards ports form your applications and makes sure that your files on the server are accessible in the container without any configuration (unlike Podman).\n\n\u003e [!important]\n\u003e Even users with sudo permissions shall not install additional software through `apt` or other package managers.\n\u003e The server is declaratively managed through Nix, so this may interfere with the configuration.\n\u003e Instead, refer to the section _consuming packages_ in this document.\n\n## Access\n\nThe server can only be accessed when connected to the VPN of Trier University.\nTo start, open a terminal and execute:\n\n```shell\nssh USERNAME@IP\n```\n\nAfter your first login, please change your password using the command `passwd` (if not prompted anyway).\n\n**Note for Windows users:**\nRecent versions of Windows 10 and 11 already contain an SSH client, so you can use Microsoft's new Windows Terminal to connect.\nPlease find more details on the corresponding [help page](https://learn.microsoft.com/en-us/windows/terminal/tutorials/ssh).\n\n## General Usage\n\nFor an introduction to the Linux command line, we refer to [Mozilla's command line crash course](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Understanding_client-side_tools/Command_line).\nWe found the following utilities to be useful for everyday use:\n\n- `htop`: Show the active processes on the server.\n- `nvidia-smi` and `gpustat`: Show the active processes on the GPUs.\n- `tmux`: Create a new terminal session that can be detached and reattached later.\n\n### Custom Packages\n\nWe use the [Nix package manager](https://nixos.org) to declaratively manage the server configuration.\nThis allows you to spawn a temporary shell with additional packages easily.\nFor instance, to create a shell with Python, execute:\n\n```shell\nnix shell pkgs#python3\n```\n\nMultiple packages can be specified as follows:\n\n```shell\nnix shell pkgs#{python3,nodejs}\n```\n\nYou may also provide a command that shall be run in the shell:\n\n```shell\nnix shell pkgs#python3 --command python --version\n```\n\nYou can search for available packages on [search.nixos.org](https://search.nixos.org/packages).\n\nThese temporary shells are useful to quickly test a new package before using it in production in a containerized environment like Apptainer.\nIf you would like to install a package permanently, please open an issue on GitHub.\n\n### GPU Selection\n\nAll GPUs of the server are set to exclusive compute mode, meaning that only one process can use a GPU at a time.\nTo select a GPU, you can set the `CUDA_VISIBLE_DEVICES=$GPU_ID` environment variable.\nPlease use the aforementioned `nvidia-smi` command to check which GPUs are currently in use.\n\n\u003e [!important]\n\u003e The system has eight GPUs, but two NUMA nodes with 4 GPUs each: GPUs 0,1,2,3 are connected with NVLink and GPUs 5,6,7,8 are connected with NVLink.\n\u003e For performance reasons, we **strongly** recommend to stay within one of these nodes when selecting GPUs.\n\n### tmux\n\nThe `tmux` command allows you to spawn a \"virtual\" shell that persists even after you disconnect from the server.\nAmong others, this allows you to start a long-running process and then close the session without interrupting it.\nTo create a new session, simply execute `tmux`.\nYou can then start your process and close the terminal if needed.\nTo reattach to the session later, execute `tmux attach`.\nThe tmux session can be closed by running `exit` or `tmux kill-session` in the shell.\n\n## Apptainer Usage\n\nThere are three main commands in Apptainer:\n\n- `apptainer shell`: Open an interactive shell in the container.\n- `apptainer run [$ARGS...]`: Run the default command of the container. You may optionally add one or more arguments to the command.\n- `apptainer exec $COMMAND [$ARGS...]`: Run a custom command inside the container. You may optionally add one or more arguments to the command.\n\nSome images (e.g., `nvidia/cuda` or `ubuntu`) provide no default command, meaning it will do nothing by default.\nIn this special case, `apptainer run` and `apptainer exec` are equivalent and can be used interchangeably.\n\n### Image Selection\n\nApptainer supports a variety of container image formats, including Docker and its own SIF format.\nTo run a container from a Docker registry, use the `docker://` prefix:\n\n```shell\napptainer run docker://$IMAGE [$ARGS...]\n```\n\nIt is recommended to convert such images to the SIF format, which is more efficient and allows for faster loading.\n\n```shell\napptainer build $IMAGE.sif docker://$IMAGE\n```\n\nIn later runs, you can then use the SIF image directly:\n\n```shell\napptainer run $IMAGE.sif [$ARGS...]\n```\n\n### GPU Selection\n\nTo access the GPUs from within the container, you need to set the `CUDA_VISIBLE_DEVICES` environment variable and pass the `--nv` flag to the Apptainer command.\nHere is the general call signature for the `apptainer exec` command:\n\n```shell\nCUDA_VISIBLE_DEVICES=$GPU_ID apptainer exec --nv $IMAGE $COMMAND\n```\n\nYou may omit the `CUDA_VISIBLE_DEVICES` environment variable, in which case Apptainer will automatically select a GPU that is currently not in use.\nFor example, to run the `nvidia-smi` command on any available GPU, execute\n\n```shell\napptainer exec --nv docker://ubuntu nvidia-smi\n```\n\n### [File Access](https://apptainer.org/docs/user/main/bind_paths_and_mounts.html)\n\nBy default, Apptainer mounts both (i) your home directory and (ii) your current working directory into the container, so no additional configuration is needed.\nIn case you only need your working directory and not your home folder, pass the option `--no-home` to the Apptainer command.\nWhen needing access to other file locations, you can use the `--bind source:target` option to mount them into the container.\n\n### Port Forwarding\n\nWhen starting a server in a container, it is directly accessible without the need to forward ports.\nThis also means that in case two users want to run two instances of the same app on the server, you are responsible for choosing different ports.\nPlease consult the documentation of the corresponding app for more details on how to change the default port.\n\n\u003e [!important]\n\u003e Regular users are allowed to use ports above `1024`.\n\u003e If connecting via the university VPN, only ports in the range `6000-8000` will be accessible!\n\n### [Image Caching](https://apptainer.org/docs/user/main/cli/apptainer_cache.html)\n\nApptainer caches images on the server to speed up subsequent runs.\nThey are stored in your home folder, so you may want to clean them up from time to time:\n\n```shell\napptainer cache clean --days $DAYS\n```\n\nAll images not accessed within the last `$DAYS` days will be deleted.\n\n### [Running Services](https://apptainer.org/docs/user/main/running_services.html)\n\nIf starting a service like a Jupyter notebook, you need to keep the terminal open.\nTo mitigate this, you may either use the `tmux` command (see above) or start the container as an _instance_:\n\n```shell\napptainer instance start --nv $IMAGE $INSTANCE_NAME [$ARGS...]\n```\n\nTo see a list of all running instances, execute\n\n```shell\napptainer instance list\n```\n\nWhile an instance is running, you can execute code in it via\n\n```shell\napptainer instance exec $INSTANCE_NAME $COMMAND [$ARGS...]\n```\n\nTo stop an instance, execute\n\n```shell\napptainer instance stop $INSTANCE_NAME\n# or, to stop all instances\napptainer instance stop --all\n```\n\n## Podman Usage\n\nWe recommend Apptainer for most users, so we will only provide some notes about the particularities of our GPU-based Podman setup.\nFor more details, please refer to the [official documentation](https://docs.podman.io/en/latest/).\nPlease keep in mind that in order to work correctly, Podman requires more configuration (e.g., port forwarding and volume mounting) than is shown here.\n\n### GPU Selection\n\nWe provide support for accessing the GPUs via the officlal [Container Device Interface](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html).\nConsequently, you can use the `--device` flag to access the GPUs from within the container like so:\n\n```shell\npodman run --rm --device nvidia.com/gpu=$GPU_ID $IMAGE $COMMAND\n```\n\n### Image Caching\n\nSimilar to Apptainer, Podman caches images in your home folder to speed up subsequent runs.\nYou may delete the entire cache as follows:\n\n```shell\npodman system reset\n```\n\n## Prebuilt Container Images\n\nWe provide a number of ready-to-use container images for common use cases.\nThey are updated regularly, so we require you to store a copy of them in your home folder before using them.\nTo do so, execute the following command:\n\n```shell\nbuild-container IMAGE_NAME [OUTPUT_FOLDER]\n```\n\nwhere `OUTPUT_FOLDER` is your current working directory by default.\nThe images are stored in the `docker-archive` format (ending in `.tar.gz`).\nSince Apptainer converts the images to its SIF format anyway, we offer a streamlined integration:\n\n```shell\nbuild-apptainer IMAGE_NAME [OUTPUT_FOLDER]\n```\n\nThe image can then be run as follows:\n\n```shell\napptainer run --nv IMAGE_NAME.sif\n# or\npodman run --rm --device nvidia.com/gpu=0 docker-archive:IMAGE_NAME.tar.gz\n```\n\n\u003e [!note]\n\u003e These images install their dependencies in a virtual environment in your current working directory (i.e., `./.venv`).\n\u003e This allows to cache the dependencies and reuse them across multiple runs.\n\u003e Please make sure to add the virtual environment to your `.gitignore` file and always start the container in the same working directory.\n\n### Poetry\n\nThe image `poetry` contains Python 3 together with the package manager Poetry.\nThis image allows proper dependency specification via `pyproject.toml` and `poetry.lock` files.\nUsing `apptainer run IMAGE ARGS...` executes `poetry ARGS...`, so you could execute a Python script via `apptainer run IMAGE run python SCRIPT.py`.\n\n### uv\n\nThe image `uv` contains Python 3 together with the package manager uv.\nThis image allows proper dependency specification via `pyproject.toml` and `uv.lock` files.\nUsing `apptainer run IMAGE ARGS...` executes `uv ARGS...`, so you could execute a Python script via `apptainer run IMAGE run SCRIPT.py`.\n\n### Jupyter\n\nThe image `jupyter` contains a Jupyter Lab server with common NLP dependencies (numpy, scipy, spacy, nltk, torch, openai, transformers, sentence-transformers).\nIt allows easy interaction with the server through a browser-based interface.\nJupyter by default listens on port `8888` which is not accessible from the outside.\nThus, choose a different port within the range `6000-8000` by passing the `--port $PORT` option to the `apptainer run` command.\n\nYou may install arbitrary Python packages by running `!pip install NAME` in a notebook.\nAlternatively, open a Terminal and execute `pip install NAME` there.\nPlease note that the binaries installed in the virtual environment are only available in the Terminal when your working directory contains the `.venv` folder (i.e., using `cd` to change the folder removes `.venv/bin` from your `$PATH`).\n\n## Ollama Usage\n\nThe server is equipped with the [Ollama](https://ollama.com) library for serving LLMs.\nIt automatically selects one free GPU, but you can override this behavior by setting the `CUDA_VISIBLE_DEVICES` environment variable (just like with Apptainer).\nBy default, Ollama does not allow remote requests and uses a fixed port (potentially conflicting with other users).\nTo change this, you need to modify the `OLLAMA_HOST` environment variable:\n\n- Accessing from your computer: `OLLAMA_HOST=0.0.0.0:PORT`.\n  `PORT` must be in the range `6000-8000`).\n- Accessing from the server only: `OLLAMA_HOST=127.0.0.1:PORT`.\n  `PORT` can be arbitrarily chosen `\u003e1024`.\n\n```shell\nOLLAMA_HOST=SEE_ABOVE ollama serve\n```\n\nIn another terminal, pull the models before performing API requests:\n\n```shell\nOLLAMA_HOST=SEE_ABOVE ollama pull MODEL_NAME\n```\n\nTo make requests, either use the `ollama` library or the `openai` library.\nRemember to modify their respective endpoints to point to the IP of the GPU server and the chosen port.\n\n## LMDeploy Usage\n\nAlternatively, you can use [LMDeploy](https://github.com/InternLM/lmdeploy) to deploy and run LLMs.\nLMDeploy offers a higher throughput than Ollama due to its CUDA Kernels for the given GPUs. It also allow easy batching.\n\nTo run LMDeploy, use podman:\n\n```shell\npodman run --gpus $GPUS \\\n  -p PORT:PORT \\\n  docker.io/openmmlab/lmdeploy:latest-cu12 \\\n  lmdeploy serve api_server $HF_LLM_NAME\n```\n\nThis will start an OpenAI-compatible endpoint at the corresponding port. More options can be found [in the documentation](https://lmdeploy.readthedocs.io/en/latest/llm/api_server.html).\n\nAs an example, this command runs Qwen/Qwen3-32B on four GPUs with 32K context on port 23333:\n\n```shell\npodman run --gpus 0,1,2,3 \\\n  -p 23333:23333 \\\n  docker.io/openmmlab/lmdeploy:latest-cu12 \\\n  lmdeploy serve api_server Qwen/Qwen3-32B \\\n  --tp 4 --session-len 32768 --cache-max-entry-count 0.85\n```\n\n## Editor Integrations\n\n### Visual Studio Code\n\n- Install the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension.\n- Connect to the server by clicking on the green icon in the bottom left corner of the window and selecting `Remote-SSH: Connect to Host...`.\n- Enter the connection string of the server: `USER@IP` and add it as a new host.\n- You now have access to the server's file system and can edit files directly on the server.\n\n\u003e [!important]\n\u003e VSCode currently does not use a login shell by default, meaning that some commands like Apptainer will be missing.\n\u003e You can track this issue on [GitHub](https://github.com/microsoft/vscode-remote-release/issues/1671).\n\u003e As a workaround, open the command palette (Ctrl+Shift+P), select `Preferences: Open Remote Settings`, and paste the following snippet:\n\n```json\n{\n  \"terminal.integrated.defaultProfile.linux\": \"bash\",\n  \"terminal.integrated.profiles.linux\": {\n    \"bash\": {\n      \"path\": \"bash\",\n      \"icon\": \"terminal-bash\",\n      \"args\": [\"-l\"]\n    }\n  }\n}\n```\n\n### PyCharm\n\n- Go to `Settings \u003e Tools \u003e SSH Configurations` and create a new connection using the credentials provided to you via mail. The server uses the default SSH port 22.\n- Go to `Settings \u003e Build, Execution, Deployment \u003e Deployment`. Choose `SFTP` as the connection type and select the connection you created in the previous step. Set your home path to `/home/\u003cusername\u003e`. The option `Mappings` allows to configure where your local project is uploaded to on the server. For instance, setting `Deployment Path` to `projects/thesis` will upload your project to `/home/\u003cusername\u003e/projects/thesis`. Adding excluded paths allows to exclude files from the upload. For instance, adding `.venv` will exclude the virtual environment from the upload.\n\n## Additional Docs\n\n- [Administration](./docs/admin.md)\n- [Setup instructions](./docs/setup.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwi2trier%2Fgpu-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwi2trier%2Fgpu-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwi2trier%2Fgpu-server/lists"}