{"id":25735952,"url":"https://github.com/small-hack/smol-metal","last_synced_at":"2025-07-23T18:34:10.682Z","repository":{"id":65824040,"uuid":"557900450","full_name":"small-hack/smol-metal","owner":"small-hack","description":"Notes for setting up Debian Bookworm hosts","archived":false,"fork":false,"pushed_at":"2025-06-15T11:17:43.000Z","size":430,"stargazers_count":4,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-15T12:21:35.701Z","etag":null,"topics":["ansible","debian"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause-clear","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/small-hack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-10-26T14:12:51.000Z","updated_at":"2025-06-15T11:17:46.000Z","dependencies_parsed_at":"2025-06-15T12:30:06.581Z","dependency_job_id":null,"html_url":"https://github.com/small-hack/smol-metal","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/small-hack/smol-metal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/small-hack%2Fsmol-metal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/small-hack%2Fsmol-metal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/small-hack%2Fsmol-metal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/small-hack%2Fsmol-metal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/small-hack","download_url":"https://codeload.github.com/small-hack/smol-metal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/small-hack%2Fsmol-metal/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266732552,"owners_count":23976051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","debian"],"created_at":"2025-02-26T05:34:22.414Z","updated_at":"2025-07-23T18:34:10.672Z","avatar_url":"https://github.com/small-hack.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Smol-Metal\n\nNotes for configuring Debian Bookworm nodes for use as VPS hosts.\nThe steps below setup the system to be further controlled by ansible. Eventually most of this will move into a cloid-init or pre-seed files.\n\n## Upgrading a host from Debian11 to Debian12\n\nFix apt sources / Upgrade: https://wiki.debian.org/DebianUpgrade\n  \n  ```bash\n  cat \u003c\u003c EOF \u003e /etc/apt/sources.list\n  deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware\n  deb-src http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware\n\n  deb http://deb.debian.org/debian-security/ bookworm-security main contrib non-free\n  deb-src http://deb.debian.org/debian-security/ bookworm-security main contrib non-free\n\n  deb http://deb.debian.org/debian bookworm-updates main contrib non-free\n  deb-src http://deb.debian.org/debian bookworm-updates main contrib non-free\n\n  deb http://deb.debian.org/debian bookworm-backports main\n  deb http://deb.debian.org/debian bullseye-backports main\n  EOF\n  \n  apt-get update \u0026\u0026 \\\n  apt-get upgrade -y \u0026\u0026 \\\n  apt-get full-upgrade -y\n  \n  reboot\n  ```\n\n## Install Proxmox kernel\n\n- List available kernels\n  \n  ```bash\n  # standard debian \n  apt list linux-*image-*\n  apt list linux-*headers-*\n\n  # proxmox\n  apt list pve-kernel-*\n  ```\n\n- Installing proxmox pve kernel and headers on debian\n\n  ```bash\n  wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg\n\n  cat \u003c\u003c EOF \u003e /etc/apt/sources.list\n  deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware\n  deb-src http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware\n\n  deb http://deb.debian.org/debian-security/ bookworm-security main contrib non-free\n  deb-src http://deb.debian.org/debian-security/ bookworm-security main contrib non-free\n\n  deb http://deb.debian.org/debian bookworm-updates main contrib non-free\n  deb-src http://deb.debian.org/debian bookworm-updates main contrib non-free\n\n  deb http://deb.debian.org/debian bookworm-backports main\n  deb http://deb.debian.org/debian bullseye-backports main\n\n  deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription\n  EOF\n  \n  apt-get update \n  apt-get install pve-kernel-6.2/stable\n  apt-get install pve-headers-6.2\n\n\n  cat \u003c\u003c EOF \u003e /etc/apt/sources.list\n  deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware\n  deb-src http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware\n\n  deb http://deb.debian.org/debian-security/ bookworm-security main contrib non-free\n  deb-src http://deb.debian.org/debian-security/ bookworm-security main contrib non-free\n\n  deb http://deb.debian.org/debian bookworm-updates main contrib non-free\n  deb-src http://deb.debian.org/debian bookworm-updates main contrib non-free\n\n  deb http://deb.debian.org/debian bookworm-backports main\n  deb http://deb.debian.org/debian bullseye-backports main\n  \n  EOF\n\n  apt-get remove apparmor\n  apt-get install apparmor\n  ```\n\n- Donwload meta package from source if its not in the apt list\n  \n  - https://packages.debian.org/search?keywords=linux-image-amd64\n\n- List all installed kernels and headers\n\n  ```bash\n  dpkg --list | egrep -i --color 'linux-image|linux-headers'\n  ```\n\n- Remove undesired kernels and headers\n\n  ```bash\n  apt-get --purge remove linux-image-6.1.0-12-amd64 linux-image-amd64 \n  ```\n\n- prevent changes\n\n  ```bash\n  sudo apt-mark hold pve-kernel-6.2/stable\n  sudo apt-mark hold pve-headers-6.2\n  ```\n- reboot\n\n\n## Ubuntu alternative package mirror\n\n  ```bash\n  cat \u003c\u003c EOF \u003e /etc/apt/sources.list\n  deb http://de.archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse\n  # deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse\n\n  deb http://de.archive.ubuntu.com/ubuntu/ jammy-updates main restricted universe multiverse\n  # deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted universe multiverse\n\n  deb http://de.archive.ubuntu.com/ubuntu/ jammy-security main restricted universe multiverse\n  # deb-src http://archive.ubuntu.com/ubuntu/ jammy-security main restricted universe multiverse\n\n  deb http://de.archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse\n  # deb-src http://archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse\n\n  deb http://archive.canonical.com/ubuntu/ jammy partner\n  # deb-src http://archive.canonical.com/ubuntu/ jammy partner\n  \n  EOF\n  sudo apt-get update \u0026\u0026 \\\n  sudo apt-get upgrade -y \u0026\u0026 \\\n  sudo apt-get full-upgrade -y\n  \n  reboot\n  ```\n  \n## Initial Setup:\n\n1. install basic dependancies (Run as Root)\n\n - Apt Packages:\n \n    ```bash\n    apt-get update \u0026\u0026 \\\n      apt-get install -y wireguard \\\n      ssh-import-id \\\n      sudo \\\n      curl \\\n      fio \\\n      lshw \\\n      rsync \\\n      mdadm \\\n      tmux \\\n      netplan.io \\\n      apt-transport-https \\\n      ca-certificates \\\n      software-properties-common \\\n      htop \\\n      git-extras \\\n      rsyslog \\\n      fail2ban \\\n      vim \\\n      gpg \\\n      open-iscsi \\\n      nfs-common \\\n      ncdu \\\n      bc \\\n      zip \\\n      unzip \\\n      pkg-config \\\n      iotop \\\n      cron \\\n      pipx \u0026\u0026 \\\n      sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq \u0026\u0026 \\\n      sudo chmod +x /usr/bin/yq \u0026\u0026 \\\n      sudo systemctl enable fail2ban \u0026\u0026 \\\n      sudo systemctl start fail2ban\n\n    # VM guest utils\n    apt update \u0026\u0026 apt -y install qemu-guest-agent \u0026\u0026 \\\n    systemctl enable qemu-guest-agent\n    systemctl start qemu-guest-agent\n\n    # Optional Go + Seaweedfs cli\n    wget https://go.dev/dl/go1.24.4.linux-amd64.tar.gz\n    rm -rf /usr/local/go \u0026\u0026 tar -C /usr/local -xzf go1.24.4.linux-amd64.tar.gz\n    echo \"export PATH=$PATH:/usr/local/go/bin:/usr/bin\" \u003e\u003e /home/friend/.bash_profile\n    sudo -u friend -i go install -x github.com/seaweedfs/seaweedfs/weed@3.91\n    mv /home/friend/go/bin/weed /usr/local/bin/\n    mkdir /home/friend/shared\n    sudo weed mount -filer=seaweedfs-filer.seaweedfs.svc.cluster.local:8888 \\\n      -dir=/home/friend/shared \\\n      -filer.path=/friend/shared -volumeServerAccess=filerProxy\n\n    # Optional Basic Desktop\n    DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get install -y \\\n      xinit \\\n      xorg \\\n      firefox-esr \\\n      tasksel \\\n      xfce4-goodies \\\n      x11-utils \\\n      x11vnc \\\n      xvfb \\\n      dbus-x11 \\\n      accountsservice \\\n      xfce4 \u0026\u0026 \\\n      echo -e \"allowed_users=anybody\\nneeds_root_rights=yes\" | sudo tee /etc/X11/Xwrapper.config \u003e /dev/null \u0026\u0026 \\\n      tasksel install xfce4-desktop\n\n    # Install sunhine\n    export VERSION=\"v0.23.1\"\n    export PLATFORM=\"debian-bookworm\"\n    export ARCH=\"amd64.deb\"\n    export REPO=\"LizardByte/Sunshine/\"\n    wget https://github.com/$REPO/releases/download/$VERSION/sunshine-$PLATFORM-$ARCH\n    apt-get install -f ./sunshine-$PLATFORM-$ARCH\n\n    echo 'KERNEL==\"uinput\", SUBSYSTEM==\"misc\", OPTIONS+=\"static_node=uinput\", TAG+=\"uaccess\"' | \\\n    sudo tee /etc/udev/rules.d/60-sunshine.rules\n    sudo udevadm control --reload-rules\n    sudo udevadm trigger\n    sudo modprobe uinput\n    sudo setcap -r $(readlink -f $(which sunshine))\n\n    mkdir -p /home/friend/.config/systemd/user\n    \n    cat \u003c\u003c EOF \u003e /home/friend/.config/systemd/user/sunshine.service\n    [Unit]\n    Description=Sunshine self-hosted game stream host for Moonlight.\n    StartLimitIntervalSec=500\n    StartLimitBurst=5\n\n    [Service]\n    Restart=always\n    RestartSec=5s\n    ExecStartPre=/bin/sleep 10\n    ExecStart=/usr/bin/sunshine\n\n    [Install]\n    WantedBy=default.target\n    EOF\n\n    sudo chown -R friend:friend /home/friend/.config\n    systemctl --user enable sunshine\n    systemctl --user start sunshine\n\n    # Install nicedcv\n    export FILE_NAME=\"nice-dcv-2023.1-16388-ubuntu2204-x86_64\"\n    wget https://d1uj6qtbmh3dt5.cloudfront.net/2023.1/Servers/$FILE_NAME.tgz\n    tar -xvzf $FILE_NAME.tgz\n    sudo apt-get install -y -f ./$FILE_NAME/*.deb\n\n    sudo usermod -aG video dcv\n    sudo sed -ie 's/#owner = \"\"/owner = \"friend\"/' /etc/dcv/dcv.conf\n    sudo sed -ie 's/\"1\"/\"0\"/g' /etc/apt/apt.conf.d/20auto-upgrades\n    sudo systemctl isolate multi-user.target\n    sudo dcvgladmin enable\n    sudo systemctl isolate graphical.target\n    sudo dcvgldiag\n    sudo systemctl enable dcvserver\n    sudo systemctl start dcvserver\n\n    dcv list-sessions\n    \n    \n    \n    ```\n    \n    Prometheus (Run this as root)\n    ```bash\n    wget -O /opt/node_exporter-1.6.1.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz \u0026\u0026 \\\n    tar -xvf /opt/node_exporter-1.6.1.linux-amd64.tar.gz -C /opt \u0026\u0026 \\\n    rm /opt/node_exporter-1.6.1.linux-amd64.tar.gz \u0026\u0026 \\\n    ln -s node_exporter-1.6.1.linux-amd64 /opt/node_exporter\n    \n    wget https://raw.githubusercontent.com/small-hack/smol-metal/main/node-exporter.service \u0026\u0026 \\\n    sudo mv node-exporter.service /etc/systemd/system/node-exporter.service \u0026\u0026 \\\n    systemctl daemon-reload \u0026\u0026 \\\n    systemctl enable node-exporter \u0026\u0026 \\\n    systemctl restart node-exporter\n    ```\n\n2. Setup the user\n\n    ```bash\n    sudo useradd -s /bin/bash -d /home/friend/ -m -G sudo friend\n    sudo echo \"friend ALL=(ALL) NOPASSWD: ALL\" \u003e\u003e /etc/sudoers\n    sudo -u friend ssh-import-id-gh cloudymax\n    sudo usermod -a -G kvm friend\n    passwd friend\n    ```\n    \n 3. Install Docker and Onboardme (Run as user, not as root)\n    \n    ```bash\n    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \\\n      sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg\n    ```\n    Ubuntu\n    ```bash\n    echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\n    ```\n    \n    Debian\n    ```bash\n    echo \"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable\" | sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\n    ```\n    \n    ```bash\n    sudo apt-get update \u0026\u0026 \\\n    sudo apt-get install -y docker-ce \u0026\u0026 \\\n    sudo usermod -a -G docker friend\n    ```\n\n3. Brew and Python3.11 (Run as User)\n    ```\n    wget -O setup.sh https://raw.githubusercontent.com/jessebot/onboardme/main/setup.sh\n    . ./setup.sh \n\n    brew install bitwarden-cli b2-tools k9s neovim\n    ```\n    \n4. Docker Compose (Run as User)\n   \n   ```bash\n   mkdir -p ~/.docker/cli-plugins/\n   curl -SL https://github.com/docker/compose/releases/download/v2.17.3/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose\n   chmod +x ~/.docker/cli-plugins/docker-compose\n   docker compose version\n   ```\n\n5. Disable sleep\n\n  ```bash\n  sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target\n  ```\n\n6. set open files limits and speed up raid sync\n  \n  ```bash\n  sudo sysctl -w fs.inotify.max_user_watches=2099999999\n  sudo sysctl -w fs.inotify.max_user_instances=2099999999\n  sudo sysctl -w fs.inotify.max_queued_events=2099999999\n  \n  # As root\n  sudo echo 50000000 \u003e /sys/block/mdX/md/sync_speed_max\n  # add to /etc/sysctl.conf\n  dev.raid.speed_limit_min = 500000\n  dev.raid.speed_limit_max = 5000000\n\n  sysctl -p\n  ```\n\n## Networking\n\nbridge the network adapter (Optional)\n  \n  ```bash\n  # /etc/netplan/99-bridge.yaml\n  network:\n    bridges:\n      br0:\n        dhcp4: no\n        dhcp6: no\n        interfaces: [enp4s0]\n        addresses: [192.168.50.101/24]\n        routes:\n          - to: default\n            via: 192.168.50.1\n        mtu: 1500\n        nameservers:\n          addresses: [192.168.50.50]\n        parameters:\n          stp: true\n          forward-delay: 4\n    ethernets:\n      enp4s0:\n        dhcp4: no\n        dhcp6: no\n    renderer: networkd\n    version: 2\n\n  sudo netplan --debug generate\n  sudo netplan --debug apply\n  ```\n    \n5. Setup Wireguard (Optional) (Run as Root)\n\n    \u003cdetails\u003e\n      \u003csummary\u003eClick to expand\u003c/summary\u003e\n\n    - Enable IP forwarding\n    ```bash\n    sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf\n    ```\n\n    - Generate Server Keys\n    ```bash\n    cd /etc/wireguard\n    wg genkey | tee privatekey | wg pubkey \u003e publickey\n    ```\n    \n    - Edit the wireguard config\n    ```bash\n    nano /etc/wireguard/wg0.conf\n    ```\n\n    - Server Config\n    ```bash\n    export SERVER_PUBLIC_KEY=$(sudo cat /etc/wireguard/publickey)\n    export SERVER_PRIVATE_KEY=$(sudo cat /etc/wireguard/privatekey)\n    export NETWORK_INTERFACE=\"enp0s31f6\"\n    export WG_INTERFACE=\"wg0\"\n    export SERVER_PORT=\"51820\"\n    export WG_ADDRESS=\"10.2.0.1\"\n\n    cat \u003c\u003c EOF \u003e wg0.conf\n    [Interface]\n    Address = ${WG_ADDRESS}/24\n    ListenPort = ${SERVER_PORT}\n    PrivateKey = ${SERVER_PRIVATE_KEY}\n\n    PostUp = iptables -A FORWARD -i ${WG_INTERFACE} -j ACCEPT; iptables -t nat -A POSTROUTING -o ${NETWORK_INTERFACE} -j MASQUERADE; ip6tables -A FORWARD -i ${WG_INTERFACE} -j ACCEPT; ip6tables -t nat -A POSTROUTING -o ${NETWORK_INTERFACE} -j MASQUERADE\n    PostDown = iptables -D FORWARD -i ${WG_INTERFACE} -j ACCEPT; iptables -t nat -D POSTROUTING -o ${NETWORK_INTERFACE} -j MASQUERADE; ip6tables -D FORWARD -i ${WG_INTERFACE} -j ACCEPT; ip6tables -t nat -D POSTROUTING -o ${NETWORK_INTERFACE} -j MASQUERADE\n    EOF\n    ```\n\n    - Client Config\n    ```bash\n    mkdir client \u0026\u0026 cd client\n    wg genkey | tee privatekey | wg pubkey \u003e publickey\n    export SERVER_PUBLIC_KEY=$(cat ../publickey)\n    export CLIENT_PRIVATE_KEY=$(cat privatekey)\n    export SERVER_PUBLIC_IP=\"128.140.72.118\"\n    export SERVER_PORT=\"51820\"\n    export IP_ADDRESS=\"10.2.0.2\"\n\n    cat \u003c\u003c EOF \u003e wg0.conf\n    [Interface]\n    PrivateKey = ${PRIVATE_KEY}\n    Address = ${IP_ADDRESS}/24\n\n    [Peer]\n    PublicKey = ${SERVER_PUBLIC_KEY}\n    AllowedIPs = 10.2.0.0/24\n    Endpoint = ${SERVER_PUBLIC_IP}:${SERVER_PORT}\n    PersistentKeepalive = 15\n    EOF\n    ```\n\n    - Enable wireguard as a service\n    ```bash\n    sudo systemctl enable wg-quick@wg0\n    ```\n\n    - Start the service\n    ```bash\n    sudo systemctl restart wg-quick@wg0\n    ```\n    \u003c/details\u003e\n\n5. Disable insecure ssh login options\n\n    ```bash\n    sudo wget -O /etc/ssh/sshd_config https://raw.githubusercontent.com/cloudymax/linux_notes/main/sshd_config\n\n    sudo systemctl reload sshd\n    ```\n\n6. Setup PCI/IOMMU Passthrough (Optional)\n\n    \u003cdetails\u003e\n      \u003csummary\u003eEnable iommu via grub\u003c/summary\u003e\n  \n    ```bash\n    # /etc/default/grub\n    GRUB_DEFAULT=0\n    GRUB_TIMEOUT=5\n    GRUB_DISTRIBUTOR=`lsb_release -i -s 2\u003e /dev/null || echo Debian`\n    GRUB_CMDLINE_LINUX_DEFAULT=\"quiet preempt=voluntary iommu=pt amd_iommu=on intel_iommu=on\"\n    GRUB_CMDLINE_LINUX=\"\"\n\n    sudo update-grub\n    sudo reboot now\n    ```\n  \n    \u003c/details\u003e\n\n\n    \u003cdetails\u003e\n      \u003csummary\u003eSetup GPU-Passthrough\u003c/summary\u003e\n  \n    ```bash\n    # See: https://github.com/small-hack/smol-gpu-passthrough\n\n    wget https://raw.githubusercontent.com/small-hack/smol-gpu-passthrough/main/setup.sh\n\n    bash setup.sh full_run NVIDIA\n    sudo reboot now\n    ```\n  \n    \u003c/details\u003e\n\n  vGPU Install (run all this as root)\n\n    \n  ```bash\n  # /etc/default/grub\n  GRUB_DEFAULT=0\n  GRUB_TIMEOUT=5\n  GRUB_DISTRIBUTOR=`lsb_release -i -s 2\u003e /dev/null || echo Debian`\n  GRUB_CMDLINE_LINUX_DEFAULT=\"quiet preempt=voluntary iommu=pt amd_iommu=on intel_iommu=on\"\n  GRUB_CMDLINE_LINUX=\"\"\n\n  sudo update-grub\n\n  echo -e \"vfio\\nvfio_iommu_type1\\nvfio_pci\\nvfio_virqfd\" \u003e\u003e /etc/modules\n  echo \"blacklist nouveau\" \u003e\u003e /etc/modprobe.d/blacklist.conf\n  update-initramfs -u -k all\n\n  sudo reboot now\n  ```\n    \n  ```console\n  apt install -y git build-essential \\\n      dkms \\\n      mdevctl \\\n      firmware-misc-nonfree \\\n      linux-headers-amd64 \\\n      gcc \\\n      make \\\n      libvulkan1 \\\n      libglvnd-dev \\\n      uuid-runtime\n\n  cd /root\n  git clone https://gitlab.com/polloloco/vgpu-proxmox.git\n\n  cd /opt\n  git clone https://github.com/mbilker/vgpu_unlock-rs.git\n\n  cd /tmp\n  curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal\n  source $HOME/.cargo/env\n\n  cd /opt/vgpu_unlock-rs/\n  cargo build --release\n\n  # Create a vgpu profile\n  mkdir /etc/vgpu_unlock\n  touch /etc/vgpu_unlock/profile_override.toml\n\n  # M60\n  cat \u003c\u003c EOF \u003e /etc/vgpu_unlock/profile_override.toml\n  [profile.nvidia-18]\n  num_displays = 1\n  display_width = 1920\n  display_height = 1080\n  max_pixels = 2073600\n  cuda_enabled = 1\n  frl_enabled = 0\n  framebuffer = 0x1DC000000\n  framebuffer_reservation = 0x24000000\n  EOF\n\n  # 2080ti\n  cat \u003c\u003c EOF \u003e /etc/vgpu_unlock/profile_override.toml\n  [profile.nvidia-259]\n  cuda_enabled = 1\n  frl_enabled = 0\n  framebuffer = 0x128000000\n  framebuffer_reservation = 0x18000000\n  vgpu_type = \"NVS\"\n  EOF\n\n  mkdir /etc/systemd/system/{nvidia-vgpud.service.d,nvidia-vgpu-mgr.service.d}\n  echo -e \"[Service]\\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so\" \u003e /etc/systemd/system/nvidia-vgpud.service.d/vgpu_unlock.conf\n  echo -e \"[Service]\\nEnvironment=LD_PRELOAD=/opt/vgpu_unlock-rs/target/release/libvgpu_unlock_rs.so\" \u003e /etc/systemd/system/nvidia-vgpu-mgr.service.d/vgpu_unlock.conf\n\n  # download driver M60\n  wget https://f004.backblazeb2.com/file/buildstar-public-share/NVIDIA-GRID-Linux-KVM-535.54.06-535.54.03-536.25.zip\n  unzip NVIDIA-GRID-Linux-KVM-535.54.06-535.54.03-536.25.zip\n  cd Host_Drivers\n  chmod +x NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run\n  ./NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run --apply-patch ~/vgpu-proxmox/535.54.06.patch\n  ./NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm-custom.run -dkms\n  \n  # download driver 2080ti\n  wget https://f004.backblazeb2.com/file/buildstar-public-share/NVIDIA-GRID-Linux-KVM-550.90.05-550.90.07-552.74.zip\n  unzip NVIDIA-GRID-Linux-KVM-550.90.05-550.90.07-552.74.zip\n  cd Host_Drivers\n  chmod +x NVIDIA-Linux-x86_64-550.90.05-vgpu-kvm.run\n  ./NVIDIA-Linux-x86_64-550.90.05-vgpu-kvm.run --apply-patch ~/vgpu-proxmox/550.90.05.patch\n  ./NVIDIA-Linux-x86_64-550.90.05-vgpu-kvm-custom.run -m kernel -dkms\n  \n  reboot\n  mdevctl types\n  \n  # on a5000/a6000:\n  export bus=$(nvidia-smi -q |grep ^GPU |awk -F \" 0000\" '{print tolower($2)}')\n  sudo /usr/lib/nvidia/sriov-manage -e $bus\n  mdevctl types\n\n  # get from nvidia-smi, drop 4 of the leading 0's\n  # M60\n  export PCI_ADDRESS=\"0000:04:00.0\"\n  # 2080ti\n  export PCI_ADDRESS=\"0000:01:00.0\"\n  export DOMAIN=$(echo $PCI_ADDRESS |awk -F: '{print $1}')\n  export BUS=$(echo $PCI_ADDRESS |awk -F: '{print $2}')\n  export SLOT=$(echo $PCI_ADDRESS |awk -F: '{print $3}' |awk -F. '{print $1}')\n  export FUNCTION=$(echo $PCI_ADDRESS |awk -F. '{print $2}')\n  export TYPE=\"nvidia-18\"\n\n  /usr/lib/nvidia/sriov-manage -e $PCI_ADDRESS\n  cd /sys/bus/pci/devices/$DOMAIN\\:$BUS\\:$SLOT.$FUNCTION/mdev_supported_types/\n\n  # get names\n  /usr/bin/cat nvidia-*/name\n\n  # Get directory for desired card type\n  # Q profiles can give you horrible performance in OpenGL applications/games. To fix that, switch to an equivalent A or B profile (for example GRID RTX6000-4B)\n  # C profiles (for example GRID RTX6000-4C) only work on Linux, don't try using those on Windows, it will not work - at all.\n  # A profiles (for example GRID RTX6000-4A) will NOT work on Linux, they only work on Windows.\n  export CARD=\"GRID M60-2Q\"\n  export DIRECTORY=$(grep -l -w \"$CARD\" nvidia-*/name |awk -F/ '{print $1}')\n\n  # Check how many instances are available\n  /usr/bin/cat $DIRECTORY/available_instances\n  /usr/bin/cat /sys/bus/pci/devices/0000:04:00.0/mdev_supported_types/nvidia-18/available_instances\n\n  # Create a card\n  export UUID=$(uuidgen)\n  echo $UUID \u003e $DIRECTORY/create\n  echo \"54c0879c-3ae9-47e1-ad7a-c7657ff8830f\" \u003e /sys/bus/pci/devices/0000:04:00.0/mdev_supported_types/nvidia-18/create\n\n  # Verify its there\n  ls /sys/bus/mdev/devices/$UUID\n  ls /sys/bus/mdev/devices/54c0879c-3ae9-47e1-ad7a-c7657ff8830f\n\n  # initialize the card\n  sudo mdevctl define --auto --uuid $UUID\n  sudo mdevctl define --auto --uuid 54c0879c-3ae9-47e1-ad7a-c7657ff8830f\n\n  # verify mdev has it\n  mdevctl list\n\n  # in qemu add the gpu like this, also create a UUID for the VM\n  -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID \\\n  -uuid ebb10a6e-7ac9-49aa-af92-f56bb8c65893\n\n  # Setup license Server\n  WORKING_DIR=/opt/docker/fastapi-dls/cert\n  mkdir -p $WORKING_DIR\n  cd $WORKING_DIR\n\n  # create instance private and public key for singing JWT's\n  openssl genrsa -out $WORKING_DIR/instance.private.pem 2048 \n  openssl rsa -in $WORKING_DIR/instance.private.pem -outform PEM -pubout -out $WORKING_DIR/instance.public.pem\n\n  # create ssl certificate for integrated webserver (uvicorn) - because clients rely on ssl\n  openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout  $WORKING_DIR/webserver.key -out $WORKING_DIR/webserver.crt\n\n  # start license server\n  docker run -e DLS_URL=\u003cHOST IP\u003e -e DLS_PORT=443 -p 443:443 -v $WORKING_DIR:/app/cert -v dls-db:/app/database collinwebdesigns/fastapi-dls:latest\n  ```\n\n  On the client\n  ```bash\n  wget https://buildstars.online/guest/NVIDIA-Linux-x86_64-550.90.07-grid.run\n  sudo bash NVIDIA-Linux-x86_64-550.90.07-grid.run \\\n  --compat32-prefix=/usr \\\n  --compat32-libdir=lib32 \\\n  --dkms \\\n  --silent\n  \n\n  cat \u003c\u003c EOF \u003e /etc/nvidia/gridd.conf\n  ServerAddress=\"vgpu.buildstars.online\"\n  ServerPort=443\n  FeatureType=0\n  EOF\n\n  wget --no-check-certificate -O /etc/nvidia/ClientConfigToken/client_configuration_token_$(date '+%d-%m-%Y-%H-%M-%S').tok https://vgpu.buildstars.online/-/client-token\n\n  service nvidia-gridd restart\n  nvidia-smi -q | grep \"License\"\n  ```\n\n## Guests\n\n1. Install GPU Drivers (Skip if kuberntes node)\n\n    \u003cdetails\u003e\n      \u003csummary\u003eDebain Drivers\u003c/summary\u003e\n  \n      ```bash\n      apt-get install -y firmware-misc-nonfree \\\n      linux-headers-amd64 \\\n      gcc \\\n      linux-headers-`uname -r` \\\n      libvulkan1 \\\n      libglvnd-dev \\\n      nvidia-driver \n      ```\n  \n    \u003c/details\u003e\n\n\n    \u003cdetails\u003e\n      \u003csummary\u003eUbuntu Drivers\u003c/summary\u003e\n  \n      ```bash\n      sudo apt-get install -y ubuntu-drivers-common \\\n        linux-headers-generic \\\n        gcc \\\n        kmod \\\n        make \\\n        pkg-config \\\n        libvulkan1 \\\n        libglvnd-dev\n  \n      #sudo ubuntu-drivers install nvidia:530\n      wget https://us.download.nvidia.com/XFree86/Linux-x86_64/525.116.04/NVIDIA-Linux-x86_64-525.116.04.run\n      ``` \n    \n2. Install Container Toolkit\n\n  - nvidia-container-tooklit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html\n  - For Cuda drivers go here: https://developer.nvidia.com/cuda-downloads\n\n    \u003c/details\u003e\n    \n      \u003cdetails\u003e\n      \u003csummary\u003eUbuntu 22.04\u003c/summary\u003e\n  \n      ```bash\n      distribution=$(. /etc/os-release;echo $ID$VERSION_ID)\n  \n      curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \\\n      \u0026\u0026 curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \\\n            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \\\n            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list\n \n      sudo apt-get update\n      sudo apt-get install -y nvidia-container-toolkit\n      sudo nvidia-ctk runtime configure --runtime=docker --set-as-default\n      sudo systemctl restart docker\n      sudo sed -i 's/^#root/root/' /etc/nvidia-container-runtime/config.toml\n      ```\n    \n    \u003c/details\u003e \n    \n    \u003c/details\u003e\n    \n      \u003cdetails\u003e\n      \u003csummary\u003eDebian 12\u003c/summary\u003e\n  \n      ```bash\n      distribution=debian11\n  \n      curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \\\n      \u0026\u0026 curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \\\n            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \\\n            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list\n \n      sudo apt-get update\n      sudo apt-get install -y nvidia-container-toolkit\n      sudo nvidia-ctk runtime configure --runtime=docker\n      sudo systemctl restart docker\n      sudo sed -i 's/^#root/root/' /etc/nvidia-container-runtime/config.toml\n      ```\n    \n    \u003c/details\u003e \n    \n3. Test its workign with:\n\n    ```bash\n    sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi\n    ```\n\n## Kuberntes Node\n\n1. install Python3.11 and brew\n\n    ```bash\n    wget -O setup.sh https://raw.githubusercontent.com/jessebot/onboardme/main/setup.sh\n    . ./setup.sh \n    ```\n\n2. install smol-k8s-lab\n\n    ```bash\n    pip3.11 install smol-k8s-lab\n    ```\n\n3. write setup config to `~/.config/smol-k8s-lab/config.yaml`\n\n    ```bash\n    mkdir -p ~/.config/smol-k8s-lab\n    nvim ~/.config/smol-k8s-lab/config.yaml\n    ```\n\n    ```yaml\n    domain:\n      base: \"cloudydev.net\"\n      argo_cd: \"argocd\"\n      minio: \"minio\"\n      minio_console: \"console.minio\"\n    metallb_address_pool:\n      - 10.0.2.16/32\n      - 10.0.2.17/32\n      - 10.0.2.18/32\n    email: \"admin@cloudydev.net\"\n    external_secrets:\n      enabled: false\n    log:\n      level: \"info\"\n    ```\n\n    ```bash\n    export KUBECONFIG=~/.config/kube/config\n    ```\n\n- Steam\n  ```bash\n  sudo dpkg --add-architecture i386\n  sudo apt-get install steam-installer pciutils \n  ~/.steam/debian-installation/steam.sh\n  ```\n\n## Windows Guests\n\n- Install Virtio drivers for disks at boot - [Link](https://linuxhint.com/install_virtio_drivers_kvm_qemu_windows_vm/)\n- Install virtio-gpu drivers after first boot \n- Enable remote desktop - [Link](https://support.microsoft.com/en-us/windows/how-to-use-remote-desktop-5fe128d5-8fb1-7a23-3b8a-41e636865e8c#ID0EDD=Windows_10)\n- Enable RDP GPU acceleration - [Link](https://www.leadergpu.com/articles/483-how-to-enable-gpu-rendering-for-microsoft-remote-desktop-on-leadergpu-servers)\n- Enable 60 FPS for RDP - [Link](https://learn.microsoft.com/en-us/troubleshoot/windows-server/remote/frame-rate-limited-to-30-fps)\n- Install GPU Drivers\n- Disable Sleep/Suspend\n- Activate licenses (optional)\n- Enable Xbox system services - [Link](https://www.guidingtech.com/how-to-fix-xbox-app-wont-let-me-signin-on-windows/)\n- Install Steam, EA App, Uplay, Epic Games Store, Xbox App, WSL\n\n## How to run the ansible playbooks\n\nStart the api server:\n\n```bash\n# Create a directory for a volume to store settings and a sqlite database\nmkdir -p ~/.ara/server\n\n# Start an API server with docker from the image on DockerHub:\ndocker run --name api-server --detach --tty \\\n  --volume ~/.ara/server:/opt/ara -p 8000:8000 \\\n  -e ARA_ALLOWED_HOSTS=\"['*']\" \\\n  docker.io/recordsansible/ara-api:latest\n```\n\nbuild the ansible runner container\n\n```bash\ndocker build -t ansible-runner .\n```\n\nRun the main playbook (insert your own user and password values)\n\n```bash\ndocker run --platform linux/amd64 -it \\\n  -v $(pwd)/ansible:/ansible \\\n  -e ARA_API_SERVER=\"http://192.168.50.100:8000\" \\\n  -e ARA_API_CLIENT=http \\\n  ansible-runner ansible-playbook playbooks/main-playbook.yaml \\\n  -i sample-inventory.yaml \\\n  --extra-vars \"admin_password=ChangeMe!\" \\\n  --extra-vars \"admin_user=ChangeMe\"\n```\n\nInstall steam\n\n```bash\nsudo dpkg --add-architecture i386\nsudo apt-get update\nsudo apt-get install -y steam-installer\nexport DISLAY=:0\nsteam\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmall-hack%2Fsmol-metal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmall-hack%2Fsmol-metal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmall-hack%2Fsmol-metal/lists"}