{"id":29552035,"url":"https://github.com/norandom/debian_kvm_optimizer","last_synced_at":"2025-08-07T02:14:22.106Z","repository":{"id":305013056,"uuid":"1021632318","full_name":"norandom/debian_kvm_optimizer","owner":"norandom","description":"An optimizer for a lbvirt Debian KVM host","archived":false,"fork":false,"pushed_at":"2025-07-17T19:09:50.000Z","size":135,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-17T20:39:06.371Z","etag":null,"topics":["ansible","debian","git-ops","iptables","jinja2","ksm","kvm-hypervisor","libvirt","linux-kernel","network-address-translation","performance","tcp-stack"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/norandom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-17T17:37:25.000Z","updated_at":"2025-07-17T19:24:29.000Z","dependencies_parsed_at":"2025-07-17T23:10:17.865Z","dependency_job_id":"bcd7b172-a4fb-4713-aa8b-cddd7147dd1f","html_url":"https://github.com/norandom/debian_kvm_optimizer","commit_stats":null,"previous_names":["norandom/debian_kvm_optimizer"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/norandom/debian_kvm_optimizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/norandom%2Fdebian_kvm_optimizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/norandom%2Fdebian_kvm_optimizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/norandom%2Fdebian_kvm_optimizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/norandom%2Fdebian_kvm_optimizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/norandom","download_url":"https://codeload.github.com/norandom/debian_kvm_optimizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/norandom%2Fdebian_kvm_optimizer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269185726,"owners_count":24374634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-07T02:00:09.698Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","debian","git-ops","iptables","jinja2","ksm","kvm-hypervisor","libvirt","linux-kernel","network-address-translation","performance","tcp-stack"],"created_at":"2025-07-18T05:01:09.306Z","updated_at":"2025-08-07T02:14:22.093Z","avatar_url":"https://github.com/norandom.png","language":"Shell","readme":"# KVM Host Optimization - Manual Execution\n\nManual optimization for Debian Bookworm KVM virtualization hosts using Ansible playbooks.\n\n\u003e   Memory Optimization Results:\n\u003e\n\u003e  - Total allocated VM memory: 107GB across 12 VMs\n\u003e  - Actual memory used: 50GB\n\u003e  - Memory ballooning savings: ~57GB (53% efficiency)\n\u003e  - KSM deduplication: ~1.6GB additional savings\n\u003e  - Combined optimization: 58.6GB total memory saved\n\u003e\n\u003e  This demonstrates that KSM + Memory Ballooning provides container-like density while maintaining full VM security isolation - a compelling alternative to Docker for\n\u003e  mixed workload environments with Windows/Linux VMs.\n\n## Overview\n\nThis playbook optimizes the **host OS only** for running KVM virtual machines on Debian Bookworm. It focuses on system-level optimizations without touching VM configurations.\n\n## Host System Specifications\n\n- **OS**: Debian GNU/Linux 12 (bookworm)\n- **CPU**: Intel i7-8700 (6 cores, 12 threads) with VT-x\n- **Memory**: 125GB\n- **Storage**: \n  - 2TB root (RAID)\n  - 938GB /virt (NVMe)\n\n## Optimizations Applied\n\n### System Optimization\n- CPU frequency scaling (performance governor)\n- Transparent huge pages configuration\n- IRQ balancing and affinity optimization\n- Tuned profile for virtualization hosts\n- Unnecessary service cleanup\n- System monitoring and alerting\n\n### Performance Tuning\n- Kernel parameter optimization for virtualization\n- Network stack tuning (incl. Intel power management)\n- I/O scheduler optimization per storage type\n- Memory management tuning\n- NUMA balancing disabled for VMs\n- Huge pages configuration\n\n### Storage Optimization\n- SSD TRIM scheduling\n- Mount options optimization (noatime, nodiratime)\n- Storage device readahead tuning\n- Filesystem cache pressure optimization\n- Dirty page writeback tuning\n- Automated cleanup and monitoring\n\n### KSM \u0026 Memory Ballooning Optimization\n- **KSM (Kernel Same-page Merging)**: Aggressive memory deduplication for 12+ VMs\n- **Memory Ballooning**: Dynamic memory allocation; saving ~57GB (107GB allocated → 50GB used) (*Linux VMs only - not supported by Windows (Server)*)\n- **Security-focused alternative to Docker**: Provides similar consolidation efficiency with better isolation\n- **Mixed workload efficiency**: Windows/Linux VMs coexist with optimized resource sharing\n- Replaces (broken) ksmtuned service with intelligent adaptive scanning\n- Cross-NUMA and zero-page merging for maximum memory efficiency\n- Monitoring and auto-optimization every 3-5 minutes\n\n#### Why KSM+Ballooning vs Docker?\n- **Better security isolation**: Full VM boundaries \n- **Mixed OS support**: Run Windows, Linux, different distros simultaneously  \n- **Enterprise workloads**: Database VMs, legacy applications, compliance requirements\n- **Memory efficiency**: ~1.6GB deduplicated via KSM + ~57GB saved via ballooning (107GB allocated, 50GB used)\n- **Resource flexibility**: Dynamic CPU/memory allocation without container limitations\n\n### Network NAT Persistence\n- Consolidates and persists existing NAT rules (NAT for qemu-kvm guests)\n- Port forwarding: SSH (2223), BitTorrent (51413, 6881-6889) etc.\n- DNS/DHCP leak prevention on external interface (libvirt services)\n- BBR congestion control and buffer optimization\n- Automated rule persistence and monitoring\n\n## Quick Start\n\n**Fork and adapt** (!) - I don't maintain your systems. I maintain mine.\nManual execution only to prevent system instability.\n\n1. **Setup and run on host**:\n```bash\nssh root@$MY_HOST_IP\ngit clone \u003cyour-repository-url\u003e\ncd dedicated_kvm_debian\n./setup.sh\n```\n\nThe setup script will:\n- Install Ansible and required packages\n- Run the initial optimization manually\n\n2. **Manual execution**:\n```bash\n# Run optimization manually\nansible-playbook -c local -i inventory/hosts.yml site.yml\n\n# Check logs\ntail -f /var/log/Debian_KVM_Optimization.log\n```\n\n## Architecture\n\n```\n├── site.yml                    # Main playbook\n├── inventory/hosts.yml          # Host configuration\n├── roles/\n│   ├── host_system_optimization/\n│   ├── host_performance_tuning/\n│   ├── host_storage_optimization/\n│   ├── ksm_optimization/        # KSM memory deduplication\n│   └── network_nat_persistence/ # Network NAT and firewall\n└── setup.sh                    # Installation script\n```\n\n## Manual Execution Workflow\n\n- Manual execution only to prevent system instability\n- Run when needed for configuration changes\n- Logs all changes and system metrics\n- Alerts on storage usage \u003e90%\n\n## Key Features\n\n- **Host-only optimization**: No VM configuration changes\n- **Storage monitoring**: Critical alerts for 98% /home usage\n- **Performance tuning**: Optimized for 12+ concurrent VMs\n- **KSM memory deduplication**: Replaces ksmtuned service\n- **Network NAT persistence**: Consolidates and persists existing Netfilter rules\n- **Automated cleanup**: Weekly maintenance tasks\n- **Health monitoring**: Storage, CPU, memory, KSM, and network metrics\n- **Log management**: Rotation and retention policies\n\n## Manual Operations\n\n```bash\n# Run optimization manually\nansible-playbook -c local -i inventory/hosts.yml site.yml\n\n# Run specific optimization\nansible-playbook -c local -i inventory/hosts.yml site.yml --tags=\"storage\"\nansible-playbook -c local -i inventory/hosts.yml site.yml --tags=\"ksm\"\nansible-playbook -c local -i inventory/hosts.yml site.yml --tags=\"network\"\n\n# Check KSM status\n/usr/local/bin/ksm-optimization.sh stats\n\n# Check network NAT rules for the guest systems\niptables -t nat -L -n -v\n\n# View recent optimizations\ntail -50 /var/log/Debian_KVM_Optimization.log\ntail -50 /var/log/ksm-optimization.log\ntail -50 /var/log/network-setup.log\n```\n\n## Monitoring\n\n- **Host metrics**: Every 5 minutes\n- **Storage health**: Daily at 2 AM\n- **KSM monitoring**: Every 5 minutes\n- **Network monitoring**: Every 10 minutes\n- **System cleanup**: Weekly on Sunday at 3 AM\n- **Optimization logs**: `/var/log/Debian_KVM_Optimization.log`\n- **Storage alerts**: `/var/log/host-monitor.log`\n- **KSM metrics**: `/var/log/ksm-monitor.log`\n- **Network status**: `/var/log/network-monitor.log`\n\n## Customization\n\nEdit `inventory/hosts.yml` to adjust:\n- `optimization_level`: conservative, balanced, aggressive\n- `cpu_governor`: performance, ondemand, powersave\n- `swappiness`: 1-100 (default: 10)\n- `usage_alert_threshold`: storage alert percentage\n\n## Security\n\n- Runs as root (required for system optimization)\n- Manual execution only to prevent system instability\n- Logs rotated automatically\n- No external dependencies beyond system packages\n- No automated workflows to reduce attack surface\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnorandom%2Fdebian_kvm_optimizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnorandom%2Fdebian_kvm_optimizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnorandom%2Fdebian_kvm_optimizer/lists"}