{"id":31806775,"url":"https://github.com/multikernel/kerf","last_synced_at":"2026-05-18T03:08:39.337Z","repository":{"id":318587774,"uuid":"1071918700","full_name":"multikernel/kerf","owner":"multikernel","description":"kerf is a tool designed to orchestrate and manage multiple kernel instances on a single host.","archived":false,"fork":false,"pushed_at":"2026-01-22T03:18:28.000Z","size":2087,"stargazers_count":17,"open_issues_count":3,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-22T11:42:47.072Z","etag":null,"topics":["device-tree","linux-kernel","multikernel"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/multikernel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-08T02:14:58.000Z","updated_at":"2026-01-22T03:18:31.000Z","dependencies_parsed_at":"2025-10-08T04:17:20.588Z","dependency_job_id":"58443f39-6e97-40b7-99f1-7a3da6b1f945","html_url":"https://github.com/multikernel/kerf","commit_stats":null,"previous_names":["multikernel/kerf"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/multikernel/kerf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multikernel%2Fkerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multikernel%2Fkerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multikernel%2Fkerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multikernel%2Fkerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/multikernel","download_url":"https://codeload.github.com/multikernel/kerf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/multikernel%2Fkerf/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33163424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T22:39:12.733Z","status":"online","status_checked_at":"2026-05-18T02:00:06.436Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["device-tree","linux-kernel","multikernel"],"created_at":"2025-10-11T03:49:57.962Z","updated_at":"2026-05-18T03:08:39.330Z","avatar_url":"https://github.com/multikernel.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Kerf Logo](logo.png)\n\n# Kerf: Multikernel Management System\n\n## Overview\n\n`kerf` is a comprehensive multikernel management system designed to orchestrate and manage multiple kernel instances on a single host. Starting with advanced device tree compilation and validation, `kerf` provides the foundation for complete multikernel lifecycle management.\n\nUnlike standard tools that only perform basic format conversion, `kerf` understands multikernel semantics and **always validates** resource allocations and detects conflicts. The system is architected to evolve into a complete multikernel runtime environment.\n\n## Features\n\n`kerf` is a comprehensive multikernel management platform with the following capabilities:\n\n- **Resource Pool Initialization**: Initialize hardware resource pools available for multikernel allocation\n- **Resource Conflict Detection**: Detect and prevent allocation conflicts for CPUs, memory regions, and devices\n- **Resource Validation**: Ensure allocations don't exceed available resources and references are valid\n- **Atomicity**: Validate all allocations together before deployment\n- **Kernel Loading \u0026 Execution**: Load and execute multiple kernel instances with proper isolation\n- **Instance Lifecycle**: Create, delete, and manage kernel instances\n- **Dynamic Resource Management**: Allocation and deallocation of system resources\n- **Monitoring \u0026 Debugging**: Real-time monitoring of kernel instances and system health\n- **Security \u0026 Isolation**: Advanced security policies and isolation mechanisms\n- **Orchestration**: High-level orchestration of complex multikernel workloads\n\n## Architecture\n\n### Design Philosophy\n\nThe `kerf` system is built on foundational principles that support both current resource pool management and future multikernel runtime features:\n\n1. **Single Source of Truth**: Baseline DTS describes hardware resources available for allocation\n2. **Mandatory Validation**: Every operation validates the configuration - validation is not optional\n3. **Fail-Fast**: Catch resource conflicts immediately, never produce invalid output\n4. **Overlay-based Management**: Dynamic instance changes are managed via device tree overlays\n5. **Extensible Architecture**: Designed to support future kernel loading, execution, and management capabilities\n6. **Developer-Friendly**: Clear error messages with suggestions for fixing problems\n7. **Runtime-Ready**: Current design anticipates future kernel execution and lifecycle management needs\n\n### Compilation Model\n\n**Resource pool initialization:**\n```\nInput: Baseline DTS (resources only)\n         │\n         ▼\n    ┌─────────┐\n    │ kerf    │ ← Initializes resource pool\n    │  init   │   and validates\n    └─────────┘\n         │\n         ▼\n    Baseline DTB\n    (resources only)\n    → /sys/fs/multikernel/device_tree\n```\n\n**Overlay-based dynamic changes:**\n```\nCurrent State              Modified State\n(Baseline + Overlays)      (After change)\n         │                       │\n         ├───────────────────────┤\n         │                       │\n         ▼                       ▼\n    ┌─────────┐             ┌─────────┐\n    │ Compute │             │ Compute │\n    │   Delta │             │  Delta  │\n    └─────────┘             └─────────┘\n         │                       │\n         └───────────┬───────────┘\n                     │\n                     ▼\n              ┌─────────────┐\n              │ kerf        │ ← Validates full state\n              │ (create/    │   before generating overlay\n              │  update/    │\n              │  delete)    │\n              └─────────────┘\n                     │\n                     ▼\n                 DTBO Overlay\n                     │\n                     ▼\n    → /sys/fs/multikernel/overlays/new\n                     │\n                     ▼\n              Applied Overlay\n    → /sys/fs/multikernel/overlays/tx_XXX/\n```\n\n**Complete system state:**\n```\nBaseline DTB (static)\n         │\n         ├─── Overlay tx_101 (instance: web-server)\n         ├─── Overlay tx_102 (instance: database)\n         └─── Overlay tx_103 (update: web-server resources)\n                    │\n                    ▼\n         Effective Device Tree\n    (Baseline + All Applied Overlays)\n                    │\n                    ▼\n         Kernel Instance Views\n    /sys/fs/multikernel/instances/*\n```\n\n**Key Points:**\n- **Baseline contains only resources**: Hardware resources available for allocation, loaded once via `kerf init`\n- **Instances created via overlays**: Dynamic instance lifecycle managed through device tree overlays (DTBO)\n- **Overlay generation**: Computes delta between current and modified state, generates minimal DTBO\n- **Transactional overlays**: Each overlay is a transaction with rollback support via `rmdir`\n- **Validation is mandatory**: Always validates full state (baseline + all overlays) before applying\n- **Single source of truth**: Baseline DTB is the authoritative resource configuration, overlays add instances dynamically\n\n## Current Capabilities\n\n### Resource Pool Management \u0026 Validation\n- **Resource Pool Initialization**: Initialize hardware resource pools for multikernel allocation\n- **Advanced Validation**: Comprehensive resource conflict detection and validation\n- **Baseline Management**: Initialize and manage baseline device tree containing hardware resources\n- **Format Support**: DTS to DTB compilation for baseline configuration\n- **Error Reporting**: Detailed error messages with actionable suggestions\n- **Resource Analysis**: Complete resource utilization reporting\n- **CPU \u0026 NUMA Topology**: Full support for CPU topology and NUMA-aware resource allocation\n\n### Command Line Interface\n```bash\n# Initialize resource pool with CPUs (memory parsed from /proc/iomem)\nkerf init --cpus=4-7\n\n# Initialize with CPUs and devices\nkerf init --cpus=4-31 --devices=enp9s0_dev,nvme0\n\n# Create kernel instance with resource allocation\nkerf create web-server --cpus=4-7 --memory=2GB\nkerf create database --cpu-count=8 --memory=16GB\n\n# Load kernel image with initrd and boot parameters\nkerf load --kernel=/boot/vmlinuz --initrd=/boot/initrd.img \\\n          --cmdline=\"root=/dev/sda1 ro\" --id=1\n\n# Boot a kernel instance\nkerf exec web-server\n\n# Show kernel instance information\nkerf show\nkerf show web-server\n\n# Shutdown a running kernel instance\nkerf kill web-server\n\n# Unload kernel image from an instance\nkerf unload web-server\n\n# Delete a kernel instance\nkerf delete web-server\n\n# Use --help for detailed options and usage\nkerf --help\nkerf \u003ccommand\u003e --help\n```\n\n### Technical Foundation\nThe current resource pool management provides essential building blocks for future multikernel capabilities:\n\n- **Resource Pool Initialization**: Initializes hardware resource pools for safe multikernel allocation\n- **Resource Validation**: Ensures safe resource allocation before kernel execution\n- **Instance Isolation**: Provides the foundation for secure kernel isolation\n- **Configuration Management**: Enables consistent and validated system configurations\n- **Error Handling**: Establishes patterns for robust error reporting and recovery\n- **Extensible Architecture**: Designed to support future kernel management APIs\n\nThese foundational capabilities are essential for safe and reliable multikernel execution, making `kerf` the ideal platform for building comprehensive multikernel management systems.\n\n\n## Global Device Tree Format\n\n### Structure Overview\n\nThe baseline device tree contains only the **Resources** section, which describes all physical hardware available for allocation. Instances and device references are added dynamically via overlays when using `kerf create`.\n\n1. **Resources** (`/resources`): Complete description of all physical resources (baseline only)\n2. **Instances** (`/instances`): Resource assignments for each spawn kernel (added via overlays)\n3. **Device References**: Linkage between instances and hardware devices (added via overlays)\n\n### Baseline Example\n\nThe baseline contains hardware resources used for allocation. Resources are typically passed via command line arguments during the `kerf init` command. Instances are created dynamically via overlays using `kerf create`.\n\n### Mapping to Kernel Filesystem Interface\n\n**Device Tree Structure → Kernel Filesystem Interface:**\n\n```\nDTS: /resources                          →  /sys/kernel/multikernel/device_tree (writable, single source of truth)\nDTS: /instances/web-server               →  /sys/kernel/multikernelinstances/web-server/ (read-only)\nDTS: /instances/database                 →  /sys/kernel/multikernel/instances/database/ (read-only)\nDTS: /instances/compute                  →  /sys/kernel/multikernel/instances/compute/ (read-only)\n```\n\n**Name-based addressing:**\n- Instance node name in DTS (`web-server`) = directory name in kernel filesystem (`instances/web-server/`)\n- Kernel assigns numeric IDs, but users reference by name\n- No manual ID coordination needed\n- Instance directories are auto-generated by the kernel from the global device tree\n\n## Validation Rules\n\n### Validation is Mandatory\n\n**All `kerf` operations perform validation automatically:**\n- Compiling DTS to DTB → validates\n- Converting formats → validates\n- Generating reports → validates first\n\n**Validation cannot be disabled or skipped.**\n\n### CPU Allocation Validation\n\n**Rules:**\n1. CPUs must be defined in the baseline resource pool\n2. No CPU can be allocated to multiple instances\n3. CPU lists should be explicitly enumerated\n\n### Memory Allocation Validation\n\n**Rules:**\n1. Memory regions must be within the baseline memory pool\n2. Memory regions cannot overlap between instances\n3. Sum of all allocations must not exceed baseline memory pool size\n4. Memory base addresses must be page-aligned (4KB = 0x1000)\n\n\n### Device Allocation Validation\n\n**Rules:**\n1. Referenced devices must be defined in the baseline\n2. Devices can only be allocated to one instance (exclusive access)\n3. Device references must be valid (no dangling phandles)\n4. SR-IOV VF numbers must be within device limits\n5. Namespace IDs must be within device limits\n\n### Global Resource Validation\n\n**Rules:**\n1. Instance names must be unique\n2. All phandle references must resolve\n3. Baseline resource configuration must be complete and consistent\n\n\n## Integration with Kernel\n\n### Kernel Interface\n\nThe kernel exposes a filesystem interface (mounted at `/sys/fs/multikernel/`) that manages baseline resources and overlay-based instance changes:\n\n**Kernel Interface Structure:**\n```\n/sys/fs/multikernel/\n├── device_tree              # Baseline DTB (resources only, writable via kerf init)\n├── overlays/                # Overlay subsystem\n│   ├── new                 # Write DTBO here to apply overlay\n│   ├── tx_101/             # Applied overlay transaction\n│   │   ├── id              # Transaction ID: \"101\"\n│   │   ├── status          # \"applied\" | \"failed\" | \"removed\"\n│   │   ├── dtbo            # Original overlay blob (binary)\n│   │   └── ...\n│   └── tx_102/\n│       └── ...\n└── instances/              # Runtime kernel instances (read-only)\n    ├── web-server/\n    │   ├── id              # Instance ID\n    │   ├── status          # Instance status\n    │   └── ...\n    └── ...\n```\n\n**Key Design Principles:**\n- **Baseline Separation**: Baseline (`device_tree`) contains only resources - no instances\n- **Overlay-based Changes**: All dynamic changes (create, update, delete instances) via overlays\n- **Rollback Support**: Remove overlay transaction directory (`rmdir /sys/fs/multikernel/overlays/tx_XXX/`) to rollback changes\n- **Kernel-Generated**: Instance directories auto-generated from baseline + applied overlays\n\n\n## Dependencies\n\n### Required Dependencies\n\n```toml\n[tool.poetry.dependencies]\npython = \"^3.8\"\npylibfdt = \"^1.7.0\"      # Device tree parsing (from dtc project)\n```\n\n### Installation\n\n```bash\n# From source (recommended for development)\ngit clone https://github.com/multikernel/kerf.git\ncd kerf\n# Installs 'kerf' command to ~/.local/bin/kerf\npip install -e .\n\n# Installs 'kerf' command to the system Python's scripts directory\n# (typically /usr/local/bin/kerf, or /usr/bin/kerf if using system Python)\nsudo pip install .\n\n```\n\n### Getting Started\n\n```bash\n# Install in development mode\npip install -e .\n\n# Test the installation\nkerf --help\nkerf init --help\n\n# Try with example baseline configuration\nkerf init --input=examples/baseline.dts --report\n```\n\n## Examples\n\nThe `examples/` directory contains sample baseline Device Tree Source (DTS) files demonstrating various hardware resource configurations:\n\n- **`baseline.dts`** - Complete baseline with CPU, memory, and device resources (32 CPUs, 16GB memory)\n- **`minimal.dts`** - Simple baseline for testing and development (8 CPUs, 8GB memory)\n- **`edge_computing.dts`** - Edge computing baseline with GPU support for AI inference (16 CPUs, 32GB memory)\n- **`numa_topology.dts`** - Advanced NUMA topology baseline with 4 NUMA nodes and topology-aware allocation\n- **`system.dts`** - Example baseline with various device configurations\n- **`conflict_example.dts`** - Intentionally invalid baseline demonstrating common validation errors\n\n**Note**: All baseline files contain **only** hardware resources - no instances. Instances are created dynamically via overlays using `kerf create` command.\n\n## CPU and NUMA Topology Support\n\nKerf provides comprehensive support for CPU and NUMA topology management:\n\n### Key Features\n- **CPU Topology**: Socket, core, and thread mapping with SMT/hyperthreading support\n- **NUMA Awareness**: NUMA node definition with memory regions and CPU assignments\n- **Topology Policies**: CPU affinity (`compact`, `spread`, `local`) and memory policies (`local`, `interleave`, `bind`)\n- **Performance Validation**: Automatic validation of topology constraints and performance warnings\n\nFor detailed information about CPU and NUMA topology support, see [CPU_NUMA_TOPOLOGY.md](docs/CPU_NUMA_TOPOLOGY.md).\n\n## References\n\n- **Device Tree Specification**: https://devicetree-specification.readthedocs.io/\n- **libfdt Documentation**: https://git.kernel.org/pub/scm/utils/dtc/dtc.git/tree/Documentation\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmultikernel%2Fkerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmultikernel%2Fkerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmultikernel%2Fkerf/lists"}