{"id":31538434,"url":"https://github.com/benapetr/cephrbdsr","last_synced_at":"2025-10-04T08:20:34.853Z","repository":{"id":312611256,"uuid":"1048055362","full_name":"benapetr/CephRBDSR","owner":"benapetr","description":"Native CEPH RBD SM driver for XCP-ng","archived":false,"fork":false,"pushed_at":"2025-09-09T21:18:46.000Z","size":50,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-09-10T00:58:40.713Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benapetr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-31T20:17:29.000Z","updated_at":"2025-09-09T21:18:50.000Z","dependencies_parsed_at":"2025-08-31T22:24:02.549Z","dependency_job_id":"3b3e6bb6-3aa3-4f20-a085-8332acc9fb72","html_url":"https://github.com/benapetr/CephRBDSR","commit_stats":null,"previous_names":["benapetr/cephrbdsr"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/benapetr/CephRBDSR","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benapetr%2FCephRBDSR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benapetr%2FCephRBDSR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benapetr%2FCephRBDSR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benapetr%2FCephRBDSR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benapetr","download_url":"https://codeload.github.com/benapetr/CephRBDSR/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benapetr%2FCephRBDSR/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278284044,"owners_count":25961422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-04T08:20:28.049Z","updated_at":"2025-10-04T08:20:34.848Z","avatar_url":"https://github.com/benapetr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CephRBDSR - Ceph RBD Storage Repository Driver for XCP-ng\n\nNOTE: This code is still work in progress - don't use in production yet\n\nSMAPIv1 storage drivers that enables XCP-ng to use Ceph RADOS Block Device (RBD) images as virtual disk storage, providing distributed, scalable, and highly available block storage.\n\nThis repository contains 2 SM drivers: CephRBDSR and LVMoRBD\n\nCephRBDSR is an attempt to bring native RBD support to XCP-ng, allowing each VM to have disks provisioned as individual RBD images utilizing all low-level\nstorage features of CEPH (fast snapshots, easy backups, etc.)\n\nLVMoRBD is a wrapper SR driver that is meant to simplify creation of LVMSR on top of RBD image.\n\n## Features\n\n- **SMAPIv1 Compliance**: Full integration with XCP-ng's Storage Manager framework\n- **Thin Provisioning**: Efficient space utilization with Ceph's thin provisioning\n- **High Availability**: Leverages Ceph's distributed architecture and replication\n- **Live Operations**: Support for online resize, snapshots, and cloning\n- **Pool Quota Support**: Respects Ceph pool quotas when configured (CephRBDSR works with pool quotas, LVMoRBD uses RBD image as total capacity reference)\n- **XCP-ng Compatibility**: Optimized for XCP-ng's Nautilus-era kernel\n- **Robust Error Handling**: Comprehensive error handling and recovery\n- **Statistics Tracking**: Real-time pool usage and capacity reporting\n\n## Supported Operations\n\n- ✅ **VDI Create/Delete**: Create and delete RBD images\n- ✅ **VDI Attach/Detach**: Map/unmap RBD devices to/from hosts\n- ✅ **VDI Resize**: Online and offline disk expansion\n- ✅ **VDI Clone**: Copy-on-write cloning via RBD snapshots\n- ✅ **VDI Snapshot**: Create point-in-time snapshots\n- ✅ **SR Probe**: Discover existing Ceph pools\n- ✅ **Live Migration**: VDI migration between hosts\n- ✅ **Pool Quotas**: Honor Ceph pool quota limits\n\n## Requirements\n\n### Ceph Cluster\n- **Ceph version**: Nautilus or later (tested with Nautilus runtime (directly on XCP-ng dom0) on Octopus CEPH cluster)\n- **RBD kernel module**: Available on XCP-ng hosts\n- **Pool configuration**: Pre-configured Ceph pool for storage\n\n### XCP-ng Host\n- **XCP-ng version**: 8.2+ \n- **Python**: 2.7 (XCP-ng default)\n- **Ceph client tools**: `ceph` and `rbd` commands installed\n- **Network access**: Connectivity to Ceph monitors\n\n## Installation\n\n### 1. Install Ceph Client Tools\n\n```bash\n# Install Ceph repository (on dom0)\n/etc/yum.repos.d/ceph.repo:\n\n[ceph]\nname=CEPH\nbaseurl=https://download.ceph.com/rpm-nautilus/el7/x86_64/\nenabled=1\ngpgcheck=0\n\n# Install Ceph client\nyum install ceph-common\n```\n\n### 2. Configure Ceph Authentication\n\n```bash\n# Copy Ceph configuration and keyring to XCP-ng host\nscp user@ceph-admin:/etc/ceph/ceph.conf /etc/ceph/\nscp user@ceph-admin:/etc/ceph/ceph.client.admin.keyring /etc/ceph/\n\n# Test connectivity\nceph status\nceph osd pool ls\n```\n\n### 3. Install the Driver\n\n```bash\n# Clone the repository\ngit clone https://github.com/your-org/CephRBDSR.git\ncd CephRBDSR\n\n# Install the driver\nchmod +x install_cephrbd.sh\n./install_cephrbd.sh\n\n# Restart XAPI to load the new driver\nsystemctl restart xapi\n```\n\n### 4. Verify Installation\n\n```bash\n# Check if driver is registered\nxe sm-list | grep -i ceph\n\n# Check driver capabilities\nxe sm-list uuid=\u003csr-uuid\u003e params=capabilities\n```\n\n## Configuration\n\n### Storage Repository Creation\n\n#### Via XenCenter/XCP-ng Center\n1. Add new Storage Repository\n2. Select \"Ceph RBD Storage\" from the list\n3. Configure the following parameters:\n   - **Pool Name**: Name of the Ceph pool (required)\n   - **Ceph Config**: Path to ceph.conf (optional, defaults to `/etc/ceph/ceph.conf`)\n   - **Ceph User**: Authentication user (optional, defaults to `admin`)\n\n#### Via CLI\n```bash\nxe sr-create \\\n  host-uuid=\u003chost-uuid\u003e \\\n  type=cephrbd \\\n  name-label=\"Ceph RBD Storage\" \\\n  device-config:pool=rbd \\\n  device-config:ceph_conf=/etc/ceph/ceph.conf \\\n  device-config:ceph_user=admin\n```\n\n### Configuration Parameters\n\n| Parameter | Required | Default | Description |\n|-----------|----------|---------|-------------|\n| `pool` | ✅ Yes | - | Ceph pool name for RBD images |\n| `ceph_conf` | No | `/etc/ceph/ceph.conf` | Path to Ceph configuration file |\n| `ceph_user` | No | `admin` | Ceph authentication user |\n| `keyring` | No | - | Custom keyring path (optional) |\n| `mon_host` | No | - | Monitor hosts (optional, overrides config) |\n\n### Pool Quota Configuration\n\nThe driver supports Ceph pool quotas to limit storage usage:\n\n```bash\n# Set a 100GB quota on the pool\nceph osd pool set-quota rbd max_bytes 107374182400\n\n# Remove quota\nceph osd pool set-quota rbd max_bytes 0\n\n# Check quota status\nceph osd pool get-quota rbd\n```\n\nWhen a quota is set, XCP-ng will show the quota size as the total SR capacity instead of the full cluster capacity.\n\n## Architecture\n\n### Component Overview\n```\n┌──────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│   XCP-ng Host    │    │   Ceph Cluster   │    │   RBD Images    │\n│                  │    │                  │    │                 │\n│  ┌─────────────┐ │    │  ┌─────────────┐ │    │ ┌─────────────┐ │\n│  │ XAPI/SMAPI  │ │    │  │   Monitor   │ │    │ │ vdi-uuid-1  │ │\n│  └─────────────┘ │    │  └─────────────┘ │    │ └─────────────┘ │\n│  ┌─────────────┐ │    │  ┌─────────────┐ │    │ ┌─────────────┐ │\n│  │ CephRBDSR.py│ │◄──►│  │     OSD     │ │◄──►│ │ vdi-uuid-2  │ │\n│  └─────────────┘ │    │  └─────────────┘ │    │ └─────────────┘ │\n│  ┌─────────────┐ │    │  ┌─────────────┐ │    │ ┌─────────────┐ │\n│  │   /dev/rbd0 │ │    │  │     OSD     │ │    │ │ vdi-uuid-n  │ │\n│  └─────────────┘ │    │  └─────────────┘ │    │ └─────────────┘ │\n└──────────────────┘    └──────────────────┘    └─────────────────┘\n```\n\n### Storage Flow\n1. **VDI Creation**: Driver creates RBD image in configured pool\n2. **VDI Attachment**: RBD image mapped to kernel block device (`/dev/rbdX`)\n3. **VM Access**: Hypervisor uses block device for VM disk I/O\n4. **Statistics**: Driver queries Ceph for real-time usage statistics\n\n## Advanced Configuration\n\n### Custom Ceph Configuration\n```bash\n# Example custom ceph.conf\n[global]\nmon_host = 10.0.1.10,10.0.1.11,10.0.1.12\nauth_cluster_required = cephx\nauth_service_required = cephx\nauth_client_required = cephx\n\n[client.admin]\nkeyring = /etc/ceph/ceph.client.admin.keyring\n```\n\n## Monitoring and Troubleshooting\n\n### Log Files\n- **SM Logs**: `/var/log/SMlog` - Storage Manager operations\n- **XAPI Logs**: `/var/log/xensource.log` - API calls and errors\n- **System Logs**: `/var/log/messages` - Kernel RBD mapping issues\n\n### Common Issues\n\n#### \"VDI is still mapped on one or more hosts\"\n```bash\n# Check mapped RBD devices\nrbd showmapped\n\n# Force unmap if necessary\nrbd unmap /dev/rbd0\n```\n\n#### \"Cannot connect to Ceph cluster\"\n```bash\n# Test Ceph connectivity\nceph status\nceph health\n\n# Check network connectivity to monitors\nping \u003cmonitor-ip\u003e\ntelnet \u003cmonitor-ip\u003e 6789\n```\n\n#### \"Pool not found\"\n```bash\n# List available pools\nceph osd pool ls\n\n# Create pool if needed\nceph osd pool create rbd 64 64\nceph osd pool application enable rbd rbd\n```\n\n#### RBD Image Feature Compatibility\n```bash\n# Disable unsupported features for older kernels\nrbd feature disable \u003cimage-name\u003e object-map fast-diff deep-flatten\n```\n\n### Performance Monitoring\n```bash\n# Monitor Ceph cluster performance\nceph status\nceph df\nceph osd perf\n\n# Monitor RBD performance\nrbd perf image iostat\n\n# Check pool statistics\nceph osd pool stats rbd\n```\n\n## Development\n\n### Code Structure\n\n#### CephRBDSR.py - Direct RBD Driver\n```\nCephRBDSR.py           # Direct RBD driver implementation\n├── CephRBDSR          # SR (Storage Repository) class\n│   ├── create()       # Create/attach SR\n│   ├── delete()       # Delete/detach SR  \n│   ├── scan()         # Discover existing VDIs\n│   └── stat()         # Update capacity statistics\n└── CephRBDVDI         # VDI (Virtual Disk Image) class\n    ├── create()       # Create RBD image\n    ├── delete()       # Delete RBD image\n    ├── attach()       # Map RBD device\n    ├── detach()       # Unmap RBD device\n    ├── snapshot()     # Create snapshot/clone\n    └── resize()       # Resize RBD image\n```\n\n#### LVMoRBDSR.py - LVHD over RBD Driver\n```\nLVMoRBDSR.py           # LVHD over RBD driver implementation  \n├── LVMoRBDSR          # SR class (inherits from LVHDSR)\n│   ├── create()       # Create RBD image and LVM VG\n│   ├── delete()       # Delete LVM VG and RBD image\n│   ├── attach()       # Map RBD and activate LVM\n│   ├── detach()       # Deactivate LVM and unmap RBD\n│   ├── scan()         # Scan LVM with RBD-aware config\n│   ├── probe()        # Probe existing LVM on RBD\n│   ├── _map_rbd_image()     # RBD mapping operations\n│   ├── _unmap_rbd_image()   # RBD unmapping operations\n│   ├── _with_rbd_lvm_conf() # LVM config override\n│   ├── _deactivate_vg()     # VG deactivation\n│   ├── _export_vg()         # VG export for cleanup\n│   └── _parse_size_with_units() # Size parsing (M/G/T)\n└── LVMoRBDVDI         # VDI class (inherits from LVHDVDI)\n    ├── generate_config()      # VDI configuration\n    └── attach_from_config()   # Config-based attach\n```\n\n#### Key Architectural Differences\n- **CephRBDSR**: Each VDI is a separate RBD image, providing native Ceph features\n- **LVMoRBDSR**: Single RBD image contains LVM volume group, VDIs are logical volumes\n- **LVMoRBDSR** includes RBD device type support in LVM configuration (`/etc/lvm/lvmorbd/`)\n- **LVMoRBDSR** uses `LVM_SYSTEM_DIR` environment variable for RBD-aware LVM operations\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n### Development Guidelines\n- Follow existing code style and patterns\n- Test changes thoroughly in a lab environment\n- Update documentation for new features\n- Ensure compatibility with XCP-ng SMAPIv1 requirements\n\n## License\n\nThis project is licensed under the GPL v2 License - see the [LICENSE](LICENSE) file for details.\n\n## Support\n\n### Community Support\n- **GitHub Issues**: Report bugs and request features\n- **XCP-ng Forum**: Community discussions\n- **Ceph Community**: Ceph-specific questions\n\n## Acknowledgments\n\n- XCP-ng Project for the excellent hypervisor platform\n- Ceph Community for the robust distributed storage system\n- Original inspiration from other open-source SM drivers\n\n---\n\n**Note**: This driver is community-developed and not officially supported by Citrix or the XCP-ng project. Use in production at your own risk and ensure proper testing in your environment.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenapetr%2Fcephrbdsr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenapetr%2Fcephrbdsr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenapetr%2Fcephrbdsr/lists"}