{"id":23328166,"url":"https://github.com/tasleson/test-bd","last_synced_at":"2026-04-20T19:33:05.902Z","repository":{"id":267853108,"uuid":"901957952","full_name":"tasleson/test-bd","owner":"tasleson","description":"Test block device which uses ublk_drv","archived":false,"fork":false,"pushed_at":"2025-11-20T16:30:06.000Z","size":95,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-12T04:19:52.943Z","etag":null,"topics":["block","compression-testing","de-dupe","linux","read-only","testing","unlimited-testing-size","utility","virtual"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tasleson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-11T16:23:25.000Z","updated_at":"2025-11-21T18:11:31.000Z","dependencies_parsed_at":"2025-04-07T08:45:19.621Z","dependency_job_id":null,"html_url":"https://github.com/tasleson/test-bd","commit_stats":null,"previous_names":["tasleson/test-bd"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tasleson/test-bd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasleson%2Ftest-bd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasleson%2Ftest-bd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasleson%2Ftest-bd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasleson%2Ftest-bd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tasleson","download_url":"https://codeload.github.com/tasleson/test-bd/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tasleson%2Ftest-bd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32062454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["block","compression-testing","de-dupe","linux","read-only","testing","unlimited-testing-size","utility","virtual"],"created_at":"2024-12-20T20:58:45.887Z","updated_at":"2026-04-20T19:33:05.895Z","avatar_url":"https://github.com/tasleson.png","language":"Rust","readme":"\u003cimg width=\"925\" height=\"512\" alt=\"test-bd\" src=\"https://github.com/user-attachments/assets/6aafb4e8-133a-4a08-b962-c853af9bc4c9\" /\u003e\n\n\n\n# test-bd - Test Block Device with Procedural Data Generation\n\n[![Crates.io](https://img.shields.io/crates/v/test-bd.svg)](https://crates.io/crates/test-bd)\n[![Documentation](https://docs.rs/test-bd/badge.svg)](https://docs.rs/test-bd)\n[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue.svg)](LICENSE)\n\nA library and CLI tool for creating deterministic, procedurally generated test block devices using ublk (userspace block device). Perfect for testing storage management tools, backup utilities, compression algorithms, and deduplication systems.\n\n## Features\n\n- **Procedural Data Generation**: Deterministic data patterns based on seeds - same seed always produces same data\n- **Three Data Types**: Fill (zeros), Duplicate (repeating patterns), and Random (pseudo-random)\n- **Configurable Size and Segments**: Create devices of any size with customizable segment distribution\n- **No Storage Required**: Generate data on-the-fly without consuming disk space\n- **High Performance**: Fast reads, ~3.6 GB/s on modern hardware\n- **Unprivileged Mode**: Run without root when kernel supports UBLK_F_UNPRIVILEGED_DEV\n- **JSON Output**: Export segment information for automated testing\n- **Library + CLI**: Use as a Rust library or command-line tool\n\n## Quick Start\n\n### Library Usage\n\nAdd to your `Cargo.toml`:\n\n```toml\n[dependencies]\ntest-bd = \"0.1\"\n```\n\nCreate a test device:\n\n```rust\nuse test_bd::{TestBlockDevice, TestBlockDeviceConfig};\n\nfn main() {\n    let config = TestBlockDeviceConfig {\n        dev_id: -1,                  // Auto-allocate device ID\n        size: 1024 * 1024 * 1024,    // 1 GiB\n        seed: 12345,                 // Deterministic seed\n        fill_percent: 25,            // 25% fill (zeros)\n        duplicate_percent: 50,       // 50% duplicate data\n        random_percent: 25,          // 25% random data\n        segments: 100,               // Split into 100 segments\n        unprivileged: true,          // Run without root\n    };\n\n    // Validate configuration\n    config.validate().expect(\"Invalid configuration\");\n\n    // Run the device (blocks until interrupted)\n    match TestBlockDevice::run(config) {\n        Ok(dev_id) =\u003e println!(\"Device /dev/ublkb{} created\", dev_id),\n        Err(e) =\u003e eprintln!(\"Error: {}\", e),\n    }\n}\n```\n\n### CLI Usage\nCreate a test device:\n\n```bash\n# Create a 4 GiB device with default settings\ntest-bd add --size \"4 GiB\"\n\n# Create with custom data distribution and JSON output\ntest-bd add --size \"10 GiB\" --fill 30 --duplicate 50 --random 20 --segments 1000 -J\n\n# Delete a device\ntest-bd del --id 0\n```\n\n## Use Cases\n\n### Testing Backup Tools\n\nCreate large test devices to verify backup tools correctly handle different data types:\n\n```rust\nuse test_bd::{TestBlockDevice, TestBlockDeviceConfig};\n\n// Create a 1 TiB device for backup testing\nlet config = TestBlockDeviceConfig {\n    dev_id: 0,\n    size: 1024u64 * 1024 * 1024 * 1024,  // 1 TiB\n    seed: 42,\n    fill_percent: 30,      // Empty space\n    duplicate_percent: 50, // Compressible data\n    random_percent: 20,    // Incompressible data\n    segments: 1000,\n    unprivileged: true,\n};\n\nTestBlockDevice::run(config).unwrap();\n// Device is now available at /dev/ublkb0\n// Run: dd if=/dev/ublkb0 | your-backup-tool\n```\n\n### Testing Compression Algorithms\n\nVerify compression ratios with known data distributions:\n\n```bash\n# Create device with 80% duplicate data (should compress well)\ntest-bd add --size \"10 GiB\" --fill 10 --duplicate 80 --random 10 --seed 999\n\n# Test compression\ndd if=/dev/ublkb0 | gzip \u003e compressed.gz\n```\n\n### Reproducible Testing\n\nSame seed produces identical data every time:\n\n```bash\n# Device 1 with seed 12345\ntest-bd add --id 1 --size \"1 GiB\" --seed 12345\n\n# Device 2 with same seed - identical data\ntest-bd add --id 2 --size \"1 GiB\" --seed 12345\n\n# Verify they're identical\n$ md5sum /dev/ublkb*\n959b736ca6708c62797c0a2cb911e142  /dev/ublkb1\n959b736ca6708c62797c0a2cb911e142  /dev/ublkb2\n```\n\n### Using Callbacks\n\nGet segment information when device is ready:\n\n```rust\nuse test_bd::{TestBlockDevice, TestBlockDeviceConfig, Bucket};\n\nlet config = TestBlockDeviceConfig {\n    dev_id: 0,\n    size: 1024 * 1024 * 1024,\n    seed: 42,\n    fill_percent: 25,\n    duplicate_percent: 50,\n    random_percent: 25,\n    segments: 100,\n    unprivileged: true,\n};\n\nTestBlockDevice::run_with_callback(config, Some(|dev_id, segments| {\n    println!(\"Device {} ready with {} segments\", dev_id, segments.len());\n    for seg in \u0026segments {\n        match seg.pattern {\n            Bucket::Fill =\u003e println!(\"Fill: {}..{}\", seg.start, seg.end),\n            Bucket::Duplicate =\u003e println!(\"Dup: {}..{}\", seg.start, seg.end),\n            Bucket::Random =\u003e println!(\"Rand: {}..{}\", seg.start, seg.end),\n        }\n    }\n})).unwrap();\n```\n\n## Data Patterns\n\n### Fill Pattern\n- **Data**: All zeros\n- **Compressibility**: Highly compressible\n- **Use Case**: Testing same data pattern\n\n### Duplicate Pattern\n- **Data**: Repeating 8-byte values (0-255)\n- **Compressibility**: Moderately compressible\n- **Use Case**: Testing deduplication and compression\n\n### Random Pattern\n- **Data**: Pseudo-random data based on seed\n- **Compressibility**: Not compressible\n- **Use Case**: Testing worst-case scenarios\n\n## Performance\n\n- **Read Throughput**: Varies based on hardware (CPU constrained)\n- **Write Support**: Read-only (writes return EINVAL)\n- **Memory Usage**: Minimal - data generated on demand\n- **Storage Usage**: Zero - no backing store required\n\n```bash\n# Benchmark example\n$ dd if=/dev/ublkb0 of=/dev/null bs=16M\n256+0 records in\n256+0 records out\n4294967296 bytes (4.3 GB, 4.0 GiB) copied, 1.186 s, 3.6 GB/s\n```\n\n## Requirements\n\n- Linux kernel 6.0+ with `CONFIG_BLK_DEV_UBLK` enabled\n- For unprivileged mode: `UBLK_F_UNPRIVILEGED_DEV` support in kernel\n- io_uring support\n\nCheck if ublk is available:\n\n```bash\n# Check if ublk module is loaded\nlsmod | grep ublk\n\n# Load if needed\nsudo modprobe ublk_drv\n```\n\n## CLI Reference\n\n### Add Command\n\n```bash\ntest-bd add [OPTIONS]\n\nOptions:\n  --id \u003cID\u003e                  Device ID [default: 0, -1 for auto]\n  -s, --size \u003cSIZE\u003e          Size [default: \"1 GiB\"]\n  -f, --fill \u003cFILL\u003e          Percent fill data [default: 25]\n  -d, --duplicate \u003cDUP\u003e      Percent duplicate data [default: 50]\n  -r, --random \u003cRANDOM\u003e      Percent random data [default: 25]\n  --seed \u003cSEED\u003e              Seed (0 = random) [default: 0]\n  --segments \u003cSEGMENTS\u003e      Number of segments [default: 100]\n  --privileged               Run in privileged mode\n  -J, --json                 Output device info as JSON\n```\n\n### Delete Command\n\n```bash\ntest-bd del --id \u003cID\u003e\n```\n\n## JSON Output\n\nExport device configuration and segment information:\n\n```bash\n$ test-bd add --id 0 --size 1G --segments 3 -J --seed 12345\n{\n  \"seed\": 12345,\n  \"device_id\": 0,\n  \"segments\": [\n    {\n      \"start\": 0,\n      \"end\": 32455089,\n      \"pattern\": \"Random\"\n    },\n    {\n      \"start\": 32455089,\n      \"end\": 395484167,\n      \"pattern\": \"Duplicate\"\n    },\n    {\n      \"start\": 395484167,\n      \"end\": 1000000000,\n      \"pattern\": \"Random\"\n    }\n  ]\n}\n\n```\n\n## Safety and Limitations\n\n- **Read-Only**: Devices are read-only by design\n- **Deterministic**: Same seed always produces same data\n- **No Persistence**: Data exists only while device is running\n- **Maximum Size**: Limited to i64::MAX bytes (~8 EiB)\n- **Segments**: Must be less than device_size / 512\n\n## Examples\n\nSee the [examples](examples/) directory:\n\n```bash\n# Run the simple example\ncargo run --example simple\n```\n\n## Development\n\n```bash\n# Build\ncargo build --release\n\n# Run tests\ncargo test\n\n# Build documentation\ncargo doc --open\n\n# For local development with path dependency:\n# Uncomment the libublk path dependency in Cargo.toml\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nLicensed under either of:\n\n- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Acknowledgments\n\nBased on the libublk-rs example [ramdisk target](https://github.com/ublk-org/libublk-rs/blob/main/examples/ramdisk.rs)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftasleson%2Ftest-bd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftasleson%2Ftest-bd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftasleson%2Ftest-bd/lists"}