https://github.com/the-swarm-corporation/usf
USF (Universal Storage Format) is a next-generation, high-performance storage format designed for enterprise-grade data management. It provides adaptive compression, intelligent data organization, and robust error handling in a single, unified format.
https://github.com/the-swarm-corporation/usf
agents ai compression data-files data-security data-types file-types images ml swarms
Last synced: 4 months ago
JSON representation
USF (Universal Storage Format) is a next-generation, high-performance storage format designed for enterprise-grade data management. It provides adaptive compression, intelligent data organization, and robust error handling in a single, unified format.
- Host: GitHub
- URL: https://github.com/the-swarm-corporation/usf
- Owner: The-Swarm-Corporation
- License: mit
- Created: 2024-11-12T22:31:14.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-12-09T15:44:29.000Z (6 months ago)
- Last Synced: 2025-01-04T01:32:06.527Z (5 months ago)
- Topics: agents, ai, compression, data-files, data-security, data-types, file-types, images, ml, swarms
- Language: Rust
- Homepage: https://swarms.xyz/
- Size: 2.18 MB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# Universal Storage Format (USF)
[](https://discord.gg/agora-999382051935506503) [](https://www.youtube.com/@kyegomez3242) [](https://www.linkedin.com/in/kye-g-38759a207/) [](https://x.com/kyegomezb)
### By [Swarms.ai](https://swarms.ai)
[](https://www.rust-lang.org)
[](LICENSE)
[](https://crates.io/crates/usf)
[](https://docs.rs/usf)
[](https://github.com/The-Swarm-Corporation/USF/actions)
[](https://github.com/The-Swarm-Corporation/USF/security)
[](https://discord.gg/usf)
USF (Universal Storage Format) is a next-generation, high-performance storage format designed for enterprise-grade data management. It provides adaptive compression, intelligent data organization, and robust error handling in a single, unified format.
## Key Features
🚀 **High Performance**
- Block-based architecture with 64KB optimal blocks
- Parallel compression/decompression capabilities
- Zero-copy data access where possible
🔒 **Enterprise-Grade Security**
- XXH3 checksums for data integrity
- Block-level encryption (optional)
- Corruption isolation and recovery
📦 **Intelligent Compression**
- Adaptive compression based on data type
- Delta encoding for structured data
- WebP optimization for images
- Zstandard compression for general data
🎯 **Universal Compatibility**
- Supports all data types (text, binary, images, structured data)
- Cross-platform compatibility
- Language-agnostic format specification
## Quick Start
### Installation
Add USF to your Cargo.toml:
```toml
[dependencies]
usf = "1.0.0"
```
### Basic Usage
```rust
use usf::UniversalStorage;
use usf::DataType;
// Create new storage
let mut storage = UniversalStorage::create("data.usf")?;
// Store different types of data
storage.store("text", "Hello world".as_bytes(), DataType::Text)?;
storage.store("image", image_bytes, DataType::Image)?;
storage.store("json", json_bytes, DataType::Json)?;
// Retrieve data
let data = storage.retrieve("text")?;
```
## Performance Benchmarks
| Operation | Size | USF | ZIP | TAR |
|-----------|------|-----|-----|-----|
| Write | 1GB | 0.8s| 2.1s| 1.9s|
| Read | 1GB | 0.3s| 1.2s| 0.9s|
| Compress | 1GB | 65% | 70% | 71% |
*Benchmarks performed on AMD Ryzen 9 5950X, 64GB RAM, NVMe SSD*
## Enterprise Features
### High Availability
- Built-in replication support
- Automatic corruption recovery
- Hot backup capabilities
### Monitoring
- Prometheus metrics export
- Detailed performance analytics
- Health check endpoints
### Security
- AES-256 encryption support
- Role-based access control
- Audit logging
## Documentation
- [Full Documentation](https://docs.rs/usf)
- [Architecture Guide](docs/ARCHITECTURE.md)
- [Security Overview](docs/SECURITY.md)
- [Performance Tuning](docs/PERFORMANCE.md)
- [Enterprise Deployment](docs/ENTERPRISE.md)
## Examples
### Large File Handling
```rust
// Efficiently handle large files with automatic blocking
let large_file = vec![0u8; 1_000_000];
storage.store("large.dat", &large_file, DataType::Binary)?;
```
### Structured Data
```rust
// Automatic delta encoding for sequential data
let numbers = vec![1, 2, 3, 4, 5];
storage.store("nums.dat", &bincode::serialize(&numbers)?, DataType::Structured)?;
```
### Image Optimization
```rust
// Automatic WebP conversion and optimization
storage.store("image.png", png_bytes, DataType::Image)?;
```
## Contributing
We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.
### Development Setup
```bash
# Clone the repository
git clone https://github.com/The-Swarm-Corporation/USF.git
cd USF
# Install development dependencies
cargo install --path .
# Run tests
cargo test
# Run benchmarks
cargo bench
```
## Enterprise Support
Enterprise support, custom development, and consulting services are available through [Swarms.ai](https://swarms.ai/enterprise).
- 24/7 Support
- SLA guarantees
- Custom feature development
- Performance optimization
- Deployment assistance
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
## Acknowledgments
- The Rust Community
- Contributors and maintainers
- Our enterprise users and partners
## Status
---
Made with ❤️ by Swarms.ai