Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/appaquet/extsort-rs
External sorting (i.e. on disk sorting) capability on arbitrarily sized iterator
https://github.com/appaquet/extsort-rs
Last synced: 13 days ago
JSON representation
External sorting (i.e. on disk sorting) capability on arbitrarily sized iterator
- Host: GitHub
- URL: https://github.com/appaquet/extsort-rs
- Owner: appaquet
- License: apache-2.0
- Created: 2018-12-06T02:44:01.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-03-25T02:01:29.000Z (8 months ago)
- Last Synced: 2024-08-08T18:32:57.933Z (3 months ago)
- Language: Rust
- Size: 50.8 KB
- Stars: 24
- Watchers: 3
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# extsort
[![crates.io](https://img.shields.io/crates/v/extsort.svg)](https://crates.io/crates/extsort)
[![dependency status](https://deps.rs/repo/github/appaquet/extsort-rs/status.svg)](https://deps.rs/repo/github/appaquet/extsort-rs)Exposes external sorting (i.e. on-disk sorting) capability on arbitrarily sized iterators, even if the
generated content of the iterator doesn't fit in memory. Once sorted, it returns a new sorted iterator.To remain efficient for all implementations, the crate doesn't handle serialization but leaves that to the user.
The sorter can optionally use [`rayon`](https://crates.io/crates/rayon) to sort the in-memory buffer. It is generally
faster when the buffer size is big enough for parallelism to have an impact on its overhead.## Example
```rust
extern crate extsort;
extern crate byteorder;use extsort::*;
use byteorder::{ReadBytesExt, WriteBytesExt};
use std::io::{Read, Write};#[derive(Debug, Eq, PartialEq, Ord, PartialOrd)]
struct MyStruct(u32);impl Sortable for MyStruct {
fn encode(&self, write: &mut W) -> std::io::Result<()> {
write.write_u32::(self.0)?;
Ok(())
}fn decode(read: &mut R) -> std::io::Result {
read.read_u32::().map(MyStruct)
}
}fn main() {
let sorter = ExternalSorter::new();
let reversed_data = (0..1000).rev().map(MyStruct).into_iter();
let sorted_iter = sorter.sort(reversed_data).unwrap();
let sorted_data = sorted_iter.collect::>>().unwrap();let expected_data = (0..1000).map(MyStruct).collect::>();
assert_eq!(sorted_data, expected_data);
}
```