https://github.com/appaquet/extsort-rs

External sorting (i.e. on disk sorting) capability on arbitrarily sized iterator
https://github.com/appaquet/extsort-rs

Last synced: 3 months ago
JSON representation

External sorting (i.e. on disk sorting) capability on arbitrarily sized iterator

• Host: GitHub
• URL: https://github.com/appaquet/extsort-rs
• Owner: appaquet
• License: apache-2.0
• Created: 2018-12-06T02:44:01.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2025-01-28T10:02:23.000Z (5 months ago)
• Last Synced: 2025-03-31T07:08:04.170Z (3 months ago)
• Language: Rust
• Size: 53.7 KB
• Stars: 25
• Watchers: 2
• Forks: 5
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# extsort

[![crates.io](https://img.shields.io/crates/v/extsort.svg)](https://crates.io/crates/extsort)

[![dependency status](https://deps.rs/repo/github/appaquet/extsort-rs/status.svg)](https://deps.rs/repo/github/appaquet/extsort-rs)

Exposes external sorting (i.e. on-disk sorting) capability on arbitrarily sized iterators, even if the

generated content of the iterator doesn't fit in memory. Once sorted, it returns a new sorted iterator.

To remain efficient for all implementations, the crate doesn't handle serialization but leaves that to the user.

The sorter can optionally use [`rayon`](https://crates.io/crates/rayon) to sort the in-memory buffer. It is generally

faster when the buffer size is big enough for parallelism to have an impact on its overhead.

## Example

```rust

extern crate extsort;

extern crate byteorder;

use extsort::*;

use byteorder::{ReadBytesExt, WriteBytesExt};

use std::io::{Read, Write};

#[derive(Debug, Eq, PartialEq, Ord, PartialOrd)]

struct MyStruct(u32);

impl Sortable for MyStruct {

    fn encode(&self, write: &mut W) -> std::io::Result<()> {

        write.write_u32::(self.0)?;

        Ok(())

    }

    fn decode(read: &mut R) -> std::io::Result {

        read.read_u32::().map(MyStruct)

    }

}

fn main() {

    let sorter = ExternalSorter::new();

    let reversed_data = (0..1000).rev().map(MyStruct).into_iter();

    let sorted_iter = sorter.sort(reversed_data).unwrap();

    let sorted_data = sorted_iter.collect::>>().unwrap();

    let expected_data = (0..1000).map(MyStruct).collect::>();

    assert_eq!(sorted_data, expected_data);

}

```