{"id":15579035,"url":"https://github.com/dapper91/ext-sort-rs","last_synced_at":"2025-04-24T01:28:05.219Z","repository":{"id":45066263,"uuid":"430347812","full_name":"dapper91/ext-sort-rs","owner":"dapper91","description":"rust external sort algorithm implementation","archived":false,"fork":false,"pushed_at":"2023-04-16T18:44:04.000Z","size":32,"stargazers_count":13,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-04-25T08:20:42.207Z","etag":null,"topics":["algorithms","bigdata","external-sort","external-sorting","rust","sort","sorting"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/ext-sort","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dapper91.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-11-21T11:32:37.000Z","updated_at":"2024-02-25T05:06:38.000Z","dependencies_parsed_at":"2023-11-10T14:33:07.564Z","dependency_job_id":null,"html_url":"https://github.com/dapper91/ext-sort-rs","commit_stats":{"total_commits":28,"total_committers":2,"mean_commits":14.0,"dds":0.0357142857142857,"last_synced_commit":"c312021275262b435800d58fa13d5860cc379ec9"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapper91%2Fext-sort-rs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapper91%2Fext-sort-rs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapper91%2Fext-sort-rs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dapper91%2Fext-sort-rs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dapper91","download_url":"https://codeload.github.com/dapper91/ext-sort-rs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250543295,"owners_count":21447866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithms","bigdata","external-sort","external-sorting","rust","sort","sorting"],"created_at":"2024-10-02T19:13:39.380Z","updated_at":"2025-04-24T01:28:05.192Z","avatar_url":"https://github.com/dapper91.png","language":"Rust","readme":"[![Crates.io][crates-badge]][crates-url]\n[![License][licence-badge]][licence-url]\n[![Test Status][test-badge]][test-url]\n[![Documentation][doc-badge]][doc-url]\n\n[crates-badge]: https://img.shields.io/crates/v/ext-sort.svg\n[crates-url]: https://crates.io/crates/ext-sort\n[licence-badge]: https://img.shields.io/badge/license-Unlicense-blue.svg\n[licence-url]: https://github.com/dapper91/ext-sort-rs/blob/master/LICENSE\n[test-badge]: https://github.com/dapper91/ext-sort-rs/actions/workflows/test.yml/badge.svg?branch=master\n[test-url]: https://github.com/dapper91/ext-sort-rs/actions/workflows/test.yml\n[doc-badge]: https://docs.rs/ext-sort/badge.svg\n[doc-url]: https://docs.rs/ext-sort\n\n\n# Rust external sort\n\n`ext-sort` is a rust external sort algorithm implementation.\n\nExternal sorting is a class of sorting algorithms that can handle massive amounts of data. External sorting \nis required when the data being sorted do not fit into the main memory (RAM) of a computer and instead must be \nresided in slower external memory, usually a hard disk drive. Sorting is achieved in two passes. During the \nfirst pass it sorts chunks of data that each fit in RAM, during the second pass it merges the sorted chunks together. \nFor more information see [External Sorting](https://en.wikipedia.org/wiki/External_sorting).\n\n## Overview\n\n`ext-sort` supports the following features:\n\n* **Data agnostic:**\n  it supports all data types that implement `serde` serialization/deserialization by default,\n  otherwise you can implement your own serialization/deserialization mechanism.\n* **Serialization format agnostic:**\n  the library uses `MessagePack` serialization format by default, but it can be easily substituted by your custom one\n  if `MessagePack` serialization/deserialization performance is not sufficient for your task.\n* **Multithreading support:**\n  multi-threaded sorting is supported, which means data is sorted in multiple threads utilizing maximum CPU resources\n  and reducing sorting time.\n* **Memory limit support:**\n  memory limited sorting is supported. It allows you to limit sorting memory consumption\n  (`memory-limit` feature required). \n\n# Basic example\n\nActivate `memory-limit` feature of the ext-sort crate on Cargo.toml:\n\n```toml\n[dependencies]\next-sort = { version = \"^0.1.4\", features = [\"memory-limit\"] }\n```\n\n```rust\nuse std::fs;\nuse std::io::{self, prelude::*};\nuse std::path;\n\nuse bytesize::MB;\nuse env_logger;\nuse log;\n\nuse ext_sort::{buffer::mem::MemoryLimitedBufferBuilder, ExternalSorter, ExternalSorterBuilder};\n\nfn main() {\n    env_logger::Builder::new().filter_level(log::LevelFilter::Debug).init();\n\n    let input_reader = io::BufReader::new(fs::File::open(\"input.txt\").unwrap());\n    let mut output_writer = io::BufWriter::new(fs::File::create(\"output.txt\").unwrap());\n\n    let sorter: ExternalSorter\u003cString, io::Error, MemoryLimitedBufferBuilder\u003e = ExternalSorterBuilder::new()\n        .with_tmp_dir(path::Path::new(\"./\"))\n        .with_buffer(MemoryLimitedBufferBuilder::new(50 * MB))\n        .build()\n        .unwrap();\n\n    let sorted = sorter.sort(input_reader.lines()).unwrap();\n\n    for item in sorted.map(Result::unwrap) {\n        output_writer.write_all(format!(\"{}\\n\", item).as_bytes()).unwrap();\n    }\n    output_writer.flush().unwrap();\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdapper91%2Fext-sort-rs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdapper91%2Fext-sort-rs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdapper91%2Fext-sort-rs/lists"}