https://github.com/acj/file-chunker

Divide a file into evenly-sized chunks
https://github.com/acj/file-chunker

chunking concurrency parallel text-processing

Last synced: 3 months ago
JSON representation

Divide a file into evenly-sized chunks

• Host: GitHub
• URL: https://github.com/acj/file-chunker
• Owner: acj
• License: mit
• Created: 2022-02-09T12:29:06.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2024-10-21T11:55:52.000Z (over 1 year ago)
• Last Synced: 2025-04-20T09:43:55.169Z (about 1 year ago)
• Topics: chunking, concurrency, parallel, text-processing
• Language: Rust
• Homepage:
• Size: 8.79 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# file-chunker

[](https://crates.io/crates/file-chunker)

[](https://github.com/acj/file-chunker/actions?query=branch%3Amain)

This crate provides the `FileChunker` type, which is useful for efficiently reading a file

in (approximately) equally-sized parts.

The original use case was to process a log file in chunks, one thread per chunk, and to

guarantee that each chunk ended with a full line of text.

## Example

```rust,no_run

use file_chunker::FileChunker;

let file = std::fs::File::open("/path/to/file").unwrap();

let chunker = FileChunker::new(&file).unwrap();

chunker.chunks(1024, Some('\n'))

    .unwrap()

    .iter()

    .for_each(|chunk| {

        println!("{:?}", chunk);

    });

```