https://github.com/sharkdp/content_inspector

Fast inspection of binary buffers to guess/determine the type of content
https://github.com/sharkdp/content_inspector

binary encoding rust text

Last synced: 2 months ago
JSON representation

Fast inspection of binary buffers to guess/determine the type of content

• Host: GitHub
• URL: https://github.com/sharkdp/content_inspector
• Owner: sharkdp
• License: apache-2.0
• Created: 2018-06-03T15:10:01.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2018-11-04T16:26:29.000Z (about 6 years ago)
• Last Synced: 2024-05-08T22:46:30.515Z (9 months ago)
• Topics: binary, encoding, rust, text
• Language: Rust
• Homepage:
• Size: 22.5 KB
• Stars: 32
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE-APACHE

Awesome Lists containing this project

README

        
# content_inspector

[![Crates.io](https://img.shields.io/crates/v/content_inspector.svg)](https://crates.io/crates/content_inspector)

[![Documentation](https://docs.rs/content_inspector/badge.svg)](https://docs.rs/content_inspector)

A simple library for *fast* inspection of binary buffers to guess the type of content.

This is mainly intended to quickly determine whether a given buffer contains "binary"

or "text" data. Programs like `grep` or `git diff` use similar mechanisms to decide whether

to treat some files as "binary data" or not.

The analysis is based on a very simple heuristic: Searching for NULL bytes

(indicating "binary" content) and the detection of special [byte order

marks](https://en.wikipedia.org/wiki/Byte_order_mark) (indicating a particular kind of textual

encoding). Note that **this analysis can fail**. For example, even if unlikely, UTF-8-encoded

text can legally contain NULL bytes. Conversely, some particular binary formats (like binary

[PGM](https://en.wikipedia.org/wiki/Netpbm_format)) may not contain NULL bytes. Also, for

performance reasons, only the first 1024 bytes are checked for the NULL-byte (if no BOM was

detected).

If this library reports a certain type of encoding (say `UTF_16LE`), there is **no guarantee** that

the binary buffer can actually be decoded as UTF-16LE.

## Usage

```rust

use content_inspector::{ContentType, inspect};

assert_eq!(ContentType::UTF_8, inspect(b"Hello"));

assert_eq!(ContentType::BINARY, inspect(b"\xFF\xE0\x00\x10\x4A\x46\x49\x46\x00"));

assert!(inspect(b"Hello").is_text());

```

## CLI example

This crate also comes with a small example command-line program (see [`examples/inspect.rs`](examples/inspect.rs)) that demonstrates the usage:

```bash

> inspect

USAGE: inspect FILE [FILE...]

> inspect testdata/*

testdata/create_text_files.py: UTF-8

testdata/file_sources.md: UTF-8

testdata/test.jpg: binary

testdata/test.pdf: binary

testdata/test.png: binary

testdata/text_UTF-16BE-BOM.txt: UTF-16BE

testdata/text_UTF-16LE-BOM.txt: UTF-16LE

testdata/text_UTF-32BE-BOM.txt: UTF-32BE

testdata/text_UTF-32LE-BOM.txt: UTF-32LE

testdata/text_UTF-8-BOM.txt: UTF-8-BOM

testdata/text_UTF-8.txt: UTF-8

```

If you only want to detect whether something is a binary or text file, this is about a factor of 250 faster than `file --mime ...`.

## License

Licensed under either of

 * Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)

 * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.