https://github.com/emcd/python-detextive
https://github.com/emcd/python-detextive
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/emcd/python-detextive
- Owner: emcd
- License: apache-2.0
- Created: 2025-08-12T17:39:02.000Z (11 months ago)
- Default Branch: master
- Last Pushed: 2025-08-29T04:38:37.000Z (10 months ago)
- Last Synced: 2025-08-29T06:42:43.020Z (10 months ago)
- Language: Python
- Size: 224 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE.txt
Awesome Lists containing this project
README
.. vim: set fileencoding=utf-8:
.. -*- coding: utf-8 -*-
.. +--------------------------------------------------------------------------+
| |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| |
+--------------------------------------------------------------------------+
*******************************************************************************
detextive
*******************************************************************************
.. image:: https://img.shields.io/pypi/v/detextive
:alt: Package Version
:target: https://pypi.org/project/detextive/
.. image:: https://img.shields.io/pypi/status/detextive
:alt: PyPI - Status
:target: https://pypi.org/project/detextive/
.. image:: https://github.com/emcd/python-detextive/actions/workflows/tester.yaml/badge.svg?branch=master&event=push
:alt: Tests Status
:target: https://github.com/emcd/python-detextive/actions/workflows/tester.yaml
.. image:: https://emcd.github.io/python-detextive/coverage.svg
:alt: Code Coverage Percentage
:target: https://github.com/emcd/python-detextive/actions/workflows/tester.yaml
.. image:: https://img.shields.io/github/license/emcd/python-detextive
:alt: Project License
:target: https://github.com/emcd/python-detextive/blob/master/LICENSE.txt
.. image:: https://img.shields.io/pypi/pyversions/detextive
:alt: Python Versions
:target: https://pypi.org/project/detextive/
🕵️ A Python library which provides consolidated text detection
capabilities for reliable content analysis. Offers MIME type detection,
character set detection, and line separator processing.
Key Features ⭐
===============================================================================
🔍 **MIME Type Detection**
Intelligent content-based detection using magic bytes with file extension
fallback for comprehensive format identification.
📝 **Character Encoding Detection**
Statistical analysis with UTF-8 optimization and validation through decode
operations for reliable text processing.
📄 **Line Separator Processing**
Cross-platform line ending detection and normalization supporting CR, LF,
and CRLF formats with mixed-content handling.
✅ **Textual Content Validation**
Smart classification of MIME types and content reasonableness assessment
using control character and printability heuristics.
Installation 📦
===============================================================================
Method: Install Python Package
-------------------------------------------------------------------------------
Install via `uv `_ ``pip``
command:
::
uv pip install detextive
Or, install via ``pip``:
::
pip install detextive
Examples 💡
===============================================================================
Basic Usage
-------------------------------------------------------------------------------
**MIME Type and Charset Detection**:
Load your content as bytes:
.. code-block:: python
import detextive
with open( 'document.txt', 'rb' ) as file:
content = file.read( )
You can detect MIME type and charset individually:
.. code-block:: python
mimetype = detextive.detect_mimetype( content, location = 'document.txt' )
charset = detextive.detect_charset( content )
Or use combined inference for better accuracy:
.. code-block:: python
mimetype, charset = detextive.infer_mimetype_charset(
content, location = 'document.txt' )
print( "Detected: {mimetype} with {charset} encoding".format(
mimetype = mimetype, charset = charset ) )
**Line Separator Processing**:
Detect line separators in mixed content:
.. code-block:: python
import detextive
content = 'Line 1\r\nLine 2\rLine 3\n'
separator = detextive.LineSeparators.detect_bytes( content.encode( ) )
Normalize line separators to Python standard:
.. code-block:: python
normalized = detextive.LineSeparators.normalize_universal( content )
Convert to platform-specific line separators:
.. code-block:: python
native = detextive.LineSeparators.CRLF.nativize( normalized )
**Content Classification**:
Check if MIME types represent textual content:
.. code-block:: python
import detextive
detextive.is_textual_mimetype( 'application/json' ) # True
detextive.is_textual_mimetype( 'image/jpeg' ) # False
Validate that decoded text content is reasonable:
.. code-block:: python
text = "Hello world!"
detextive.is_valid_text( text ) # True
Binary data that might decode as text but isn't valid fails validation:
.. code-block:: python
binary_as_text = "Config file\x00\x00\x00data"
detextive.is_valid_text( binary_as_text ) # False
**High-Level Decoding**:
For complete bytes-to-text processing with automatic charset detection and validation:
.. code-block:: python
import detextive
with open( 'document.txt', 'rb' ) as file:
content = file.read( )
text = detextive.decode( content, location = 'document.txt' )
print( f"Decoded text: {text}" )
Contribution 🤝
===============================================================================
Contribution to this project is welcome! However, it must follow the `code of
conduct
`_
for the project.
Please file bug reports and feature requests in the `issue tracker
`_ or submit `pull
requests `_ to
improve the source code or documentation.
For development guidance and standards, please see the `development guide
`_.
Additional Indicia
===============================================================================
.. image:: https://img.shields.io/github/last-commit/emcd/python-detextive
:alt: GitHub last commit
:target: https://github.com/emcd/python-detextive
.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/copier-org/copier/master/img/badge/badge-grayscale-inverted-border-orange.json
:alt: Copier
:target: https://github.com/copier-org/copier
.. image:: https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg
:alt: Hatch
:target: https://github.com/pypa/hatch
.. image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit
:alt: pre-commit
:target: https://github.com/pre-commit/pre-commit
.. image:: https://microsoft.github.io/pyright/img/pyright_badge.svg
:alt: Pyright
:target: https://microsoft.github.io/pyright
.. image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
:alt: Ruff
:target: https://github.com/astral-sh/ruff
.. image:: https://img.shields.io/pypi/implementation/detextive
:alt: PyPI - Implementation
:target: https://pypi.org/project/detextive/
.. image:: https://img.shields.io/pypi/wheel/detextive
:alt: PyPI - Wheel
:target: https://pypi.org/project/detextive/
Other Projects by This Author 🌟
===============================================================================
* `python-absence `_ (`absence `_ on PyPI)
🕳️ A Python library package which provides a **sentinel for absent values** - a falsey, immutable singleton that represents the absence of a value in contexts where ``None`` or ``False`` may be valid values.
* `python-accretive `_ (`accretive `_ on PyPI)
🌌 A Python library package which provides **accretive data structures** - collections which can grow but never shrink.
* `python-classcore `_ (`classcore `_ on PyPI)
🏭 A Python library package which provides **foundational class factories and decorators** for providing classes with attributes immutability and concealment and other custom behaviors.
* `python-dynadoc `_ (`dynadoc `_ on PyPI)
📝 A Python library package which bridges the gap between **rich annotations** and **automatic documentation generation** with configurable renderers and support for reusable fragments.
* `python-falsifier `_ (`falsifier `_ on PyPI)
🎭 A very simple Python library package which provides a **base class for falsey objects** - objects that evaluate to ``False`` in boolean contexts.
* `python-frigid `_ (`frigid `_ on PyPI)
🔒 A Python library package which provides **immutable data structures** - collections which cannot be modified after creation.
* `python-icecream-truck `_ (`icecream-truck `_ on PyPI)
🍦 **Flavorful Debugging** - A Python library which enhances the powerful and well-known ``icecream`` package with flavored traces, configuration hierarchies, customized outputs, ready-made recipes, and more.
* `python-librovore `_ (`librovore `_ on PyPI)
🐲 **Documentation Search Engine** - An intelligent documentation search and extraction tool that provides both a command-line interface for humans and an MCP (Model Context Protocol) server for AI agents. Search across Sphinx and MkDocs sites with fuzzy matching, extract clean markdown content, and integrate seamlessly with AI development workflows.
* `python-mimeogram `_ (`mimeogram `_ on PyPI)
📨 A command-line tool for **exchanging collections of files with Large Language Models** - bundle multiple files into a single clipboard-ready document while preserving directory structure and metadata... good for code reviews, project sharing, and LLM interactions.