https://github.com/tktech/mutf8
Pure-python and optional C encoders/decoders for MUTF-8/CESU-8.
https://github.com/tktech/mutf8
Last synced: 3 months ago
JSON representation
Pure-python and optional C encoders/decoders for MUTF-8/CESU-8.
- Host: GitHub
- URL: https://github.com/tktech/mutf8
- Owner: TkTech
- License: mit
- Created: 2020-08-26T06:04:00.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2023-06-27T10:49:45.000Z (almost 3 years ago)
- Last Synced: 2025-09-06T02:31:43.031Z (10 months ago)
- Language: Python
- Size: 38.1 KB
- Stars: 12
- Watchers: 2
- Forks: 3
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
Awesome Lists containing this project
README
# mutf-8
This package contains simple pure-python as well as C encoders and decoders for
the MUTF-8 character encoding. In most cases, you can also parse the even-rarer
CESU-8.
These days, you'll most likely encounter MUTF-8 when working on files or
protocols related to the JVM. Strings in a Java `.class` file are encoded using
MUTF-8, strings passed by the JNI, as well as strings exported by the object
serializer.
This library was extracted from [Lawu][], a Python library for working with JVM
class files.
## 🎉 Installation
Install the package from PyPi:
```
pip install mutf8
```
Binary wheels are available for the following:
| | py3.6 | py3.7 | py3.8 | py3.9 |
| ---------------- | ----- | ----- | ----- | ----- |
| OS X (x86_64) | y | y | y | y |
| Windows (x86_64) | y | y | y | y |
| Linux (x86_64) | y | y | y | y |
If binary wheels are not available, it will attempt to build the C extension
from source with any C99 compiler. If it could not build, it will fall back
to a pure-python version.
## Usage
Encoding and decoding is simple:
```python
from mutf8 import encode_modified_utf8, decode_modified_utf8
unicode = decode_modified_utf8(byte_like_object)
bytes = encode_modified_utf8(unicode)
```
This module *does not* register itself globally as a codec, since importing
should be side-effect-free.
## 📈 Benchmarks
The C extension is significantly faster - often 20x to 40x faster.
### MUTF-8 Decoding
| Name | Min (μs) | Max (μs) | StdDev | Ops |
|------------------------------|------------|------------|----------|---------------|
| cmutf8-decode_modified_utf8 | 0.00009 | 0.00080 | 0.00000 | 9957678.56358 |
| pymutf8-decode_modified_utf8 | 0.00190 | 0.06040 | 0.00000 | 450455.96019 |
### MUTF-8 Encoding
| Name | Min (μs) | Max (μs) | StdDev | Ops |
|------------------------------|------------|------------|----------|----------------|
| cmutf8-encode_modified_utf8 | 0.00008 | 0.00151 | 0.00000 | 11897361.05101 |
| pymutf8-encode_modified_utf8 | 0.00180 | 0.16650 | 0.00000 | 474390.98091 |
## C Extension
The C extension is optional. If a binary package is not available, or a C
compiler is not present, the pure-python version will be used instead. If you
want to ensure you're using the C version, import it directly:
```python
from mutf8.cmutf8 import decode_modified_utf8
decode_modified_utf(b'\xED\xA1\x80\xED\xB0\x80')
```
[Lawu]: https://github.com/tktech/lawu