https://github.com/etrepum/pyutf8

Python extension for dealing with validation and cleanup of UTF-8 strings
https://github.com/etrepum/pyutf8

Last synced: 3 days ago
JSON representation

Python extension for dealing with validation and cleanup of UTF-8 strings

• Host: GitHub
• URL: https://github.com/etrepum/pyutf8
• Owner: etrepum
• License: mit
• Created: 2010-03-13T21:20:32.000Z (almost 15 years ago)
• Default Branch: master
• Last Pushed: 2018-10-11T17:37:15.000Z (about 6 years ago)
• Last Synced: 2024-04-24T21:43:55.816Z (8 months ago)
• Language: Python
• Homepage:
• Size: 675 KB
• Stars: 25
• Watchers: 3
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst
  • Changelog: CHANGES.txt
  • License: LICENSE.txt

Awesome Lists containing this project

• starred-awesome - pyutf8 - Python extension for dealing with validation and cleanup of UTF-8 strings (Python)

README

        
pyutf8 provides tools to efficiently deal with the validation

and cleanup of UTF-8 strings. The primary use case is "garbage in utf-8 out".

Usage::

    >>> from pyutf8 import valid_utf8_bytes

    >>> SNOWMAN = u'\N{SNOWMAN}'

    >>> SNOWMAN_BYTES = SNOWMAN.encode('utf-8')

    >>> valid_utf8_bytes(SNOWMAN) == SNOWMAN_BYTES

    True

    >>> valid_utf8_bytes(SNOWMAN_BYTES) == SNOWMAN_BYTES

    True

    >>> valid_utf8_bytes('\xff' + SNOWMAN_BYTES + '\xff') == SNOWMAN_BYTES

    True