https://github.com/kaniini/libguess

character set guessing library
https://github.com/kaniini/libguess

Last synced: 16 days ago
JSON representation

character set guessing library

• Host: GitHub
• URL: https://github.com/kaniini/libguess
• Owner: kaniini
• License: bsd-3-clause
• Created: 2012-05-16T03:37:05.000Z (over 12 years ago)
• Default Branch: master
• Last Pushed: 2015-06-09T14:43:03.000Z (over 9 years ago)
• Last Synced: 2024-05-23T00:08:20.633Z (6 months ago)
• Language: C
• Size: 215 KB
• Stars: 30
• Watchers: 3
• Forks: 8
• Open Issues: 2
• Metadata Files:
  • Readme: README
  • License: COPYING

Awesome Lists containing this project

README

        
libguess - a high-speed character set detection library

-------------------------------------------------------

libguess is free, but copyrighted software.  See COPYING for details.

Using libguess in your programs:

--------------------------------

libguess has two functions:

  - libguess_determine_encoding(const char *inbuf, int length, const char *region);

  This detects a character set.  Returns an appropriate charset name

  that can be passed to iconv_open().  Region is the name of the language

  or region that the data is related to, e.g. 'Baltic' for the Baltic states,

  or 'Japanese' for Japan.

  - libguess_validate_utf8(const char *inbuf, int length);

  This employs libguess's DFA-based character set validation rules to ensure

  that a string is pure UTF-8.  GLib's UTF-8 validation functions are broken,

  for example.

Just include libguess.h and link to libguess to get these functions in your

program.  For your convenience, a pkg-config file is also supplied.

How does libguess work:

-----------------------

libguess employs discrete-finite automata to deduce the character set of the

input buffer.  The advantage of this is that all character sets can be checked

in parallel, and quickly.  Right now, libguess passes a byte to each DFA on the

same pass, meaning that the winning character set can be deduced as efficiently

as possible.

libguess is fully reentrant, using only local stack memory for DFA operations.

I found a bug:

--------------

Please report your bug using GitHub issues.