{"id":16689548,"url":"https://github.com/sigpwned/chardet4j","last_synced_at":"2025-03-17T00:33:14.237Z","repository":{"id":42195582,"uuid":"476544649","full_name":"sigpwned/chardet4j","owner":"sigpwned","description":"Simple, compact charset detection for Java 8+","archived":false,"fork":false,"pushed_at":"2024-09-29T01:55:08.000Z","size":260,"stargazers_count":21,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-13T15:48:29.924Z","etag":null,"topics":["chardet","chardetect","charset-detector","icu4j","java"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sigpwned.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-01T02:17:27.000Z","updated_at":"2024-09-29T01:55:11.000Z","dependencies_parsed_at":"2024-01-28T20:30:10.006Z","dependency_job_id":"5ce54a2e-c943-42fc-896a-33031ec0edac","html_url":"https://github.com/sigpwned/chardet4j","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sigpwned%2Fchardet4j","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sigpwned%2Fchardet4j/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sigpwned%2Fchardet4j/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sigpwned%2Fchardet4j/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sigpwned","download_url":"https://codeload.github.com/sigpwned/chardet4j/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221669471,"owners_count":16860888,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chardet","chardetect","charset-detector","icu4j","java"],"created_at":"2024-10-12T15:48:33.003Z","updated_at":"2024-10-27T11:36:31.190Z","avatar_url":"https://github.com/sigpwned.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CHARDET4J [![tests](https://github.com/sigpwned/chardet4j/actions/workflows/tests.yml/badge.svg)](https://github.com/sigpwned/chardet4j/actions/workflows/tests.yml) [![Maven Central](https://img.shields.io/maven-central/v/com.sigpwned/chardet4j)](https://central.sonatype.com/search?q=g%3Acom.sigpwned%20a%3Achardet4j) [![javadoc](https://javadoc.io/badge2/com.sigpwned/chardet4j/javadoc.svg)](https://javadoc.io/doc/com.sigpwned/chardet4j)\n\n## Introduction\n\nThe state-of-the-art character set detection library for Java is\n[icu4j](https://github.com/unicode-org/icu). However, the icu4j JAR\nfile is about 13MB. This is a hefty price to pay for programs that\nonly require charset detection! There should be a smaller option of\nthe same quality.\n\nThe chardet4j library pulls the `CharsetDetector` feature from icu4j\nand repackages it as this standalone library. This allows programs to\nmake good use of this important feature without bloating their\nJARs. At the time of this writing, the chardet4j JAR comes in around\n85KB. There are no dependencies.\n\nThis library also implements some other important components of\ncharacter set detection and decoding, namely byte order mark handling.\n\n## Features\n\nThe library assists the user with detecting character set encodings for byte\nstreams and decoding them into character streams. It offers specific\nabstractions for byte order marks (BOMs) and specific methods for identifying\nand decoding character encodings for byte arrays and input streams.\n\nThe library uses the following algorithm to determine character encoding of\nbinary data:\n\n1. Check for a BOM. If one is present, then trust it, and use the corresponding\n   charset to decode the data.\n2. Use a battery of bespoke character set detectors to guess which charset is\n   most likely. Users may provide a declared encoding, which provides a boost\n   to the given charset in this estimation process. If a charset is identified\n   with sufficient confidence, then use it to decode the data.\n3. The default charset is used to decode the data, if one is given.\n\n## Installation\n\nThe library can be found in Maven Central with the following coordinates:\n\n    \u003cdependency\u003e\n        \u003cgroupId\u003ecom.sigpwned\u003c/groupId\u003e\n        \u003cartifactId\u003echardet4j\u003c/artifactId\u003e\n        \u003cversion\u003e75.1.2\u003c/version\u003e\n    \u003c/dependency\u003e\n\nIt is compatible with Java versions 8 and later. chardet4j has no dependencies.\n\nThe `$major.$minor.$patch` version of the library is determined by the underlying\nicu4j version and the local release version. The `$major` and `$minor` are taken\nfrom the icu4j version, and `$patch` is the release number of this library for\nthe icu4j version, starting with 0.\n\n## Getting Started\n\nTo decode an `InputStream` to a `Reader` by detecting its character set:\n\n    try (Reader chars=Chardet.decode(bytes, StandardCharsets.UTF_8)) {\n        // Process chars here\n    }\n\nCharset detection is important when dealing with content of unknown provenance,\nlike content downloaded from the internet or text files uploaded by users. In\nsuch cases, users often have a declared encoding, typically from a content type.\nThe name of the declared encoding can be provided as a hint to charset\ndetection:\n\n    try (Reader chars=Chardet.decode(bytes, declaredEncoding, StandardCharsets.UTF_8)) {\n        // Process chars here\n    }\n\nByte arrays can be converted directly to Strings as well:\n\n    String chars=Chardet.decode(bytes, declaredEncoding, StandardCharsets.UTF_8);\n    \nUsers only interested in detection can detect the charset directly, or by name\nin case the detected charset is not supported by the JVM:\n\n    // Throws an UnsupportedCharsetException if the charset is not supported by JVM\n    Optional\u003cCharset\u003e maybeCharset = Chardet.detect(bytes, declaredEncoding);\n    \n    // Never throws\n    Optional\u003cString\u003e maybeCharsetName = Chardet.detectName(bytes, declaredEncoding);\n\n## Advanced Usage\n\nThe following are more sophisticated use cases and edge cases that most users\nwill not need to worry about.\n\n### Working with BOMs Directly\n\nThe easiest way to work with byte order marks directly is with the\n`BomAwareInputStream` class:\n\n    try (BomAwareInputStream bomed=BomAwareInputStream.detect(in)) {\n        if(bomed.bom().isPresent()) {\n            // A BOM was detected in this byte stream, and can be accessed using\n            // bomed.bom()\n        } else {\n            // No BOM was detected in this byte stream.\n        }\n    }\n\nIt is not typically required to work with BOMs directly, but it can be useful\nwhen creating a custom decode pipeline.\n\n### Accessing Character Encoding\n\nThe easiest way to determine which character encoding is in use is with the\n`DecodedInputStreamReader` class:\n\n    try (DecodedInputStreamReader chars=Chardet.decode(bytes, StandardCharsets.UTF_8)) {\n        // The charset that was detected and is being used to decode the given byte\n        // stream can be accessed using chars.charset()\n        Charset charset = chars.charset();\n    }\n\n### Handling Unsupported Charsets\n\nThe Java Standard only requires that distributions support the\n[standard charsets](https://docs.oracle.com/javase/8/docs/api/index.html?java/nio/charset/StandardCharsets.html)\nISO-8859-1, US-ASCII, UTF-8, UTF-16BE, and UTF-16LE. This library detects those\ncharsets and many more besides, so there is a possibility that the detected\ncharset is not supported by the current JVM.\n\nUsers are unlikely to hit this situation in the wild, since (a) Java generally\nsupports almost all of the charsets this library detects, and (b) the\nunsupported charsets are scarce in the wild, and getting more scarce every year.\n\nRegardless, there are a couple ways to manage this situation.\n\n#### Catch UnsupportedCharsetException\n\nThe library throws a `UnsupportedCharsetException` when the detected charset is not\nsupported by the current JVM. Users are free to catch this exception and handle\nas desired.\n\n    try (Reader chars=Chardet.decode(bytes, StandardCharsets.UTF_8)) {\n        // Process chars here\n    } catch(UnsupportedCharsetException e) {\n        // The charset was detected, but is not supported by current JVM. There are a\n        // few ways this is typically handled:\n        // \n        // - Propagate as an IOException, since the content cannot be decoded properly\n        // - Ignore the error and use a default charset\n    }\n\n#### Detect Charset Names\n\nRather than working with charsets, work with charset names instead. This will\nnever throw an exception.\n\n    Optional\u003cString\u003e maybeCharsetName = Chardet.detectCharsetName(bytes);\n    if(maybeCharsetName.isPresent()) {\n        // The charset was detected successfully, and the name can be accessed using\n        // maybeCharsetName.get()\n    } else {\n        // The charset could not be detected\n    }\n\n### Using Custom Charsets\n\nUsers who wish to add new charsets to the JVM should follow the instructions\non the\n[CharsetProvider](https://docs.oracle.com/javase/8/docs/api/java/nio/charset/spi/CharsetProvider.html)\nclass. The library will automatically pick up any such new charsets.\n    \n## Configuration\n\nThe following configuration variables are available to customize the working of\nthe library.\n\n### System Property chardet4j.detect.bufsize\n\nOne way the library detects character encodings is by analyzing the leading\nbytes of a binary file. The more data the library analyzes, the more accurate\nthe estimates will be, but the longer it will take. By default, this value is\n8192 bytes, or 8KiB. Users can change this value by setting the\n`chardet4j.detect.bufsize` system property. For example, to set this value to \n16KiB, use:\n\n    java -Dchardet.detect.bufsize=16384 ...\n\nAdjusting the buffer size can be useful when dealing with particularly large\nfiles where detection accuracy or performance might be a concern.\n\n## Supported Character Encodings\n\nThe chardet4j library and Java in general supports the following character\nencodings at the following levels:\n\n|     Name     | Standard | ICU4J | BOM | Laptop |\n|:------------:|:--------:|:-----:|:---:|:------:|\n| Big5         |          |   ✔   |     |    ✔   |\n| EUC-JP       |          |   ✔   |     |    ✔   |\n| EUC-KR       |          |   ✔   |     |    ✔   |\n| GB18030      |          |   ✔   |  ✔  |    ✔   |\n| ISO-2022-CN  |          |   ✔   |     |    ✔   |\n| ISO-2022-JP  |          |   ✔   |     |    ✔   |\n| ISO-2022-KR  |          |   ✔   |     |    ✔   |\n| ISO-8859-1   |          |   ✔   |     |    ✔   |\n| ISO-8859-2   |          |   ✔   |     |    ✔   |\n| ISO-8859-5   |          |   ✔   |     |    ✔   |\n| ISO-8859-6   |          |   ✔   |     |    ✔   |\n| ISO-8859-7   |          |   ✔   |     |    ✔   |\n| ISO-8859-8   |          |   ✔   |     |    ✔   |\n| ISO-8859-8-I |          |   ✔   |     |        |\n| ISO-8859-9   |          |   ✔   |     |    ✔   |\n| KOI8-R       |          |   ✔   |     |    ✔   |\n| Shift_JIS    |          |   ✔   |     |    ✔   |\n| US-ASCII     |     ✔    |   ✔*  |     |    ✔   |\n| UTF-1        |          |       |  ✔  |        |\n| UTF-16BE     |     ✔    |   ✔   |  ✔  |    ✔   |\n| UTF-16LE     |     ✔    |   ✔   |  ✔  |    ✔   |\n| UTF-32BE     |          |   ✔   |  ✔  |    ✔   |\n| UTF-32LE     |          |   ✔   |  ✔  |    ✔   |\n| UTF-8        |     ✔    |   ✔   |  ✔  |    ✔   |\n| UTF-EBCDIC   |          |       |  ✔  |        |\n| windows-1250 |          |   ✔   |     |    ✔   |\n| windows-1251 |          |   ✔   |     |    ✔   |\n| windows-1252 |          |   ✔   |     |    ✔   |\n| windows-1253 |          |   ✔   |     |    ✔   |\n| windows-1254 |          |   ✔   |     |    ✔   |\n| windows-1255 |          |   ✔   |     |    ✔   |\n| windows-1256 |          |   ✔   |     |    ✔   |\n\nNotes:  \n`*`: ICU4J detects US-ASCII as ISO-8859-1, a superset of US-ASCII\n\nThe support levels have the following meanings:\n\n* `Standard` -- The Java Standard requires that all JVMs support this\n   character encoding\n* `ICU4J` -- The ICU4J project has a bespoke charset recognizer for this\n  character encoding\n* `BOM` -- The character encoding can be detected by Byte Order Mark\n* `Laptop` -- The character sets supported by `java version \"1.8.0_321\"` on my\n   laptop (Obviously, this test is completely unscientific. If you have a\n   better suggestion, please open an issue!)\n\n## Licensing\n\nThe icu library is released under the ICU license. The chardet4j library is\nreleased under the Apache license. For more details, see the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsigpwned%2Fchardet4j","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsigpwned%2Fchardet4j","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsigpwned%2Fchardet4j/lists"}