https://github.com/lemire/unicode_lipsum
https://github.com/lemire/unicode_lipsum
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/lemire/unicode_lipsum
- Owner: lemire
- Created: 2021-04-29T16:16:19.000Z (about 5 years ago)
- Default Branch: main
- Last Pushed: 2025-03-21T09:01:53.000Z (over 1 year ago)
- Last Synced: 2025-03-21T10:24:13.746Z (over 1 year ago)
- Language: HTML
- Size: 18.1 MB
- Stars: 5
- Watchers: 4
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# unicode_lipsum
Tests files encoded with UTF-8, UTF-16LE and UTF-32LE.
By convention, all UTF-8 files end with `.utf8.txt` while all UTF-16LE files end with `.utf16.txt` and
all UTF-32LE end with `.utf32.txt`.
A small number of files are encoded using Latin 1 (ISO-8859-1): esperanto.latin1.txt, french.latin1.txt, german.latin1.txt, portuguese.latin1.txt
in the wikipedia_mars directory. They are not exactly equivalent to the Unicode files: e.g., it is not possible to reproduce the equivalent Unicode files from the Latin 1 files. However, we have have modified Unicode files with the suffixes `.utflatin8.txt` (UTF-8 recovered from Latin 1), `.utflatin16.txt` (UTF-16LE recovered from Latin 1), `.utflatin32.txt` (UTF-32LE recovered from Latin 1).
The wikipedia_mars files are derived from the Mars wikipedia article in different languages.
Wikipedia is licensed under a Creative Commons license.
The `html2text` Python program is used to convert them to text, by stripping HTML codes.
The lipsum file come from the package [https://github.com/rusticstuff/simdutf8](simdutf8)
by Hans Kratz (licensed under both MIT and Apache).
These files are provided for research purposes.