{"id":31067295,"url":"https://github.com/willwade/worldalphabets","last_synced_at":"2026-01-20T17:52:03.442Z","repository":{"id":311006337,"uuid":"1042077702","full_name":"willwade/WorldAlphabets","owner":"willwade","description":"A python and node tool to get alphabets of the world: uppercase, lowercase, alphabetical and frequency order","archived":false,"fork":false,"pushed_at":"2025-09-13T20:48:48.000Z","size":90166,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-13T22:21:31.096Z","etag":null,"topics":["alphabet","alphabets","multilingual"],"latest_commit_sha":null,"homepage":"https://willwade.github.io/WorldAlphabets/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/willwade.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-08-21T12:58:45.000Z","updated_at":"2025-09-13T20:48:53.000Z","dependencies_parsed_at":"2025-09-13T22:23:50.723Z","dependency_job_id":null,"html_url":"https://github.com/willwade/WorldAlphabets","commit_stats":null,"previous_names":["willwade/worldalphabets"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/willwade/WorldAlphabets","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willwade%2FWorldAlphabets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willwade%2FWorldAlphabets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willwade%2FWorldAlphabets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willwade%2FWorldAlphabets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/willwade","download_url":"https://codeload.github.com/willwade/WorldAlphabets/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willwade%2FWorldAlphabets/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275312966,"owners_count":25442563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-15T02:00:09.272Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alphabet","alphabets","multilingual"],"created_at":"2025-09-15T19:57:59.403Z","updated_at":"2026-01-20T17:52:03.431Z","avatar_url":"https://github.com/willwade.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WorldAlphabets\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"web/public/logo.png\" alt=\"World Alphabets Logo\" width=\"200\" height=\"auto\"\u003e\n\n  [![npm version](https://img.shields.io/npm/v/worldalphabets.svg)](https://www.npmjs.com/package/worldalphabets)\n  [![PyPI version](https://img.shields.io/pypi/v/worldalphabets.svg)](https://pypi.org/project/worldalphabets/)\n  [![GitHub release](https://img.shields.io/github/v/release/willwade/WorldAlphabets)](https://github.com/willwade/WorldAlphabets/releases/latest)\n  [![C Library](https://img.shields.io/badge/C_Library-Download-blue)](https://github.com/willwade/WorldAlphabets/releases/latest)\n  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\u003c/div\u003e\n\nA tool to access alphabets of the world with Python, Node.js, and C interfaces.\n\n## Usage\n\n### Python\n\nInstall the package:\n\n```bash\npip install worldalphabets\n```\n\nTo load the data in Python (omitting ``script`` uses the first script listed):\n\n```python\nfrom worldalphabets import (\n    get_available_codes,\n    get_scripts,\n    load_alphabet,\n    load_frequency_list,\n)\n\ncodes = get_available_codes()\nprint(\"Loaded\", len(codes), \"alphabets\")\n\nalphabet = load_alphabet(\"en\")  # defaults to first script (Latn)\nprint(\"English uppercase:\", alphabet.uppercase[:5])\nprint(\"English digits:\", alphabet.digits)\n\nscripts = get_scripts(\"mr\")\nprint(\"Marathi scripts:\", scripts)\n\nalphabet_mr = load_alphabet(\"mr\", script=scripts[0])\nprint(\"Marathi uppercase:\", alphabet_mr.uppercase[:5])\nprint(\"Marathi frequency for 'a':\", alphabet_mr.frequency[\"a\"])\n\n# Example with Arabic digits\nalphabet_ar = load_alphabet(\"ar\", \"Arab\")\nprint(\"Arabic digits:\", alphabet_ar.digits)\n\n# Language detection (see Language Detection section for details)\nfrom worldalphabets import optimized_detect_languages\nresults = optimized_detect_languages(\"Hello world\")  # Automatic detection\nprint(\"Detected languages:\", results)\n\nfreq_en = load_frequency_list(\"en\")\nprint(\"English tokens (first 5):\", freq_en.tokens[:5])\nprint(\"Token mode:\", freq_en.mode)\n```\n\n### Node.js\n\n#### From npm\n\nInstall the package from npm:\n\n```bash\nnpm install worldalphabets\n```\n\nThen, you can use the functions in your project:\n\n```javascript\nconst {\n  getUppercase,\n  getLowercase,\n  getFrequency,\n  getDigits,\n  getAvailableCodes,\n  getScripts,\n  loadFrequencyList,\n} = require('worldalphabets');\n\nasync function main() {\n  const codes = await getAvailableCodes();\n  console.log('Available codes (first 5):', codes.slice(0, 5));\n\n  const scriptsSr = await getScripts('sr');\n  console.log('Serbian scripts:', scriptsSr);\n\n  const uppercaseSr = await getUppercase('sr', scriptsSr[0]);\n  console.log('Serbian uppercase:', uppercaseSr);\n\n  const lowercaseFr = await getLowercase('fr');\n  console.log('French lowercase:', lowercaseFr);\n\n  const frequencyDe = await getFrequency('de');\n  console.log('German frequency for \"a\":', frequencyDe['a']);\n\n  const digitsAr = await getDigits('ar', 'Arab');\n  console.log('Arabic digits:', digitsAr);\n\n  const freqEn = await loadFrequencyList('en');\n  console.log('English tokens (first 5):', freqEn.tokens.slice(0, 5));\n  console.log('Token mode:', freqEn.mode);\n}\n\nmain();\n```\n\nTypeScript projects receive typings automatically via `index.d.ts`.\n\n\n#### ES Modules (Browser / Node ESM)\n\nIf your project uses ES modules (e.g. Vite/webpack/Next.js or `\"type\": \"module\"` in Node), you can import directly. The ES module build also supports automatic candidate selection for language detection.\n\n```javascript\nimport { getAvailableCodes, getScripts, detectLanguages } from 'worldalphabets';\n\nconst codes = await getAvailableCodes();\nconsole.log('Available codes (first 5):', codes.slice(0, 5));\n\n// Automatic candidate selection (pass null). Also works in the browser.\nconst textKo = '나는 매우 행복하고 돈이 많이 있습니다.';\nconst top = await detectLanguages(textKo, null, {}, 3);\nconsole.log(top);\n// e.g. [['ko', 0.1203], ['ja', ...], ...]\n```\n\nNotes:\n- CommonJS (`require`) API requires `candidateLangs` (array) for detection.\n- ES Module (`import`) API supports `candidateLangs = null` and will select smart candidates via character analysis and embedded word/bigram ranks.\n- Browser ESM build ships data with the package and uses static JSON imports — no `fs`, `path`, or `fetch` required.\n- The published ESM entry is `index.mjs`, so bundlers pick it up without extra configuration. `index.esm.js` remains as a compatibility re-export.\n\n#### Browser (ESM) usage\n\nModern bundlers (Webpack 5, Vite, Rollup, Next.js) will automatically pick the browser ESM entry via conditional exports. The browser build is fully static and contains the alphabets, keyboard layouts, and frequency ranks.\n\n```javascript\n// Works in the browser with ESM bundlers\nimport {\n  getIndexData,\n  getLanguage,\n  getAvailableCodes,\n  getScripts,\n  detectLanguages,\n  getAvailableLayouts,\n  loadKeyboard,\n  getUnicode,\n  extractLayers,\n  detectDominantScript,\n} from 'worldalphabets';\n\n// Detect language without specifying candidates (short phrases supported)\nconst res = await detectLanguages('Bonjour comment allez-vous?', null, {}, 3);\nconsole.log(res);\n\n// Use keyboard layouts\nconst layouts = await getAvailableLayouts();\nconst kb = await loadKeyboard('fr-french-standard-azerty');\nconsole.log(getUnicode(kb.keys[1], 'base'));\n\n// Inspect modifier layers (Shift, AltGr, etc.)\nconst layers = extractLayers(kb, ['base', 'shift', 'altgr', 'shift_altgr']);\nconsole.log(layers.shift_altgr.Digit1); // 'À'\n\n// Optional: determine dominant script in an input\nconsole.log(detectDominantScript('Здраво, како си?')); // 'Cyrl'\n```\n\nNotes:\n- No network fetches are performed at runtime; data is packaged and statically imported.\n- For ambiguous greetings shared by multiple languages, detection may return either (both are acceptable). Use domain context to disambiguate.\n\n\n#### Local Usage\n\nIf you have cloned the repository, you can use the module directly:\n\n\n```javascript\nconst { getUppercase } = require('./index');\n\nasync function main() {\n    const uppercaseSr = await getUppercase('sr', 'Latn');\n    console.log('Serbian Latin uppercase:', uppercaseSr);\n}\n\nmain();\n```\n\n### Diacritic Utilities\n\nBoth interfaces provide helpers to work with diacritic marks.\n\n#### Python\n\n```python\nfrom worldalphabets import strip_diacritics, has_diacritics\n\nstrip_diacritics(\"café\")  # \"cafe\"\nhas_diacritics(\"é\")       # True\n```\n\n#### Node.js\n\n```javascript\nconst { stripDiacritics, hasDiacritics } = require('worldalphabets');\n\nstripDiacritics('café'); // 'cafe'\nhasDiacritics('é');      // true\n```\n\nUse `characters_with_diacritics`/`charactersWithDiacritics` to extract letters\nwith diacritic marks from a list.\n\nUse `get_diacritic_variants`/`getDiacriticVariants` to list base letters and\ntheir diacritic forms for a given language.\n\n```python\nfrom worldalphabets import get_diacritic_variants\n\nget_diacritic_variants(\"pl\", \"Latn\")[\"L\"]  # [\"L\", \"Ł\"]\n```\n\n```javascript\nconst { getDiacriticVariants } = require('worldalphabets');\n\ngetDiacriticVariants('pl').then((v) =\u003e v.L); // ['L', 'Ł']\n```\n\n### Language Detection\n\nThe library provides two language detection approaches:\n\n1. **Word-based detection** (primary): Uses Top-1000 frequency lists for languages with available word frequency data\n2. **Character-based fallback**: For languages without frequency data, analyzes character sets and character frequencies from alphabet data\n\n#### Automatic Detection (Recommended)\n\nThe optimized detection automatically selects candidate languages using character analysis:\n\n```python\nfrom worldalphabets import optimized_detect_languages\n\n# Automatic detection - analyzes ALL 310+ languages intelligently\noptimized_detect_languages(\"Hello world\")\n# [('en', 0.158), ('de', 0.142), ('fr', 0.139)]\n\noptimized_detect_languages(\"Аҧсуа бызшәа\")  # Abkhazian\n# [('ab', 0.146), ('ru', 0.136), ('bg', 0.125)]\n\noptimized_detect_languages(\"ⲧⲙⲛⲧⲣⲙⲛⲕⲏⲙⲉ\")  # Coptic\n# [('cop', 0.077)]\n\n# Still supports manual candidate specification\noptimized_detect_languages(\"Hello world\", candidate_langs=['en', 'de', 'fr'])\n# [('en', 0.158), ('de', 0.142), ('fr', 0.139)]\n```\n\n#### Manual Candidate Selection\n\nThe original detection requires you to specify candidate languages:\n\n```python\nfrom worldalphabets import detect_languages\n\n# Must specify candidate languages\ndetect_languages(\"Hello world\", candidate_langs=['en', 'de', 'fr'])\n# [('en', 0.158), ('de', 0.142), ('fr', 0.139)]\n\ndetect_languages(\"Аҧсуа бызшәа\", candidate_langs=['ab', 'ru', 'bg'])\n# [('ab', 0.146), ('ru', 0.136), ('bg', 0.125)]\n```\n\n#### Node.js (Manual Candidates Required)\n\n```javascript\nconst { detectLanguages } = require('worldalphabets');\n\n// Must specify candidate languages\ndetectLanguages('Hello world', ['en', 'de', 'fr']).then(console.log);\n// [['en', 0.158], ['de', 0.142], ['fr', 0.139]]\n\ndetectLanguages('ⲧⲙⲛⲧⲣⲙⲛⲕⲏⲙⲉ', ['cop', 'el', 'ar']).then(console.log);\n// [['cop', 0.077], ['el', 0.032], ['ar', 0.021]]\n```\n\nThe detection system supports **310+ languages** total: 86 with word frequency data and 224+ with character-based analysis.\n\n### Examples\n\nThe `examples/` directory contains small scripts demonstrating the library:\n\n- `examples/python/` holds Python snippets for printing alphabets, collecting\n  stats, listing scripts, and more.\n- `examples/node/` includes similar examples for Node.js.\n\n### Audio Samples\n\nAudio recordings are stored under `data/audio/` and named\n`{langcode}_{engine}_{voiceid}.wav`. Available voices are listed in\n`data/audio/index.json`.\n\n### Web Interface\n\nThe Vue app under `web/` compiles to a static site with `npm run build`.\nTo work on the interface locally, install its dependencies and start the\ndevelopment server:\n\n```bash\ncd web\nnpm install\nnpm run dev\n```\nGitHub Pages publishes the contents of `web/dist` through a workflow that\nruns on every push to `main`.\n\nEach language view is addressable at `/\u003ccode\u003e`, allowing pages to be\nbookmarked directly.\n\n### Alphabet Index\n\nThis library also provides an index of all available alphabets with additional metadata.\n\n#### Python\n\n```python\nfrom worldalphabets import get_index_data, get_language, get_scripts\n\n# Get the entire index\nindex = get_index_data()\nprint(f\"Index contains {len(index)} languages.\")\n\n# Show available scripts for Serbian\nscripts = get_scripts(\"sr\")\nprint(f\"Serbian scripts: {scripts}\")\n\n# Load Marathi in the Latin script\nmarathi_latn = get_language(\"mr\", script=\"Latn\")\nprint(f\"Script: {marathi_latn['script']}\")\nprint(f\"First letters: {marathi_latn['alphabetical'][:5]}\")\n```\n\n#### Node.js\n\n```javascript\nconst { getIndexData, getLanguage, getScripts } = require('worldalphabets');\n\nasync function main() {\n  // Get the entire index\n  const index = await getIndexData();\n  console.log(`Index contains ${index.length} languages.`);\n\n  // Show available scripts for Serbian\n  const scripts = await getScripts('sr');\n  console.log(`Serbian scripts: ${scripts}`);\n\n  // Load Marathi in the Latin script\n  const marathiLatn = await getLanguage('mr', 'Latn');\n  console.log(`Script: ${marathiLatn.script}`);\n  console.log(`First letters: ${marathiLatn.alphabetical.slice(0, 5)}`);\n}\n\nmain();\n```\n\nPrefer to work directly with the alphabet JSON instead of the higher-level helpers? Both the CommonJS and ESM builds export `loadAlphabet(code, script)` which resolves to the same object used above, letting you opt into manual handling of the raw alphabet data when needed.\n\n### Keyboard Layouts\n\nKey entries expose `pos` (a [`KeyboardEvent.code`](https://developer.mozilla.org/docs/Web/API/KeyboardEvent/code) when available) along with `row`, `col`, and size information.\n\n#### Python\n\nThe script `examples/python/keyboard_md_table.py` demonstrates rendering a\nlayout as a Markdown table. Copy the `layout_to_markdown` helper into your\nproject and use it like this:\n\n```python\nfrom keyboard_md_table import layout_to_markdown\n\nprint(layout_to_markdown(\"en-united-kingdom\"))\n```\n\nOutput:\n\n\n| ` | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 0 | - | = |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| q | w | e | r | t | y | u | i | o | p | [ | ] |  |\n| a | s | d | f | g | h | j | k | l | ; | ' | # |  |\n| z | x | c | v | b | n | m | , | . | / |  |  |  |\n| ␠ |  |  |  |  |  |  |  |  |  |  |  |  |\n\nor with --offset flag\n\n| ` | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 0 | - | = |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | q | w | e | r | t | y | u | i | o | p | [ | ] |\n|  | a | s | d | f | g | h | j | k | l | ; | ' | # |\n|  |  | z | x | c | v | b | n | m | , | . | / |  |\n|  |  |  |  |  | ␠ |  |  |  |  |  |  |  |\n\nProgrammatically, the same data is accessible from Python:\n\n```python\nfrom worldalphabets import load_keyboard, extract_layers\n\nlayout = load_keyboard(\"fr-french-standard-azerty\")\nlayers = extract_layers(layout, [\"base\", \"shift\", \"altgr\", \"shift_altgr\"])\nprint(layers[\"shift_altgr\"][\"Digit1\"])\n```\n\n#### Node.js\n\n```javascript\nconst {\n  getAvailableLayouts,\n  loadKeyboard,\n  getUnicode,\n  extractLayers,\n} = require('worldalphabets');\n\nasync function main() {\n  const layouts = await getAvailableLayouts();\n  console.log('Available layouts (first 5):', layouts.slice(0, 5));\n\n  const kb = await loadKeyboard('fr-french-standard-azerty');\n  console.log('First key Unicode:', getUnicode(kb.keys[1], 'base'));\n  const layers = extractLayers(kb, ['base', 'shift', 'altgr', 'shift_altgr']);\n  console.log('Shift+AltGr on Digit1:', layers.shift_altgr.Digit1);\n}\n\nmain();\n```\n\n### Export layouts to C headers\n\nGenerate an ESP32-friendly header that maps keyboard layers to USB HID usage\ncodes:\n\n```javascript\nconst { generateCHeader } = require('worldalphabets');\n\nasync function main() {\n  const header = await generateCHeader('fr-french-standard-azerty', {\n    layers: ['base', 'shift', 'altgr', 'shift_altgr'],\n  });\n  console.log(header);\n}\n\nmain();\n```\n\nThe Python API provides the same helper:\n\n```python\nfrom worldalphabets import generate_c_header\n\nheader = generate_c_header(\n    \"fr-french-standard-azerty\",\n    layers=[\"base\", \"shift\", \"altgr\", \"shift_altgr\"],\n)\nprint(header)\n```\n\nThe emitted header defines `keyboard_layout_t` plus per-layer mappings keyed by\nHID usage codes (e.g. `0x04` for `KeyA`). Pass `guard=False` if you need to\ninline the output into an existing header.\n\n### C library\n\nA native C library is generated from the same JSON assets and ships prebuilt\ndata tables for alphabets, frequency lists, detection, and keyboard layers.\n\nGenerate the C sources and build:\n\n```bash\nuv run python scripts/generate_c_library_data.py\ncmake -S c -B c/build\ncmake --build c/build\nctest --test-dir c/build\n```\n\nPublic API (see `c/include/worldalphabets.h`):\n\n```c\n#include \"worldalphabets.h\"\n\nconst wa_alphabet *alpha = wa_load_alphabet(\"fr\", \"Latn\");\nconst wa_frequency_list *freq = wa_load_frequency_list(\"fr\");\nwa_prior priors[] = {{\"fr\", 0.5}};\nwa_detect_result_array r = wa_detect_languages(\n    \"bonjour\", NULL, 0, priors, 1, 3);\nconst wa_keyboard_layout *kb = wa_load_keyboard(\"fr-french-standard-azerty\");\nwa_keyboard_layer base = wa_extract_layer(kb, \"base\");\nwa_free_detect_results(\u0026r);\n```\n\nArtifacts can be published as GitHub release assets; CMake installs both static\nand shared builds plus headers. CI builds for Linux, macOS, and Windows and\nuploads release assets automatically.\n\n### Quick layout hint from a keycode\n\nYou can ask for layouts that use a given keycode (DOM `code`, scan code, VK, or\nHID usage) on a specific layer. A common trick is the key immediately right of\nLeft Shift (`IntlBackslash`/scan code `56`) to distinguish ISO layouts.\n\nJavaScript / Node:\n\n```javascript\nconst { findLayoutsByKeycode } = require('worldalphabets');\nconst matches = await findLayoutsByKeycode('IntlBackslash', 'base');\nconsole.log(matches[0]); // { id, name, legend, layer }\n```\n\nPython:\n\n```python\nfrom worldalphabets import find_layouts_by_keycode\nmatches = find_layouts_by_keycode(\"IntlBackslash\", layer=\"base\")\nprint(matches[0])\n```\n\nC (using HID code 0x64 for `IntlBackslash`):\n\n```c\nwa_layout_match_array arr = wa_find_layouts_by_hid(0x64, \"base\");\n// arr.items points into static data; free with wa_free_layout_matches when done.\n```\n\n## Supported Languages\n\nFor a detailed list of supported languages and their metadata, including available\nkeyboard layouts, see the [Alphabet Table](table.md).\n\n## Developer Guide\n\nOlder versions of this project relied on a Java repository and assorted helper\nscripts to scrape alphabets and estimate letter frequencies. Those utilities\nhave been deprecated in favor of a cleaner pipeline based on Unicode CLDR and\nWikidata. The remaining scripts focus on fetching language–script mappings and\nbuilding alphabet JSON files directly from CLDR exemplar characters, enriching\nthem with frequency counts from the Simia dataset or OpenSubtitles when\navailable.\n\nThe alphabet builder preserves the ordering from CLDR exemplar lists and\nplaces diacritic forms immediately after their base letters when the CLDR\nindex omits them. For languages with tonal variants such as Vietnamese,\ncommon tone marks are stripped before deduplication to avoid generating\nseparate entries for every tone combination.\n\nEach JSON file includes:\n\n- `language` – English language name\n- `iso639_3` – ISO 639-3 code\n- `iso639_1` – ISO 639-1 code when available\n- `alphabetical` – letters of the alphabet (uppercase when the script has\n  case)\n- `uppercase` – uppercase letters\n- `lowercase` – lowercase letters\n- `frequency` – relative frequency of each lowercase letter (zero when no\n  sample text is available)\n\nExample JSON snippet:\n\n```json\n{\n  \"language\": \"English\",\n  \"iso639_3\": \"eng\",\n  \"iso639_1\": \"en\",\n  \"alphabetical\": [\"A\", \"B\"],\n  \"uppercase\": [\"A\", \"B\"],\n  \"lowercase\": [\"a\", \"b\"],\n  \"frequency\": {\"a\": 0.084, \"b\": 0.0208}\n}\n```\n\n### Setup\n\nThis project uses `uv` for dependency management. To set up the development\nenvironment:\n\n```bash\n# Install uv\npipx install uv\n\n# Create and activate a virtual environment\nuv venv\nsource .venv/bin/activate\n\n# Install dependencies\nuv pip install -e '.[dev]'\n```\n\n### Data Generation\n\n#### Consolidated Pipeline (Recommended)\n\nThe WorldAlphabets project uses a unified Python-based data collection pipeline:\n\n```bash\n# Build complete dataset with all stages\nuv run scripts/build_data_pipeline.py\n\n# Build with verbose output\nuv run scripts/build_data_pipeline.py --verbose\n\n# Run specific pipeline stage\nuv run scripts/build_data_pipeline.py --stage build_alphabets\n\n# Build single language\nuv run scripts/build_data_pipeline.py --language mi --script Latn\n```\n\n**Pipeline Stages:**\n1. **collect_sources** - Download CLDR, ISO 639-3, frequency data\n2. **build_language_registry** - Create comprehensive language database\n3. **build_alphabets** - Generate alphabet files from CLDR + fallbacks\n4. **build_translations** - Add \"Hello, how are you?\" translations\n5. **build_keyboards** - Generate keyboard layout files\n6. **build_top1000** - Generate Top-1000 token lists for detection\n7. **build_tts_index** - Index available TTS voices\n8. **build_audio** - Generate audio files using TTS\n9. **build_index** - Create searchable indexes and metadata\n10. **validate_data** - Comprehensive data validation\n\nFor detailed pipeline documentation, see [docs/DATA_PIPELINE.md](docs/DATA_PIPELINE.md).\n\n#### Legacy Individual Scripts (Deprecated)\n\nThe following individual scripts are deprecated in favor of the consolidated pipeline:\n\n**Add ISO language codes**\n```bash\nuv run scripts/add_iso_codes.py  # Use: --stage build_language_registry\n```\n\n**Fetch language-script mappings**\n```bash\nuv run scripts/fetch_language_scripts.py  # Use: --stage collect_sources\n```\n\n**Build alphabets from CLDR**\n```bash\nuv run scripts/build_alphabet_from_cldr.py  # Use: --stage build_alphabets\n```\n\n**Generate translations**\n\nPopulate a sample translation for each alphabet using Google Translate. The\nscript iterates over every language and script combination, writing a\n`hello_how_are_you` field to `data/alphabets/\u003ccode\u003e-\u003cscript\u003e.json`.\n\n```bash\nGOOGLE_TRANS_KEY=\u003ckey\u003e uv run scripts/generate_translations.py\n```\n\nTo skip languages that already have translations:\n\n```bash\nGOOGLE_TRANS_KEY=\u003ckey\u003e uv run scripts/generate_translations.py --skip-existing\n```\n\n**Populate keyboard layouts**\n\nTo refresh keyboard layout references after restructuring, run:\n\n```bash\nuv run scripts/populate_layouts.py\n```\n\nTo skip languages that already have keyboard data:\n\n```bash\nuv run scripts/populate_layouts.py --skip-existing\n```\n\n### Linting and type checking\n\n```bash\nruff check .\nmypy .\n```\n\n### Top-1000 token lists\n\nThe language detection helpers rely on comprehensive frequency lists for each language.\nThese lists contain the 1000 most common words per language, sourced from real external\ndata sources like Leipzig Corpora, HermitDave FrequencyWords, CommonVoice, and Tatoeba sentences.\nThis provides significantly improved accuracy over the previous top-200 approach.\n\nFor major languages (French, English, Spanish), the expanded lists include:\n- Common vocabulary: \"heureux\" (happy), \"argent\" (money), \"travail\" (work)\n- Numbers, colors, family terms, everyday objects\n- Improved word coverage: 60% → 80% for typical sentences\n\nThe lists are generated using a unified 5-priority pipeline that maximizes\ncoverage across as many languages as we can:\n\n```bash\n# Generate expanded frequency lists\nuv run python scripts/expand_to_top1000.py\n```\n\n**Priority Sources (in order):**\n1. **Leipzig Corpora Collection** - High-quality news/web corpora (CC-BY)\n2. **HermitDave FrequencyWords** - OpenSubtitles/Wikipedia sources (CC-BY)\n3. **Tatoeba sentences** - Sentence-based extraction (CC-BY 2.0 FR)\n4. **Existing alphabet frequency data** - Character-level fallback\n5. **Simia unigrams** - CJK character data\n\nThe script writes results to ``data/freq/top1000`` with build reports in\n``BUILD_REPORT_UNIFIED.json``. The unified pipeline also runs within the\nconsolidated data pipeline as the ``build_top1000`` stage.\n\n\n## Sources\n\n- [kalenchukov/Alphabet](https://github.com/kalenchukov/Alphabet)\n- [Simia unigrams dataset](http://simia.net/letters/)\n- [Wikipedia](https://wikipedia.org)\n- [ICU locale data](http://site.icu-project.org/)\n- [Unicode](https://unicode.org/)\n- [Kbdlayout](https://kbdlayout.info)\n\n\n## Licence Info\n\n- This project is licensed under the MIT License.\n- **All external data sources permit redistribution** under their respective open licenses.\n- For comprehensive licensing information and attribution requirements, see **[Data Sources and Licenses](docs/DATA_SOURCES_LICENSES.md)**.\n\n### Key Data Sources\n\n- **Leipzig Corpora Collection** (CC-BY) - High-quality frequency data for 77 languages\n- **HermitDave FrequencyWords** (CC-BY-SA-4.0) - OpenSubtitles/Wikipedia frequency data for 48 languages\n- **Mozilla CommonVoice** (CC0) - Speech transcription data for 130+ languages via Mozilla Data Collective\n- **Tatoeba** (CC-BY 2.0 FR) - Sentence data for 73 languages\n- **Simia unigrams** (CC-BY-SA) - CJK character frequency data from Wiktionary\n- **Kalenchukov/Alphabet** (Apache 2.0) - Alphabet definitions\n- **Unicode CLDR** (Unicode License) - Locale and character data\n- **ISO 639-3** (ODC-By) - Language codes from SIL International\n- **Wikidata** (CC0) - Language-script mappings\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillwade%2Fworldalphabets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwillwade%2Fworldalphabets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillwade%2Fworldalphabets/lists"}