{"id":48987348,"url":"https://github.com/theskyc/l10n-expansion-data","last_synced_at":"2026-04-18T13:10:20.310Z","repository":{"id":328358416,"uuid":"1115261621","full_name":"TheSkyC/l10n-expansion-data","owner":"TheSkyC","description":"Data-driven text expansion ratios for software localization (l10n) and UI planning.","archived":false,"fork":false,"pushed_at":"2025-12-13T14:26:23.000Z","size":146,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-12-14T06:58:22.320Z","etag":null,"topics":["dataset","expansion-ratio","i18n","l10n","localization","opus-100"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheSkyC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-12T15:26:45.000Z","updated_at":"2025-12-13T14:26:26.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/TheSkyC/l10n-expansion-data","commit_stats":null,"previous_names":["theskyc/l10n-expansion-data"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/TheSkyC/l10n-expansion-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheSkyC%2Fl10n-expansion-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheSkyC%2Fl10n-expansion-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheSkyC%2Fl10n-expansion-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheSkyC%2Fl10n-expansion-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheSkyC","download_url":"https://codeload.github.com/TheSkyC/l10n-expansion-data/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheSkyC%2Fl10n-expansion-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31969981,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","expansion-ratio","i18n","l10n","localization","opus-100"],"created_at":"2026-04-18T13:10:08.757Z","updated_at":"2026-04-18T13:10:20.298Z","avatar_url":"https://github.com/TheSkyC.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# l10n-expansion-data\n\n![License: CC0-1.0](https://img.shields.io/badge/License-CC0%201.0-lightgrey.svg)\n![Data Source: OPUS-100](https://img.shields.io/badge/Data%20Source-OPUS--100-blue)\n![Format: JSON/CSV/YAML](https://img.shields.io/badge/Format-JSON%20%7C%20CSV%20%7C%20YAML-orange)\n\n**Data-driven text expansion ratios for robust software localization and UI planning.**\n\nStop guessing how much space to leave for German or how much Vietnamese text will expand. This repository provides highly granular statistical expansion ratios derived from massive parallel corpora (currently **OPUS-100**), helping developers and designers prevent UI overflows before translation begins.\n\nAll data is generated using the [Expansion Rate Generator (ERG)](https://github.com/TheSkyC/expansion-ratio-generator) tool.\n\n## 📊 Why Use This Data?\n\nWhen translating software, text length changes unpredictably.\n*   **English → Chinese:** Text usually shrinks (~0.6x).\n*   **English → German:** Text often expands (~1.3x).\n*   **Short Strings:** UI labels (e.g., \"OK\", \"New\") behave very differently from long paragraphs.\n\nThis dataset offers:\n1.  **Granularity:** Ratios are grouped by source string length (0-20 chars, 20-50 chars, etc.).\n2.  **Reliability:** Based on millions of sentence pairs, not just heuristics.\n3.  **Flexibility:** Provides multiple statistical metrics (mean, median, percentiles) to fit different risk tolerances.\n\n## 📂 Directory Structure\n\nThe data is organized by **Source Corpus** -\u003e **Export Strategy** -\u003e **File**.\n\n```text\ndata/\n└── opus-100/                  # Derived from the Helsinki-NLP OPUS-100 corpus\n    ├── detailed/              # Bucketed data with full statistics\n    │   ├── opus-100_detailed_mean.json\n    │   ├── opus-100_detailed_med.json\n    │   └── ... (and CSV/YAML variants)\n    │\n    └── simple/                # Single global value per language pair\n        ├── opus-100_simple_wgt_mean.json\n        ├── opus-100_simple_wgt_p75.json\n        └── ... (and CSV/YAML variants)\n```\n\n## 📝 Data Formats \u0026 Strategies\n\nWe provide two main export strategies: **Simple** and **Detailed**.\n\n### 1. Simple Strategy (`/simple`)\n\nProvides a **single value** per language pair. Ideal for quick estimates or runtime checks.\n\n*   **File Naming:** `corpus_simple_\u003cstrategy\u003e_\u003cmetric\u003e.json`\n    *   `\u003cstrategy\u003e`: `wgt` (weighted average), `max` (worst-case), `bkt0-20` (specific bucket).\n    *   `\u003cmetric\u003e`: `mean`, `med` (median), `p25`, `p75`, `rng25-75` (range).\n\n**Example (`opus-100_simple_bkt0-20_mean.json`):**\n```json\n{\n  \"en-sh\": 1.1321,\n  \"en-se\": 1.9319,\n  \"en-ro\": 1.1275,\n  \"en-zh\": 0.5633\n}\n```\n\n**Example (`opus-100_simple_wgt_rng25-75.json`):**\n```json\n{\n  \"en-sh\": \"0.88-1.09\",\n  \"en-se\": \"0.97-1.98\",\n  \"en-ro\": \"0.83-1.15\",\n  \"en-zh\": \"0.26-0.40\"\n}\n```\n\n### 2. Detailed Strategy (`/detailed`)\n\nProvides **bucketed data** with rich statistics. Ideal for dynamic UI layout engines or in-depth analysis.\n\n*   **File Naming:** `corpus_detailed_\u003cmetric\u003e.json`\n    *   `\u003cmetric\u003e`: The primary metric used for the `val` key (e.g., `mean`, `med`).\n\n**Example (`opus-100_detailed_mean.json`):**\n```json\n{\n  \"en-zh\": {\n    \"0.0-20.0\": {\n      \"val\": 0,\n      \"count\": 176354,\n      \"std\": 0.5426569950687576,\n      \"min\": 0.1,\n      \"max\": 10.0,\n      \"mean\": 0.563295776173526,\n      \"median\": 0.42857142857142855,\n      \"p25\": 0.3333333333333333,\n      \"p75\": 0.6\n    },\n    \"20.0-50.0\": {\n      \"val\": 0,\n      \"count\": 286113,\n      \"std\": 0.2895320290537651,\n      \"min\": 0.1,\n      \"max\": 9.954545454545455,\n      \"mean\": 0.38877682122354107,\n      \"median\": 0.32558139534883723,\n      \"p25\": 0.25925925925925924,\n      \"p75\": 0.41379310344827586\n    }\n  }\n}\n```\n\n## 🤔 Which File Should I Use?\n\nWith so many files, here’s a quick guide:\n\n*   **For general UI development (buttons, labels):**\n    *   Use `simple/opus-100_simple_bkt0-20.json`. This gives you a based on short strings (0-20 characters), which is the most common scenario for UI text.\n\n*   **For a single, balanced ratio for your entire app:**\n    *   Use `simple/opus-100_simple_wgt_mean.json`. This provides a weighted average across all text lengths.\n\n*   **For dynamic layout engines that adapt to text length:**\n    *   Use `detailed/opus-100_detailed_mean.json`. This allows you to look up the appropriate ratio based on the source text's character count.\n\n*   **For data analysis or academic research:**\n    *   Use the `.csv` files in the `detailed/` directory, which can be easily imported into Excel or Pandas.\n\n## 🚀 Usage Example\n\n### Python\n\n```python\nimport json\n\n# For simple, quick checks, use the 'simple' weighted mean data.\nwith open('data/opus-100/simple/opus-100_simple_wgt_mean.json', 'r', encoding='utf-8') as f:\n    ratios = json.load(f)\n\ndef get_estimated_length(text, target_lang_code):\n    \"\"\"Get a simple estimated length.\"\"\"\n    ratio = ratios.get(f\"en-{target_lang_code}\", 1.0) # Fallback to 1.0\n    return len(text) * ratio\n\nprint(f\"Estimated length for 'Save File' in German: {get_estimated_length('Save File', 'de'):.2f}\")\n```\n\n## 🔬 Methodology\n\n1.  **Source:** We utilize the `train` split from the [OPUS-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100) corpus, which contains 1 million sentence pairs per language.\n2.  **Filtering:**\n    *   Pairs with extreme expansion ratios (\u003c0.1 or \u003e10.0) are discarded as likely alignment errors.\n    *   Source strings with zero length are excluded.\n3.  **Calculation:**\n    *   `Ratio = CharacterLength(Target) / CharacterLength(Source)`\n    *   All calculations are based on character counts.\n\n## ⚖️ License \u0026 Attribution\n\n### Dataset License\nThe statistical data in this repository is released under the **CC0 1.0 Universal (Public Domain Dedication)**. You are free to use, modify, and distribute it in any commercial or open-source software without restriction.\n\n### Disclaimer\nThis repository contains **statistical metadata only**. It does **not** contain any original sentence pairs or text content from the source corpora.\n\n### Source Attribution\nThis data is derived from the **OPUS-100** corpus. If you use this data in academic work, please cite the original paper:\n\n\u003e **OPUS-100**  \n\u003e *Zhang, Biao et al. \"Improving Massively Multilingual Neural Machine Translation in Zero-Shot Scenarios.\" ACL (2020).*\n\n```bibtex\n@inproceedings{zhang-etal-2020-improving,\n    title = \"Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation\",\n    author = \"Zhang, Biao  and\n      Williams, Philip  and\n      Titov, Ivan  and\n      Sennrich, Rico\",\n    editor = \"Jurafsky, Dan  and\n      Chai, Joyce  and\n      Schluter, Natalie  and\n      Tetreault, Joel\",\n    booktitle = \"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics\",\n    month = jul,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2020.acl-main.148\",\n    doi = \"10.18653/v1/2020.acl-main.148\",\n    pages = \"1628--1639\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheskyc%2Fl10n-expansion-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheskyc%2Fl10n-expansion-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheskyc%2Fl10n-expansion-data/lists"}