{"id":51580877,"url":"https://github.com/starrocks/doc-translator","last_synced_at":"2026-07-11T06:02:43.652Z","repository":{"id":357653535,"uuid":"1237064884","full_name":"StarRocks/doc-translator","owner":"StarRocks","description":"Use Anthropic to translate markdown files","archived":false,"fork":false,"pushed_at":"2026-05-19T23:17:48.000Z","size":656,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-30T15:40:09.280Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StarRocks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-12T20:57:40.000Z","updated_at":"2026-05-14T13:41:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/StarRocks/doc-translator","commit_stats":null,"previous_names":["starrocks/doc-translator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/StarRocks/doc-translator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StarRocks%2Fdoc-translator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StarRocks%2Fdoc-translator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StarRocks%2Fdoc-translator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StarRocks%2Fdoc-translator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StarRocks","download_url":"https://codeload.github.com/StarRocks/doc-translator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StarRocks%2Fdoc-translator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35352623,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-11T06:02:40.336Z","updated_at":"2026-07-11T06:02:43.641Z","avatar_url":"https://github.com/StarRocks.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Markdown Translator\n\nA powerful command-line tool that uses Claude AI to translate markdown and MDX files from English to any specified language while preserving formatting and structure.\n\n## Usage at StarRocks\n\nThis code and most of the README are from the team at [PlayCanvas](https://github.com/playcanvas/markdown-translator). The only changes are:\n- StarRocks specific prompt\n- StarRocks specific dictionary\n- StarRocks specific words that should always be in English\n- the `-s, --source` option to allow specifying the source language as we translate from both English and Chinese.\n\n## Options\n\n```\n  -i, --input \u003cpattern\u003e   Input file path or glob pattern (e.g., \"*.md\",\n                          \"docs/**/*.md\")\n  -l, --language \u003clang\u003e   Target language (e.g., Spanish, French, German)\n  -s, --source \u003clang\u003e     Source language (default: English)\n  -o, --output \u003cfile\u003e     Output file path (for single file translation)\n  -d, --output-dir \u003cdir\u003e  Output directory (for batch translation or single\n                          file)\n  -k, --key \u003capikey\u003e      Anthropic API key (or set ANTHROPIC_API_KEY env var)\n  --flat                  Use flat structure in output directory (default:\n                          preserve structure)\n  --suffix \u003csuffix\u003e       Custom suffix for output files (default: language\n                          name)\n   --log-chunk-metadata    Log API metadata for each chunk\n   --trace                 Log per-ID source text sent and translated text\n                                       received (full content, no truncation)\n  -h, --help              display help for command\n```\n\nThe translator now uses the AST pipeline by default.\n\nWhen `--trace` is enabled, the tool logs one JSON trace record per ID and includes the full `sourceText` and `translatedText` values. The only masking applied is replacing occurrences of the actual API key value with `***`.\n\n### Interpreting AST parse failures\n\nIn AST mode, each chunk asks the model to return a strict JSON array of `{ id, text }` items.\n\n- Parse errors such as `Expected ',' or '}'` or `Expected ':' after property name` usually mean the model returned malformed JSON for that chunk.\n- These are response-format failures, not semantic translation failures.\n- `finishReason: STOP` with parse errors means the output completed, but the JSON structure was invalid.\n- When you see `json repair retry`, the tool requested a strict JSON retry and recovered automatically.\n- When you see `split fallback recovered X/Y missing ids`, the tool retried unresolved IDs in smaller sub-batches and merged recovered results back into the chunk.\n\nHow to read the outcome:\n\n- `AST completeness check: Translated IDs N/N - ✅ PASS` means the chunk is fully recovered, even if repair notes are present.\n- Missing IDs after all retries are the only case that indicates unresolved chunk-level translation for those specific items.\n\n## Quick Start\n\n1. cd into the root of this repo\n2. Get an Anthropic API Key\n3. Export your Anthropic API Key like so:\n   ```sh\n   export ANTHROPIC_API_KEY=\"\u003cyour key here\u003e\"\n   ```\n4. Install the prerequisites:\n   ```sh\n   npm install\n   ```\n5. Translate an example file:\n   ```sh\n   npm run demo\n   ```\n6. Check the source and destination example files (names are in the output from `npm run demo`). Look for our key phrases that are in our dictionaries and the terms that should always be left in English.\n7. List the options:\n   ```sh\n   node bin/cli.js translate -h\n   ```\n   \n## Example use on your workstation with the StarRocks repo\n   ```sh\n   # Export your Anthropic API key\n   export ANTHROPIC_API_KEY=\"sk-ant-xxxxxxxxxxxxxxxx\"\n   \n   # in the doc-translator repo directory install the translator globally on your system:\n   npm install\n   npm link\n   \n   # now in the starrocks/starrocks repo dir\n   # view the options:\n   doc-translate translate -h\n\n   # Example, translate the English architecture doc to Japanese:\n   doc-translate translate -s en -i docs/en/introduction/Architecture.md -l ja -o docs/ja/introduction/Architecture.md\n   ```\n## Example use in GitHub PRs\n\n1. check the boxes\n\n    \u003cimg width=\"907\" height=\"622\" alt=\"image\" src=\"https://github.com/user-attachments/assets/7a1198f6-a9f5-4dd2-b57d-aa07d0c5b9ff\" /\u003e\n\n2. Add a /translate comment (docs-maintainers only at the moment)\n\n    \u003cimg width=\"567\" height=\"211\" alt=\"image\" src=\"https://github.com/user-attachments/assets/621bbd58-bb1b-438e-9c5c-fef44857e308\" /\u003e\n\n## Features\n\n- 🌍 **Multi-language support** - Translate to 40+ languages\n- 📝 **Markdown-aware** - Preserves all markdown formatting (headers, links, code blocks, tables, etc.)\n- 🔄 **Smart chunking** - Handles large files by splitting content intelligently\n- 🎯 **Selective translation** - Only translates text content, keeps code and URLs intact\n- 📂 **Batch processing** - Translate multiple files using glob patterns (e.g., `docs/**/*.md`)\n- 🏗️ **Structure preservation** - Maintain directory structure or flatten output as needed\n- 📊 **Progress tracking** - Real-time progress indication with spinners for single files and batches\n- 🎨 **Beautiful CLI** - Colorful, user-friendly command-line interface\n- ⚡ **Fast processing** - Optimized for speed with high-performance Claude model\n\n## Installation\n\n### Prerequisites\n\n- Node.js 16.0.0 or higher\n- Anthropic API key ([Get one here](https://console.anthropic.com/))\n\n\u003e **Note**: This tool uses ES modules (ESM) and requires Node.js 16+ for full compatibility.\n\n### Install dependencies\n\n```bash\nnpm install\n```\n\n### Make CLI globally available (optional)\n\n```bash\nnpm link\n```\n\nOr run directly with Node:\n\n```bash\nnode bin/cli.js\n```\n\n## Setup\n\n### 1. Get Anthropic API Key\n\n1. Visit [Anthropic Console](https://console.anthropic.com/)\n2. Create a new API key\n3. Copy the generated key\n\n### 2. Set API Key\n\n**Option A: Environment Variable (Recommended)**\n\n```bash\nexport ANTHROPIC_API_KEY=\"your-api-key-here\"\n```\n\n**Option B: Command Line Argument**\n\n```bash\ndoc-translate translate -i file.md -l Spanish --key your-api-key-here\n```\n\n## Usage\n\n### Basic Translation\n\n```bash\n# Translate README.md to Spanish\ndoc-translate translate -i README.md -l Spanish\n\n# Translate with custom output file\ndoc-translate translate -i docs/guide.md -l French -o docs/guide_fr.md\n\n# Translate using API key argument\ndoc-translate translate -i file.md -l German --key your-api-key\n\n# Translate with AST mode (default)\ndoc-translate translate -i examples/External_table.md -l Japanese\n```\n\n### Batch Processing\n\nThe tool supports batch processing of multiple markdown files using glob patterns:\n\n```bash\n# Translate all .md files in current directory\ndoc-translate translate -i \"*.md\" -l Spanish -d ./spanish/\n\n# Translate all markdown files in docs folder and subfolders\ndoc-translate translate -i \"docs/**/*.md\" -l French -d ./translations/\n\n# Batch translate with flat structure (no subdirectories)\ndoc-translate translate -i \"content/**/*.md\" -l German -d ./output/ --flat\n\n# Batch translate with custom suffix\ndoc-translate translate -i \"*.md\" -l ja -d ./translated/ --suffix \"ja\"\n```\n\n### Available Commands\n\n#### `translate` - Translate a markdown or MDX file\n\n```bash\ndoc-translate translate [options]\n\nOptions:\n  -i, --input \u003cpattern\u003e    Input file path or glob pattern (required)\n                          Examples: \"file.md\", \"*.md\", \"docs/**/*.md\"\n  -l, --language \u003clang\u003e    Target language (required)\n  -o, --output \u003cfile\u003e      Output file path (for single file translation)\n  -d, --output-dir \u003cdir\u003e   Output directory (for batch translation or single file)\n  -k, --key \u003capikey\u003e       Anthropic API key (optional)\n  --flat                   Use flat structure in output directory (default: preserve structure)\n  --suffix \u003csuffix\u003e        Custom suffix for output files (default: language name)\n   --log-chunk-metadata     Log API metadata for each chunk\n   --trace                  Log per-ID source text sent and translated text received\n```\n\n#### `languages` - List supported languages\n\n```bash\ndoc-translate languages\n```\n\n#### `setup` - Show setup guide\n\n```bash\ndoc-translate setup\n```\n\n#### `--help` - Show help\n\n```bash\ndoc-translate --help\n```\n\n## Supported Languages\n\nThe tool supports 40+ languages including:\n\n- **European**: Spanish, French, German, Italian, Portuguese, Dutch, Russian, Polish, Swedish, Norwegian, Danish, Finnish, Greek, Ukrainian, Czech, Hungarian, Romanian, Bulgarian, Croatian, Serbian, Slovak, Slovenian, Estonian, Latvian, Lithuanian, Catalan, Basque, Welsh, Irish\n- **Asian**: Chinese, Japanese, Korean, Hindi, Thai, Vietnamese, Indonesian, Malay\n- **Middle Eastern**: Arabic, Hebrew, Turkish\n\n\u003e Tip\n\u003e\n\u003e Use the two letter short code for the language if you like. For example, `zh` instead of \"Simplified Chinese\".\n\n## Examples\n\n### Single File Translation\n\n#### Example 1: Basic Translation\n\n```bash\ndoc-translate translate -i README.md -l es\n```\n\n**Output**: Creates `README_spanish.md` with Spanish translation\n\n#### Example 2: Custom Output Path\n\n```bash\ndoc-translate translate -i docs/api.md -l fr -o docs/fr/api.md\n```\n\n**Output**: Creates `docs/fr/api.md` with French translation\n\n#### Example 3: Using API Key Argument\n\n```bash\ndoc-translate translate -i guide.md -l German --key AIzaSyC...\n```\n\n#### Example 4: Large File Translation\n\nThe tool automatically handles large files by splitting them into chunks:\n\n```bash\ndoc-translate translate -i large-document.md -l ja\n```\n\n### Batch Translation\n\n#### Example 5: Translate All Markdown Files\n\n```bash\ndoc-translate translate -i \"*.md\" -l Spanish -d ./spanish/\n```\n\n**Output**: Translates all `.md` files in current directory to `./spanish/` folder\n\n#### Example 6: Recursive Translation with Structure Preservation\n\n```bash\ndoc-translate translate -i \"docs/**/*.md\" -l French -d ./translations/\n```\n\n**Output**: Translates all markdown files in `docs/` and preserves directory structure in `./translations/`\n\n```\ndocs/\n├── guide.md\n├── api/\n│   └── reference.md\n└── tutorials/\n    └── getting-started.md\n\n# Becomes:\ntranslations/\n├── guide_french.md\n├── api/\n│   └── reference_french.md\n└── tutorials/\n    └── getting-started_french.md\n```\n\n#### Example 7: Flat Structure Batch Translation\n\n```bash\ndoc-translate translate -i \"content/**/*.md\" -l German -d ./output/ --flat\n```\n\n**Output**: Translates all files but places them in a flat structure (no subdirectories)\n\n```\ncontent/\n├── intro.md\n├── chapters/\n│   ├── chapter1.md\n│   └── chapter2.md\n└── appendix/\n    └── notes.md\n\n# Becomes:\noutput/\n├── intro_german.md\n├── chapter1_german.md\n├── chapter2_german.md\n└── notes_german.md\n```\n\n#### Example 8: Custom Suffix\n\n```bash\ndoc-translate translate -i \"*.md\" -l ja -d ./translated/ --suffix \"ja\"\n```\n\n**Output**: Uses \"ja\" instead of \"japanese\" as the file suffix\n\n\n\n## What Gets Translated\n\n✅ **Translated**:\n- Heading text\n- Paragraph text\n- List items\n- Table content\n- Link text\n- Image alt text\n- Quote text\n\n❌ **Preserved**:\n- Code blocks and inline code\n- URLs and file paths\n- Markdown syntax characters\n- HTML tags\n- Mathematical expressions\n- Technical terms and proper nouns (when appropriate)\n\n## Output\n\nThe tool provides detailed progress feedback for both single file and batch processing:\n\n### Single File Translation Output\n\n```\n╔═══════════════════════════════════════╗\n║        Markdown Translator            ║\n║       Powered by Claude AI            ║\n╚═══════════════════════════════════════╝\n\n📋 Translation Details:\n   Input:    /path/to/README.md\n   Output:   /path/to/README_spanish.md\n   Language: Spanish\n\n⠋ Translating chunk 2/3...\n✅ Translation completed successfully!\n\n📊 Summary:\n   Original length:  2,845 characters\n   Translated length: 3,120 characters\n   Language:         Spanish\n   Output file:      /path/to/README_spanish.md\n```\n\n### Batch Translation Output\n\n```\n╔═══════════════════════════════════════╗\n║        Markdown Translator            ║\n║       Powered by Claude AI            ║\n╚═══════════════════════════════════════╝\n\n📋 Batch Translation Details:\n   Pattern:  docs/**/*.md\n   Output:   /path/to/translations/\n   Language: Spanish\n   Structure: Preserved\n\n⠋ [2/5] reference.md - chunk 1/2...\n✅ All translations completed successfully!\n\n📊 Summary:\n   Files processed: 5\n   Successful: 5\n   Failed: 0\n   Output directory: /path/to/translations/\n```\n\n## Error Handling\n\nThe tool provides clear error messages for common issues:\n\n- Missing or invalid API key\n- File not found\n- Invalid file format\n- Network connectivity issues\n- API rate limiting\n\n## Testing\n\nThe `examples/` directory contains a test corpus and an automated checker.\n\n### Test corpus: `examples/StarRocksTest.md`\n\nA curated set of patterns drawn from real StarRocks documentation that have caused translation problems in the past:\n\n| Pattern | Why it matters |\n|---------|---------------|\n| YAML frontmatter | Must be preserved exactly |\n| HTML in Markdown table cells (`\u003cul\u003e\u003cli\u003e`, `\u003cbr /\u003e`, `\u003ccode class=\"...\"\u003e`) | Tags must not be translated or restructured |\n| Tilde fence code blocks (`~~~SQL`) | Must be converted to backtick fences cleanly |\n| MDX `import` statements and `\u003cTabs\u003e`/`\u003cTabItem\u003e` JSX | Must be preserved unchanged |\n| Template variables in code (`{{ data_interval_start }}`) | Airflow/dbt syntax must not be touched |\n| HTML comparison tables with `colspan` | Full HTML blocks must pass through untranslated |\n| Admonitions indented inside numbered lists | Indentation must survive translation |\n| `\u003cdetails\u003e` collapsible blocks | Content indentation must be preserved |\n| Cross-references with relative paths and anchors | Only the display text is translated; the URL is not |\n\n### Automated checker: `examples/check_translation.js`\n\nAfter translation, the checker runs 13 static checks against the source/output pair and reports PASS/FAIL for each:\n\n- No `__MTX_` placeholder leaks\n- Heading count\n- Code block count and non-comment content\n- Link URL preservation\n- HTML tags in table cells\n- Frontmatter preserved exactly\n- Import statements preserved\n- Admonition marker count\n- Admonition indentation (catches the \"indented :::note gets unindented\" bug)\n- Never-translate term spot-check\n- Unordered list item count\n- Table column counts\n\n### npm scripts\n\n```bash\nnpm test          # Translate StarRocksTest.md → zh, then run all checks\nnpm run test:ja   # Translate StarRocksTest.md → ja, then run all checks\nnpm run check:zh  # Re-run checks on an already-translated StarRocksTest_zh.md\nnpm run check:ja  # Re-run checks on an already-translated StarRocksTest_ja.md\n```\n\n`check:zh` and `check:ja` are useful for iterating on the system prompt or dictionaries without calling the API again.\n\n## Development\n\n### Project Structure\n\n```\ndoc-translator/\n├── bin/\n│   └── cli.js                    # CLI entry point\n├── src/\n│   ├── translator.js             # Base class and shared utilities\n│   ├── translator_ast_mvp.js     # AST-based translator (default)\n│   └── configs/\n│       ├── system_prompt.txt     # Translation instructions for the model\n│       ├── never_translate.yaml  # Terms that must never be translated\n│       └── language_dicts/       # Per-language translation dictionaries\n├── examples/\n│   ├── StarRocksTest.md          # Test corpus\n│   └── check_translation.js     # Automated output checker\n├── package.json\n└── README.md\n```\n\n### Architecture\n\nThis project uses **ES modules (ESM)** for modern JavaScript development:\n\n- All files use `import`/`export` syntax instead of `require`/`module.exports`\n- `package.json` includes `\"type\": \"module\"` for ESM support\n- Compatible with the latest versions of dependencies (chalk 5.x, ora 8.x)\n- Requires Node.js 16+ for full ESM compatibility\n\n### Key Dependencies\n\n- `@anthropic-ai/sdk` - Anthropic Claude AI SDK\n- `commander` - Command-line interface framework\n- `chalk` - Terminal styling\n- `ora` - Progress spinners\n- `fs-extra` - Enhanced file system operations\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Troubleshooting\n\n### API Key Issues\n\n- Ensure your API key is valid and active\n- Check that you have sufficient quota in your Anthropic account\n- Verify the API key is active in the Anthropic Console\n\n### Large File Processing\n\n- The tool automatically chunks large files\n- Each chunk is processed with a small delay to avoid rate limiting\n- Very large files may take several minutes to process\n\n### Batch Processing\n\n- Use quotes around glob patterns to prevent shell expansion: `\"*.md\"` not `*.md`\n- The `--output-dir` option is required for batch translation\n- Large batches may take considerable time; use progress indicators to monitor\n- Failed files in a batch are reported individually without stopping the process\n\n### Network Issues\n\n- Ensure you have a stable internet connection\n- The tool will retry failed requests automatically\n- Check firewall settings if you encounter connection issues\n\n## Support\n\nIf you encounter any issues or have questions:\n\n1. Check the troubleshooting section above\n2. Run `doc-translate setup` for configuration help\n3. Create an issue on the project repository\n\n---\n\n**Happy translating! 🌍✨** \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarrocks%2Fdoc-translator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstarrocks%2Fdoc-translator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstarrocks%2Fdoc-translator/lists"}